Lake warehouse metadata tag creation method and lake warehouse metadata tag query method

Abstract

The application discloses a lake warehouse metadata label creation method, a lake warehouse metadata label query method, a system, a device, equipment and a medium. The method comprises the following steps: after a metadata service gateway determines a metadata label creation request according to metadata label creation parameters provided by a data engine, the metadata service gateway sends the metadata label creation request to a metadata label service system, so that the metadata label service system executes label creation logic according to metadata label creation parameters carried by the metadata label creation request, realizes the labeling demand for one or more metadata, and thus can realize more fine-grained labeling processing. In this way, defects caused by the labeling processing realized by adding new parameters in attribute information of a metadata database, a metadata table or a metadata partition can be effectively avoided, and the labeling effect can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for creating lake warehouse metadata tags, a method for querying lake warehouse metadata tags, a system, a device, equipment, and a medium. Background Technology

[0002] For some big data application scenarios, there may be the following requirements: tagging the metadata of the lake warehouse (for example, a user has access rights to a certain metadata) so that the tag can describe the characteristics of the metadata.

[0003] However, how to meet the above requirements is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method for creating lake warehouse metadata tags, a method for querying lake warehouse metadata tags, a system, an apparatus, equipment, and a medium.

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides a method for creating lake warehouse metadata tags, the method being applied to a metadata tag service system, the method comprising:

[0007] Receive a metadata tag creation request sent by the metadata service gateway; the metadata tag creation request is determined by the metadata service gateway based on the metadata tag creation parameters provided by the data engine; the metadata tag creation parameters include a first tag name, a first tag value, and a description of the first tag's target object;

[0008] Based on the metadata tag creation parameters carried in the metadata tag creation request, the tag creation logic is executed.

[0009] In one possible implementation, the tag creation logic includes:

[0010] Write the tag name record content corresponding to the first tag name into the tag name record table. The tag name record content includes the tag name identifier corresponding to the first tag name.

[0011] Write the tag target record content corresponding to the first tag target description information into the tag target record table. The tag target record content includes the tag target identifier corresponding to the first tag target description information.

[0012] Write the tag value record content corresponding to the first tag value into the tag value record table. The tag value record content is determined based on the first tag value, the tag name identifier, and the tag target identifier.

[0013] In one possible implementation, the metadata tag creation parameters also include the name of the first tag set;

[0014] Before writing the tag name record content corresponding to the first tag name into the tag name record table, the tag creation logic further includes:

[0015] Write the tag set record content corresponding to the first tag set name into the tag set record table. The tag set record content includes the set identifier corresponding to the first tag set name.

[0016] Based on the set identifier and the first tag name, determine the tag name record content corresponding to the first tag name.

[0017] In one possible implementation, writing the tag target record content corresponding to the first tag target description information into the tag target record table includes:

[0018] If there is no tag action object record in the tag action object record table that corresponds to the first tag action object description information, then the tag action object record corresponding to the first tag action object description information is written into the tag action object record table.

[0019] In one possible implementation, the method further includes:

[0020] If the completion of the tag creation logic is detected, feedback information corresponding to the metadata tag creation request is generated.

[0021] The feedback information is sent to the metadata service gateway, which then forwards the feedback information to the data engine.

[0022] In one possible implementation, the metadata tag creation parameters include a first tag set name, a first tag name, a first tag value, and a description of the object to which the first tag applies;

[0023] The feedback information corresponding to the metadata tag creation request includes:

[0024] The feedback information is obtained by combining the first tag value, the set identifier corresponding to the first tag set name, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the first tag target description information.

[0025] In one possible implementation, the metadata tag creation request is used to request the creation of a tag corresponding to one or more metadata.

[0026] The one or more metadata items are managed by a metadata storage system;

[0027] The metadata service gateway is used to route the metadata processing request sent by the data engine to the metadata storage system, and the metadata storage system is used to execute the data processing logic corresponding to the metadata processing request.

[0028] This application provides a method for querying lake warehouse metadata tags. The method is applied to a metadata tag service system, which is used to perform query processing on tags created using the lake warehouse metadata tag creation method provided in this application.

[0029] The query method includes:

[0030] Receive a metadata tag query request sent by the metadata service gateway; the metadata tag query request is determined by the metadata service gateway based on the metadata tag query parameters provided by the data engine;

[0031] Based on the metadata tag query parameters carried in the metadata tag query request, tag query processing is performed to obtain the tag query result corresponding to the metadata tag query request;

[0032] The tag query results are sent to the metadata service gateway, which then feeds back the tag query results to the data engine.

[0033] In one possible implementation, the metadata tag query parameters include the second tag name and the description information of the object on which the second tag applies;

[0034] The process of determining the tag query results includes:

[0035] Search the label name record table for the label name corresponding to the second label name;

[0036] Search the label target record table for the label target description information corresponding to the label target identifier.

[0037] The second tag value is obtained by querying the tag value record table based on the tag name identifier and the tag target identifier;

[0038] The tag query result is determined based on the second tag value.

[0039] In one possible implementation, the metadata tag creation parameters also include a second tag set name;

[0040] Before searching the tag name identifier corresponding to the second tag name in the tag name record table, the process of determining the tag query result further includes:

[0041] Find the set identifier corresponding to the name of the second tag set from the tag set record table;

[0042] The step of searching for the tag name identifier corresponding to the second tag name in the tag name record table includes:

[0043] Based on the set identifier and the second tag name, query the tag name record table to obtain the tag name identifier corresponding to the second tag name.

[0044] In one possible implementation, determining the tag query result based on the second tag value includes:

[0045] The tag query result is obtained by combining the set identifier, the tag name identifier, the tag target identifier, and the second tag value.

[0046] This application provides a lake warehouse metadata management system, which includes a metadata service gateway, at least one metadata storage system, and a metadata tagging service system;

[0047] The metadata service gateway is used to route requests sent by the data analysis engine to the metadata tag service system or any of the metadata storage systems;

[0048] The metadata storage system is used to manage at least one piece of metadata according to the request;

[0049] The metadata tagging service system is used to manage tags corresponding to some or all of the metadata in the at least one metadata according to the request.

[0050] In one possible implementation, the metadata tagging service system is used to execute the lake warehouse metadata tagging creation method provided in this application, or to execute the lake warehouse metadata tagging creation method provided in this application.

[0051] This application provides a lake warehouse metadata tag creation device, including:

[0052] The first receiving unit is configured to receive a metadata tag creation request sent by the metadata service gateway; the metadata tag creation request is determined by the metadata service gateway based on the metadata tag creation parameters provided by the data engine; the metadata tag creation parameters include a first tag name, a first tag value, and a description of the first tag's target object;

[0053] The tag creation unit is used to execute tag creation logic based on the metadata tag creation parameters carried in the metadata tag creation request.

[0054] This application provides a lake warehouse metadata tag query device, which is used to query tags created using the lake warehouse metadata tag creation method provided in this application;

[0055] The device includes:

[0056] The second receiving unit is used to receive a metadata tag query request sent by the metadata service gateway; the metadata tag query request is determined by the metadata service gateway based on the metadata tag query parameters provided by the data engine;

[0057] The tag query unit is used to perform tag query processing based on the metadata tag query parameters carried in the metadata tag query request, and obtain the tag query result corresponding to the metadata tag query request.

[0058] The query feedback unit is used to send the tag query results to the metadata service gateway, and the metadata service gateway is used to feed back the tag query results to the data engine.

[0059] This application provides an electronic device, the device comprising: a processor and a memory;

[0060] The memory is used to store instructions or computer programs;

[0061] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device executes the lake warehouse metadata tag creation method or the lake warehouse metadata tag query method provided in this application.

[0062] This application provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform the lake warehouse metadata tag creation method or the lake warehouse metadata tag query method provided in this application.

[0063] This application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the lake warehouse metadata tag creation method or the lake warehouse metadata tag query method provided in this application.

[0064] Compared with related technologies, this application has at least the following advantages:

[0065] In the technical solution provided in this application, after the metadata service gateway determines the metadata tag creation request based on the metadata tag creation parameters (such as tag collection name, tag name, tag target, tag value, etc.) provided by a data engine (such as Spark), the metadata service gateway sends the metadata tag creation request to the metadata tag service system. This enables the metadata tag service system to execute the tag creation logic based on the metadata tag creation parameters carried in the metadata tag creation request, thereby fulfilling the metadata tag creation requirements described by the metadata tag creation request. This allows for tagging requirements for one or more metadata items, enabling finer-grained tagging processing. This effectively avoids the defects caused by adding new parameters to the metadata database, metadata table, or metadata partition properties to achieve tagging processing, thus effectively improving the tagging effect. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A schematic diagram of a metadata service architecture provided in an embodiment of this application;

[0068] Figure 2 This application provides a schematic diagram of the structure of a lake warehouse metadata management system.

[0069] Figure 3 A schematic diagram of another metadata service architecture provided in an embodiment of this application;

[0070] Figure 4 A flowchart illustrating a method for creating lake warehouse metadata tags, provided in an embodiment of this application;

[0071] Figure 5A flowchart illustrating a method for querying metadata tags in a lake warehouse, as provided in this application embodiment;

[0072] Figure 6 A schematic diagram illustrating a tag data storage method provided in an embodiment of this application;

[0073] Figure 7 A schematic diagram of a lake warehouse metadata tag creation device provided in this application embodiment;

[0074] Figure 8 This is a schematic diagram of the structure of a lake warehouse metadata tag query device provided in an embodiment of this application;

[0075] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] Research has found that in some application scenarios (e.g., using...) Figure 1 In the architecture scenario shown, to meet the tagging requirements mentioned above, tag information can be recorded by adding new parameters to the metadata database, metadata table, or metadata partition properties. However, this tagging method has the following drawbacks: ① The permissions for these tags are the same as the metadata permissions; anyone with metadata permissions can operate on the tags, making it difficult to achieve business isolation, and metrics can affect each other. ② It is only suitable for coarse-grained tagging of databases, tables, and partitions, and cannot support tagging of rows or columns, resulting in limited tagging granularity; moreover, for unified metadata services that support multiple data sources, it cannot tag specific data types. ③ A large amount of tag data can easily put significant pressure on the stability and performance of the metadata storage system, affecting all users.

[0077] Based on the above findings, in order to improve the tagging effect, this application provides a method for creating metadata tags in a lake warehouse. This method includes: after a metadata service gateway determines a metadata tag creation request based on metadata tag creation parameters (such as tag collection name, tag name, tag target, tag value, etc.) provided by a data engine (e.g., Spark engine), the metadata service gateway sends the metadata tag creation request to a metadata tag service system. This allows the metadata tag service system to execute tag creation logic based on the metadata tag creation parameters carried in the request, thereby fulfilling the metadata tag creation requirements described by the request. This enables tagging of one or more metadata items, achieving finer-grained tagging processing. This effectively avoids the defects caused by adding new parameters to the metadata database, metadata tables, or metadata partition properties for tagging, thus effectively improving the tagging effect.

[0078] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0079] To better understand the technical solution provided in this application, the lake warehouse metadata management system provided in this application will be described below with reference to some accompanying drawings. For example... Figure 2 As shown in the embodiments of this application, the Lakeware Metadata Management System 200 may include a metadata service gateway 201, at least one metadata storage system 202, and a metadata tagging service system 203. The metadata service gateway routes requests sent by the data analysis engine to the metadata tagging service system 203 or any of the metadata storage systems 202; the metadata storage system 202 manages at least one metadata based on the request; and the metadata tagging service system 203 manages tags corresponding to some or all of the at least one metadata based on the request. For ease of understanding, the data analysis engine, metadata service gateway 201, metadata storage system 202, and metadata tagging service system 203 are described below in sequence.

[0080] For the data engine mentioned above, it refers to an engine that can use the Lakehouse Metadata Management System 200 to implement certain processing procedures (such as metadata processing, metadata tagging, etc.) so that the data engine can be used to represent the upper-level services of the Lakehouse Metadata Management System 200 (such as clients, etc.), thereby enabling the data engine to use the Lakehouse Metadata Management System 200 to implement certain processing procedures through certain methods (such as interface calls, etc.).

[0081] Furthermore, this application does not limit the implementation method of the data engine mentioned above. For example, the data engine can be implemented using any existing or future engine that can access the Lakewareh metadata management system 200 (e.g., any data analysis engine, any data query engine, or any data calculation engine). Therefore, in one possible implementation, when the metadata storage system 202 in the Lakewareh metadata management system 200 is implemented using HiveMetastore, the data engine can be implemented using the HiveServer2 engine. This allows the data engine to access the metadata service gateway 201 via an interface call. The metadata service gateway 201 can then route requests provided by the data engine to the metadata tagging service system 203 or any metadata storage system 202 in the Lakewareh metadata management system 200, and the metadata tagging service system 203 or the metadata storage system 202 will process the requests to implement metadata-related processes.

[0082] It should be noted that this application does not limit the implementation of the HiveServer2 engine mentioned above. For example, the HiveServer2 engine may include one or more of the following: at least one Structured Query Language (SQL) engine, at least one batch stream processing engine, and at least one intelligent analysis platform engine.

[0083] It should also be noted that this application does not limit the implementation method of the SQL engine mentioned above. For example, the SQL engine can be implemented using any existing or future SQL engine (such as Hive or Presto). Similarly, this application does not limit the implementation method of the batch / stream processing engine mentioned above. For example, the batch / stream processing engine can be implemented using any existing or future batch / stream processing engine (such as Spark or Flink). Furthermore, this application does not limit the implementation method of the intelligent analysis platform engine mentioned above. For example, the intelligent analysis platform engine can be implemented using any existing or future intelligent analysis platform engine, such as a Business Intelligence (BI) analysis platform engine.

[0084] Furthermore, this application does not limit the implementation method of the requests sent by the data engine mentioned above. For example, in some application scenarios (such as adding, deleting, or modifying metadata), the requests sent by the data engine may refer to metadata processing requests. These metadata processing requests are used to request certain processing (such as adding, deleting, or modifying) of one or more metadata items; and this application does not limit the implementation method of such metadata processing requests. As another example, in some application scenarios (such as tagging metadata), the requests sent by the data engine may be metadata tagging requests. These metadata tagging requests are used to request tagging processing of one or more metadata items; and this application does not limit the implementation method of such metadata tagging requests.

[0085] Furthermore, the aforementioned data engine and the metadata service gateway 201 in the aforementioned Lake Warehouse metadata management system 200 can communicate with each other; and this application does not limit the communication method of this data communication. For example, it can be implemented using any existing or future method that enables data communication between the data engine and the metadata service gateway 201. For instance, in some application scenarios, the data engine can access the metadata service gateway 201 by calling an interface, so that the metadata service gateway 201 can route the request provided by the data engine through the interface to the metadata tag service system 203 or any metadata storage system 202 in the Lake Warehouse metadata management system 200, and the metadata tag service system 203 or the metadata storage system 202 can then process the request accordingly.

[0086] Based on the above information about the data engine, it can be used to represent the client of the aforementioned Lake Warehouse Metadata Management System 200. This allows the data engine to provide users with metadata-related services (such as metadata deletion and metadata tagging services) through the Lake Warehouse Metadata Management System 200. Specifically, the implementation process for these metadata-related services is as follows: First, the data engine generates a request based on user actions (such as a metadata processing request or a metadata tagging request). Then, the data engine sends this request to the metadata service gateway 201 within the Lake Warehouse Metadata Management System 200 via a software development kit (SDK) already deployed within the data engine. The metadata service gateway 201 then routes the request to the metadata tagging service system 203 or any metadata storage system 202 within the Lake Warehouse Metadata Management System 200, enabling the metadata tagging service system 203 or the metadata storage system 202 to process the request. This fulfills the metadata service requirements described by the user's actions (such as deleting metadata or tagging metadata).

[0087] Regarding the metadata service gateway 201 mentioned above (for example, Figure 3 As shown in the metadata service gateway, the metadata service gateway 201 can be used to route the request sent by the data analysis engine to the metadata tag service system 203 or any metadata storage system 202, so that the metadata tag service system 203 or the metadata storage system 202 can process the request. For ease of understanding, two scenarios are described below.

[0088] Scenario 1: In some application scenarios (such as adding, deleting, or modifying metadata), when the data analysis engine generates a metadata processing request based on user operations, and this metadata processing request requests data processing for one or more metadata (such as metadata deletion, metadata modification, or metadata addition), the working principle of the metadata service gateway 201 can be as follows: After receiving the metadata processing request sent by the data analysis engine, the metadata service gateway 201 can first determine a metadata storage system 202 from at least one metadata storage system 202 to process the metadata processing request, based on a pre-built routing relationship, as the target system; then, the metadata service gateway 201 sends the metadata processing request to the target system so that the target system can subsequently process the metadata processing request. The routing relationship is pre-defined for the metadata service gateway 201, enabling it to guide the gateway in selecting which downstream system (e.g., metadata tagging service system 203 or a metadata storage system 202) to use to process metadata-related requests received by the gateway. The target system represents the metadata storage system selected by the gateway for processing the metadata processing request.

[0089] Based on the above content, in some application scenarios, the metadata service gateway 201 can be used to route the metadata processing request sent by the data analysis engine to the target system in at least one metadata storage system 202, so that the target system can process the request.

[0090] Scenario 2: In some application scenarios (such as tagging metadata), when the data analysis engine generates a metadata tagging request based on user operations, and the metadata tagging request is used to request tagging processing for one or more metadata, the working principle of the metadata service gateway 201 can be as follows: After receiving the metadata tagging request sent by the data analysis engine, the metadata service gateway 201 can send the metadata tagging request to the metadata tagging service system 203 according to the pre-built routing relationship, so that the metadata tagging service system 203 can process the metadata processing request subsequently.

[0091] Based on the above content, in some application scenarios, the above-mentioned metadata service gateway 201 can be used to route the metadata tagging request sent by the above-mentioned data analysis engine to the above-mentioned metadata tagging service system 203, so that the metadata tagging service system 203 can process the request.

[0092] Furthermore, this application does not limit the implementation method of the metadata service gateway 201 mentioned above. For example, it can be implemented using any existing or future metadata service gateway. Also, in some application scenarios, when all metadata storage systems 202 in the aforementioned Lake Warehouse metadata management system 200 are implemented using Hive Metastore, in order to better improve compatibility, the metadata service gateway 201 can also be implemented using Hive Metastore, thus enabling the metadata service gateway 201 to be fully compatible with the Hive Metastore interface.

[0093] Based on the above content regarding the metadata service gateway 201, it can be seen that the metadata service gateway 201 in the above-mentioned Hucang metadata management system 200 is mainly used to forward information sent by some upstream services (such as at least one data engine) to the corresponding processing system (such as the metadata tag service system 203 or a metadata storage system 202), which is beneficial to improving the processing effect of metadata.

[0094] Regarding the metadata storage system 202 mentioned above, it is used to manage at least one piece of metadata (e.g., deleting metadata, adding metadata, modifying metadata, etc.). For example, the metadata storage system 202 can be used to manage at least one piece of metadata based on a request (e.g., a data processing request) sent by the metadata service gateway 201 mentioned above. Therefore, in one possible implementation, the metadata storage system 202 can be used to process the metadata processing request sent by the metadata service gateway 201 accordingly (e.g., metadata deletion processing, metadata addition processing, metadata modification processing, etc.) to achieve the purpose of managing one or more pieces of metadata, thus improving the metadata management effect.

[0095] Furthermore, this application does not limit the implementation of the metadata storage system 202 described above. For example, it can be implemented using any existing or future system capable of managing metadata (such as Hive Metastore).

[0096] Based on the metadata storage system 202 mentioned above, for any metadata storage system 202 in the Lake Warehouse Metadata Management System 200, the metadata storage system 202 can be mainly used to manage Lake Warehouse metadata (e.g., adding new metadata, deleting existing metadata, modifying existing metadata, etc.), so that the metadata storage system 202 can realize the relevant management of metadata by processing the requests sent by the metadata service gateway 201 in the Lake Warehouse Metadata Management System 200, which is conducive to improving the metadata management effect.

[0097] Regarding the metadata tagging service system 203 mentioned above, it is used to manage tags corresponding to some or all of the metadata in at least one metadata set. For example, the metadata tagging service system 203 can be used to manage tags corresponding to some or all of the metadata in the at least one metadata set based on requests sent by the metadata service gateway 201 (e.g., metadata tagging requests). Therefore, in one possible implementation, the metadata tagging service system 203 can be used to process the metadata tagging requests sent by the metadata service gateway 201 (e.g., metadata tagging processing) to achieve the purpose of managing the relevant tags of one or more metadata sets, thus improving the effectiveness of metadata tagging management.

[0098] Furthermore, this application does not limit the implementation method of the metadata tagging service system 203 mentioned above. For example, when each metadata storage system 202 in the aforementioned lake warehouse metadata management system 200 is implemented using Hive Metastore, and the metadata service gateway 201 in the lake warehouse metadata management system 200 is also implemented using Hive Metastore, in order to better reduce the deployment difficulty, the metadata tagging service system 203 can be implemented using Hive Metastore. This is beneficial to reduce the cost required to deploy the metadata tagging service system 203 in the lake warehouse metadata management system 200, thereby facilitating access to tag services at a lower cost without affecting the service stability of each metadata storage system 202 in the lake warehouse metadata management system 200.

[0099] Furthermore, this application does not limit the working principle of the aforementioned metadata tagging service system 203. For example, to better improve tag management, the metadata tagging service system 203 can adopt any implementation of the lake warehouse metadata tagging creation method provided in this application embodiment to implement tagging processing, and adopt any implementation of the lake warehouse metadata tagging query method provided in this application embodiment to implement tag query processing. Therefore, in one possible implementation, the metadata tagging service system 203 can be used to execute any implementation of the lake warehouse metadata tagging creation method provided in this application embodiment, or execute any implementation of the lake warehouse metadata tagging creation method provided in this application embodiment. It should be noted that the relevant content of the lake warehouse metadata tagging creation method is described below.

[0100] Based on the aforementioned metadata tagging service system 203, it can be understood that the metadata tagging service system 203 in the aforementioned Hucang metadata management system 200 is mainly used for tag management of Hucang metadata (such as tagging, tag querying, etc.). This allows the metadata tagging service system 203 to manage metadata tags by processing requests sent by the metadata service gateway 201 in the Hucang metadata management system 200, thereby improving the effectiveness of metadata tag management.

[0101] Based on the above content regarding the Lake Warehouse Metadata Management System 200, it is known that the Lake Warehouse Metadata Management System 200 can implement metadata tag management by means of a metadata tag service system 203 that is completely independent of all metadata storage systems 202. This ensures that there is no mutual interference between the processing operations related to metadata tags and the processing operations related to metadata itself. This facilitates the access to the metadata tag service at a lower cost without affecting the service stability of the metadata storage system 202. As a result, it can effectively avoid the defects caused by adding new parameters to the attribute information (properties) of the metadata database, metadata table, or metadata partition to implement tagging processing, thereby effectively improving the tagging effect.

[0102] Furthermore, to better understand the technical solution provided in this application, the method for creating lake warehouse metadata tags provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 4 As shown, the lake warehouse metadata tag creation method provided in this application embodiment includes S401-S402 below. Wherein, the Figure 4 A flowchart illustrating a method for creating lake warehouse metadata tags, provided in an embodiment of this application.

[0103] S401: The metadata tag service system receives a metadata tag creation request sent by the metadata service gateway; the metadata tag creation request is determined by the metadata service gateway based on the metadata tag creation parameters provided by the data engine; the metadata tag creation parameters include a first tag name, a first tag value, and a description of the first tag's target object.

[0104] The metadata tag creation request refers to a request sent by the metadata service gateway mentioned above to request tagging of one or more metadata items.

[0105] Based on the foregoing, in one possible implementation, the metadata tag creation request can be used to request the creation of tags corresponding to one or more metadata items. This metadata item is managed by the metadata storage system. Furthermore, the metadata service gateway routes the metadata processing request sent by the data engine to the metadata storage system, enabling the metadata storage system to execute the data processing logic corresponding to the metadata processing request. This data processing logic refers to the logic executed by the data storage system when processing the metadata processing request; however, this application does not limit the data processing logic. For example, the data processing logic can refer to the processing logic pre-configured for the data storage system to respond to the metadata processing request.

[0106] Furthermore, regarding the metadata tag creation request mentioned above, this request is determined by the metadata service gateway based on the metadata tag creation parameters provided by the data engine. This ensures that the metadata tag creation request carries these parameters, allowing it to specify which metadata is being tagged. The metadata tag creation parameters are those provided by the data engine and used when tagging one or more metadata items (e.g., tag set name, tag name, tag value, tag target, etc.). These parameters represent the elements required for tag creation (e.g., tag name, tag value, and tag target; or tag set name, tag name, tag value, and tag target).

[0107] Furthermore, this application does not limit the implementation of the metadata tag creation parameters in the above paragraph. For ease of understanding, the following description combines two scenarios.

[0108] Scenario 1: In some application scenarios (such as when the number of tags is small), if the three elements constituting the tag of the metadata include the tag name, tag value, and the object to which the tag applies, then the relevant tags of the metadata can be stored directly according to these three elements, without the need to store them by constructing a tag collection (such as a library).

[0109] Based on the relevant content in Situation 1 above, in one possible implementation, the metadata tag creation parameters mentioned above may include a first tag name, a first tag value, and a first tag target object description. The first tag target object description refers to the information used to describe the tag target object when tagging one or more metadata; and the first tag target object description can be used to describe the first tag target object. The first tag target object refers to the object that needs to be tagged (that is, the object described by the first tag target object description); and this application does not limit the first tag target object; for example, the first tag target object may include one or more metadata. The first tag name refers to the tag name used when tagging the first tag target object. The first tag value refers to the tag value configured for the first tag name when tagging the first tag target object.

[0110] Furthermore, this application does not limit the implementation method of the description information of the first tag target object in the preceding paragraph. For example, in some application scenarios, the description information of the first tag target object may include one or more of the following: data source name (namespace), database name (db_name), table name (tbl_name), column name (col_name), row expression (row_expression), and partition name (part_name). The data source name is used to identify the data source to which the first tag target object belongs; and this application does not limit the implementation method of the data source name. For example, the data source name can be hive or clickhouse. The database name indicates which database the first tag target object resides in; the table name indicates which table the first tag target object resides in; the column name indicates which columns the first tag target object occupies in the table; the row expression indicates which rows the first tag target object occupies in the table; and this application does not limit the row expression. For example, the row expression can be implemented using an expression similar to "score=90". The partition name is used to indicate which partition the first label target is located in; and this application does not limit the partition name. For example, the partition name can be implemented using a string like date='20230612'.

[0111] Scenario 2: In some application scenarios (such as when there are a large number of tags), the relevant tags of metadata may need to be stored by building a tag collection (such as a library), which is beneficial to improving the tag storage effect and the subsequent tag query efficiency.

[0112] Based on the relevant content in scenario 2 above, in one possible implementation, the metadata tag creation parameters mentioned above may include a first tag set name, a first tag name, a first tag value, and a first tag target description. Here, the first tag set name refers to the name of the first tag set, enabling it to indicate which set the tag composed of the first tag name, the first tag value, and the first tag target description belongs to. The first tag set refers to the set described by the first tag set name; and the first tag set is used to record one or more tag data.

[0113] Furthermore, this application does not limit the implementation method used by the data engine to provide metadata tag creation parameters as described above. For ease of understanding, the following explanation is provided with reference to examples.

[0114] Example 1: In some application scenarios, the aforementioned data engine can provide data tag creation parameters to the aforementioned metadata service gateway via a request sending method. Based on this, in one possible implementation, the data engine can be used to send a metadata tagging request to the metadata service gateway, and this metadata tagging request carries the data tag creation parameters, so that the metadata service gateway can extract the data tag creation parameters from the metadata tagging request. The metadata tagging request is used to request tagging processing for one or more metadata items; and this application does not limit the implementation method of the metadata tagging request.

[0115] Example 2: In some application scenarios, the data engine mentioned above can directly provide data tag creation parameters to the metadata service gateway using a certain communication protocol (e.g., the Thrift protocol). Based on this, in one possible implementation, the data engine can be used to pass these data tag creation parameters as a communication message (e.g., a Thrift message) directly to the metadata service gateway through the SDK already deployed in the data engine.

[0116] Based on the relevant content of S401 above, in some application scenarios, when at least one data engine has been connected to the metadata service gateway in the above-mentioned Hucang metadata management system, if a data engine sends a metadata tagging request to the metadata service gateway, the metadata service gateway can determine the metadata tag creation request based on the metadata tagging request (for example, directly determine the metadata tagging request as the metadata tag creation request, etc.), and send the metadata tag creation request to the metadata tag service system so that the metadata tag service system can subsequently perform tag creation processing based on the metadata tag creation parameters carried by the metadata tag creation request.

[0117] S402: The metadata tag service system executes the tag creation logic based on the metadata tag creation parameters carried in the metadata tag creation request.

[0118] The tag creation logic refers to the logic pre-configured for the metadata tag service system mentioned above, which is used to perform tag creation processing based on the metadata tag creation parameters.

[0119] In addition, this application does not limit the logic for creating the above tags. For ease of understanding, the following explanation combines two scenarios.

[0120] In some application scenarios (such as when the number of tags is small), when the metadata tag creation request carries metadata tag creation parameters including the first tag name, the first tag value, and the description information of the first tag's target object, the above tag creation logic may specifically include steps 11-13 below.

[0121] Step 11: The metadata tagging service system writes the tag name record content corresponding to the first tag name into the tag name record table. The tag name record content includes the tag name identifier corresponding to the first tag name.

[0122] The tag name record table is used to record the relevant content of the tag names involved in some metadata tags.

[0123] Furthermore, this application does not limit the implementation method of the aforementioned tag name record table. For example, each record in the tag name record table can be implemented using a binary tuple of (tag name, tag name identifier corresponding to the tag name). Here, the tag name identifier is used to uniquely identify the tag name; and this application does not limit the implementation method of the tag name identifier. For example, it can be implemented using an identification code (ID) corresponding to the tag name.

[0124] Furthermore, this application does not limit the storage location of the aforementioned tag name record table. For example, the tag name record table can be stored in the tag data storage space corresponding to the aforementioned metadata tag service system (e.g., Figure 3 (as shown in the tag database).

[0125] Furthermore, regarding the first tag name mentioned above, the tag name record content corresponding to the first tag name refers to the record content that needs to be written when recording the relevant content of the first tag name in the tag name record table above, so that the "tag name record content corresponding to the first tag name" can represent a record related to the first tag name existing in the tag name record table; and the "tag name record content corresponding to the first tag name" can at least include the tag name identifier corresponding to the first tag name, so that the first tag value corresponding to the first tag name can be queried based on the tag name identifier in the future. Among them, the tag name identifier corresponding to the first tag name is used to uniquely identify the first tag name.

[0126] Based on the relevant content of step 11 above, in some application scenarios, for the metadata tag service system mentioned above, after receiving the metadata tag creation request sent by the metadata service gateway, the metadata tag service system can first create a tag name identifier corresponding to the first tag name carried in the metadata tag creation request; then, based on the tag name identifier corresponding to the first tag name, write the tag name record content corresponding to the first tag name into the tag name record table, so that the tag name record content includes the tag name identifier corresponding to the first tag name, so that the first tag value corresponding to the first tag name can be queried based on the tag name identifier in the future.

[0127] Step 12: The metadata tagging service system writes the tag target record content corresponding to the first tag target description information into the tag target record table. The tag target record content includes the tag target identifier corresponding to the first tag target description information.

[0128] The tag-based object record table is used to record metadata related to tags.

[0129] Furthermore, this application does not limit the implementation method of the tag target record table mentioned above. For example, each record in the tag target record table can be implemented using a binary tuple of (tag target description information, tag target identifier corresponding to the tag target description information). The tag target description information is used to describe one or more metadata, and its implementation method is similar to that of the "first tag target description information" mentioned above; for the sake of brevity, it will not be repeated here. The tag target identifier is used to uniquely identify the tag target described by the tag target description information; and this application does not limit the implementation method of the tag target identifier. For example, it can be implemented using the ID corresponding to the tag target described by the tag target description information.

[0130] Furthermore, this application does not limit the storage location of the aforementioned tag target record table. For example, the tag target record table can be stored in the tag data storage space corresponding to the aforementioned metadata tag service system (e.g., Figure 3 (as shown in the tag database).

[0131] Furthermore, regarding the description information of the first tag's target object mentioned above, if this description information is used to describe the characteristics of the first tag's target object, then the tag's target object record content corresponding to this description information refers to the record content that needs to be written when recording the relevant content of the first tag's target object in the aforementioned tag's target object record table. This ensures that the "tag's target object record content corresponding to the first tag's target object description information" can represent a record related to the first tag's target object existing in the tag's target object record table. Moreover, the "tag's target object record content corresponding to the first tag's target object description information" can at least include the tag's target object identifier corresponding to the first tag's target object description information. The tag's target object identifier is used to uniquely identify the first tag's target object.

[0132] Furthermore, in some application scenarios, to avoid the defect caused by repeatedly writing the same tag object related content into the tag object record table above, this application also provides a possible implementation of step 12 above. In this implementation, step 12 can specifically be: if there is no tag object record content corresponding to the first tag object description information above in the tag object record table above, then write the tag object record content corresponding to the first tag object description information into the tag object record table.

[0133] It should be noted that this application does not limit the method of obtaining the conclusion in the preceding paragraph that "there is no tag target record in the tag target record table corresponding to the first tag target description information". For example, it can be specifically: for each existing record in the tag target record table, determine whether the first tag target description information appears in the record. When it is determined that the first tag target description information does not appear in any of the existing records in the tag target record table, it can be determined that there is no tag target record in the tag target record table corresponding to the first tag target description information. Alternatively, this conclusion can also be obtained by: first, creating a tag target identifier corresponding to the first tag target description information; then, for each existing record in the tag target record table, determining whether the tag target identifier appears in the record. When it is determined that the tag target identifier does not appear in any of the existing records in the tag target record table, it can be determined that there is no tag target record in the tag target record table corresponding to the first tag target description information.

[0134] Based on the above two paragraphs, in some application scenarios, for the metadata tagging service system mentioned above, after receiving the metadata tag creation request sent by the metadata service gateway, the metadata tagging service system can first create a tag target identifier corresponding to the first tag target description information carried in the metadata tag creation request; then, it can determine whether there is a record in the tag target record table that includes the tag target identifier. If it exists, it can be determined that there is a tag target record in the tag target record table that corresponds to the first tag target description information, and thus it can be determined that the tag target record table has already been written with the tag target description information. Therefore, to avoid duplicate writing, step 13 below can be directly executed. However, if there is no record in the tag target record table that includes the tag target identifier, it can be determined that there is no tag target record in the tag target record table that corresponds to the first tag target description information. Therefore, the tag target record in the tag target record table that corresponds to the first tag target description information can be directly written to the tag target record table so that the tag target record content includes the tag target identifier.

[0135] Based on the relevant content of step 12 above, in some application scenarios, for the metadata tagging service system mentioned above, after the metadata tagging service system receives the metadata tag creation request sent by the metadata service gateway, the metadata tagging service system can first create a tag target identifier corresponding to the first tag target description information carried in the metadata tag creation request; then, based on the tag target identifier, write the tag target record content corresponding to the first tag target description information into the tag name record table, so that the tag target record content includes the tag target identifier corresponding to the first tag target description information, so that the first tag value corresponding to the first tag target description information can be queried based on the tag target identifier in the future.

[0136] Step 13: The metadata tag service system writes the tag value record content corresponding to the first tag value into the tag value record table. The tag value record content is determined based on the first tag value, the tag name identifier corresponding to the first tag name above, and the tag target identifier corresponding to the tag target description information above.

[0137] The tag value record table is used to record the relevant content of tag values ​​involved in some metadata tags.

[0138] Furthermore, this application does not limit the implementation method of the tag value record table mentioned above. For example, each record in the tag value record table can be implemented using a triplet format: (tag value, tag name identifier associated with the tag value, tag target identifier associated with the tag value). Here, the "tag name identifier associated with the tag value" uniquely identifies the tag name corresponding to the tag value. The "tag target identifier associated with the tag value" uniquely identifies the tag target corresponding to the tag value.

[0139] Furthermore, this application does not limit the storage location of the aforementioned tag value record table. For example, the tag value record table can be stored in the tag data storage space corresponding to the aforementioned metadata tag service system (e.g., Figure 3 (as shown in the tag database).

[0140] Furthermore, regarding the first tag value mentioned above, the tag value record content corresponding to the first tag value refers to the record content that needs to be written when recording the relevant content of the first tag value in the tag value record table above, so that the "tag value record content corresponding to the first tag value" can represent a record related to the first tag value in the tag value record table; and the "tag value record content corresponding to the first tag value" is determined based on the first tag value, the tag name identifier corresponding to the first tag name mentioned above, and the tag target identifier corresponding to the tag target description information mentioned above, so that the tag value record content corresponding to the first tag value can at least include the triplet (the first tag value, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the tag target description information), so that the first tag value can be quickly retrieved from the tag value record table in the future based on the tag name identifier and the tag target identifier.

[0141] Based on the relevant content of steps 11 to 13 above, in some application scenarios, for the metadata tagging service system mentioned above, after the metadata tagging service system receives a metadata tag creation request sent by the metadata service gateway, if the metadata tag creation request carries a first tag name, a first tag value, and a description of the first tag's target object, the working principle of the metadata tagging service system can include: creating a tag name identifier corresponding to the first tag name based on the first tag name, and writing the tag name record content corresponding to the first tag name into the tag name record table so that the tag name record content includes the tag name identifier; creating a tag target object identifier corresponding to the first tag's target object description information based on the first tag's target object description information, and writing the tag target object record content corresponding to the first tag's target object description information into the tag name record table so that the tag target object record content includes the tag target object identifier; then, writing the tag value record content corresponding to the first tag value into the tag value record table based on the tag name identifier, the tag target object identifier, and the first tag value, thus realizing tag creation processing.

[0142] Scenario 2: In some application scenarios (such as scenarios with a large number of tags), when the metadata tag creation request carried by the above metadata tag creation request includes the first tag set name, the first tag name, the first tag value, and the description information of the first tag's target object, the above tag creation logic may specifically include steps 21-25 below.

[0143] Step 21: The metadata tag service system writes the tag set record content corresponding to the first tag set name into the tag set record table. The tag set record content includes the set identifier corresponding to the first tag set name.

[0144] The tag set record table is used to record the relevant content of the tag set to which a certain metadata tag belongs.

[0145] Furthermore, this application does not limit the implementation method of the tag set record table mentioned above. For example, each record in the tag set record table can be implemented using a binary tuple of (tag set name, set identifier corresponding to the tag set name). Here, the tag set name describes a tag set. The set identifier uniquely identifies the tag set described by the tag set name; and this application does not limit the implementation method of the set identifier, for example, it can be implemented using the ID corresponding to the tag set described by the tag set name.

[0146] Furthermore, this application does not limit the storage location of the aforementioned tag set record table. For example, the tag set record table can be stored in the tag data storage space corresponding to the aforementioned metadata tag service system (e.g., Figure 3 (as shown in the tag database).

[0147] Furthermore, regarding the first tag set name mentioned above, if the first tag set name is used to describe a first tag set, then the tag set record content corresponding to the first tag set name refers to the record content that needs to be written when recording the relevant content of the first tag set in the tag set record table mentioned above, so that the "tag set record content corresponding to the first tag set name" can represent a record related to the first tag set existing in the tag set record table; and the "tag set record content corresponding to the first tag set name" includes the set identifier corresponding to the first tag set name. Here, the first tag set refers to the tag set described by the first tag set name; the set identifier corresponding to the first tag set name is used to uniquely identify the first tag set.

[0148] Based on the relevant content of step 21 above, in some application scenarios, for the metadata tag service system mentioned above, after the metadata tag service system receives the metadata tag creation request sent by the metadata service gateway, the metadata tag service system can first create a set identifier corresponding to the first tag set name carried in the metadata tag creation request; then, based on the set identifier, write the tag set record content corresponding to the first tag set name into the tag name record table, so that the tag set record content includes the set identifier corresponding to the first tag set name.

[0149] Step 22: The metadata tagging service system determines the tag name record content corresponding to the first tag name based on the set identifier corresponding to the first tag set name above and the first tag name above.

[0150] It should be noted that this application does not limit the implementation of step 22 above. For example, in one possible implementation, step 22 may specifically include steps 221-222 below.

[0151] Step 221: The metadata tagging service system creates a tag name identifier corresponding to the first tag name mentioned above, so that the tag name identifier can be used to uniquely identify the first tag name.

[0152] It should be noted that this application does not limit the implementation of step 221 above. For example, step 221 may specifically be: the metadata tagging service system creates a tag name identifier corresponding to the first tag name based on the first tag name above, so that the tag name identifier can represent the characteristics of the first tag name. Alternatively, step 221 may specifically be: the metadata tagging service system creates a tag name identifier corresponding to the first tag name based on the set identifier corresponding to the first tag set name above and the first tag name, so that the tag name identifier can represent the characteristics of the set identifier and the characteristics of the first tag name. Yet another example is: step 221 may specifically be: randomly generate an ID as the tag name identifier corresponding to the first tag name.

[0153] It should be noted that this application does not limit the implementation of the step "creating a tag name identifier corresponding to the first tag name based on the set identifier corresponding to the first tag set name and the first tag name mentioned above" in the preceding paragraph. For example, this step can specifically be: creating a tag name identifier corresponding to the first tag name based on the set identifier corresponding to the first tag set name and the first tag name mentioned above, so that the tag name identifier is implemented using the tuple (the set identifier corresponding to the first tag set name, the ID corresponding to the first tag name). Alternatively, this step can specifically be: merging the set identifier corresponding to the first tag set name and the first tag name to obtain the tag name identifier corresponding to the first tag name.

[0154] Step 222: Based on the first tag name above, the tag name identifier corresponding to the first tag name, and the set identifier corresponding to the first tag set name above, determine the tag name record content corresponding to the first tag name, so that the tag name record content includes the first tag name, the tag name identifier corresponding to the first tag name, and the set identifier.

[0155] It should be noted that this application does not limit the implementation of step 222 above. For example, it can specifically be: determining the tag name record content corresponding to the first tag name based on the triple constructed by the first tag name above, the tag name identifier corresponding to the first tag name, and the set identifier corresponding to the first tag set name above, so that the tag name record content can include the triple, such as a triple similar to (the set identifier corresponding to the first tag set name, the first tag name, the tag name identifier corresponding to the first tag name), so that after the tag name record content is written into the tag name record table above, the tag name record content can be quickly located from the tag name record table above based on the set identifier and the first tag name.

[0156] Based on the relevant content of steps 221 to 222 above, it can be seen that in some application scenarios, for the metadata tag service system above, after the metadata tag service system obtains the set identifier corresponding to the first tag set name above, the metadata tag service system can determine the tag name record content corresponding to the first tag name based on the set identifier, the first tag name above, and the tag name identifier corresponding to the first tag name. This ensures that the tag name record content includes the first tag name, the tag name identifier corresponding to the first tag name, and the set identifier. This allows the system to quickly locate the tag name record content from the tag name record table above based on the set identifier and the first tag name after the tag name record content is written into the tag name record table above.

[0157] Step 23: The metadata tagging service system writes the tag name record content corresponding to the first tag name into the tag name record table. The tag name record content includes the tag name identifier corresponding to the first tag name.

[0158] It should be noted that the relevant content of step 23 can be found in step 11 above.

[0159] Step 24: The metadata tagging service system writes the tag target record content corresponding to the first tag target description information into the tag target record table. The tag target record content includes the tag target identifier corresponding to the first tag target description information.

[0160] It should be noted that the relevant content of step 24 can be found in step 12 above.

[0161] Step 25: The metadata tag service system writes the tag value record content corresponding to the first tag value into the tag value record table. The tag value record content is determined based on the first tag value, the tag name identifier corresponding to the first tag name above, and the tag target identifier corresponding to the tag target description information above.

[0162] It should be noted that the relevant content of step 25 can be found in step 13 above.

[0163] Based on the relevant content of steps 21 to 25 above, in some application scenarios, for the metadata tag service system mentioned above, after the metadata tag service system receives a metadata tag creation request sent by the metadata service gateway, if the metadata tag creation request carries a first tag set name, a first tag name, a first tag value, and a description of the first tag's target object, then the working principle of the metadata tag service system may include: creating a set identifier corresponding to the first tag set name based on the first tag set name, and writing the tag set record content corresponding to the first tag set name into the tag name record table, so that the tag set record content includes the set identifier; then, based on the set identifier and the first tag's target object description information... For each tag name, a tag name record is created, including the tag name identifier, and written to the tag name record table. Based on the description information of the first tag's target object, a tag target object identifier is created, and written to the tag name record table, including the tag target object identifier. Then, based on the tag name identifier, the tag target object identifier, and the first tag value, a tag value record is written to the tag value record table. This completes the tag creation process.

[0164] Based on the relevant content in S401 to S402 above, it is clear that the metadata tagging service system provided in this application (e.g., Figure 2 The metadata tagging service system 203 shown is... Figure 3 The metadata tagging service system shown herein operates as follows: When the metadata service gateway determines a metadata tag creation request based on metadata tag creation parameters (such as tag collection name, tag name, tag target, tag value, etc.) provided by a data engine (e.g., Spark engine), the gateway sends the request to the metadata tagging service system. This allows the system to execute tag creation logic based on the metadata tag creation parameters carried in the request, thus fulfilling the metadata tag creation requirements described in the request. This enables tagging of one or more metadata items, allowing for finer-grained tagging processing. This effectively avoids the defects caused by adding new parameters to the metadata database, metadata tables, or metadata partition properties, thereby significantly improving tagging performance.

[0165] In addition, in some application scenarios, in order to better improve the tagging experience, this application also provides a possible implementation of the above-described lake warehouse metadata tag creation method. In this implementation, the lake warehouse metadata tag creation method not only includes S401-S402 above, but may also include steps 31-32 below.

[0166] Step 31: If the metadata tag service system detects that the above tag creation logic has been completed, the metadata tag service system generates feedback information corresponding to the above metadata tag creation request.

[0167] The feedback information is used to indicate that the metadata tag service system has completed the metadata tag creation request, so that the feedback information can indicate that the metadata tag service system has completed the metadata tag creation requirement described by the metadata tag creation request.

[0168] Furthermore, this application does not limit the implementation method of the above feedback information. For example, when the metadata tag creation parameters carried in the above metadata tag creation request include the first tag set name, the first tag name, the first tag value, and the first tag target description information, the feedback information can be implemented using a tag composed of the first tag value, the set identifier corresponding to the first tag set name, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the first tag target description information.

[0169] Based on the above, in one possible implementation, when the metadata tag creation parameters carried in the metadata tag creation request include a first tag set name, a first tag name, a first tag value, and a first tag target description, step 31 above can specifically be: if the metadata tag service system detects that the metadata tag creation logic has been completed, the metadata tag service system combines the first tag value, the set identifier corresponding to the first tag set name, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the first tag target description to obtain feedback information, so that the feedback information carries a tag composed of the first tag value, the set identifier corresponding to the first tag set name, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the first tag target description, thereby enabling the feedback information to convey which tag was created for which metadata during the processing of the metadata tag creation request.

[0170] Step 32: The metadata tagging service system sends the feedback information to the metadata service gateway, which forwards the feedback information to the data engine.

[0171] In this application, for the metadata tagging service system mentioned above, after the metadata tagging service system generates the feedback information corresponding to the metadata tag creation request, the metadata tagging service system can send the feedback information to the metadata service gateway, so that the metadata service gateway forwards the feedback information to the data engine. This allows both the metadata service gateway and the data engine to know from the feedback information that the metadata tagging service system has completed the metadata tag creation request, which helps to improve the tagging experience.

[0172] Based on the relevant content of steps 31 to 32 above, it is clear that for the metadata tagging service system provided in this application (e.g., Figure 2 The metadata tagging service system 203 shown is... Figure 3 As shown in the metadata tagging service system, this metadata tagging service system can not only process the metadata tag creation request based on the above metadata tag creation request, but also feed back the processing status of the metadata tag creation request (such as the above feedback information) to the data engine through the above metadata service gateway, so that the data engine can know in a timely manner that the metadata tagging service system has completed the metadata tag creation request, which is conducive to improving the tagging experience.

[0173] In addition, based on the above-described method for creating lake warehouse metadata tags, this application also provides a method for querying lake warehouse metadata tags, such as... Figure 5 As shown, the lake warehouse metadata tag query method provided in this application embodiment includes S501-S503 below. Wherein, Figure 5 A flowchart illustrating a method for querying lake warehouse metadata tags provided in this application embodiment.

[0174] S501: The metadata tag service system receives a metadata tag query request sent by the metadata service gateway; the metadata tag query request is determined by the metadata service gateway based on the metadata tag query parameters provided by the data engine.

[0175] The metadata tagging service system can be used not only for tag creation processing using any implementation of the lake warehouse metadata tagging creation method provided in this application, but also for querying tags created using the lake warehouse metadata tagging creation method; and for related content on the query processing, please refer to the relevant content on the lake warehouse metadata tagging query method provided in this application.

[0176] A metadata tag query request refers to a request sent by the metadata service gateway mentioned above to query the tags associated with one or more metadata. Therefore, in one possible implementation, this data tag creation request can be used to query the tags corresponding to one or more metadata.

[0177] Furthermore, regarding the metadata tag query request mentioned above, this request is determined by the metadata service gateway based on the metadata tag query parameters provided by the data engine. This ensures that the metadata tag query request carries these parameters, allowing it to indicate which metadata is being queried for which tag. The metadata tag query parameters are parameters provided by the data engine required when performing tag query processing on one or more metadata items (e.g., tag set name, tag name, tag target, etc.). These parameters are used to represent the elements required for the tag query (e.g., tag name and tag target, or tag set name, tag name, and tag target).

[0178] Furthermore, this application does not limit the implementation of the metadata tag query parameters in the above paragraph. For ease of understanding, the following description combines two scenarios.

[0179] Scenario 1: In some application scenarios (such as when the number of tags is small), if the three elements constituting the tag of the metadata include the tag name, tag value, and the object to which the tag applies, then the relevant tags of the metadata can be stored directly according to these three elements, without the need to store them by constructing a tag collection (such as a library).

[0180] Based on the relevant content of Situation 1 above, in one possible implementation, the metadata tag query parameters mentioned above may include a second tag name and a second tag target description information. The second tag target description information refers to the information used to describe the tag target object when performing tag query processing on one or more metadata; moreover, the second tag target description information can be used to describe the second tag target object. The second tag target object refers to the object that needs to be tag-queried (that is, the object described by the second tag target description information); and this application does not limit the second tag target object, for example, the second tag target object may include one or more metadata. The second tag name refers to the tag name used when performing tag query processing on the second tag target object. It should be noted that the implementation method of the second tag target description information is similar to the implementation method of the first tag target description information mentioned above, and for the sake of brevity, it will not be repeated here.

[0181] Scenario 2: In some application scenarios (such as when there are a large number of tags), the relevant tags of metadata may need to be stored by building a tag collection (such as a library), which is beneficial to improving the tag storage effect and the subsequent tag query efficiency.

[0182] Based on the relevant content in scenario 2 above, in one possible implementation, the metadata tag query parameters mentioned above may include a second tag set name, a second tag name, and a description of the second tag's target object. Here, the second tag set name refers to the name of the second tag set, used to indicate which set the tag described by the second tag name and the second tag's target object description information belongs to. The second tag set refers to the set described by the second tag set name; and the second tag set records at least one tag described by the second tag name and the second tag's target object description information.

[0183] Furthermore, this application does not limit the implementation method adopted by the data engine to provide metadata tag query parameters. For example, the implementation method adopted by the data engine to provide metadata tag query parameters is similar to the implementation method adopted by the data engine to provide metadata tag creation parameters. For the sake of brevity, it will not be described in detail here.

[0184] Based on the content related to S501 above, in some application scenarios, when at least one data engine has been connected to the metadata service gateway in the aforementioned Lake Warehouse metadata management system, if a data engine sends a metadata tag retrieval request to the metadata service gateway, the metadata service gateway can determine a metadata tag query request based on the metadata tag retrieval request (for example, directly determining the metadata tag retrieval request as the metadata tag query request), and send the metadata tag query request to the metadata tag service system, so that the metadata tag service system can subsequently perform tag query processing based on the metadata tag query parameters carried in the metadata tag query request. The metadata tag retrieval request is used to request tag query processing for one or more metadata items; and the metadata tag retrieval request carries the metadata tag query parameters. Furthermore, this application does not limit the implementation method of the metadata tag retrieval request.

[0185] S502: The metadata tag service system performs tag query processing based on the metadata tag query parameters carried in the metadata tag query request, and obtains the tag query result corresponding to the metadata tag query request.

[0186] The tag query result corresponding to the metadata tag query request refers to the processing result obtained by the metadata tag service system mentioned above through processing the metadata tag query request, so that the tag query result can represent the tag requested by the metadata tag query request.

[0187] Furthermore, this application does not limit the implementation of S502 above. For ease of understanding, the following description combines two scenarios.

[0188] In some application scenarios (such as when the number of tags is small), when the metadata tag query parameters carried in the metadata tag query request above include the second tag name and the description information of the second tag's target object, the above S502 may specifically include steps 41-44 below.

[0189] Step 41: The metadata tagging service system searches for the tag name identifier corresponding to the second tag name mentioned above in the tag name record table.

[0190] In this application, for the metadata tag service system, after receiving a metadata tag query request sent by the metadata service gateway, the metadata tag service system can query the metadata tag name record table based on the second tag name carried in the metadata tag query request to obtain the tag name identifier corresponding to the second tag name, so that the tag name identifier can be used to uniquely identify the second tag name.

[0191] Step 42: The metadata tagging service system searches for the tag target identifier corresponding to the tag target description information in the second tag target record table above.

[0192] In this application, for the metadata tagging service system, after receiving a metadata tagging query request sent by the metadata service gateway, the metadata tagging service system can query the metadata tagging object record table based on the second tagging object description information carried in the metadata tagging query request to obtain the tagging object identifier corresponding to the second tagging object description information, so that the tagging object identifier can be used to uniquely identify the second tagging object described by the second tagging object description information.

[0193] Step 43: The metadata tagging service system queries the tag value record table based on the tag name identifier corresponding to the second tag name mentioned above and the tag target identifier corresponding to the description information of the second tag target mentioned above, and obtains the second tag value.

[0194] In this application, for the metadata tagging service system, after the metadata tagging service system queries the tag name identifier corresponding to the second tag name and the tag target identifier corresponding to the description information of the second tag target, the metadata tagging service system queries the tag value record table based on the tag name identifier and the tag target identifier to obtain the second tag value, so that the second tag value can represent the tag value recorded in the tag value record table that is associated with the tag name identifier and the tag target identifier.

[0195] Step 44: Based on the second tag value above, determine the tag query result corresponding to the metadata tag query request above.

[0196] It should be noted that this application does not limit the implementation of step 44 above. For example, step 44 may specifically be: determining the second tag value above as the tag query result corresponding to the metadata tag query request above. Alternatively, step 44 may specifically be: combining the second tag value, the second tag name above, and the description information of the second tag's target object above to obtain the tag query result corresponding to the metadata tag query request, so that the tag query result can more intuitively represent what the queried tag is. Yet another example, step 44 may specifically be: combining the second tag value, the tag name identifier corresponding to the second tag name, and the tag target identifier corresponding to the description information of the second tag's target object into a tag, which serves as the tag query result corresponding to the metadata tag query request, so that the tag query result can more comprehensively represent what information was queried during the processing of the metadata tag query request.

[0197] Based on the relevant content of steps 41 to 44 above, in some application scenarios, for the metadata tag service system, after receiving a metadata tag query request sent by the metadata service gateway, if the metadata tag service system carries a second tag name and a description of the second tag's target object, the working principle of the metadata tag service system can include: First, querying the metadata tag name record table based on the second tag name to obtain the tag name identifier corresponding to the second tag name, and querying the metadata tag target object record table based on the description of the second tag's target object to obtain the tag target object identifier corresponding to the description of the second tag's target object; then, querying the tag value record table based on the tag name identifier and the tag target object identifier to obtain the second tag value; finally, combining the second tag value, the tag name identifier, and the tag target object identifier into a tag, which serves as the tag query result corresponding to the metadata tag query request, so that the tag query result can more comprehensively represent the information obtained from the data tag query request, thus improving the query experience.

[0198] In some application scenarios (such as when there are many tags), when the metadata tag query parameters carried in the metadata tag query request above include the name of the second tag set, the name of the second tag, and the description information of the object on which the second tag is applied, S502 above may specifically include steps 51-55 below.

[0199] Step 51: The metadata tag service system searches for the set identifier corresponding to the second tag set name mentioned above in the tag set record table.

[0200] In this application, for the aforementioned metadata tagging service system, after receiving a metadata tag query request sent by the aforementioned metadata service gateway, the metadata tagging service system can query the aforementioned tag set record table based on the second tag set name carried in the metadata tag query request to obtain the set identifier corresponding to the second tag set name, so that the set identifier can be used to uniquely identify the second tag set described by the second tag set name. Here, the second tag set refers to the set with the second tag set name.

[0201] Step 52: The metadata tag service system queries the tag name record table based on the set identifier corresponding to the second tag set name mentioned above and the second tag name mentioned above to obtain the tag name identifier corresponding to the second tag name.

[0202] In this application, for the metadata tag service system mentioned above, after the metadata tag service system receives a metadata tag query request sent by the metadata service gateway and the metadata tag service system finds the set identifier corresponding to the second tag set name mentioned above, the metadata tag service system can query the tag name record table based on the set identifier and the second tag name carried in the metadata tag query request to obtain the tag name identifier corresponding to the second tag name, so that the tag name identifier can be used to uniquely identify the second tag name.

[0203] Step 53: The metadata tagging service system searches for the tag target identifier corresponding to the tag target description information in the second tag target record table above.

[0204] It should be noted that for the relevant content of step 53, please refer to the relevant content of step 42 above.

[0205] Step 54: The metadata tagging service system queries the tag value record table based on the tag name identifier corresponding to the second tag name mentioned above and the tag target identifier corresponding to the second tag target description information mentioned above, and obtains the second tag value.

[0206] It should be noted that for the relevant content of step 54, please refer to the relevant content of step 43 above.

[0207] Step 55: The metadata tag service system determines the tag query result corresponding to the metadata tag query request above based on the second tag value mentioned above.

[0208] It should be noted that this application does not limit the implementation of step 55 above. For example, step 55 may specifically be: determining the second tag value above as the tag query result corresponding to the metadata tag query request above. Alternatively, step 55 may specifically be: combining the second tag value, the name of the second tag set above, the name of the second tag above, and the description information of the object of the second tag above to obtain the tag query result corresponding to the metadata tag query request, so that the tag query result can more intuitively represent what the queried tag is. Furthermore, step 44 may specifically be: combining the second tag value, the set identifier corresponding to the name of the second tag set, the tag name identifier corresponding to the name of the second tag, and the tag object identifier corresponding to the description information of the object of the second tag into a tag, which serves as the tag query result corresponding to the metadata tag query request.

[0209] As can be seen, in one possible implementation, step 55 above can specifically be: the metadata tag service system combines the set identifier corresponding to the second tag set name above, the tag name identifier corresponding to the second tag name above, and the tag target identifier corresponding to the description information of the second tag target above into a tag, and combines it with the second tag value above to obtain the tag query result corresponding to the metadata tag query request above, so that the tag query result can more comprehensively represent the information obtained from the query for the data tag query request, which is beneficial to improving the query experience.

[0210] Based on the content of steps 51 to 55 above, in some application scenarios, for the metadata tag service system mentioned above, after receiving a metadata tag query request sent by the metadata service gateway, if the metadata tag service system carries the second tag set name, the second tag name, and the description information of the second tag's target object, the working principle of the metadata tag service system can include: querying the above tag set record table based on the second tag set name to obtain the set identifier corresponding to the second tag set name; and querying the above tag name record table based on the set identifier and the second tag name to obtain the tag name identifier corresponding to the second tag name. Based on the description information of the second tag's target object, the above tag target object record table is queried to obtain the tag target object identifier corresponding to the description information of the second tag's target object; based on the tag name identifier and the tag target object identifier, the tag value record table is queried to obtain the second tag value; the second tag value, the set identifier corresponding to the second tag set name, the tag name identifier corresponding to the second tag name, and the tag target object identifier corresponding to the description information of the second tag's target object are combined into a tag, which serves as the tag query result corresponding to the metadata tag query request, so that the tag query result can more comprehensively represent the information obtained from the query for the data tag query request, thus improving the query experience.

[0211] Based on the content related to S502 above, for the metadata tagging service system mentioned above, after receiving a metadata tag query request sent by the metadata service gateway, the metadata tagging service system can execute tag query logic (such as the logic described in steps 41-44 above or the logic described in steps 51-55 above, etc.) according to the metadata tag query parameters carried in the metadata tag query request, and obtain the tag query result corresponding to the metadata tag query request. Here, the tag query logic refers to the logic pre-configured for the metadata tagging service system to implement tag query processing.

[0212] S503: The metadata tagging service system sends the tag query results to the metadata service gateway, which is used to feed back the tag query results to the data engine.

[0213] In this application, for the aforementioned Hucang metadata management system, after the metadata tag service system in the Hucang metadata management system obtains the tag query result by processing the aforementioned metadata tag query request, the metadata tag service system sends the tag query result to the metadata service gateway in the Hucang metadata management system, so that the metadata service gateway feeds back the tag query result to the data engine, thereby enabling both the metadata service gateway and the data engine to know the processing status of the metadata tag query request from the tag query result.

[0214] Based on the relevant content in S501 to S503 above, it is clear that for the metadata tagging service system provided in this application (e.g., Figure 2 The metadata tagging service system 203 shown is... Figure 3 In the case of the metadata tagging service system shown, after the metadata tagging service system stores some metadata-related tags in the tag data storage space corresponding to the metadata tagging service system by executing the lake warehouse metadata tagging creation method provided in this application, the metadata tagging service system can query these tags from the storage space by executing the lake warehouse metadata tagging query method provided in this application. This can effectively avoid the defects caused by adding new parameters to the attribute information (properties) of the metadata database, metadata table, or metadata partition to implement tagging processing and corresponding tag query processing, thereby effectively improving the tag query effect.

[0215] In addition, this application does not limit the tag recording method in the metadata tag service system mentioned above. For ease of understanding, the following explanation is provided with examples.

[0216] As an example, in some application scenarios, for the metadata tag service system mentioned above, the tags in the metadata tag service system can meet the constraints shown in (1)-(3) below.

[0217] (1) For the tags in the metadata tag service system mentioned above (for example, tags described by the first tag set name, first tag name, first tag value and first tag target description information mentioned above), the tags can be stored in MySQL and can be horizontally scaled through a database sharding scheme.

[0218] (2) For the tags in the metadata tag service system mentioned above, the elements involved in the tag may include the tag collection name, tag name, tag value, and tag target. Moreover, these four elements correspond to the four data elements TagGroup, TagKey, TagValue, and TagSubject in the metadata tag service system, respectively.

[0219] Regarding the TagGroup data element mentioned above, this TagGroup represents a collection of tags, and the number of TagKeys in each TagGroup can be limited as needed, for example, not exceeding 1000. Furthermore, the TagGroup name (i.e., the tag collection name) is globally unique, and all access must be performed under the specified TagGroup name. Moreover, this application does not limit the implementation method of this TagGroup; for example, the TagGroup can be implemented using a collection identifier corresponding to a tag collection.

[0220] Regarding the TagKey data element mentioned above, this TagKey represents the name of a tag (i.e., the tag name), and it is required that the TagKey be unique within the specified TagGroup, but global uniqueness is not required. Furthermore, this application does not limit the implementation method of this TagKey; for example, the TagKey can be implemented using a tag name identifier corresponding to a tag name.

[0221] Regarding the TagValue data element mentioned above, this TagValue represents a specific value of a tag (i.e., the tag value), and it needs to specify information about the tag name and its target (e.g., associating the TagValue with a tag name identifier and a tag target identifier). Furthermore, this TagValue can only be associated with a specific TagKey. Moreover, this application does not limit the implementation of this TagValue; for example, the TagValue can be implemented using a tag value.

[0222] For the TagSubject data element mentioned above, this TagSubject represents an object that is subject to a tag (i.e., the object on which the tag is applied). This object can be one or more metadata entries. The object's description information can include namespace (used to distinguish data sources, such as Hive, ClickHouse, etc.), db_name (database name), tbl_name (table name), col_name (list), row_expression (row expression, such as score=90), and part_name (partition name, such as date='20230612'). Furthermore, this TagSubject must be globally unique. In addition, a TagSubject can be tagged by multiple TagKeys and associated with multiple TagValues. A TagKey cannot be associated with the same TagSubject (e.g., ...). Figure 6The TagSubject1 shown is bound multiple times. Furthermore, this application does not limit the implementation of the TagSubject; for example, the TagSubject can be implemented using a tag object identifier corresponding to the object on which the tag is applied.

[0223] (3) For the tags in the above metadata tag service system, the tags can be recorded and stored using the following four underlying tables: tag_subject (e.g., the above tag target record table), tag_group (e.g., the above tag set record table), tag_key (e.g., the above tag name record table), and tag_value (e.g., the above tag value record table).

[0224] The `tag_subject` table mentioned above records the relevant content of each tag's target object (e.g., tag object identifier, tag object description, etc.). Because the data volume in this `tag_subject` table may be quite large, it needs to be sharded and partitioned. Additionally, a unique index needs to be created for the relevant content of each tag's target object in the `tag_subject` table to ensure uniqueness and accelerate access.

[0225] The `tag_group` table mentioned above records information about each tag set (e.g., set identifier, tag set name, etc.). Since the amount of data in the `tag_group` table is usually not large, and it needs to support querying tag sets that meet certain conditions, there is no need to use a database sharding or table partitioning scheme for it. The tag set name needs to be globally unique, and a unique key is used to ensure uniqueness and accelerate access.

[0226] The `tag_key` table mentioned above records information related to each tag name (e.g., tag identifier, tag name, etc.). Since a `TagKey` can only belong to one `TagGroup`, to prevent the data volume in the `tag_key` table from becoming too large, there is no need to use a database sharding or table partitioning scheme. Furthermore, duplicate `TagKey` names are not allowed within a `TagGroup`; a unique key is used to ensure uniqueness and accelerate access.

[0227] The `tag_value` table mentioned above records the relevant information for each tag value (e.g., the tag value, the tag name identifier, and the target object identifier). Because the `tag_value` table contains a large amount of data, a database sharding / partitioning scheme can be used. Furthermore, since a tag value in the `tag_value` table needs to be associated with both `TagKey` and `TagSubject`, indexes need to be created on the foreign keys associated with `TagSubject` and `TagKey` respectively to speed up access. Additionally, a `TagValue` can only belong to one `TagKey`, and a `TagKey` belongs to one `TagGroup`. Also, the same tag cannot be used repeatedly on the same `TagSubject`; that is, a `TagSubject` cannot be associated with the same `TagKey` multiple times, using unique key constraints.

[0228] Based on the relevant content of the above tags, it can be seen that for the metadata tag service system in the above-mentioned Hucang metadata management system, since the tags in the metadata tag service system meet the constraints shown in (1)-(3) above, the creation process and query process of the tags will not cause any interference to the metadata storage system in the Hucang metadata management system. In this way, it is possible to access the tag service at a lower cost without affecting the service stability of the metadata storage system.

[0229] Based on the lake warehouse metadata tag creation method provided in the embodiments of this application, the embodiments of this application also provide a lake warehouse metadata tag creation device, which is described below in conjunction with... Figure 7 Explanation and clarification will be provided. Among them, Figure 7 This is a schematic diagram of a lake warehouse metadata tag creation device provided in an embodiment of this application. It should be noted that for technical details of the lake warehouse metadata tag creation device provided in this embodiment, please refer to the relevant content of the lake warehouse metadata tag creation method described above.

[0230] like Figure 7 As shown, the lake warehouse metadata tag creation device 700 provided in this application embodiment includes:

[0231] The first receiving unit 701 is used to receive a metadata tag creation request sent by the metadata service gateway; the metadata tag creation request is determined by the metadata service gateway based on the metadata tag creation parameters provided by the data engine; the metadata tag creation parameters include a first tag name, a first tag value, and a description of the first tag's target object;

[0232] The tag creation unit 702 is used to execute tag creation logic based on the metadata tag creation parameters carried in the metadata tag creation request.

[0233] In one possible implementation, the metadata tag creation parameters include a first tag name, a first tag value, and a description of the object to which the first tag applies;

[0234] The tag creation unit 702 includes:

[0235] The first writing subunit is used to write the tag name record content corresponding to the first tag name into the tag name record table. The tag name record content includes the tag name identifier corresponding to the first tag name.

[0236] The second writing subunit is used to write the tag target record content corresponding to the first tag target description information into the tag target record table. The tag target record content includes the tag target identifier corresponding to the first tag target description information.

[0237] The third writing subunit is used to write the tag value record content corresponding to the first tag value into the tag value record table. The tag value record content is determined based on the first tag value, the tag name identifier, and the tag target identifier.

[0238] In one possible implementation, the metadata tag creation parameters also include the name of the first tag set;

[0239] The tag creation unit 702 further includes:

[0240] The fourth writing subunit is used to write the tag set record content corresponding to the first tag set name into the tag set record table. The tag set record content includes the set identifier corresponding to the first tag set name.

[0241] The first determining subunit is used to determine the tag name record content corresponding to the first tag name based on the set identifier and the first tag name.

[0242] In one possible implementation, the second writing subunit is specifically used to: if there is no tag action object record content corresponding to the first tag action object description information in the tag action object record table, then write the tag action object record content corresponding to the first tag action object description information into the tag action object record table.

[0243] In one possible implementation, the lake warehouse metadata tag creation device 700 further includes:

[0244] An information generation unit is used to generate feedback information corresponding to the metadata tag creation request if it is detected that the tag creation logic has been completed.

[0245] An information sending unit is used to send the feedback information to the metadata service gateway, and the metadata service gateway is used to forward the feedback information to the data engine.

[0246] In one possible implementation, the metadata tag creation parameters include a first tag set name, a first tag name, a first tag value, and a description of the object to which the first tag applies;

[0247] The information generation unit is specifically used to: if it is detected that the tag creation logic has been completed, combine the first tag value, the set identifier corresponding to the first tag set name, the tag name identifier corresponding to the first tag name, and the tag target identifier corresponding to the first tag target description information to obtain the feedback information.

[0248] In one possible implementation, the metadata tag creation request is used to request the creation of a tag corresponding to one or more metadata; the one or more metadata are managed by a metadata storage system; the metadata service gateway is used to route the metadata processing request sent by the data engine to the metadata storage system, and the metadata storage system is used to execute the data processing logic corresponding to the metadata processing request.

[0249] In one possible implementation, the lake warehouse metadata tag creation device 700 is deployed within a metadata tag service system.

[0250] Based on the aforementioned content regarding the Lakeware Metadata Tag Creation Device 700, it is understood that, for the Lakeware Metadata Tag Creation Device 700 provided in this application embodiment, after the metadata service gateway determines the metadata tag creation request based on the metadata tag creation parameters (e.g., tag set name, tag name, tag target, tag value, etc.) provided by a data engine (e.g., Spark engine), the metadata service gateway sends the metadata tag creation request to the Lakeware Metadata Tag Creation Device 700. This allows the Lakeware Metadata Tag Creation Device 700 to execute tag creation logic based on the metadata tag creation parameters carried in the metadata tag creation request, thereby fulfilling the metadata tag creation requirements described by the metadata tag creation request. This enables tagging requirements for one or more metadata items, resulting in finer-grained tagging processing. This effectively avoids the defects caused by adding new parameters to the metadata database, metadata table, or metadata partition's property information to achieve tagging processing, thus effectively improving the tagging effect.

[0251] Based on the lake warehouse metadata tag query method provided in the embodiments of this application, the embodiments of this application also provide a lake warehouse metadata tag query device, which is described below in conjunction with... Figure 8 Explanation and clarification will be provided. Among them, Figure 8 This is a schematic diagram of a lake warehouse metadata tag query device provided in an embodiment of this application. It should be noted that for technical details of the lake warehouse metadata tag query device provided in this embodiment, please refer to the relevant content of the lake warehouse metadata tag query method described above.

[0252] like Figure 8 As shown, the lake warehouse metadata tag query device 800 provided in this application embodiment includes:

[0253] The second receiving unit 801 is used to receive a metadata tag query request sent by the metadata service gateway; the metadata tag query request is determined by the metadata service gateway based on the metadata tag query parameters provided by the data engine;

[0254] The tag query unit 802 is used to perform tag query processing based on the metadata tag query parameters carried in the metadata tag query request, and obtain the tag query result corresponding to the metadata tag query request.

[0255] The query feedback unit 803 is used to send the tag query results to the metadata service gateway, and the metadata service gateway is used to feed back the tag query results to the data engine.

[0256] In one possible implementation, the metadata tag query parameters include the second tag name and the description information of the object on which the second tag applies;

[0257] The tag query unit 802 includes:

[0258] The first lookup subunit is used to look up the tag name identifier corresponding to the second tag name from the tag name record table;

[0259] The second lookup subunit is used to look up the tag target identifier corresponding to the tag target description information from the tag target record table.

[0260] The third lookup subunit is used to query the tag value record table based on the tag name identifier and the tag target identifier to obtain the second tag value;

[0261] The second determining subunit is used to determine the tag query result based on the second tag value.

[0262] In one possible implementation, the metadata tag creation parameters also include a second tag set name;

[0263] The tag query unit 802 further includes:

[0264] The fourth lookup subunit is used to look up the set identifier corresponding to the name of the second tag set from the tag set record table;

[0265] The first lookup subunit is specifically used to: query the tag name record table based on the set identifier and the second tag name to obtain the tag name identifier corresponding to the second tag name.

[0266] In one possible implementation, the second determining subunit is specifically used to: combine the set identifier, the tag name identifier, the tag target identifier, and the second tag value to obtain the tag query result.

[0267] In one possible implementation, the lake warehouse metadata tag query device 800 is deployed within the metadata tag service system.

[0268] Based on the above-mentioned content of the lake warehouse metadata tag query device 800, it can be seen that if some metadata-related tags are stored in the storage space in advance by executing the lake warehouse metadata tag creation method provided in this application, the lake warehouse metadata tag query device 800 can be used to query these tags from the storage space. This can effectively avoid the defects caused by adding new parameters to the attribute information (properties) of the metadata database, metadata table, or metadata partition to implement tagging and corresponding tag querying, thereby effectively improving the tag query effect.

[0269] In addition, this application embodiment also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the lake warehouse metadata tag creation method provided in this application embodiment, or executes any implementation of the lake warehouse metadata tag query method provided in this application embodiment.

[0270] See Figure 9 This document illustrates a structural schematic diagram of an electronic device 900 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0271] like Figure 9 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0272] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0273] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0274] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0275] This application also provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform any implementation of the lake warehouse metadata tag creation method provided in this application, or any implementation of the lake warehouse metadata tag query method provided in this application.

[0276] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0277] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0278] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0279] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0280] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0281] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0282] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0283] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0284] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0285] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0286] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0287] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0288] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0289] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for creating lake warehouse metadata tags, characterized in that, The method is applied to a metadata tag service system, and the method comprises: receiving a metadata tag creation request sent by a metadata service gateway; the metadata tag creation request is determined by the metadata service gateway according to metadata tag creation parameters provided by a data engine; the metadata tag creation parameters comprise a first tag name, a first tag value, and first tag action object description information; executing tag creation logic according to the metadata tag creation parameters carried by the metadata tag creation request; wherein the tag creation logic comprises: writing tag name record content corresponding to the first tag name into a tag name record table, the tag name record content comprising a tag name identifier corresponding to the first tag name; writing tag action object record content corresponding to the first tag action object description information into a tag action object record table, the tag action object record content comprising a tag action object identifier corresponding to the first tag action object description information; writing tag value record content corresponding to the first tag value into a tag value record table, the tag value record content being determined according to the first tag value, the tag name identifier, and the tag action object identifier; wherein the metadata tag creation parameters further comprise a first tag set name; the tag creation logic further comprises, before writing the tag name record content corresponding to the first tag name into the tag name record table: writing tag set record content corresponding to the first tag set name into a tag set record table, the tag set record content comprising a set identifier corresponding to the first tag set name; determining the tag name record content corresponding to the first tag name according to the set identifier and the first tag name.

2. The method of claim 1, wherein, The writing of the tag action object record content corresponding to the first tag action object description information into the tag action object record table comprises: if there is no tag action object record content corresponding to the first tag action object description information in the tag action object record table, then writing the tag action object record content corresponding to the first tag action object description information into the tag action object record table.

3. The method of claim 1, wherein, The method further comprises: if it is detected that the tag creation logic is executed completely, then generating feedback information corresponding to the metadata tag creation request; sending the feedback information to the metadata service gateway, the metadata service gateway being configured to forward the feedback information to the data engine.

4. The method of claim 3, wherein, The metadata tag creation parameters comprise a first tag set name, a first tag name, a first tag value, and first tag action object description information; The generation of the feedback information corresponding to the metadata tag creation request comprises: combining the first tag value, a set identifier corresponding to the first tag set name, a tag name identifier corresponding to the first tag name, and a tag action object identifier corresponding to the first tag action object description information to obtain the feedback information.

5. The method according to any one of claims 1 to 4, characterized in that, The metadata tag creation request is used to request the creation of a tag corresponding to one metadata or multiple metadata. The one or more metadata are managed by a metadata storage system; The metadata service gateway is configured to route a metadata processing request sent by the data engine to the metadata storage system, and the metadata storage system is configured to execute data processing logic corresponding to the metadata processing request.

6. A method of querying metadata tags for a lake store, the method comprising: receiving a query for metadata tags associated with a lake store; and returning metadata tags associated with the lake store. The method is applied to a metadata tag service system configured to process a query for a tag created by the lake warehouse metadata tag creation method of any one of claims 1-5; The query method comprises: receiving a metadata tag query request sent by a metadata service gateway, wherein the metadata tag query request is determined by the metadata service gateway based on metadata tag query parameters provided by a data engine; performing tag query processing based on metadata tag query parameters carried by the metadata tag query request to obtain a tag query result corresponding to the metadata tag query request; sending the tag query result to the metadata service gateway, wherein the metadata service gateway is configured to feed back the tag query result to the data engine; wherein the metadata tag query parameters comprise a second tag name and second tag object description information; and the determination process of the tag query result comprises: finding a tag name identifier corresponding to the second tag name from a tag name record table; finding a tag object identifier corresponding to the second tag object description information from a tag object record table; querying a tag value record table based on the tag name identifier and the tag object identifier to obtain a second tag value; determining the tag query result based on the second tag value; wherein the metadata tag creation parameters further comprise a second tag set name; and the determination process of the tag query result further comprises, before finding the tag name identifier corresponding to the second tag name from the tag name record table, finding a set identifier corresponding to the second tag set name from a tag set record table; finding the tag name identifier corresponding to the second tag name from the tag name record table comprises: querying the tag name record table based on the set identifier and the second tag name to obtain the tag name identifier corresponding to the second tag name.

7. The method of claim 6, wherein, determining the tag query result based on the second tag value comprises: combining the set identifier, the tag name identifier, the tag object identifier, and the second tag value to obtain the tag query result.

8. A lake bin metadata management system characterized by, The lake warehouse metadata management system comprises a metadata service gateway, at least one metadata storage system, and a metadata tag service system; The metadata service gateway is configured to route a request sent by a data analysis engine to the metadata tag service system or any one of the metadata storage systems; The metadata storage system is configured to manage at least one metadata based on the request; The metadata tag service system is configured to manage tags corresponding to part or all of the at least one metadata based on the request, The metadata tag service system is configured to execute the lake warehouse metadata tag creation method of any one of claims 1-5 or the lake warehouse metadata tag creation method of any one of claims 6-7.

9. A lake bin metadata tag creation apparatus characterized by comprising: Comprise: A first receiving unit configured to receive a metadata tag creation request sent by a metadata service gateway; The metadata tag creation request is determined by the metadata service gateway according to metadata tag creation parameters provided by a data engine; the metadata tag creation parameters comprise a first tag name, a first tag value, and first tag object description information; A tag creation unit configured to execute tag creation logic according to metadata tag creation parameters carried by the metadata tag creation request; The tag creation logic comprises: Writing tag name record content corresponding to the first tag name into a tag name record table, wherein the tag name record content comprises a tag name identifier corresponding to the first tag name; Writing tag object record content corresponding to the first tag object description information into a tag object record table, wherein the tag object record content comprises a tag object identifier corresponding to the first tag object description information; Writing tag value record content corresponding to the first tag value into a tag value record table, wherein the tag value record content is determined according to the first tag value, the tag name identifier, and the tag object identifier; The metadata tag creation parameters further comprise a first tag set name; the tag creation logic further comprises, before writing the tag name record content corresponding to the first tag name into the tag name record table: Writing tag set record content corresponding to the first tag set name into a tag set record table, wherein the tag set record content comprises a set identifier corresponding to the first tag set name; Determining the tag name record content corresponding to the first tag name according to the set identifier and the first tag name.

10. A lake bin metadata tag query apparatus, characterized by, The lake warehouse metadata tag query device is configured to query the tag created by the lake warehouse metadata tag creation method of any one of claims 1-5; The device comprises: A second receiving unit configured to receive a metadata tag query request sent by a metadata service gateway; the metadata tag query request is determined by the metadata service gateway according to metadata tag query parameters provided by a data engine; A tag query unit configured to perform tag query processing according to metadata tag query parameters carried by the metadata tag query request to obtain a tag query result corresponding to the metadata tag query request; A query feedback unit configured to send the tag query result to the metadata service gateway, wherein the metadata service gateway is configured to feed back the tag query result to the data engine, The metadata tag query parameters comprise a second tag name and second tag object description information; the determination process of the tag query result comprises: Finding a tag name identifier corresponding to the second tag name from a tag name record table; finding a label action object identifier corresponding to the second label action object description information from a label action object record table; querying a label value record table according to the label name identifier and the label action object identifier to obtain a second label value; determining the label query result according to the second label value; wherein the metadata label creation parameter further comprises a second label set name; and the determination process of the label query result further comprises, before finding the label name identifier corresponding to the second label name from the label name record table, finding a set identifier corresponding to the second label set name from a label set record table; finding the label name identifier corresponding to the second label name from the label name record table comprises: querying the label name record table according to the set identifier and the second label name to obtain the label name identifier corresponding to the second label name.

11. An electronic device, comprising: The device comprises a processor and a memory; The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory, so that the electronic device executes the method of any one of claims 1-7.

12. A computer readable medium characterized by The computer readable medium stores instructions or computer programs, which, when executed on a device, cause the device to execute the method of any one of claims 1-7.

Images