A kind of lake warehouse metadata service log generation method and query method

Abstract

The application discloses a kind of lake warehouse metadata service log generation method and query method, the method includes: after receiving the data processing request sent by data engine in metadata service end, first request identification information is determined by metadata service end according to the data processing request, to make the first request identification information be used to identify the data processing request;Again, the request processing logic corresponding to the data processing request is executed by the metadata service end, and the log corresponding to the request processing logic is generated according to the first request identification information, so that the log carries the first request identification information, so that the first request identification information can be used as medium to associate the log corresponding to the request processing logic with the log generated in the data engine for the data processing request, and then the log in the metadata service end and the log in the data engine can be connected by the first request identification information, so as to improve the log query effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for generating lake warehouse metadata service logs, a method for querying lake warehouse metadata service logs, an apparatus, a device, and a medium. Background Technology

[0002] In some application scenarios (such as big data scenarios), for metadata service systems (such as HiveMetastore) and data engines connected to these systems (such as Presto and Spark), the metadata service system can provide interfaces to the data engines, allowing them to access the metadata service system by calling these interfaces. For example, the data engine can send a request (such as a metadata read request) to the metadata service system through an interface, enabling the metadata service system to process the request.

[0003] However, because the logs in the data engine and the logs in the metadata service system belong to different systems, the logs are fragmented and difficult to connect, which affects the execution of some log-related tasks (such as troubleshooting, performance tuning, and log querying). Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method for generating lake warehouse metadata service logs, a method for querying lake warehouse metadata service logs, an apparatus, a device, and a medium.

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides a method for generating lake warehouse metadata service logs, the method being applied to a metadata server, the method comprising:

[0007] Receive data processing requests sent by the data engine;

[0008] A first request identifier is determined based on the request identifier information carried in the data processing request. The first request identifier is used to identify the data processing request. If the data engine provides a log identifier and generates a request tracing identifier for the data processing request, then the request identifier includes the log identifier and the request tracing identifier. If the data engine does not provide a log identifier, but generates a request tracing identifier for the data processing request, then the request identifier includes the request tracing identifier.

[0009] The request processing logic corresponding to the data processing request is executed, and a log corresponding to the request processing logic is generated based on the first request identification information, wherein the log carries the first request identification information.

[0010] In one possible implementation, the log identifier is determined by the software development kit (SDK) in the data engine based on the target parameters of the data engine;

[0011] If the data engine supports log identifier configuration, then the target parameter is a log identifier pre-configured for the data engine;

[0012] If the data engine does not support log identifier configuration, then the target parameter is the task identifier, which is used to identify the data analysis task that triggers the request generation logic of the data processing request. The data analysis task is created by the data engine in response to user operations.

[0013] In one possible implementation, the request tracking identifier is used to uniquely identify the data processing request;

[0014] And / or,

[0015] The request tracking identifier is generated by the SDK in the data engine.

[0016] In one possible implementation, the request identification information is located in a preset field in the data processing request;

[0017] And / or,

[0018] The request identification information is written by the SDK in the data engine into a preset field in the data processing request.

[0019] In one possible implementation, the method further includes:

[0020] If the data processing request does not carry request identification information, then the first request identification information is determined based on at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request.

[0021] In one possible implementation, the first request identification information includes at least one of the at least one request parameter information and a request tracking identifier generated by the metadata server for the data processing request;

[0022] And / or,

[0023] If the data processing request does not carry request identification information, then the first request identification information is determined based on at least one request parameter information of the data processing request, including:

[0024] If the data engine does not provide a log identifier and does not generate a request tracking identifier for the data processing request, then the first request identifier information is determined based on at least one request parameter information of the data processing request.

[0025] In one possible implementation, the engine description information includes at least one of the engine identifier of the data engine and the user identifier of the data engine;

[0026] And / or,

[0027] The interface description information includes at least one of the interface identifier corresponding to the data processing request and the interface parameters corresponding to the data processing request.

[0028] In one possible implementation, the metadata server is a metadata service system.

[0029] In one possible implementation, the metadata server includes a metadata service gateway and at least one metadata service system;

[0030] Determining the first request identifier information based on the request identifier information carried in the data processing request includes:

[0031] The metadata service gateway determines the first request identifier information based on the request identifier information carried in the data processing request;

[0032] The execution of the request processing logic corresponding to the data processing request, and the generation of a log corresponding to the request processing logic based on the first request identifier information, includes:

[0033] The metadata service gateway generates a data processing message carrying the first request identification information based on the data processing request. The metadata service gateway sends the data processing message to the target system in the at least one metadata service system. The metadata service gateway also generates a metadata service gateway log corresponding to the data processing message based on the first request identification information. The metadata service gateway log carries the first request identification information.

[0034] The target system executes the message processing logic corresponding to the data processing message, and the target system generates a system log corresponding to the message processing logic based on the first request identification information carried in the data processing message, wherein the system log carries the first request identification information.

[0035] In one possible implementation, the method further includes:

[0036] After executing the message processing logic corresponding to the data processing message, the target system clears the first request identifier information recorded in the target system, and the target system sends a first feedback message to the metadata service gateway.

[0037] After the metadata service gateway receives the first feedback message, the metadata service gateway clears the first request identification information recorded in the metadata service gateway, and the metadata service gateway sends a second feedback message to the data engine.

[0038] After the data engine receives the second feedback message, if the data engine records a target identifier corresponding to the first request identifier information, the data engine will clear the target identifier recorded in the data engine.

[0039] In one possible implementation, if the data engine supports log identifier configuration and the first request identifier information recorded in the data engine includes a log identifier and a request tracing identifier, then the target identifier is the request tracing identifier.

[0040] In one possible implementation, if the data processing request carries the first request identification information, the data engine is used to generate an engine log for the data processing request, and the engine log carries the first request identification information.

[0041] This application provides a method for querying lake warehouse metadata service logs. The method is applied to a metadata server and includes:

[0042] The system receives a log query request sent by a log request device. The log query request is used to request a query for the logs corresponding to the data processing request sent by the data engine to the metadata server.

[0043] The request identifier information corresponding to the data processing request carried in the log query request is determined as the second request identifier information, which is used to identify the data processing request.

[0044] Based on the logs containing the second request identifier information in the log records of the metadata server, the log query result corresponding to the log query request is determined; the logs in the log records are generated using the Lakewarehouse metadata service log generation method provided in this application.

[0045] In one possible implementation, the method further includes:

[0046] If the log query request does not carry the request identifier information corresponding to the data processing request, but the log query request carries a log query time range and at least one request parameter information of the data processing request, then the second request identifier information is generated based on the at least one request parameter information carried by the log query request; or, the second request identifier information is determined from the log records of the metadata server based on the log query time range carried by the log query request and at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request.

[0047] Based on the logs existing in the log records of the metadata server that conform to the log query time range and carry the second request identifier information, the log query result corresponding to the log query request is determined.

[0048] In one possible implementation, the log request device is used to integrate the log query results fed back by the metadata server and the logs recorded in the data engine for the data processing request to obtain an integrated log, which is used to describe the process executed by the data engine and the metadata server for the data processing request.

[0049] In one possible implementation, if the log query request carries request identification information corresponding to the data processing request, then the log recorded in the data engine for the data processing request refers to the log in the data engine's log records that carries the request identification information.

[0050] In one possible implementation, the log request device is the data engine.

[0051] This application provides a lakeware metadata service log generation device, including:

[0052] The first receiving unit is used to receive data processing requests sent by the data engine.

[0053] The first determining unit is configured to determine first request identification information based on the request identification information carried in the data processing request, wherein the first request identification information is used to identify the data processing request; if the data engine provides a log identifier and the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the log identifier and the request tracing identifier; if the data engine does not provide a log identifier, but the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the request tracing identifier.

[0054] The request processing unit is used to execute the request processing logic corresponding to the data processing request, and generate a log corresponding to the request processing logic based on the first request identification information, wherein the log carries the first request identification information.

[0055] This application provides a lake warehouse metadata service log query device, including:

[0056] The second receiving unit is used to receive a log query request sent by the log request device. The log query request is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server.

[0057] The second determining unit is used to determine the request identification information corresponding to the data processing request carried in the log query request as the second request identification information, which is used to identify the data processing request.

[0058] The third determining unit is used to determine the log query result corresponding to the log query request based on the logs carrying the second request identifier information existing in the log records of the metadata server; the logs in the log records are generated using the Lakewarehouse metadata service log generation method provided in this application.

[0059] This application provides an electronic device, the device comprising: a processor and a memory;

[0060] The memory is used to store instructions or computer programs;

[0061] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device executes the lake warehouse metadata service log generation method or the lake warehouse metadata service log query method provided in this application.

[0062] This application provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform the lakeware metadata service log generation method or the lakeware metadata service log query method provided in this application.

[0063] This application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the lake warehouse metadata service log generation method or the lake warehouse metadata service log query method provided in this application.

[0064] Compared with related technologies, this application has at least the following advantages:

[0065] In the technical solution provided in this application, for the metadata server (e.g., a metadata server composed of a metadata service gateway and multiple metadata service systems), after receiving a data processing request sent by the data engine, the metadata server determines a first request identifier based on the request identifier information (e.g., log identifier, request tracking identifier, etc.) carried in the data processing request. This first request identifier is used to identify the data processing request, thereby enabling it to represent the characteristics of the data processing request. Then, the metadata server executes the request processing logic corresponding to the data processing request and generates a log corresponding to the request processing logic based on the first request identifier, so that the log carries the first request identifier. This allows the first request identifier information to act as a medium to associate the logs corresponding to the request processing logic with the logs generated in the data engine for the data processing request. This enables the connection between the logs in the metadata server and the logs in the data engine, allowing subsequent queries to retrieve complete logs related to any data processing request in the data engine (i.e., the logs recorded by the metadata server for the data processing request and the logs recorded by the data engine for the data processing request). This effectively overcomes the shortcomings caused by the separation between the logs in the data engine and the logs in the metadata service system, thereby improving log query performance (e.g., increasing query accuracy and efficiency). Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A flowchart illustrating a method for generating lake warehouse metadata service logs provided in this application embodiment;

[0068] Figure 2A schematic diagram illustrating a communication process provided in an embodiment of this application;

[0069] Figure 3 A schematic diagram illustrating another communication process provided in an embodiment of this application;

[0070] Figure 4 A schematic diagram illustrating yet another communication process provided in an embodiment of this application;

[0071] Figure 5 A schematic diagram of a communication protocol provided for an embodiment of this application;

[0072] Figure 6 A flowchart illustrating a method for querying lake warehouse metadata service logs provided in this application embodiment;

[0073] Figure 7 A schematic diagram of a lake warehouse metadata service log generation device provided in an embodiment of this application;

[0074] Figure 8 This is a schematic diagram of the structure of a lake warehouse metadata service log query device provided in an embodiment of this application;

[0075] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0077] To better understand the technical solution provided in this application, the method for generating lake warehouse metadata service logs provided in this application will be explained below with reference to some accompanying figures. For example... Figure 1 As shown, the lake warehouse metadata service log generation method provided in this application embodiment includes S101-S103 below. Wherein, the Figure 1 A flowchart illustrating a method for generating lake warehouse metadata service logs, provided in an embodiment of this application.

[0078] S101: The metadata server receives data processing requests sent by the data engine.

[0079] The metadata server is used to process requests sent by a data engine (such as metadata modification requests, metadata deletion requests, etc.).

[0080] Furthermore, this application does not limit the implementation method of the metadata server described above. For ease of understanding, the following description combines two scenarios.

[0081] Scenario 1: For some application scenarios (e.g., scenarios with relatively low data processing pressure), these application scenarios can use only one metadata service system (e.g., metadata service system similar to Hive Metastore, etc.) to provide data services (e.g., metadata modification, metadata deletion, metadata addition, etc.) to its upstream data engine.

[0082] Based on the above situation 1, it can be seen that in one possible implementation, the metadata service terminal mentioned above can be a metadata service system, so that S101-S103 implemented based on the metadata service terminal can be used to describe the use of a metadata service system and at least one data engine (e.g., Figure 2 Data engines 1 to N (where N is a positive integer, etc.) are shown to perform related processing (e.g., request generation processing, execution of the request processing logic, etc.) for a certain request (e.g., metadata modification request, metadata deletion request, etc.). The metadata service system refers to a system capable of managing data from data sources (e.g., adding new data, deleting existing data, modifying existing data, etc.) to provide certain data services (e.g., metadata modification, metadata deletion, metadata addition, etc.).

[0083] It should be noted that this application does not limit the implementation method of the metadata service system described above. For example, in some application scenarios (such as data lake warehouses), the metadata service system can be implemented using any existing or future metadata service system capable of managing metadata (such as Hive Metastore instances), enabling the metadata service system to manage metadata in the data lake warehouse. Furthermore, this application does not limit the implementation method of the data source described above. For example, it can be implemented using any existing or future data source (such as data lakes, data warehouses, integrated data lake warehouses, etc.). Therefore, in one possible implementation, the metadata service system can be used to manage metadata within at least some data sources.

[0084] Based on the above two paragraphs, it can be seen that in some application scenarios, the metadata service mentioned above can be a metadata service system (such as a Hive Metastore instance, etc.) so that S101-S103 implemented based on the metadata service can be used to describe the use of the metadata service system and at least one data engine (such as some upstream services of the metadata service system, etc.) to complete the relevant processing for a certain metadata processing request (such as request generation processing, execution of the request processing logic corresponding to the request, etc.).

[0085] Scenario 2: For some application scenarios (e.g., scenarios with high data processing pressure), these application scenarios can utilize multiple metadata service systems (e.g., Figure 3 The metadata service systems 1 to M (where M is a positive integer, etc.) provide data services (such as metadata modification, metadata deletion, metadata addition, etc.).

[0086] Based on the above situation 2, it can be seen that in one possible implementation, the metadata service mentioned above may include a metadata service gateway (e.g., Figure 3 The metadata service gateway shown) and at least one metadata service system (e.g., Figure 3 Metadata service systems 1 to M are shown. The metadata service gateway receives requests sent by a data engine and routes (e.g., forwards) the requests to a metadata service system. Therefore, the metadata service gateway is deployed between at least one data engine and at least one metadata service system to facilitate communication between these data engines and systems. Furthermore, this application does not limit the implementation of the metadata service gateway; for example, it can be implemented using any existing or future device capable of facilitating communication between at least one data engine and at least one metadata service system (e.g., a metadata service gateway based on Hive Metastore).

[0087] Based on the above content, in some application scenarios, the metadata service mentioned above may include a metadata service gateway and at least one metadata service system, so that S101-S103 implemented based on the metadata service can be used to describe the use of the at least one metadata service system, the metadata service gateway and at least one data engine to complete the relevant processing for a certain metadata processing request (e.g., request generation processing, execution of the request processing logic corresponding to the request, etc.).

[0088] A data engine is an engine that can use the metadata server mentioned above to perform certain metadata processing procedures, so that the data engine can be used to represent the upstream service of the metadata server, thereby enabling the data engine to use the metadata server to perform certain data processing procedures in some way (such as interface calls).

[0089] Furthermore, this application does not limit the implementation method of the aforementioned data engine. For example, the data engine can specifically adopt any existing or future engine capable of accessing the aforementioned metadata server (e.g., Figure 3 or Figure 4 The implementation uses data engines 1, 2, ..., N (as shown). Alternatively, in some application scenarios, the data engine can be implemented as a data analysis engine, data computation engine, or data query engine. Therefore, in one possible implementation, when the metadata service system in the metadata server uses a Hive Metastore instance, the data engine can be implemented using the HiveServer2 engine. This allows the data engine to access the metadata server via API calls, enabling the metadata server to perform data processing by handling requests provided by the data engine.

[0090] It should be noted that this application does not limit the implementation of the HiveServer2 engine mentioned above. For example, the HiveServer2 engine may include one or more of the following: at least one Structured Query Language (SQL) engine, at least one batch stream processing engine, and at least one intelligent analysis platform engine.

[0091] It should also be noted that this application does not limit the implementation method of the SQL engine mentioned above. For example, the SQL engine can be implemented using any existing or future SQL engine (such as Hive or Presto). Similarly, this application does not limit the implementation method of the batch / stream processing engine mentioned above. For example, the batch / stream processing engine can be implemented using any existing or future batch / stream processing engine (such as Spark or Flink). Furthermore, this application does not limit the implementation method of the intelligent analysis platform engine mentioned above. For example, the intelligent analysis platform engine can be implemented using any existing or future intelligent analysis platform engine, such as a Business Intelligence (BI) analysis platform engine.

[0092] Furthermore, for the aforementioned metadata server and data engine, data communication is possible between the data engine and the metadata server. This application does not limit the communication method used; for example, it can be implemented using any existing or future method capable of enabling communication between the data engine and the metadata server. In some application scenarios, the data engine can access the metadata server by calling an interface, enabling the metadata server to process data processing requests (e.g., requests to modify table names) sent by the data engine through the interface. It should be noted that this application does not limit the implementation method of the interface. For example, in some application scenarios, the interface can be implemented using a Remote Procedure Call (RPC) interface. In other application scenarios, the interface can be implemented using a Thrift interface.

[0093] Furthermore, this application does not limit the structure of the aforementioned data engine. For example, it can be implemented using any existing or future engine capable of performing certain data processing tasks using the aforementioned metadata server. Additionally, in some application scenarios, the data engine can access the metadata server through its pre-deployed Software Development Kit (SDK). Here, the SDK refers to a toolkit pre-deployed within the data engine for communicating with the metadata server; moreover, this application does not limit the working principle of the SDK. For ease of understanding, some scenarios will be used to illustrate this below.

[0094] Scenario 1: If the data engine mentioned above supports log identification (e.g., Figure 4 The logID shown is used, and the SDK deployed in this data engine has request tracing identifiers (e.g., logID). Figure 4 Given the traceID generation capability shown, the working principle of the SDK deployed in this data engine can be as follows: When the data engine wants to access the metadata server mentioned above through this SDK, the SDK can read the log identifier from the data engine and put the log identifier into a Thrift message (e.g., Figure 5The SDK automatically generates a globally unique request tracing identifier in field 0 of the Thrift message shown; and the request tracing identifier is used to uniquely identify the Thrift request carrying the Thrift message (e.g., the data processing request mentioned above), and puts the request tracing identifier into field 0 of the Thrift message, so that the Thrift request ultimately sent by the data engine to the metadata server through the SDK carries the log identifier and the request tracing identifier, thus enabling the data engine to transparently transmit the log identifier and the request tracing identifier to the metadata server through the SDK.

[0095] It should be noted that this application does not limit the implementation of the step "the SDK can read the log identifier from the data engine" in the above paragraph. For example, if the data engine supports Mapped Diagnostic Context (MDC), it can be determined that the data engine supports log identifier configuration, so the SDK deployed in the data engine can automatically read the log identifier from the MDC of the data engine; if the data engine does not support MDC, it can be determined that the data engine does not support log identifier configuration, so the SDK deployed in the data engine can obtain the log identifier according to the relevant configuration in the data engine. Furthermore, this application does not limit the implementation of the step "obtaining the log identifier according to the relevant configuration in the data engine". As an example, when the relevant configuration of the Spark data engine indicates that the data analysis task identifier applicationId can be used as a log identifier, the SDK deployed in Spark can automatically obtain the applicationId from Spark and use it as a log identifier, so that the SDK can subsequently put the log identifier into certain fields of the Thrift request generated based on the data analysis task (e.g., ...). Figure 5 Within field 0 shown, the Thrift request is sent to the metadata server.

[0096] Scenario 2: If the data engine does not support log identifiers, but the SDK deployed in the data engine has the ability to generate request tracing identifiers, the working principle of the SDK deployed in the data engine can be as follows: When the data engine wants to access the metadata server through the SDK, the SDK will automatically generate a globally unique request tracing identifier for the data analysis task provided by the data engine. This request tracing identifier is used to uniquely identify the Thrift request generated based on the data analysis task (e.g., the data processing request mentioned above). The request tracing identifier is placed in field 0 of the Thrift message carried by the Thrift request, so that the Thrift request sent by the data engine to the metadata server through the SDK carries the request tracing identifier. This achieves the data engine's transparent transmission of the request tracing identifier to the metadata server through the SDK.

[0097] It should be noted that this application does not limit the correspondence between the data analysis task and the Thrift request in the above paragraph. If the data analysis task describes only one data processing requirement (e.g., deleting a table partition), then there is a one-to-one correspondence between the data analysis task and the Thrift request, so that the Thrift request conveys the data processing requirement to the metadata server. However, if the data analysis task describes multiple data processing requirements (e.g., deleting multiple table partitions), then multiple Thrift requests can be generated based on the data analysis task, so that each Thrift request can be used to convey a data processing requirement (e.g., deleting a table partition) to the metadata server. This results in the data analysis task corresponding to multiple Thrift requests, and thus the data engine mentioned above needs to access the metadata server multiple times through the SDK to complete the data analysis task.

[0098] Scenario 3: If the data engine does not support log identifiers and the SDK deployed in the data engine does not have the ability to generate request tracing identifiers, the working principle of the SDK deployed in the data engine can be as follows: When the data engine wants to access the metadata server through the SDK, the SDK only needs to generate a Thrift request based on the data analysis task provided by the data engine. This Thrift request should represent a data processing requirement described by the data analysis task. After the SDK sends the Thrift request to the metadata server, in order to better connect the logs between the data engine and the metadata server, the metadata server needs to obtain and record some parameters of the Thrift request, such as the interface name, parameters, client's Internet Protocol Address (IP address), username, etc. This allows the metadata server to perform log generation processing based on these parameters. In the future, this information can be used to determine that the logs recorded by the data engine for the Thrift request and the logs generated by the metadata server based on these parameters belong to the same Thrift request.

[0099] Based on the aforementioned data engine, in certain application scenarios, after detecting a user operation (e.g., entering the SQL query "alter table test_db.test_tbl rename to test_db.test_tbl_new"), the data engine can create a data analysis task for that user operation. This task can represent the data processing requirements conveyed by the user operation, enabling the SDK deployed within the data engine (e.g., ...) to... Figure 4 Any of the SDKs shown can generate a data processing request based on the data analysis of the biological data, and send the data processing request to the metadata server (e.g., the metadata service gateway in the metadata server), so that the metadata server can realize the data processing requirement by processing the data processing request accordingly.

[0100] A data processing request refers to a request sent by the data engine to the metadata server to request certain processing (e.g., modifying a table name, deleting a column, adding a new column) of a certain data object (e.g., a database, table, partition, metadata). This application does not limit the implementation method of the data processing request; for example, it can be implemented using any existing or future request sent by the data engine to the metadata server for requesting certain data processing. The data object refers to an object that can be processed by the metadata server; and this application does not limit the implementation method of the data object; for example, it can be implemented using a database, table, partition, or metadata.

[0101] In fact, in some application scenarios (for example, upgrading all SDKs deployed in the data engine to...), Figure 4 In scenarios such as the new SDK shown, to better facilitate log communication between the data engine and the metadata server, this application provides a possible implementation of the data processing request described above. In this implementation, the data processing request carries request identification information (e.g., Figure 4 The logID and traceID shown are used to identify the data processing request so that the metadata server can obtain the request identification information from the data processing request. This request identification information is used to identify the data processing request so that it represents the characteristics of the data processing request. Furthermore, this application does not limit the implementation of the request identification information; for ease of understanding, two scenarios are described below.

[0102] In some application scenarios, if the data engine supports log identifiers and the SDK deployed in the data engine has the ability to generate request tracing identifiers, then the requests generated by the SDK (such as Thrift requests) can carry log identifiers and request tracing identifiers. Therefore, these two identifiers can be used to reflect the characteristics of the request.

[0103] Based on the above scenario, in one possible implementation, if the aforementioned data engine is used to provide log identifiers (e.g., the data engine supports log identifiers), and the data engine is also used to generate request tracking identifiers for the aforementioned data processing requests, then the request identifier information carried by the data processing request may include log identifiers (e.g., ...). Figure 4 The logID shown) and request tracking identifier (e.g., Figure 4The traceID shown is used to better represent the characteristics of the data processing request. This allows the data processing request to not only convey the data processing requirements to the metadata server, but also to inform the metadata server of the identification information involved in the data processing request in the data engine. This is beneficial for connecting the logs in the metadata server with the logs in the data engine.

[0104] Log identifiers are identifiers used by the data engine to create logs for data processing requests (e.g., ...). Figure 4 The log ID shown is used to identify the object (e.g., a task, a thread, etc.) that triggers the request generation logic within the data engine. The request generation logic refers to the processing logic executed by the data engine to generate the data processing request (e.g., the logic involved in the data processing request generation process shown above), enabling the data engine to generate the data processing request by executing this request generation logic.

[0105] Furthermore, this application does not limit the implementation method of the log identifier mentioned above. For example, in some application scenarios, the log identifier can be adopted as follows: Figure 4 Implement the logID shown.

[0106] Furthermore, this application does not limit the method of obtaining the log identifier mentioned above. For example, it can be implemented using any existing or future method that can obtain the log identifier (e.g., logID).

[0107] Furthermore, to further improve the efficiency of log identifier acquisition, this application also provides a method for obtaining the log identifier. In this implementation, the log identifier is determined by the SDK in the data engine mentioned above based on the target parameters of the data engine. Here, the SDK refers to a toolkit already deployed in the data engine for communicating with the metadata server. The target parameter refers to the parameter required by the SDK to obtain the log identifier from the data engine. Furthermore, this application does not limit the implementation method of the target parameter. For example, if the data engine is used to provide log identifiers and supports log identifier configuration (e.g., the data engine supports MDC), then the target parameter can be a log identifier pre-configured for the data engine, so that the SDK can directly use the target parameter as the log identifier (e.g., the SDK can read the log identifier from the data engine's MDC). If the data engine is used to provide log identifiers, but the data engine does not support log identifier configuration (e.g., the data engine does not support MDC), then the target parameter can be the task identifier (e.g., applicationId), so that the SDK can directly use the target parameter as the log identifier (e.g., the SDK needs to treat applicationId as the log identifier). The task identifier is used to identify the data analysis task that triggers the request generation logic of the data processing request. This data analysis task is created by the data engine in response to user operations. It should be noted that the relevant content of this data analysis task is described above.

[0108] Based on the above, in one possible implementation, the working principle of the data engine described above can be as follows: After the data engine detects a user operation, it can create a data analysis task for the user operation, so that the data analysis task can represent the data analysis requirements conveyed by the user operation; then, the SDK deployed in the data engine obtains the target parameters of the data engine (for example, reading the logID from MDC or obtaining the task identifier of the data analysis task, etc.), and uses the target parameters as log identifiers, so that the SDK can subsequently generate data processing requests based on the log identifiers, so that the data processing requests carry the log identifiers.

[0109] Furthermore, this application does not limit the implementation of the content "the data processing request carries the log identifier" mentioned above. For example, in some application scenarios, in order to minimize the transformation cost, the log identifier can be recorded using an idle field in the existing communication protocol (e.g., Thrift protocol) between the data engine and the metadata server. Based on this, it can be seen that in one possible implementation, the SDK in the data engine mentioned above can be used to write the log identifier into the first field of the data processing request during the generation of the data processing request, so that the log identifier can be transmitted (e.g., through transmission) using the first field. Here, the first field refers to the field in the data processing request that is used to record the log identifier; and this application does not limit the implementation of the first field. For example, the first field can be implemented using an idle field in the data processing request. It can be seen that when the data processing request uses Figure 5 When implementing the Thrift protocol format shown, the first field can be implemented using field 0 (e.g., Field0), which is in an idle state.

[0110] The request tracking identifier is used to uniquely identify the preceding data processing request, so that the request tracking identifier can be used to identify a single access process of the preceding data engine to the metadata server; moreover, this application does not limit the implementation method of the request tracking identifier. For example, in some application scenarios, the request tracking identifier can adopt... Figure 4 Implement the traceID shown.

[0111] Furthermore, this application does not limit the method of obtaining the above request tracking identifier. For example, in some application scenarios, the request tracking identifier may be generated by the SDK in the above data engine. Moreover, this application does not limit the generation process. For example, it may be that the SDK can automatically generate the request tracking identifier during the generation of the above data processing request, so that the request tracking identifier can uniquely identify the data processing request, so that the SDK can subsequently generate the data processing request based on the request tracking identifier, so that the data processing request carries the request tracking identifier.

[0112] It should be noted that this application does not limit the implementation of the content "the data processing request carries the request tracking identifier" in the above paragraph. For example, in some application scenarios, in order to minimize the transformation cost, the request tracking identifier can be recorded using an idle field in the existing communication protocol (e.g., Thrift protocol) between the data engine and the metadata server. Based on this, it can be seen that in one possible implementation, the SDK in the data engine mentioned above can be used to write the request tracking identifier into the second field of the data processing request during the generation of the data processing request, so that the request tracking identifier can be transmitted (e.g., through transmission) using the second field. Here, the second field refers to the field in the data processing request that is used to record the request tracking identifier; and this application does not limit the implementation of the second field. For example, the second field can be implemented using an idle field in the data processing request. It can be seen that when the data processing request uses Figure 5 When implementing the Thrift protocol format shown, the second field can be implemented using field 0 (e.g., Field0), which is in an idle state.

[0113] It should also be noted that this application does not limit the relationship between the first field and the second field mentioned above. For example, in some application scenarios, they refer to different fields. Furthermore, in some application scenarios, to better improve the amount of information recorded in each field, the first field and the second field can refer to the same field (e.g., ...). Figure 5 The field 0 shown is used to ensure that the log identifier and the request trace identifier share the same field for transmission.

[0114] Based on the above descriptions of Scenario 1, Log Identifier, and Request Tracking Identifier, it is known that in some application scenarios, a data processing request sent by the data engine to the metadata service can carry request identification information. This request identification information can include both a log identifier and a request tracking identifier. This allows the data engine to provide the metadata service with these identifiers by sending the data processing request. Subsequently, the metadata service can use these identifiers to generate logs. This enables the data engine to query all logs related to the data processing request from both the data engine and the metadata service using these identifiers, thus facilitating the communication between the data engine and the metadata service.

[0115] Scenario 2: In some application scenarios, if the data engine mentioned above does not support log identifiers, but the SDK deployed in the data engine has the ability to generate request tracing identifiers, then the requests generated by the SDK (such as Thrift requests) can carry the request tracing identifiers, so this identifier can be used to reflect the characteristics of the request.

[0116] Based on the above scenario two, in one possible implementation, if the data engine does not provide a log identifier, but the data engine is used to generate a request tracking identifier for the data processing request, then the request identification information carried by the data processing request includes the request tracking identifier, so that the request identification information can better represent the characteristics of the data processing request.

[0117] As can be seen, for the data processing request mentioned above, in some application scenarios, the request identification information carried by the data processing request can include a request tracking identifier. This allows the data engine to provide the request tracking identifier to the metadata server by sending the data processing request. Subsequently, the metadata server can use the request tracking identifier to perform log generation processing. This enables the data engine and the metadata server to query all logs related to the data processing request in the future, thereby facilitating the communication between the data engine and the metadata server.

[0118] Based on the aforementioned content regarding the data processing request, it is understood that, in one possible implementation, the data processing request may carry request identification information, which is used to identify the data processing request. Furthermore, this application does not limit the method in which the data processing request carries the request identification information. For example, in some application scenarios, the request identification information is located in a preset field within the data processing request (e.g., ...). Figure 4 The field shown is 0). Here, the preset field refers to the field present in the data processing request used to record the request identification information. Furthermore, this application does not limit the implementation method of writing the request identification information into the data processing request. For example, in some application scenarios, the request identification information can be written by the SDK in the data engine mentioned above into the preset field in the data processing request, so that the preset field in the data processing request records the request identification information, so that the metadata server can subsequently read the request identification information from the preset field.

[0119] Furthermore, the data engine mentioned above can not only send data processing requests to the metadata server, but also generate engine logs for these requests. These engine logs can represent events related to the data processing request that occur within the data engine (e.g., the generation process of the data processing request). The engine logs refer to logs generated within the data engine that record various events occurring within it. This application does not limit the implementation of the engine logs; for ease of understanding, examples are provided below.

[0120] As an example, in some application scenarios (e.g., the SDK deployed in this data engine is...), Figure 4 Under the new SDK shown, in order to better connect the logs in the data engine with the logs in the metadata server, this application provides a possible implementation of the engine logs mentioned above. In this implementation, if the data engine can also provide request identification information to the metadata server when sending a data processing request (e.g., ...), Figure 4 If the data processing request carries request identification information (e.g., the first request identification information below), and the data engine can be used to generate engine logs for the data processing request, so that the engine logs carry the request identification information, so that the logs can be queried based on the request identification information in the future. This is beneficial for connecting the logs in the data engine with the logs in the metadata server.

[0121] It should be noted that this application does not limit the specific working principle of the data engine in the above paragraph. For example, it can be that when the SDK in the data engine generates and sends a data processing request based on the request identification information above (e.g., the first request identification information below), the SDK can record the request identification information in the log created by the data engine for the data processing request, so that the log can be queried based on the request identification information later. This is beneficial for connecting the log in the data engine with the log in the metadata server.

[0122] Furthermore, this application does not limit the implementation method of S101 above. For example, in some application scenarios (e.g., when the connection method between the data engine and the metadata service system is similar to...), the implementation method may vary. Figure 2 When the connection method shown above is used, if the metadata service includes a metadata service system, then S101 can specifically be: the metadata service system receives a data processing request sent by the data engine so that the metadata service system can subsequently process the data processing request accordingly.

[0123] For example, in some application scenarios (such as when the connection between the data engine and the metadata service system is similar to...), Figure 3 When the connection method shown is used, if the metadata service includes a metadata service gateway and at least one metadata service system, then S101 can specifically be: the metadata service gateway receives a data processing request sent by the data engine, so that the metadata service gateway can subsequently route the data processing request to a metadata service system among the at least one metadata service system according to a preset routing rule, so that the routed metadata service system can process the data processing request accordingly.

[0124] Based on the above content related to S101, in some application scenarios (such as big data scenarios), for metadata servers and data engines that can communicate with each other, the data engine can access the metadata server (such as the metadata service gateway in the metadata server) through the SDK already deployed in the data engine, so that the metadata server can subsequently process the data processing requests (such as the Thrift requests mentioned above) sent by the SDK.

[0125] S102: The metadata server determines the first request identifier information based on the request identifier information carried in the data processing request. The first request identifier information is used to identify the data processing request. If the data engine provides a log identifier and the data engine generates a request tracing identifier for the data processing request, then the request identifier information includes the log identifier and the request tracing identifier. If the data engine does not provide a log identifier, but the data engine generates a request tracing identifier for the data processing request, then the request identifier information includes the request tracing identifier.

[0126] The first request identification information refers to the identification information determined by the metadata server based on the data processing request above, so that the first request identification information can be used to identify the data processing request.

[0127] Furthermore, this application does not limit the process for determining the first request identifier information mentioned above. For example, in some application scenarios, if the data processing request mentioned above carries request identifier information (e.g., Figure 4 If the logID and traceID shown are used as examples, then the process of determining the first request identifier information can be specifically as follows: the metadata server determines the first request identifier information based on the request identifier information carried by the data processing request.

[0128] Furthermore, this application does not limit the implementation of the step "the metadata server determines the first request identifier information based on the request identifier information carried by the data processing request" in the preceding paragraph. For example, it can specifically be that the metadata server can directly determine the request identifier information carried by the data processing request as the first request identifier information. It is evident that in some application scenarios, if the request identifier information carried by the data processing request includes a log identifier and a request tracking identifier, the metadata server can determine the log identifier and request tracking identifier carried by the data processing request as the first request identifier information, so that the first request identifier information includes the log identifier and the request tracking identifier. This allows subsequent log generation processing based on the log identifier and the request tracking identifier, ensuring that the final generated log carries the log identifier and the request tracking identifier, thereby enabling future log query processing based on the log identifier and the request tracking identifier. However, in some application scenarios, if the request identification information carried by the data processing request above includes a request tracking identifier, the metadata server can determine the request tracking identifier carried by the data processing request as the first request identification information, so that the first request identification information includes the request tracking identifier, so that log generation processing can be performed based on the request tracking identifier in the future. This ensures that the final generated log carries the request tracking identifier, and thus log query processing can be performed based on the request tracking identifier in the future.

[0129] Furthermore, in some application scenarios (e.g., when the SDK deployed in the data engine has not been upgraded), the data engine may not be able to provide log identifiers and request tracing identifiers. Therefore, to better facilitate log communication between the data engine and the metadata server in such scenarios, this application also provides a possible implementation of the process for determining the first request identifier information mentioned above. In this implementation, if the data processing request does not carry request identifier information (e.g., the data engine does not provide log identifiers, and the data engine does not generate request tracing identifiers for the data processing request), the process for determining the first request identifier information can specifically be as follows: the metadata server generates the first request identifier information based on at least one request parameter information of the data processing request, so that the first request identifier information can represent the characteristics of the data processing request. The at least one request parameter information is used to describe the characteristics of the data processing request.

[0130] Furthermore, this application does not limit the implementation method of the above-mentioned at least one request parameter information. For example, in some application scenarios, the at least one request parameter information may include at least one of the above-mentioned data engine description information and the above-mentioned interface description information corresponding to the data processing request. These two types of information are described below.

[0131] Regarding the engine description information of the aforementioned data engine, this engine description information is used to describe the relevant content of the data engine that sent the aforementioned data processing request, so that the engine description information can represent the characteristics of the data engine that sent the data processing request. Furthermore, this application does not limit the implementation method of the engine description information. For example, the engine description information may include at least one of the data engine's engine identifier and the data engine's user identifier. The engine identifier is used to uniquely identify the data engine that sent the data processing request. Moreover, this application does not limit the implementation method of the engine identifier. For example, the engine identifier can be implemented using any existing or future information that can identify the data engine (e.g., the data engine's IP address). The user identifier is used to uniquely identify the user of the data engine that sent the data processing request. Moreover, this application does not limit the user identifier. For example, the user identifier can be implemented using any existing or future information that can identify the user (e.g., username, the data engine's login account, etc.).

[0132] Regarding the interface description information corresponding to the data processing request mentioned above, this interface description information is used to describe the relevant content of the interface (e.g., the Thrift interface) used when the data engine sends the data processing request, so that the interface description information can represent the characteristics presented by the interface; moreover, this application does not limit the implementation method of the interface description information. For example, the interface description information may include at least one of the interface identifier corresponding to the data processing request and the interface parameters corresponding to the data processing request. The interface identifier is used to uniquely identify the interface used when the data engine sends the data processing request; moreover, this application does not limit the implementation method of the interface identifier. For example, it can be implemented using any existing or future information that can identify an interface (e.g., interface name). The interface parameters are used to describe the parameters that the data engine needs to provide to the interface when using the interface to send the data processing request (e.g., the parameters passed to the interface); moreover, this application does not limit the implementation method of the interface parameters.

[0133] Furthermore, this application does not limit the implementation of the step "generating first request identification information based on at least one request parameter information of the data processing request" described above. For example, it can specifically be: generating first request identification information based on at least one request parameter information of the data processing request, such that the first request identification information includes the at least one request parameter information and a request tracking identifier generated by the metadata server for the data processing request (e.g., Figure 4The first request identification information can represent the characteristics of the data processing request by at least one of the following: the "request trace ID generated by the metadata service gateway for the data processing request" (e.g., the trace ID generated by the metadata service gateway for the data processing request). Figure 4 The metadata service gateway shown generates a trace identifier (e.g., traceID) to uniquely identify the data processing request based on at least one request parameter information; and this application does not limit the implementation of the process of generating the "request trace identifier generated by the metadata service for the data processing request", for example, it can be similar to the implementation of the SDK generating trace identifier (e.g., traceID) above.

[0134] Based on the above four paragraphs, it can be seen that in some application scenarios, if the data engine supports neither log identifiers nor request tracing identifiers (for example, the SDK deployed in the data engine is...), Figure 4 As shown in the old SDK, when the data engine sends a data processing request to the metadata server, it does not actively provide any information that reflects the characteristics of the data processing request (such as logID, traceID, etc.). Therefore, in order to overcome the defects caused by the separation of logs in the data engine and logs in the metadata server as much as possible, the metadata server can automatically obtain some parameters of the data processing request (such as the interface name of the data processing request, the interface parameters of the data processing request, the IP address of the data engine that sent the data processing request, the number of parameters sent by the data engine, etc.). Based on parameters such as the engine's username, the metadata server can generate identification information corresponding to the data processing request (e.g., these parameters + traceID, etc.). This identification information can accurately represent the characteristics of the data processing request, ensuring that each log generated by the metadata server for the data processing request carries this identification information. This allows for quick and accurate retrieval of each log generated by the metadata server for the data processing request based on the log query scope and all or part of the identification information, thus improving log query efficiency.

[0135] Based on the above information regarding the first request identifier, it can be seen that the first request identifier obtained by the metadata server can be used in some application scenarios (e.g., the SDK deployed in the data engine). Figure 4 In the new SDK shown, the first request identifier information can be the request identifier information carried in the data processing request above (e.g., logID, traceID, etc.). In other application scenarios (e.g., the SDK deployed in the data engine is...), the first request identifier information can be the request identifier information carried in the data processing request above (e.g., logID, traceID, etc.). Figure 4 In the old SDK shown, the first request identification information can be determined based on at least one request parameter information of the data processing request (e.g., the interface name of the data processing request, the interface parameters of the data processing request, the IP address of the data engine that sent the data processing request, the username of the data engine that sent the data processing request, etc.).

[0136] In fact, the metadata server mentioned above may have different structures in different application scenarios, so that S102 above may have different implementation methods. For ease of understanding, two examples are used to illustrate this below.

[0137] Example 1: When the metadata server mentioned above is a metadata service system (e.g., Figure 2 When the metadata service system shown above receives the data processing request sent by the data engine, the metadata service system can determine the first request identification information based on the data processing request, so that the first request identification information can be used to identify the data processing request, thereby enabling the first request identification information to represent the characteristics of the data processing request.

[0138] Example 2, when the metadata server mentioned above includes a metadata service gateway (e.g., Figure 3 The metadata service gateway shown) and at least one metadata service system (e.g., Figure 3 When the metadata service system 1 to metadata service system M are shown, S102 above can specifically be: after the metadata service gateway receives the data processing request sent by the data engine, the metadata service gateway can determine the first request identification information based on the data processing request, so that the first request identification information can be used to identify the data processing request, thereby enabling the first request identification information to represent the characteristics of the data processing request.

[0139] Furthermore, this application does not limit the implementation of the step "the metadata service gateway can determine the first request identification information based on the data processing request" in the preceding paragraph. For example, it can specifically be: if the data processing request carries request identification information, then the metadata service gateway can determine the first request identification information based on the request identification information carried by the data processing request; however, if the data processing request does not carry request identification information (for example, the data engine does not provide log identification, and the data engine does not generate request tracking identification for the data processing request), then the metadata service gateway can generate the first request identification information based on at least one request parameter information of the data processing request.

[0140] Based on the relevant content of S102 above, for the metadata service, after receiving the data processing request sent by the data engine, the metadata service can determine the first request identification information according to the data processing request, so that the first request identification information can be used to identify the data processing request, thereby enabling the first request identification information to represent the characteristics of the data processing request, so that the metadata service can subsequently perform log generation processing according to the first request identification information, so that the final generated log carries the first request identification information.

[0141] S103: The metadata server executes the request processing logic corresponding to the data processing request, and generates a log corresponding to the request processing logic based on the first request identification information. The log carries the first request identification information.

[0142] The request processing logic corresponding to the data processing request refers to the processing logic that the metadata server needs to execute when processing the data processing request, so that the metadata server can realize the data processing requirements represented by the data processing request by executing the request processing logic. In addition, this application does not limit the implementation of the request processing logic. For example, the request processing logic may refer to a sequence of steps pre-set for the data processing request.

[0143] Furthermore, regarding the request processing logic corresponding to the aforementioned data processing request, the log corresponding to this request processing logic is used to describe the relevant content of the request processing logic (e.g., what the request that triggered the request processing logic is, what specific steps the request processing logic includes, etc.), so that the log can indicate which steps the metadata server performed when processing the data processing request, thereby enabling the log to indicate what kind of processing the metadata server performed on the data processing request; moreover, this application does not limit the implementation method of the log corresponding to the request processing logic. For example, the log corresponding to the request processing logic may at least include the execution log of the request processing logic. Here, the execution log refers to the log generated when the request processing logic is executed.

[0144] In addition, the logs corresponding to the above request processing logic carry the identification information of the first request above, so that the logs can be found in the future based on the identification information of the first request. This is beneficial for connecting the logs between the data engine and the metadata server.

[0145] Furthermore, this application does not limit the implementation method of S103 above. For example, in some application scenarios, when the metadata server mentioned above is a metadata service system (e.g., Figure 2When the metadata service system shown above receives the data processing request sent by the data engine, the metadata service system executes the request processing logic corresponding to the data processing request, and generates a log corresponding to the request processing logic based on the first request identification information above, so that the log carries the first request identification information, so that the log can be queried based on the first request identification information in the future. This is conducive to connecting the log in the metadata service system with the log in the data engine.

[0146] For example, in some application scenarios, when the metadata service mentioned above includes a metadata service gateway (e.g., Figure 3 The metadata service gateway shown) and at least one metadata service system (e.g., Figure 3 When the metadata service system 1 to metadata service system M are shown, the above S103 may specifically include steps 11-12 below.

[0147] Step 11: The metadata service gateway generates a data processing message carrying the first request identification information based on the data processing request mentioned above. The metadata service gateway sends the data processing message to the target system in at least one of the metadata service systems mentioned above. The metadata service gateway also generates a metadata service gateway log corresponding to the data processing message based on the first request identification information. The metadata service gateway log carries the first request identification information.

[0148] Here, a data processing message refers to the message used when a metadata service gateway transmits the data processing requirements described by the data processing request to a metadata service system, so that the data processing message carries the data processing requirements described by the data processing request.

[0149] Furthermore, this application does not limit the implementation method of the data processing message mentioned above. For example, the data processing message can be implemented using the Thrift protocol format.

[0150] Furthermore, this application does not limit the generation process of the aforementioned data processing message. For example, if the aforementioned data processing request carries first request identification information, the specific generation process of the data processing message can be: the metadata service gateway directly sends the data processing request as the data processing message to the corresponding metadata service system. As another example, if the data processing request does not carry the first request identification information, the specific generation process of the data processing message can be: the metadata service gateway writes the first request identification information into the third field of the data processing request. Here, the third field refers to a field existing in the data processing request that can be used to record the first request identification information; and this application does not limit the implementation method of the third field. For example, the third field can be implemented using a field in the data processing request that is in an idle state. It can be seen that when the data processing request adopts... Figure 5 When implementing the Thrift protocol format shown, this third field can be implemented using field 0 (e.g., Field0), which is in an idle state.

[0151] Furthermore, for the data processing message mentioned above, the metadata service gateway log corresponding to the data processing message is used to record matters related to the data processing message that occur within the metadata service gateway; moreover, the metadata service gateway log carries first request identification information so that the metadata service gateway log can be queried based on the first request identification information in the future, which is beneficial for connecting the logs in the metadata service gateway with the logs in the data engine.

[0152] The target system refers to the metadata service system selected by the metadata service gateway from at least one metadata service system mentioned above, which is used to process the data processing request mentioned above; and this application does not limit the method of determining the target system. For example, it can be implemented using metadata service system selection rules pre-configured for the metadata service gateway.

[0153] Based on the relevant content of step 11 above, for the metadata service gateway in the metadata service, after receiving the data processing request sent by the data engine, the metadata service gateway can generate a data processing message carrying the first request identification information according to the data processing request. The metadata service gateway can then send the data processing message to the target system in at least one of the metadata service systems mentioned above. In addition, the metadata service gateway can generate a metadata service gateway log corresponding to the data processing message (e.g., a log describing the generation process, sending process, etc. of the data processing message) according to the first request identification information, so that the metadata service gateway log carries the first request identification information. This allows the metadata service gateway log to be queried based on the first request identification information, which is beneficial for connecting the logs in the metadata service gateway with the logs in the data engine.

[0154] Step 12: The target system executes the message processing logic corresponding to the data processing message, and the target system generates a system log corresponding to the message processing logic based on the first request identification information carried by the data processing message. The system log carries the first request identification information.

[0155] The message processing logic corresponding to the data processing message refers to the processing logic that the target system needs to execute when processing the data processing message, so that the target system can realize the data processing requirements represented by the data processing message by executing the message processing logic. Furthermore, this application does not limit the implementation method of the message processing logic; for example, the message processing logic may refer to a pre-defined sequence of steps for the data processing message.

[0156] Furthermore, regarding the system log corresponding to the aforementioned message processing logic, this system log is used to describe the relevant content of the message processing logic (e.g., what the Thrift message that triggered the message processing logic is, what specific steps the message processing logic includes, etc.), so that the system log can indicate which steps the target system performed when processing the aforementioned data processing message; and this application does not limit the implementation method of the system log, for example, the system log may at least include the execution log of the message processing logic. Here, the execution log refers to the log generated when the message processing logic is executed.

[0157] In addition, the system log corresponding to the message processing logic mentioned above carries the identification information of the first request mentioned above, so that the log can be found in the future based on the identification information of the first request. This is beneficial for connecting the logs in the data engine, the logs in the metadata service gateway, and the logs in the metadata service system.

[0158] Based on the relevant content of steps 11 to 12 above, in some application scenarios, if the metadata service includes a metadata service gateway and at least one metadata service system, after the metadata service gateway receives the data processing request sent by the data engine, it can generate a data processing message carrying a first request identifier based on the data processing request. The metadata service gateway then sends the data processing message to the target system in at least one metadata service system and generates a metadata service gateway log corresponding to the data processing message (e.g., a log describing the generation and sending process of the data processing message) based on the first request identifier. Then, the target system executes the message processing logic corresponding to the data processing message and generates a system log corresponding to the message processing logic based on the first request identifier carried by the data processing message. This system log carries the first request identifier so that the metadata service gateway log and the system log can be located based on the first request identifier. This facilitates the connection between the logs in the data engine, the logs in the metadata service gateway, and the logs in the metadata service system.

[0159] Based on the relevant content of S101 to S103 above, for the lake warehouse metadata service log generation method provided in this application embodiment, after the metadata service (e.g., a metadata service consisting of a metadata service gateway and multiple metadata service systems) receives a data processing request sent by the data engine, the metadata service determines a first request identification information based on the data processing request, so that the first request identification information is used to identify the data processing request, thereby enabling the first request identification information to represent the characteristics of the data processing request (e.g., log identification, request tracking identification, etc.); then the metadata service executes the request processing logic corresponding to the data processing request, and generates a log corresponding to the request processing logic based on the first request identification information, so that the log carries the first request... The system requests identification information, which serves as a medium to associate the logs corresponding to the request processing logic with the logs generated by the data engine for the data processing request. This allows the system to connect the logs in the metadata server with the logs in the data engine, enabling subsequent queries to retrieve complete logs related to any data processing request in the data engine (i.e., the logs recorded by the metadata server for the data processing request and the logs recorded by the data engine for the data processing request). This effectively overcomes the shortcomings caused by the separation between the logs in the data engine and the logs in the metadata service system, thereby improving log query performance (e.g., increasing query accuracy and efficiency).

[0160] In fact, in order to better avoid currently recorded identification information (e.g., Figure 4 The logID and traceID shown may interfere with the next data processing. This application also provides a possible implementation of the above-described method for generating logs for the Lake Warehouse metadata service. For ease of understanding, the following is an explanation with examples.

[0161] As an example, when the metadata service mentioned above includes a metadata service gateway and at least one metadata service system, the method for generating metadata service logs in the above-mentioned Lake Warehouse may include steps 21-27 below.

[0162] Step 21: The metadata service gateway receives the data processing request sent by the data engine.

[0163] It should be noted that the relevant content of step 21 is similar to the relevant content in S101 above.

[0164] Step 22: The metadata service gateway determines the first request identification information based on the data processing request above. The first request identification information is used to identify the data processing request.

[0165] It should be noted that the implementation method of determining the first request identifier information in step 22 is similar to the implementation method of determining the first request identifier information in S102 above.

[0166] Step 23: The metadata service gateway generates a data processing message carrying the first request identification information based on the data processing request above. The metadata service gateway sends the data processing message to the target system in at least one metadata service system above. The metadata service gateway also generates a metadata service gateway log corresponding to the data processing message based on the first request identification information. The metadata service gateway log carries the first request identification information.

[0167] It should be noted that for the relevant content of step 23, please refer to the relevant content in step 11 above.

[0168] Step 24: The target system executes the message processing logic corresponding to the data processing message, and the target system generates a system log corresponding to the message processing logic based on the first request identification information carried by the data processing message. The system log carries the first request identification information.

[0169] It should be noted that the relevant content of step 24 can be found in the relevant content of step 12 above.

[0170] Step 25: After the target system executes the message processing logic corresponding to the data processing message above, the target system clears the first request identification information recorded in the target system, and the target system sends the first feedback message to the metadata service gateway.

[0171] The first feedback message refers to the message sent by the target system to the metadata service gateway to convey that the message processing logic corresponding to the data processing message above has been executed, so that the first feedback message can inform the metadata service gateway that the target system has processed the data processing message.

[0172] Furthermore, this application does not limit the implementation of the step "the target system clears the first request identification information recorded in the target system" in step 25 above. For example, this step can specifically be: the target system clears the first request identification information recorded in the target system's MDC to avoid the first request identification information interfering with the next data processing process of the target system.

[0173] As can be seen, for the target system described above, in one possible implementation, if the target system's MDC records the first request identification information (e.g., Figure 4 If the logID and traceID are shown, then the target system can be used to: after determining the message processing logic corresponding to the data processing message above, clean up the first request identification information recorded in the MDC of the target system.

[0174] Based on the relevant content in step 25 above, it can be seen that for the target system mentioned above (for example, Figure 3 For the metadata service system 1, metadata service system 2, ..., or metadata service system M shown, when the target system detects the message processing logic corresponding to the data processing message above, the target system clears the first request identification information recorded in the target system (for example, clearing the log identifier and request tracking identifier recorded in the target system's MDC, etc.), and the target system sends a first feedback message to the metadata service gateway so that the metadata service gateway can know from the first feedback message that the target system has processed the data processing message.

[0175] Step 26: After the metadata service gateway receives the first feedback message mentioned above, the metadata service gateway clears the first request identification information recorded in the metadata service gateway and sends the second feedback message to the data engine.

[0176] The second feedback message refers to the message sent by the metadata service gateway to the data engine to convey that the request processing logic corresponding to the data processing request has been completed, so that the second feedback message can inform the data engine that the metadata service gateway has processed the data processing request.

[0177] Furthermore, this application does not limit the implementation of the step "the metadata service gateway clears the first request identification information recorded in the metadata service gateway" in step 26 above. For example, this step can specifically be: the metadata service gateway clears the first request identification information recorded in the MDC of the metadata service gateway to avoid the first request identification information interfering with the next data processing process of the metadata service gateway.

[0178] As can be seen, for the metadata service gateway mentioned above, in one possible implementation, if the metadata service gateway's MDC records the first request identifier information (e.g., Figure 4 If the logID and traceID are shown, then the metadata service gateway can be used to: clean up the first request identification information recorded in the MDC of the metadata service gateway after receiving the first feedback message sent by the target system mentioned above.

[0179] Based on the relevant content in step 26 above, it can be seen that for the metadata service gateway mentioned above (for example, Figure 3 For the metadata service gateway shown, after receiving the first feedback message sent by the target system above, the metadata service gateway clears the first request identification information recorded in the metadata service gateway (for example, clearing the log identifier and request tracking identifier recorded in the MDC of the metadata service gateway), and sends a second feedback message to the data engine above, so that the data engine can know from the second feedback message that the metadata service gateway has processed the data processing request above.

[0180] Step 27: After the data engine receives the second feedback message, if the data engine records the target identifier corresponding to the first request identifier information, the data engine will clear the target identifier recorded in the data engine.

[0181] Wherein, the target identifier corresponding to the first request identifier information refers to the identifier in the first request identifier information that needs to be cleared; and this application does not limit the implementation method of the target identifier, for example, in some application scenarios (e.g., deployed in a data engine) Figure 4 The new SDK shown, but the data engine does not support logID), under this, the target identifier can refer to all identifiers present in the first request identifier information (e.g., Figure 4 (The logID and traceID are shown).

[0182] For example, in some application scenarios (e.g., deployed in a data engine) Figure 4 The new SDK shown, and the data engine supports logID, if the data engine supports log identifier configuration (e.g., the data engine supports MDC, etc.), and the data engine has log identifiers and request tracing identifiers, then the target identifier can be the request tracing identifier.

[0183] Based on the above content, in one possible implementation, if the data engine supports log identifier configuration (e.g., the data engine supports MDC, etc.), and the data engine records log identifiers and request tracing identifiers, then step 27 above can specifically be: after the data engine receives the second feedback message, the data engine clears the request tracing identifier recorded in the data engine (e.g., clears the request tracing identifier in the MDC of the data engine, but retains the log identifier).

[0184] Based on the content of steps 21 to 27 above, it is known that in some application scenarios, for the data engine, metadata service gateway, and metadata service system mentioned above, after processing the data processing request sent by the data engine through the cooperation between the metadata service gateway and the metadata service system, it is necessary to clean up the identification information recorded in the MDC of the metadata service system (e.g., Figure 4 The logID and traceID shown are cleared, and all or part of the identification information recorded in the MDC of the metadata service gateway and the MDC of the data engine are cleared. This can effectively avoid the interference caused by the identification information in the next data processing process.

[0185] To better understand the lake warehouse metadata service log generation method provided in this application, the following will use... Figures 3-4 The application scenarios shown are illustrated as examples.

[0186] As an example, if a data engine n (e.g., Spark) communicates with a metadata service gateway using the Thrift protocol, and this metadata service gateway communicates with a metadata service system m (e.g., ... Figure 4 If any HiveMetastore (e.g., shown) communicates with each other using the Thrift protocol, where n is a positive integer, n≤N, and m is a positive integer, m≤M, then the lake warehouse metadata service log generation scheme provided in this application may include steps 31-40 below.

[0187] Step 31: Data engine n sends a data processing request (e.g., a Thrift request) to the metadata service gateway through the SDK deployed in data engine n, so that data engine n can access the metadata service gateway through the SDK.

[0188] Step 32: For the SDK deployed in data engine n mentioned above, the SDK can obtain the logId from data engine n and put the logID into Field 0 of the Thrift message (that is, ...). Figure 5 Field 0 is shown.

[0189] It should be noted that this application does not limit the specific implementation of step 32 above. For example, step 32 may include steps 321-322 below.

[0190] Step 321: If the data engine n mentioned above does not support MDC, then the data engine n can set the logId through the SDK. For example, Spark can specify the applicationId as the logId.

[0191] Step 322: If the data engine n mentioned above supports MDC, the SDK deployed in the data engine n will automatically read the logId from MDC.

[0192] Step 33: For the data engine n mentioned above, each time it generates a Thrift request (such as the data processing request mentioned above), the SDK deployed in the data engine n will automatically generate a globally unique traceId so that the traceId can uniquely identify the current Thrift request. The SDK will also put the traceId into Field0 of the Thrift message. Subsequently, the logs of the Thrift request will automatically include the logId and traceId.

[0193] Step 34: For the SDK deployed in the data engine n mentioned above, when the SDK initiates a Thrift request, it will record the logId and traceId of the Thrift request in the log of the data engine n, which is convenient for tuning and troubleshooting data engines such as Spark / Flink that do not support MDC.

[0194] Step 35: For the metadata service gateway mentioned above, after receiving the Thrift request carrying the Thrift message sent by the data engine n mentioned above, the metadata service gateway first marks the binary stream (for example, marks the start and end positions of the Thrift message, etc.), then the metadata service gateway reads the logId and traceId from Field0 of the Thrift message, stores them in the metadata service gateway's MDC, and resets the binary stream. Subsequently, the logs related to the Thrift request in the metadata service gateway will automatically include the logId and traceId.

[0195] Step 36: For the metadata service gateway mentioned above, it begins processing the Thrift requests mentioned above, routing them to the metadata service system m (e.g., ...) as needed. Figure 4 (as shown in the example of a Hive Metastore). Here, the metadata service system m refers to the metadata service system determined by the metadata service gateway that needs to process the Thrift request.

[0196] Step 37: For the metadata service system m mentioned above, after receiving a Thrift request carrying a Thrift message, the metadata service system m first marks the binary stream, then reads the traceId and logId from Field0 of the Thrift message, and resets the binary stream. Subsequently, the logs related to the Thrift request in the metadata service system m will automatically include the logId and traceId.

[0197] Step 38: For the metadata service system m mentioned above, the metadata service system m processes the Thrift request mentioned above so that after the metadata service system m finishes processing the Thrift request, the metadata service system m sends a feedback message to the metadata service gateway, and the metadata service system m cleans up the logId and traceId in the MDC of the metadata service system m.

[0198] Step 39: For the metadata service gateway above, after receiving the feedback message sent by the metadata service system m above, the metadata service gateway sends the feedback message to the data engine n above, and cleans up the logId and traceId in the MDC of the metadata service gateway.

[0199] Step 40: For the data engine n mentioned above, after the SDK deployed in the data engine n receives the feedback message sent by the metadata service gateway mentioned above, if the data engine n supports MDC, then the data engine n will clean up the traceId in the MDC of the data engine n, while the logId remains unchanged.

[0200] Based on the content of steps 31 to 40 above, it is known that in some application scenarios, one can leverage... Figure 4 The logId and traceId shown connect the logs in the data engine, the logs in the metadata service gateway, and the logs in the metadata service system. This enables the connection of data access logs in big data scenarios at a low cost, ensuring the smooth implementation of the entire solution.

[0201] Based on the above-described method for generating lake warehouse metadata service logs, this application also provides a method for querying lake warehouse metadata service logs, which is explained and illustrated below with some accompanying figures. Figure 6 As shown, the lake warehouse metadata service log query method may include S601-S604 below. Among them, the... Figure 6 A flowchart illustrating a method for querying lake warehouse metadata service logs provided in an embodiment of this application.

[0202] S601: The metadata server receives a log query request sent by the log request device. This log query request is used to request the logs corresponding to the data processing request sent by the data engine to the metadata server.

[0203] In this context, a log request device refers to a device with log query requirements. Furthermore, this application does not limit the implementation method of the log request device. For example, in some application scenarios, the log request device may refer to a data engine capable of data communication with the aforementioned metadata server. Similarly, in some application scenarios, the log request device may refer to other devices besides the metadata server and the data engine, and these other devices may be able to obtain some log content from requests made by the metadata server and the data engine.

[0204] A log query request is a request sent by the log requesting device to the metadata server to request some log content. For example, this log query request can be used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, so that the logs can indicate what kind of processing the metadata server performed on the data processing request.

[0205] Furthermore, this application does not limit the implementation method of the log query request described above. For ease of understanding, three examples are provided below.

[0206] Example 1: In some application scenarios (such as when a new SDK is deployed in the data engine and the data engine supports log identification), when the log query request is used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, the log query request can carry the log identifier and request tracking identifier corresponding to the data processing request, so that the metadata server can subsequently query the logs related to the data processing request based on these two identifiers.

[0207] Example 2: In some application scenarios (e.g., a new SDK is deployed in the data engine but the data engine does not support log identifiers), when the log query request mentioned above is used to request the logs corresponding to the data processing request sent by the data engine to the metadata server, the log query request can carry the request tracking identifier corresponding to the data processing request, so that the metadata server can query the logs related to the data processing request based on this identifier.

[0208] Based on the above two paragraphs, it can be seen that in one possible implementation, when the log query request mentioned above is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server, the log query request can carry the request identification information corresponding to the data processing request (e.g., Figure 4 The logID and traceID shown are used to identify the characteristics of the data processing request.

[0209] Example 3: In some application scenarios (e.g., where an older SDK is deployed in the data engine), when the log query request mentioned above is used to request logs corresponding to a data processing request sent by the data engine to the metadata server, the log query request may carry a log query time range and at least one request parameter information of the data processing request. The log query time range indicates the time range required when querying logs related to the data processing request from the metadata server; moreover, this embodiment does not limit the method of obtaining the log query time range. For example, it can be manually provided by the user or automatically determined based on the sending time of the data processing request. It should be noted that the relevant content of at least one request parameter information of the data processing request is described above.

[0210] Based on the above content regarding S601, it is known that for the metadata server, after receiving a data processing request sent by a data engine, the metadata server can not only process the data processing request, but also record some logs for the data processing request. These logs can indicate what kind of processing the metadata server performed on the data processing request. In the future, when the metadata server receives a log query request sent by a log request device for the data processing request, the metadata server can feed back these logs to the log request device, so that the log request device can know from these logs what kind of processing the metadata server performed on the data processing request.

[0211] S602: The metadata server determines the request identification information corresponding to the data processing request carried in the log query request as the second request identification information, which is used to identify the data processing request.

[0212] The second request identifier refers to the identifier required when querying logs on the metadata server, so that the second request identifier can represent the characteristics of the data processing request described by the log query request above.

[0213] Furthermore, this application does not limit the implementation of the process for determining the identification information in the second request above. For ease of understanding, four examples are provided below.

[0214] Example 1: When the log query request above is used to query the log corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries the log identifier and request tracking identifier corresponding to the data processing request, the process of determining the second request identifier information above can be as follows: The metadata server extracts the log identifier and the request tracking identifier from the log query request as the second request identifier information.

[0215] Example 2: When the log query request above is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries a request tracking identifier corresponding to the data processing request, the process of determining the second request identifier information above can be as follows: the metadata server extracts the request tracking identifier from the log query request and uses it as the second request identifier information.

[0216] Based on the above two paragraphs, it can be seen that in one possible implementation, when the log query request mentioned above is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server, the log query request carries the request identification information corresponding to the data processing request (e.g., Figure 4The process of determining the second request identifier information mentioned above can be as follows: the metadata server determines the request identifier information carried by the log query request as the second request identifier information.

[0217] Example 3: When the log query request mentioned above is used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, and the log query request does not carry the request identifier information corresponding to the data processing request, but carries the log query time range and at least one request parameter information of the data processing request, the process of determining the second request identifier information mentioned above can be specifically as follows: the metadata server generates the second request identifier information based on the at least one request parameter information carried by the log query request, so that the second request identifier information is used to identify the data processing request. It should be noted that this application does not limit the implementation of the second request identifier information. For example, the implementation of the second request identifier information is similar to the implementation of the first request identifier information mentioned above; moreover, this application does not limit the implementation of the aforementioned step of "generating the second request identifier information based on the at least one request parameter information carried by the log query request". For example, the implementation of this step is similar to the implementation of the aforementioned step of "generating the first request identifier information based on the at least one request parameter information of the data processing request".

[0218] Example 4: When the log query request is used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries a log query time range and at least one request parameter information of the data processing request, the determination process of the second request identifier information can be specifically as follows: The metadata server determines the second request identifier information from the log records of the metadata server based on the log query time range carried by the log query request and at least one request parameter information of the data processing request, so that the second request identifier information is used to identify the data processing request. In this way, the metadata server can lock the second request identifier information by using the record content in its logs. The log records of the metadata server are used to record some events within the metadata server; and the logs in the log records of the metadata server are all generated using any embodiment of the Lakewaregia metadata service log generation method provided in this application, so that the logs in the log records of the metadata server all carry the identifier information of the corresponding data processing request.

[0219] It should be noted that this application does not limit the implementation of the step in the preceding paragraph where "the metadata server determines the second request identification information from the log records of the metadata server based on the log query time range carried by the log query request and at least one request parameter information of the data processing request." For example, it can specifically be as follows: First, search for target logs in the log records of the metadata server that meet the log query time range and carry at least one request parameter information of the data processing request, so that the target log carries at least one request parameter information and the time involved in the target log (e.g., log creation time) is within the log query time range; then, determine the second request identification information based on the identification information (e.g., traceID) that appears in these target logs to identify a certain data processing request, so that the second request identification information at least includes the content of "identification information for identifying a certain data processing request".

[0220] Based on the relevant content of S602 above, for the metadata server, after receiving the log query request sent by the log request device, if the log query request is used to request the query of the log corresponding to the data processing request sent by the data engine to the metadata server, the metadata server can determine the second request identification information based on the log query request, so that the second request identification information can be used to identify the data processing request, thereby enabling the second request identification information to represent the characteristics of the data processing request.

[0221] S603: The metadata server determines the log query result corresponding to the log query request based on the logs carrying the second request identifier information existing in the log records of the metadata server; the logs in the log records are generated using any implementation method of the Lake Warehouse metadata service log generation method provided in this application.

[0222] The log query result corresponding to the log query request is used to represent the log determined in the metadata server in response to the log query request, so that the log query result can indicate what kind of processing the metadata server performed in response to the data processing request mentioned above.

[0223] Furthermore, this application does not limit the content related to S603 above. For example, when the log query request above is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries the request identification information corresponding to the data processing request (e.g., Figure 4When the logId and traceId are shown, S603 can specifically be as follows: The metadata server first queries the log records of the metadata server that carry the request identification information, so that the log carrying the request identification information can indicate what kind of processing the metadata server performed for the data processing request; then, the metadata server determines the log query result corresponding to the log query request based on the log carrying the request identification information, so that the log query result includes the log carrying the request identification information, thereby enabling the log query result to indicate what kind of processing the metadata server performed for the data processing request.

[0224] For example, when the log query request mentioned above is used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries the log query time range and at least one request parameter information of the data processing request, and the second request identifier information mentioned above is determined based on some or all of the information in the log query request, then S603 can specifically be: the metadata server determines the log query result corresponding to the log query request based on the logs in the log records of the metadata server that meet the log query time range and carry the second request identifier information (e.g., the target log mentioned above), so that the log query result includes the "logs that meet the log query time range and carry the second request identifier information". This enables the rapid location of logs related to a certain data processing request in the metadata server even when the data engine does not support log identifiers and request tracking identifiers, which is beneficial to improving the log query effect.

[0225] Based on the relevant content of S603 above, for the metadata service, the metadata service can use the logs containing the second request identifier information from its log records as the log query result corresponding to the log query request. This allows the log query result to accurately indicate what kind of processing the metadata service performed on the data processing request involved in the log query request. Subsequently, the metadata service can feed back the log query result to the log request device so that the log request device can learn from the log query result what kind of processing the metadata service performed on the data processing request.

[0226] Furthermore, this application does not limit the working principle of the aforementioned log request device. For example, it may at least include: integrating the log query results fed back by the aforementioned metadata server and the logs recorded in the aforementioned data engine for the aforementioned data processing request to obtain an integrated log. The integrated log includes the log query results fed back by the metadata server and the logs recorded in the data engine for the data processing request. Thus, the integrated log includes the logs recorded by the metadata server and the data engine respectively for the data processing request. In this way, the integrated log is used to describe the process executed by the data engine and the metadata server for the data processing request. This allows the log request device to integrate all logs related to the data processing request, thereby improving the log query effect for the data processing request. The “logs recorded in the data engine for the data processing request” refers to the logs related to the data processing request that exist in the log records of the data engine; and this application does not limit the method of obtaining the “logs recorded in the data engine for the data processing request”. For example, it can be implemented using any existing or future log acquisition method.

[0227] For example, to improve log retrieval efficiency, when the log query request mentioned above is used to query the logs corresponding to the data processing request sent by the data engine to the metadata server, and the log query request carries the request identification information corresponding to the data processing request (e.g., Figure 4 When the logId and traceId are shown, the "logs recorded in the data engine for this data processing request" mentioned above can refer to the logs in the data engine's log records that carry the request identification information. This enables the use of identification information to quickly query the logs recorded by each device for this data request, thereby improving log query efficiency.

[0228] Based on the relevant content in S601 to S604 above, for the metadata server mentioned above, each log record in the metadata server's log records carries the identification information of the data processing request corresponding to that log. This allows the logs related to a data processing request to be quickly found from the metadata server's log records based on the identification information of a data processing request. In this way, the log information can be used to connect the logs in the metadata server and the logs in the data engine, thereby improving the log query effect for a specific data processing request (e.g., log query efficiency, log integrity, etc.).

[0229] Based on the above-mentioned methods for generating and querying lake warehouse metadata service logs, it is clear that the technical solution provided in this application has the advantages shown in ①-② below.

[0230] ① For data engines that can upgrade their SDKs (i.e., data engines with deployed new SDKs), the advantages are as follows: If the data engine already supports logId through MDC, and the SDK deployed in the data engine can automatically generate traceId, then the data engine can pass the logId and traceId through the SDK to the metadata service gateway. The metadata service gateway then passes the logId and traceId to the corresponding metadata service system, ensuring that the logs in the data engine, the metadata service gateway, and the metadata service system all carry the logId and traceId. This allows for quick querying of logs recorded by these three devices for the same data request using the logId and traceId, thus achieving interconnection at a lower cost. In big data scenarios, data access logs are crucial for improving log query performance. However, if the data engine (e.g., Spark, Flink) does not support logId via MDC, the SDK deployed within the data engine can still automatically generate a traceId and pass it through to the metadata service gateway. The metadata service gateway then passes the traceId to the corresponding metadata service system, ensuring that the logs in the data engine, the metadata service gateway, and the metadata service system all carry the traceId. This allows for quick querying of logs recorded by these three devices for a single data request, achieving a low-cost connection of data access logs in big data scenarios and thus improving log query performance.

[0231] ② For data engines that cannot upgrade their SDK (i.e., data engines with the old SDK deployed), the advantages are as follows: Since the old SDK does not pass any logId or traceId to the metadata service gateway, in order to better connect the logs in the data engine with the logs in the metadata service gateway, the metadata service gateway obtains the parameters of each Thrift request (such as the requested interface name, parameters, client IP address, username, etc.). The metadata service gateway automatically generates a traceId based on these parameters and passes the traceId to the metadata service system (such as Hive Metastore). This ensures that both the logs in the metadata service gateway and the logs in the metadata service system carry the traceId, so that the traceId can be used to quickly query the logs recorded by the two devices for a data request. This allows for future optimization and troubleshooting by searching the metadata service gateway logs based on the time range and request parameters, identifying the traceId, and then connecting the logs in the metadata service gateway and the metadata service system based on the traceId, thus improving the log query efficiency.

[0232] Based on the lake warehouse metadata service log generation method provided in the embodiments of this application, the embodiments of this application also provide a lake warehouse metadata service log generation device, which is described below in conjunction with... Figure 7 Explanation and clarification will be provided. Among them, Figure 7 This is a schematic diagram of a lakeware metadata service log generation device provided in an embodiment of this application. It should be noted that for technical details of the lakeware metadata service log generation device provided in this embodiment, please refer to the relevant content of the lakeware metadata service log generation method described above.

[0233] like Figure 7 As shown in the embodiment of this application, the lake warehouse metadata service log generation device 700 includes:

[0234] The first receiving unit 701 is used to receive data processing requests sent by the data engine;

[0235] The first determining unit 702 is configured to determine first request identification information based on the request identification information carried in the data processing request, wherein the first request identification information is used to identify the data processing request; if the data engine provides a log identifier and the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the log identifier and the request tracing identifier; if the data engine does not provide a log identifier, but the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the request tracing identifier.

[0236] The request processing unit 703 is used to execute the request processing logic corresponding to the data processing request, and generate a log corresponding to the request processing logic based on the first request identification information, wherein the log carries the first request identification information.

[0237] In one possible implementation, the log identifier is determined by the software development kit (SDK) in the data engine based on the target parameters of the data engine;

[0238] If the data engine supports log identifier configuration, then the target parameter is a log identifier pre-configured for the data engine;

[0239] If the data engine does not support log identifier configuration, then the target parameter is the task identifier, which is used to identify the data analysis task that triggers the request generation logic of the data processing request. The data analysis task is created by the data engine in response to user operations.

[0240] In one possible implementation, the request tracking identifier is used to uniquely identify the data processing request; and / or, the request tracking identifier is generated by the SDK in the data engine.

[0241] In one possible implementation, the request identification information is located in a preset field in the data processing request; and / or, the request identification information is written by the SDK in the data engine into the preset field in the data processing request.

[0242] In one possible implementation, the first determining unit 702 is further configured to determine first request identification information based on at least one request parameter information of the data processing request if the data processing request does not carry request identification information; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request.

[0243] In one possible implementation, the first request identification information includes at least one of the at least one request parameter information and a request tracking identifier generated by the metadata server for the data processing request.

[0244] In one possible implementation, the first determining unit 702 is specifically configured to: if the data engine does not provide a log identifier and the data engine does not generate a request tracking identifier for the data processing request, then determine the first request identifier information based on at least one request parameter information of the data processing request.

[0245] In one possible implementation, the engine description information includes at least one of the engine identifier of the data engine and the user identifier of the data engine.

[0246] In one possible implementation, the interface description information includes at least one of the interface identifier corresponding to the data processing request and the interface parameters corresponding to the data processing request.

[0247] In one possible implementation, the lake warehouse metadata service log generation device 700 is a metadata service system.

[0248] In one possible implementation, the lake warehouse metadata service log generation device 700 includes a metadata service gateway and at least one metadata service system;

[0249] The first determining unit 702 is specifically used for: determining the first request identification information by the metadata service gateway based on the data processing request;

[0250] The request processing unit 703 is specifically configured to: generate a data processing message carrying the first request identifier information based on the data processing request by the metadata service gateway; send the data processing message to the target system in the at least one metadata service system by the metadata service gateway; generate a metadata service gateway log corresponding to the data processing message based on the first request identifier information; the metadata service gateway log carrying the first request identifier information; execute message processing logic corresponding to the data processing message by the target system; and generate a system log corresponding to the message processing logic based on the first request identifier information carried in the data processing message; the system log carrying the first request identifier information.

[0251] In one possible implementation, the lake warehouse metadata service log generation device 700 further includes:

[0252] The identifier clearing unit is configured to, after executing the message processing logic corresponding to the data processing message, have the target system clear the first request identifier information recorded in the target system, and have the target system send a first feedback message to the metadata service gateway; after the metadata service gateway receives the first feedback message, have the metadata service gateway clear the first request identifier information recorded in the metadata service gateway, and have the metadata service gateway send a second feedback message to the data engine, so that after the data engine receives the second feedback message, if the data engine records a target identifier corresponding to the first request identifier information, the data engine will clear the target identifier recorded in the data engine.

[0253] In one possible implementation, if the data engine supports log identifier configuration and the first request identifier information recorded in the data engine includes a log identifier and a request tracing identifier, then the target identifier is the request tracing identifier.

[0254] In one possible implementation, if the data processing request carries the first request identification information, the data engine is used to generate an engine log for the data processing request, and the engine log carries the first request identification information.

[0255] Based on the aforementioned content regarding the Lakeware Metadata Service Log Generation Device 700, it can be understood that the Lakeware Metadata Service Log Generation Device 700 provided in this application embodiment is integrated into the aforementioned metadata service. Furthermore, the working principle of the Lakeware Metadata Service Log Generation Device 700 is as follows: After receiving a data processing request sent by the data engine, the Lakeware Metadata Service Log Generation Device 700 determines a first request identification information based on the data processing request. This first request identification information is used to identify the data processing request, thereby enabling the first request identification information to represent the characteristics of the data processing request (e.g., log identification, request tracking identification, etc.). Then, the Lakeware Metadata Service Log Generation Device 700 executes the request processing logic corresponding to the data processing request and generates the request based on the first request identification information. The logs corresponding to the processing logic are processed so that they carry the first request identification information. This first request identification information can then be used as a medium to associate the logs corresponding to the request processing logic with the logs generated in the data engine for the data processing request. This allows the logs in the Lakewarehouse metadata service log generation device 700 to be connected with the logs in the data engine, so that the complete logs related to any data processing request in the data engine can be queried using the first request identification information (that is, the logs recorded by the Lakewarehouse metadata service log generation device 700 for the data processing request and the logs recorded by the data engine for the data processing request). This effectively overcomes the defects caused by the separation between the logs in the data engine and the logs in the metadata service system, thereby improving the log query effect (e.g., improving query accuracy and query efficiency).

[0256] Based on the lake warehouse metadata service log query method provided in the embodiments of this application, the embodiments of this application also provide a lake warehouse metadata service log query device, which is described below in conjunction with... Figure 8 Explanation and clarification will be provided. Among them, Figure 8 This is a schematic diagram of a lake warehouse metadata service log query device provided in an embodiment of this application. It should be noted that for technical details of the lake warehouse metadata service log query device provided in this embodiment, please refer to the relevant content of the lake warehouse metadata service log query method described above.

[0257] like Figure 8 As shown in the embodiment of this application, the lake warehouse metadata service log query device 800 includes:

[0258] The second receiving unit 801 is used to receive a log query request sent by the log request device. The log query request is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server.

[0259] The second determining unit 802 is used to determine the request identification information corresponding to the data processing request carried in the log query request as the second request identification information, and the second request identification information is used to identify the data processing request;

[0260] The third determining unit 803 is used to determine the log query result corresponding to the log query request based on the logs carrying the second request identifier information existing in the log records of the metadata server; the logs in the log records are generated using any embodiment of the Lakewarehouse metadata service log generation method provided in this application.

[0261] In one possible implementation, the second determining unit 802 is further configured to: if the log query request does not carry request identification information corresponding to the data processing request, but the log query request carries a log query time range and at least one request parameter information of the data processing request, then generate the second request identification information based on the at least one request parameter information carried by the log query request; or, determine the second request identification information from the log records of the metadata server based on the log query time range carried by the log query request and at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request;

[0262] The third determining unit 803 is further configured to: determine the log query result corresponding to the log query request based on the log records of the metadata server that exist, conform to the log query time range, and carry the second request identifier information.

[0263] In one possible implementation, the log request device is used to integrate the log query results fed back by the metadata server and the logs recorded in the data engine for the data processing request to obtain an integrated log, which is used to describe the process executed by the data engine and the metadata server for the data processing request.

[0264] In one possible implementation, if the log query request carries request identification information corresponding to the data processing request, then the log recorded in the data engine for the data processing request refers to the log in the data engine's log records that carries the request identification information.

[0265] In one possible implementation, the log request device is the data engine.

[0266] Based on the aforementioned content regarding the Lakeware Metadata Service Log Query Device 800, it is understood that the Lakeware Metadata Service Log Query Device 800 provided in this application embodiment is integrated into the metadata server. Furthermore, the working principle of the Lakeware Metadata Service Log Query Device 800 is as follows: each log entry in the Lakeware Metadata Service Log Query Device 800 carries the identification information of the corresponding data processing request. This allows for the rapid retrieval of logs related to a specific data processing request from the log records of the Lakeware Metadata Service Log Query Device 800 based on the identification information of that request. This enables the connection between the logs in the Lakeware Metadata Service Log Query Device 800 and the logs in the data engine, thereby improving the log query effect for a specific data processing request (e.g., log query efficiency, log completeness, etc.).

[0267] In addition, this application embodiment also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the lake warehouse metadata service log generation method or the lake warehouse metadata service log query method provided in this application embodiment.

[0268] See Figure 9 This document illustrates a structural schematic diagram of an electronic device 900 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0269] like Figure 9As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0270] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0271] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0272] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0273] This application also provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform any implementation of the lake warehouse metadata service log generation method or the lake warehouse metadata service log query method provided in this application.

[0274] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0275] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0276] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0277] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0278] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0279] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0280] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0281] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0282] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0283] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0284] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0285] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0286] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0287] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a log for a lake store metadata service, the method comprising: The method is applied to a metadata server, and the method includes: The system receives data processing requests sent by a data engine, where the data engine represents an upstream service of the metadata service. The data engine includes logs recorded in response to the data processing requests. The logs in the data engine and the logs in the metadata service belong to different systems. The processing of the data processing requests is completed through the metadata service and the data engine. The data processing requests are generated by the data processing engine based on data analysis tasks, which are created by the data engine in response to user operations. The data processing requests can convey data processing requirements to the metadata service. If the data processing request carries request identification information, a first request identification information is determined based on the request identification information carried by the data processing request. The log in the data engine carries the request identification information, and the first request identification information is used to identify the data processing request. If the request identification information of the data engine provides a log identifier, and the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the log identifier and the request tracing identifier. If the request identification information of the data engine does not provide a log identifier, but the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the request tracing identifier. The identifiers included in the first request identification information correspond to the identifiers included in the request identification information. The log identifier is determined by the software development kit (SDK) in the data engine based on the target parameters of the data engine. The target parameters are log identifiers or task identifiers pre-configured for the data engine. If the data processing request does not carry the request identification information, the first request identification information is determined based on at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request; the first request identification information includes at least one of the at least one request parameter information and the request tracking identifier generated by the metadata server for the data processing request. The request processing logic corresponding to the data processing request is executed, and a log corresponding to the request processing logic is generated based on the first request identification information, wherein the log carries the first request identification information.

2. The method according to claim 1, characterized in that, If the data engine supports log identifier configuration, then the target parameter is a log identifier pre-configured for the data engine; If the data engine does not support log identifier configuration, then the target parameter is the task identifier, which is used to identify the data analysis task that triggers the request generation logic of the data processing request.

3. The method of claim 1, wherein, The request tracking identifier is used to uniquely identify the data processing request; And / or, The request tracking identifier is generated by the SDK in the data engine.

4. The method of claim 1, wherein, The request identification information is located in a preset field in the data processing request; And / or, The request identification information is written by the SDK in the data engine into a preset field in the data processing request.

5. The method of claim 1, wherein, If the data processing request does not carry request identification information, then the first request identification information is determined based on at least one request parameter information of the data processing request, including: If the data engine does not provide a log identifier and does not generate a request tracking identifier for the data processing request, then the first request identifier information is determined based on at least one request parameter information of the data processing request.

6. The method of claim 1, wherein, The engine description information includes at least one of the engine identifier of the data engine and the user identifier of the data engine; And / or, The interface description information includes at least one of the interface identifier corresponding to the data processing request and the interface parameters corresponding to the data processing request.

7. The method according to claim 1, characterized in that, The metadata server is a metadata service system.

8. The method according to claim 1, characterized in that, The metadata server includes a metadata service gateway and at least one metadata service system; Determining the first request identifier information based on the request identifier information carried in the data processing request includes: The metadata service gateway determines the first request identifier information based on the request identifier information carried in the data processing request; The execution of the request processing logic corresponding to the data processing request, and the generation of a log corresponding to the request processing logic based on the first request identifier information, includes: The metadata service gateway generates a data processing message carrying the first request identification information based on the data processing request. The metadata service gateway sends the data processing message to the target system in the at least one metadata service system. The metadata service gateway also generates a metadata service gateway log corresponding to the data processing message based on the first request identification information. The metadata service gateway log carries the first request identification information. The target system executes the message processing logic corresponding to the data processing message, and the target system generates a system log corresponding to the message processing logic based on the first request identification information carried in the data processing message, wherein the system log carries the first request identification information.

9. The method according to claim 8, characterized in that, The method further includes: After executing the message processing logic corresponding to the data processing message, the target system clears the first request identifier information recorded in the target system, and the target system sends a first feedback message to the metadata service gateway. After the metadata service gateway receives the first feedback message, the metadata service gateway clears the first request identification information recorded in the metadata service gateway, and the metadata service gateway sends a second feedback message to the data engine. After the data engine receives the second feedback message, if the data engine records a target identifier corresponding to the first request identifier information, the data engine will clear the target identifier recorded in the data engine.

10. The method according to claim 9, characterized in that, If the data engine supports log identifier configuration, and the first request identifier information recorded in the data engine includes a log identifier and a request tracing identifier, then the target identifier is the request tracing identifier.

11. A method for querying lake warehouse metadata service logs, characterized in that, The method is applied to a metadata server, and the method includes: The system receives a log query request sent by a log request device. The log query request is used to request a query for the logs corresponding to the data processing request sent by the data engine to the metadata server. The request identifier information corresponding to the data processing request carried in the log query request is determined as the second request identifier information, which is used to identify the data processing request. Based on the logs containing the second request identifier information in the log records of the metadata server, the log query result corresponding to the log query request is determined; the logs in the log records are generated using the Lakewarehouse metadata service log generation method according to any one of claims 1-10.

12. The method according to claim 11, characterized in that, The method further includes: If the log query request does not carry the request identifier information corresponding to the data processing request, but the log query request carries a log query time range and at least one request parameter information of the data processing request, then the second request identifier information is generated based on the at least one request parameter information carried by the log query request; or, the second request identifier information is determined from the log records of the metadata server based on the log query time range carried by the log query request and at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request. Based on the logs existing in the log records of the metadata server that conform to the log query time range and carry the second request identifier information, the log query result corresponding to the log query request is determined.

13. The method according to claim 11, characterized in that, The log request device is used to integrate the log query results fed back by the metadata server and the logs recorded in the data engine for the data processing request to obtain an integrated log. The integrated log is used to describe the process executed by the data engine and the metadata server for the data processing request.

14. The method according to claim 13, characterized in that, If the log query request carries request identification information corresponding to the data processing request, then the log recorded in the data engine for the data processing request refers to the log in the data engine's log records that carries the request identification information.

15. The method according to any one of claims 11-14, characterized in that, The log request device is the data engine.

16. A lake warehouse metadata service log generation device, characterized in that, include: The first receiving unit is used to receive data processing requests sent by the data engine, which represents the upstream service of the metadata server. The data engine includes logs recorded in response to the data processing requests. The logs in the data engine and the logs in the metadata server belong to different systems. The processing of the data processing requests is completed through the metadata server and the data engine. The data processing requests are generated by the data processing engine based on data analysis tasks. The data analysis tasks are created by the data engine in response to user operations. The data processing requests can convey data processing requirements to the metadata server. The first determining unit is configured to determine first request identification information based on the request identification information carried by the data processing request if the data processing request carries request identification information. The log in the data engine carries the request identification information, and the first request identification information is used to identify the data processing request. If the request identification information of the data engine provides a log identifier, and the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the log identifier and the request tracing identifier; If the request identification information of the data engine does not provide a log identifier, but the data engine generates a request tracing identifier for the data processing request, then the request identification information includes the request tracing identifier. The identifier included in the first request identification information corresponds to the identifier included in the request identification information. The log identifier is determined by the software development kit (SDK) in the data engine based on the target parameters of the data engine. The target parameters are log identifiers or task identifiers pre-configured for the data engine. The first determining unit is further configured to, if the data processing request does not carry the request identification information, determine the first request identification information based on at least one request parameter information of the data processing request; the at least one request parameter information includes at least one of the engine description information of the data engine and the interface description information corresponding to the data processing request; the first request identification information includes at least one of the at least one request parameter information and the request tracking identifier generated by the metadata server for the data processing request; The request processing unit is used to execute the request processing logic corresponding to the data processing request, and generate a log corresponding to the request processing logic based on the first request identification information, wherein the log carries the first request identification information.

17. A lake warehouse metadata service log query device, characterized in that, include: The second receiving unit is used to receive a log query request sent by the log request device. The log query request is used to request a query for the log corresponding to the data processing request sent by the data engine to the metadata server. The second determining unit is used to determine the request identification information corresponding to the data processing request carried in the log query request as the second request identification information, which is used to identify the data processing request. The third determining unit is used to determine the log query result corresponding to the log query request based on the logs carrying the second request identifier information existing in the log records of the metadata server; the logs in the log records are generated using the Lake Warehouse metadata service log generation method according to any one of claims 1-10.

18. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-15.

19. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-15.

Images