Table metadata cache refreshing method, medium, device, equipment and program product

Abstract

The embodiment of the invention provides a table metadata cache refreshing method, a medium, a device, equipment and a program product, and relates to the technical field of computers, the table metadata cache refreshing method comprises the steps of obtaining first modification time of a partition directory of a table in a distributed file system, and according to the first modification time and second modification time of a partition directory of the table cached by the query engine, determining whether a target table with expired file metadata exists in the table cached by the query engine, and if yes, refreshing the file metadata in a partition of the target table. According to the method and the device, the target table with the file metadata expired in the table cached by the query engine is automatically detected and found, so that the file metadata in the partition of the target table can be timely refreshed, the problem of query abnormity caused by too old file metadata of the table cached by the query engine can be solved, and the use experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology. More specifically, the embodiments of the present disclosure relate to a table metadata cache refresh method, medium, device, equipment and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Data lake query refers to the process of retrieving and analyzing data in a data lake through a query engine. Its characteristic is that data writing and querying are provided by different services (i.e., storage and computing are separated). During the query structured query language (SQL) parsing and execution plan optimization phase, the query engine needs to obtain and cache the required table metadata from the Hive (a data warehouse tool) metadata storage (Hive MetaStore, HMS) service and the name node (NameNode, NN) service of the distributed file system (HDFS) across the network.

[0004] Currently, the metadata automatic synchronization mechanism based on the HMS change log is usually used to refresh the table metadata cached by the query engine. However, in some scenarios, there are still query anomalies caused by outdated table metadata cache. Summary of the invention

[0005] The present disclosure provides a table metadata cache refresh method, medium, device, equipment and program product to solve the query abnormality problem caused by the outdated table metadata cache that still exists in some scenarios.

[0006] In a first aspect of the embodiments of the present disclosure, a table metadata cache refresh method is provided, comprising:

[0007] Get the first modification time of the partition directory of the table in the distributed file system;

[0008] Determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine;

[0009] If so, refresh the file metadata in the partition of the target table.

[0010] In a second aspect, an embodiment of the present disclosure provides a table metadata cache refresh device, including:

[0011] An acquisition module, used to acquire the first modification time of a partition directory of a table in a distributed file system;

[0012] A determination module, used to determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine;

[0013] A refresh module is used to refresh the file metadata in the partition of the target table if yes.

[0014] In a third aspect, an embodiment of the present disclosure provides a computing device, including: a processor, and a memory communicatively connected to the processor;

[0015] Memory stores computer-executable instructions;

[0016] The processor executes the computer execution instructions stored in the memory to implement the table metadata cache refresh method as described in the first aspect of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a storage medium, in which computer program instructions are stored. When the computer program instructions are executed, the table metadata cache refresh method as described in the first aspect of the present disclosure is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the table metadata cache refresh method as described in the first aspect of the present disclosure when the computer program is executed.

[0019] The table metadata cache refresh method, medium, device, equipment and program product provided by the embodiments of the present disclosure obtain the first modification time of the partition directory of the table in the distributed file system, and determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine. If so, the file metadata in the partition of the target table is refreshed. In the present disclosure, the first modification time of the partition directory of the table in the distributed file system is automatically compared with the second modification time of the partition directory of the table cached by the query engine to detect and find the target table with expired file metadata in the table cached by the query engine, so that the file metadata in the partition of the target table can be refreshed in time, which can solve the query abnormality problem caused by the old file metadata of the table cached by the query engine and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0021] Figure 1A schematic diagram of obtaining table metadata in a data lake query scenario provided for related technologies;

[0022] Figure 2 A schematic diagram of refreshing the table metadata cached by the query engine based on the HMS change log provided for related technologies;

[0023] Figure 3 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0024] Figure 4 A flowchart of a table metadata cache refresh method provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a time interval for producing historical data provided by an embodiment of the present disclosure;

[0026] Figure 6 A flowchart of a table metadata cache refresh method provided by another embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of a source of a table in a refresh list provided by an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of the structure of a table metadata cache refresh device provided by an embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of the structure of a table metadata cache refresh device provided by another embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram of a storage medium provided in an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram of the structure of a computing device provided in one embodiment of the present disclosure.

[0032] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0033] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. The data involved in the present disclosure can be data authorized by the user or fully authorized by all parties, and the embodiments / examples of the present disclosure can be combined with each other.

[0035] According to an embodiment of the present disclosure, a table metadata cache refresh method, medium, device, equipment and program product are proposed.

[0036] In this article, it is important to understand that the terms involved are:

[0037] Separation of storage and computing is a prerequisite for data lake query; separation of storage and computing usually means that storage and computing are independent services. For example, in the big data scenario, HDFS and Kudu (a column storage management system) are independent storage systems that provide application programming interfaces (APIs) for data reading and writing operations; computing engines are mainly batch-based engines such as Spark (a distributed computing framework) or HiveServer2 (a service component in Hive), or query-based online analytical processing (OLAP) engines such as Impala (a distributed SQL query engine) or Presto (a distributed SQL query engine), or Doris (a massively parallel processing database) or StarRocks (a massively parallel processing database).

[0038] Hive table format is the most widely used table format in the open source ecosystem of Hadoop (a distributed system infrastructure) and is the industry standard for table formats. Due to the huge amount of existing users and data, most data lakes currently still use Hive table format for storage, and various mainstream computing engines support Hive table format. In typical scenarios, Hive table format uses HMS service to save table metadata information. Since the data management granularity of Hive table format is at the directory level, the file granularity information of the table needs to be obtained from storage systems such as HDFS.

[0039] Table metadata information refers to the table definition, table partitions, data files corresponding to the table, and information about these files and data blocks that the computing engine needs to use when executing SQL. The engine uses this information to determine whether the SQL statement is correct and generate a better execution plan.

[0040] Data lake query: query refers to obtaining the required data from a table through SQL commands such as select, or obtaining the required data through multi-table join, and performing various aggregation and sorting operations such as summation and averaging as required; data lake query refers to using OLAP engines such as Impala to query Hive and other table formats. The characteristic is that data writing and query are provided by different services (storage and computing separation). If no optimization is done, Impala needs to obtain the required table metadata information from the HMS service and HDFS NN service across the network during the query SQL parsing and execution plan optimization stages.

[0041] Impala is an open source massively parallel processing (MPP) query engine. Compared with other query engines, its biggest feature is speed. In terms of architecture, there is an independent catalog management (Catalog) process to cache the metadata and statistical information required for the query to accelerate the query.

[0042] The Catalog node is a role in the Impala system that caches metadata information of Hive tables, including table / partition-level metadata information obtained from HMS and data file list information under the table or partition obtained from HDFS NN. When the Coordinator node formulates a query plan, it obtains table metadata information from the Catalog without going through HMS and HDFSNN, greatly reducing query time.

[0043] The Coordinator node is a role in the Impala system. It accepts user query requests, performs lexical and grammatical analysis, formulates distributed execution plans, and sends them to executors for execution. To reduce the time spent on generating execution plans, the Coordinator node obtains table metadata information from the Catalog node on demand and caches it in the process memory.

[0044] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0046] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0048] The inventors found that in the traditional storage-computing integrated mode, the query engine obtains table metadata information from the local memory of the database process to generate an execution plan, and the time consumption is negligible. However, in the data lake query scenario, the query engine needs to obtain and cache the required table metadata from the HMS service and the NN service of the distributed file system (HDFS) across the network during the query SQL parsing and execution plan optimization phase, that is, the storage service and computing service are separated. Figure 1 A schematic diagram of obtaining table metadata in a data lake query scenario provided for related technologies, such as Figure 1 As shown in the figure, the batch running engine (such as Spark) writes data and metadata to HMS and HDFS for storage; the query engine (such as Impala) reads data and metadata from HMS and HDFS. Compared with the storage-computing integrated model, the data lake query scenario requires cross-network acquisition of table metadata information, which significantly increases the time required, and the additional impact of performance fluctuations of the network, HMS, and HDFS NN must also be considered. Especially when performing big data analysis, some tables may have a large number of partitions and data files, which will significantly increase the size of the table metadata, further increasing the time it takes for the data lake query engine to obtain table metadata, affecting query performance.

[0049] Performance is one of the key indicators for measuring OLAP engines, including data lake query engines. For example, TPC-C (a benchmark for online transaction processing), TPC-H (a benchmark for decision support systems), and TPC-DS (a more complex benchmark for decision support systems) are all based on SQL in different business scenarios to evaluate the performance of query engines. Cache is an important means for performance optimization. In the data lake query scenario, query engines such as Impala or Doris will reduce the time required to obtain table metadata by caching table metadata in the local memory of the process. Taking Impala as an example, the Catalog component is introduced in the system architecture as the table metadata cache service of the Impala cluster. Catalog obtains table metadata information from HMS and HDFS NN, and then broadcasts it to the Coordinator node through the Statestore component (a component for managing and storing system status information). In this way, when the Coordinator node executes SQL queries, it can directly obtain the table metadata information involved in the SQL from the memory cache of the process, avoiding cross-network acquisition, thereby improving query performance.

[0050] In traditional storage-computing integrated databases, data processing and querying naturally use the same computing service because storage and computing are bound together. The table metadata cached in the process is always up to date. However, in the data lake query scenario, data processing (such as Spark) and data query (such as Impala) are not on the same database engine. The Impala engine used for querying cannot perceive the changes in table metadata in time and re-acquire (refresh), making the query engine's process local metadata cache too old and unable to reflect the current metadata status of the Hive table. An old metadata cache can cause query anomalies, the main types of anomalies include query errors or returning wrong results. Query errors are generally caused by users rewriting the data of a partition, then the original data files of the partition will be deleted, and the cache in the Impala engine is still the metadata of these deleted files. Impala reading the deleted files will cause errors. Returning an error result is caused by not reading the newly added files. For example, if a user adds a new data partition, Impala does not perceive it in time and cache it locally, causing the newly added partition to be ignored when querying the table.

[0051] For performance-sensitive OLAP query engines, table metadata caching is required in data lake query scenarios. This is an effective means of performance optimization. Therefore, the cache timeliness problem (i.e., cache refresh problem) needs to be solved. At present, the table metadata cached by the query engine is usually refreshed based on the metadata automatic synchronization mechanism of the HMS change log. However, in some scenarios, there are still query anomalies caused by the old table metadata cache. Specifically, by turning on the notification events or change log function on the HMS side, HMS will write operations such as Data Definition Language (DDL) as events into the database, such as table creation, deletion or modification in the database, and partition addition, deletion or modification. Table 1 shows the events recorded by HMS, as well as event behaviors and corresponding actions.

[0052] Table 1

[0053]

[0054] With the help of HMS change logs, Impala can read various events from HMS in a timely manner, thereby perceiving changes in Hive table metadata and performing targeted metadata refreshes.

[0055] Figure 2 A schematic diagram of refreshing the table metadata cached by the query engine based on the HMS change log provided for related technologies, such as Figure 2As shown, based on the automatic metadata synchronization mechanism of the HMS change log, HMS writes DDL and other operations as events into the database; the specific refresh steps may include: Step 1, the metadata event processor (i.e., the Catalog function module) sends a request to HMS to obtain the latest 1,000 change logs; Step 2, HMS obtains events (Event) from the database table, and returns Event 1 (Event1), Event 2 (Event2), and Event 3 (Event3); Step 3, the metadata event processor refreshes metadata according to Event1 to Event3 (i.e., applies Event1 to Event3). At present, engines such as Impala, Doris, and StarRocks already support metadata refresh based on the automatic metadata synchronization mechanism of the HMS change log.

[0056] The automatic metadata synchronization mechanism based on the HMS change log can solve the metadata refresh problem of most tables, but there are still some scenarios that are not covered, resulting in Impala operation and maintenance personnel still needing to spend a lot of time to troubleshoot and analyze query anomalies caused by outdated metadata cache. The reason for the problem is that the computing engine or user's data modification operations on the Hive table do not go through HMS. The solution based on the HMS change log cannot perceive the metadata changes and thus cannot update the cache. The main scenarios include the following two: (1) Directly operate the HDFS data of the Hive table through tools such as DataX (an offline data synchronization tool) to add, delete, and overwrite data files; (2) When batch engines such as Spark produce data, if there is no need to change the metadata in HMS, they will choose to bypass HMS for performance reasons. Both of the above scenarios cause the file-level metadata (i.e., file metadata) cache of the Hive table to be outdated, without affecting the table-level or partition-level metadata of the Hive table in HMS. For Impala users, the cache stale problem caused by the above two scenarios can be avoided by setting the table-level metadata cache validity period or disabling the metadata cache of the specified table. However, this can only be done after the problem occurs and requires intervention from operation and maintenance personnel.

[0057] In addition, in the related technology, the table metadata cached by the query engine can also be refreshed by active timeout invalidation and timed refresh. Among them, for the active timeout invalidation method, the cache function is lost after the timeout, and the table metadata needs to be reloaded during query. The shorter the timeout invalidation time, the weaker the cache effect, but the longer the invalidation time, the greater the probability of query anomaly. Alternatively, a periodic refresh method can be used to avoid cache invalidation, but there is still the problem of query anomaly within the refresh cycle. In order to reduce the probability of query anomaly as much as possible, it is necessary to reduce the refresh cycle, but this will increase the metadata service pressure of HMS and HDFS NN.

[0058] Based on the above problems, the present invention provides a table metadata cache refresh method, which automatically compares the first modification time of the partition directory of the table in the distributed file system with the second modification time of the partition directory of the table cached by the query engine, so as to detect and discover the table with expired file metadata in the table cached by the query engine, so that the file metadata in the partition of the table can be refreshed in time, which can solve the query abnormality problem caused by the outdated file metadata of the table cached by the query engine, and improve the user experience.

[0059] The present disclosure provides a table metadata cache refresh method, which does not replace the current metadata automatic synchronization mechanism based on the HMS change log, but optimizes and improves the current mechanism. The main technical problems solved include the query anomaly caused by the inability to timely identify outdated file metadata cache, and the high cost and low efficiency of frequent and repeated refreshing of the file metadata cache.

[0060] Application Scenario Overview

[0061] First reference Figure 3 The application scenarios of the solutions provided in the present disclosure are illustrated by examples. Figure 3 A schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 3 As shown, in this application scenario, the user submits an SQL query request through the client 301 to obtain the required data, and the server 302 obtains the data corresponding to the SQL query request through the query engine, and displays the data to the user through the client 301. Among them, according to the table metadata cache refresh method provided by the present disclosure, the table metadata cached by the query engine is refreshed to avoid abnormal table queries caused by using expired table metadata. It should be noted that Figure 3 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 3 does not limit the equipment included in Figure 3 The positional relationship between the devices is limited.

[0062] Exemplary Methods

[0063] Combine the following Figure 3 For application scenarios, refer to Figure 4 To describe a method for refreshing a table metadata cache according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0064] First, a table metadata cache refresh method is introduced through a specific embodiment.

[0065] Figure 4 The flowchart of the table metadata cache refresh method provided by an embodiment of the present disclosure is as follows. The method of the embodiment of the present disclosure can be applied to a computing device, which can be a server or a server cluster. Figure 4 As shown, the method of the embodiment of the present disclosure includes:

[0066] S401: Obtain the first modification time of a partition directory of a table in a distributed file system.

[0067] In the embodiments of the present disclosure, exemplarily, the first modification time of the partition directory of the table in the distributed file system can be periodically obtained according to the preset detection interval, or the first modification time of the partition directory of the table in the distributed file system can be obtained based on the historical data production time of the table, or the first modification time of the partition directory of the table in the distributed file system can be obtained based on the time corresponding to the query behavior of the query engine on the table, so as to compare the first modification time with the second modification time of the partition directory of the table cached by the query engine to obtain the file metadata cache that cannot be refreshed. For specific information on how to obtain the first modification time of the partition directory of the table in the distributed file system, please refer to the subsequent embodiments.

[0068] S402: Determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine.

[0069] In this step, after obtaining the first modification time of the partition directory of the table in the distributed file system, the first modification time can be compared with the second modification time of the partition directory of the table cached by the query engine to determine whether there is a target table with expired file metadata in the table cached by the query engine. This step can be understood as a health check of the file metadata.

[0070] Optionally, determining whether there is a target table with expired file metadata in the tables cached by the query engine based on the first modification time and the second modification time of the partition directory of the table cached by the query engine may include: determining a table to be detected from the tables cached by the query engine based on the loading time of the table cached by the query engine, a preset detection interval duration, and the current time; determining whether there is a target table with expired file metadata in the tables cached by the query engine based on the first modification time and the second modification time of the partition directory of the table to be detected.

[0071] Exemplarily, the preset detection interval duration is, for example, 1 hour, which can be controlled by the configuration item hdfs_file_cache_detect_interval_s. The preset detection interval duration can be configured as needed, and the embodiments of the present disclosure do not limit this. Taking the query engine as Impala as an example, all file metadata loaded by Catalogd (i.e., the process of the Catalog node) can be regularly detected according to the preset detection interval duration. In order to improve the performance of file metadata health detection, multiple tables can be detected in parallel, and the number of tables to be detected can be controlled by the configuration item max_hdfs_tables_parallel_detect, for example. In order to avoid detecting the table just loaded by Cattalogd, the table to be detected can be determined from the table cached by Cattalogd according to the loading time of the table, the preset detection interval duration, and the current time, that is, only the table with [table loading time + hdfs_file_cache_detect_interval_s / 2] < current time is selected for detection. Considering that a large table may have many partitions, the detection of the partition directories of the table also supports parallel execution. The number of partition directories detected in parallel can be controlled by the configuration item max_hdfs_partitions_parallel_load, for example.

[0072] It can be understood that periodic detection can be performed based on the preset detection interval to determine whether there is a target table with expired file metadata in the table cached by the query engine, so as to refresh the file metadata in the partition of the target table. Considering that each detection is to traverse all cached tables, the modification time of the partition directory will be obtained by accessing the HDFS NN during the traversal. Therefore, the interval of periodic detection should not be too short to avoid causing obvious load pressure on the Catalog node and HDFS NN. For example, it is sufficient to detect several times a day.

[0073] Optionally, determining whether there is a target table with expired file metadata in the tables cached by the query engine based on the first modification time and the second modification time of the partition directory of the table cached by the query engine can include: locking the table cached by the query engine to obtain the locked table; obtaining the second modification time of the partition directory of the locked table; unlocking the locked table; and determining whether there is a target table with expired file metadata in the tables cached by the query engine based on the first modification time and the second modification time of the partition directory of the locked table.

[0074] Exemplarily, before testing each table cached by the query engine, a read lock can be added to the table to obtain the locked table, and the second modification time of the partition directory of the locked table can be obtained. Then, the locked table can be unlocked to avoid the detection process holding the lock and blocking other metadata services. For example, when locking a table, the lock is acquired in a time-limited waiting manner. If the lock is not obtained within the limited time, the table will be skipped. After obtaining the second modification time of the partition directory of the locked table, it can be determined whether there is a target table with expired file metadata in the table cached by the query engine based on the first modification time and the second modification time of the partition directory of the locked table. Optionally, if there is no need to detect a table cached by the query engine, the configuration item disable_hdfs_file_cache_detect corresponding to the table attribute can be set to true, and the table will be skipped during the detection and will not be detected.

[0075] S403: If yes, refresh the file metadata in the partition of the target table.

[0076] In this step, after determining that there is a target table with expired file metadata in the table cached by the query engine, the file metadata in the partitions of the target table can be refreshed, that is, the file metadata in the partitions of the target table can be reloaded. Exemplarily, the file metadata in the partitions of the target table can be refreshed periodically according to a preset refresh interval; or, the file metadata in the partitions of the target table can be refreshed based on the historical data production time of the table; or, the file metadata in the partitions of the target table can be refreshed based on the time corresponding to the query behavior of the query engine on the table. For specific information on how to refresh the file metadata in the partitions of the target table, please refer to the subsequent embodiments.

[0077] The table metadata cache refresh method provided by the disclosed embodiment obtains the first modification time of the partition directory of the table in the distributed file system, and determines whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine. If so, the file metadata in the partition of the target table is refreshed. In the disclosed embodiment, the first modification time of the partition directory of the table in the distributed file system is automatically compared with the second modification time of the partition directory of the table cached by the query engine to detect and find the target table with expired file metadata in the table cached by the query engine, so that the file metadata in the partition of the target table can be refreshed in time, which can solve the query anomaly problem caused by the old file metadata of the table cached by the query engine and improve the user experience.

[0078] Based on the above embodiment, optionally, obtaining the first modification time of the partition directory of the table in the distributed file system may include at least one of the following: periodically obtaining the first modification time of the partition directory of the table in the distributed file system according to a preset detection interval; obtaining the first modification time of the partition directory of the table in the distributed file system within a first time interval, and the first time interval is determined based on the historical data production time of the table; obtaining the first modification time of the partition directory of the table in the distributed file system within a second time interval, and the second time interval is determined based on the time corresponding to the query behavior of the query engine on the table.

[0079] Exemplarily, the preset detection interval is, for example, 1 hour, which can be controlled by the configuration item hdfs_file_cache_detect_interval_s. The first modification time of the partition directory of the table in the distributed file system can be periodically obtained according to the preset detection interval to periodically detect whether there is a target table with expired file metadata in the table cached by the query engine, so as to refresh the file metadata in the partition of the target table in time. The interval of periodic detection should not be too short, for example, several times a day is sufficient.

[0080] Taking into account the historical data production time of the table, the historical data production time of each partition of the table can be collected and summarized to determine at which time points in the day the table had data production behavior in history, and then divided into several possible data production time intervals, that is, determine the first time interval. Figure 5 A schematic diagram of the time interval for producing historical data provided by an embodiment of the present disclosure, as shown in FIG5, a triangle indicates that a partition produces data at this time point, "yes" indicates that the time interval corresponding to "yes" (i.e., the first time interval) requires a higher frequency of file metadata cache expiration detection, and "no" indicates that the time interval corresponding to "no" does not require a higher frequency of file metadata cache expiration detection. Specifically, the first modification time of the partition directory of the table in the distributed file system can be obtained within the first time interval to detect whether there is a target table with expired file metadata in the table cached by the query engine, so as to facilitate timely refreshing of the file metadata in the partition of the target table.

[0081] Considering the time corresponding to the query behavior of the query engine on the table, the time distribution of the query engine querying the table can be collected, and the time interval of the first query of the table by the query engine every day within a period of time (such as a week) can be analyzed. Assuming that the time interval is represented by [Atime, Btime], a higher frequency of file metadata cache expiration detection can be performed within the [Atime-n, Btime] time interval (i.e., the second time interval), where n represents n seconds and can be set as needed. Specifically, the first modification time of the partition directory of the table in the distributed file system can be obtained within the second time interval to detect whether there is a target table with expired file metadata in the table cached by the query engine, so as to facilitate timely refresh of the file metadata in the partition of the target table.

[0082] In this embodiment, it is possible to periodically and automatically detect whether there is a target table with expired file metadata in the table cached by the query engine. The detection efficiency can be improved based on the historical data production time of the table, and the detection efficiency can be improved based on the time corresponding to the query behavior of the query engine on the table.

[0083] Figure 6 This is a flowchart of a table metadata cache refresh method provided by another embodiment of the present disclosure. Based on the above embodiment, the table metadata cache refresh method of the embodiment of the present disclosure is further described. Figure 6 As shown, the method of the embodiment of the present disclosure may include:

[0084] S601: Obtain the first modification time of a partition directory of a table in a distributed file system.

[0085] The detailed description of this step can be found in Figure 4 The relevant description of S401 in the illustrated embodiment will not be repeated here.

[0086] In the present disclosure, Figure 4 The step S402 may further include the following two steps S602 and S603:

[0087] S602. Determine the target partition directory according to the first modification time and the second modification time of the partition directory of the table cached by the query engine, where the first modification time and the second modification time of the target partition directory are different; if the time difference between the first modification time of the target partition directory and the current time is greater than the duration threshold, it is determined that there is a target table with expired file metadata in the table cached by the query engine, and the target table includes the target partition directory, and continue to execute step S604.

[0088] Exemplarily, the duration threshold is controlled by the configuration item hdfs_file_cache_overdue_threshold_s, and the duration threshold can be configured as needed. When judging whether there is a target table with expired file metadata in the table cached by the query engine, three time concepts are involved, namely the current time, the first modification time of the partition directory of the table in the distributed file system (i.e., the actual modification time of the partition directory), and the second modification time of the partition directory of the table cached by the query engine. Considering that the embodiment of the present disclosure is used to handle the situation that the current metadata automatic synchronization scheme cannot cover, therefore, it is not the case that when the second modification time is inconsistent with the first modification time, it is determined that the table has a file metadata expiration problem, because the table may be repaired by the metadata automatic synchronization function later. In this step, after determining the target partition directory with different first modification time and second modification time, if the time difference between the first modification time of the target partition directory and the current time is greater than the duration threshold, that is, the first modification time + hdfs_file_cache_overdue_threshold_s < current time, it is determined that there is a target table with expired file metadata in the table cached by the query engine.

[0089] Optionally, after determining that there is a target table with expired file metadata in the table cached by the query engine, the target table can be added to the list of expired tables (such as represented by OverdueTableList). For example, the web user interface (WebUI) of Catalogd can be used to view which tables have expired file metadata. Table 2 shows the relevant information of the OverdueTableList list, where Table 1 is the target table with expired file metadata; the file metadata refresh module is used to refresh the tables in the OverdueTableList list, and the details can be seen in the subsequent embodiments.

[0090] Table 2

[0091]

[0092]

[0093] Optionally, the table includes multiple partition directories, and determining the target partition directory according to the first modification time and the second modification time of the partition directory of the table cached by the query engine can include: sorting the multiple partition directories of the table in reverse order according to the modification time to obtain the sorted partition directory; detecting whether there is a partition directory in the sorted partition directory whose first modification time is different from the second modification time; if so, determining that the target partition directory is a partition directory in the sorted partition directory whose first modification time is different from the second modification time.

[0094] Exemplarily, the detection of the partition directory of the table also supports parallel execution, and the number of partition directories detected in parallel can be controlled by the configuration item max_hdfs_partitions_parallel_load, for example. The expiration detection of each partition directory will obtain the first modification time of the partition directory by accessing the HDFS NN, and then compare it with the second modification time of the partition directory of the table cached by the query engine to determine whether the partition directory is an expired partition directory. At the same time, since it is only necessary to find the existence of expired partition directories, there is no need to find all expired partition directories. Therefore, before the partition directory expiration detection, each partition directory will be sorted in reverse order according to the order of directory modification time to obtain the sorted partition directory, and the partition with the latest modification time will be detected first. Check whether there is a partition directory with a different first modification time and a different second modification time in the sorted partition directory. If so, determine that the target partition directory is a partition directory with a different first modification time and a different second modification time in the sorted partition directory.

[0095] S603. If the second modification time of the partition directory of the table cached by the query engine is the same as the first modification time of the corresponding partition directory in the distributed file system, or the time difference between the first modification time of the target partition directory and the current time is less than or equal to the duration threshold, it is determined that there is no target table with expired file metadata in the table cached by the query engine, and the process ends.

[0096] In this step, if the second modification time is the same as the first modification time, it can be determined that there is no target table with expired file metadata in the table cached by the query engine, that is, there is no need to refresh the file metadata in the table cached by the query engine. Alternatively, after determining the target partition directory whose second modification time is different from the first modification time, if the time difference between the first modification time of the target partition directory and the current time is less than or equal to the duration threshold, it is determined that there is no target table with expired file metadata in the table cached by the query engine.

[0097] It should be noted that the embodiment of the present disclosure does not limit the execution order of S602 and S603.

[0098] In the present disclosure, Figure 4 The step S403 may further include the following two steps S604 and S605:

[0099] S604: Obtain an expired partition in the partition of the target table, wherein the first modification time and the second modification time of the partition directory of the expired partition are different, and the time difference between the first modification time and the current time of the partition directory of the expired partition is greater than the duration threshold.

[0100] It can be understood that when detecting whether there is a target table with expired file metadata in the table cached by the query engine, it is only necessary to find out the existence of expired partitions, and there is no need to find out all expired partitions. In this step, the file metadata of the expired partition needs to be refreshed, so it is necessary to obtain all expired partitions in the partitions of the target table. The judgment logic of the expired partition is the same as the judgment logic of detecting whether there is a target table with expired file metadata in the table cached by the query engine, that is, the first modification time of the partition directory of the expired partition is different from the second modification time, and the time difference between the first modification time of the partition directory of the expired partition and the current time is greater than the duration threshold.

[0101] S605: Reload the file metadata of the expired partition.

[0102] For example, taking Impala as the query engine, after obtaining all expired partitions in the partitions of the target table, the existing "REFRESH" interface of Catalogd can be called to reload the file metadata of the expired partitions, including the statistics of the expired partitions and the partition file list.

[0103] The table metadata cache refresh method provided by the embodiment of the present disclosure obtains the first modification time of the partition directory of the table in the distributed file system; determines the target partition directory according to the first modification time and the second modification time of the partition directory of the table cached by the query engine, and the first modification time of the target partition directory is different from the second modification time; if the time difference between the first modification time of the target partition directory and the current time is greater than the duration threshold, it is determined that there is a target table with expired file metadata in the table cached by the query engine, and the target table includes the target partition directory; obtains an expired partition in the partition of the target table, the first modification time of the partition directory of the expired partition is different from the second modification time, and the time difference between the first modification time of the partition directory of the expired partition and the current time is greater than the duration threshold, and reloads the file metadata of the expired partition; if the second modification time of the partition directory of the table cached by the query engine is the same as the first modification time of the corresponding partition directory in the distributed file system, or the time difference between the first modification time of the target partition directory and the current time is less than or equal to the duration threshold, it is determined that there is no target table with expired file metadata in the table cached by the query engine. In the disclosed embodiment, the first modification time of the partition directory of the table in the distributed file system is automatically compared with the second modification time of the partition directory of the table cached by the query engine. After determining the target partition directory whose first modification time is different from the second modification time and whose time difference between the first modification time and the current time is greater than a time threshold, all expired partitions in the target table containing the target partition directory are obtained, and the file metadata of the expired partitions is reloaded to refresh the file metadata of the expired partitions. This can solve the query anomaly problem caused by the outdated file metadata of the table cached by the query engine and improve the user experience.

[0104] Based on the above embodiment, optionally, refreshing the file metadata in the partition of the target table may include: adding the target table to a refresh list; and refreshing the file metadata in the partition of the table in the refresh list.

[0105] Exemplarily, after detecting that there is a target table with expired file metadata in the table cached by the query engine, the target table can be added to a refresh list (for example, represented by AutoRefreshList), and the file metadata in the partitions of the table in the refresh list can be refreshed. The file metadata refresh interval of the table can be controlled by the configuration item hdfs_file_cache_refresh_interval_s. Similarly, to speed up the refresh efficiency, the number of parallel refresh tables can also be controlled by the configuration item max_hdfs_tables_parallel_refresh, and the number of parallel detections of table partitions can still be controlled by max_hdfs_partitions_parallel_load.

[0106] Further, optionally, refreshing the file metadata in the partitions of the tables in the refresh list may include at least one of the following: periodically refreshing the file metadata in the partitions of the tables in the refresh list according to a preset refresh interval; refreshing the file metadata in the partitions of the tables in the refresh list within a first time interval; refreshing the file metadata in the partitions of the tables in the refresh list within a second time interval.

[0107] In this embodiment, the preset refresh interval can be controlled by, for example, the configuration item hdfs_file_cache_refresh_interval_s. Taking Impala as an example, the disclosed embodiment is used to identify and refresh the file metadata of the table that cannot be refreshed through the HMS change log as early as possible. Therefore, according to the conventional method, a smaller refresh cycle needs to be set, such as setting the configuration item hdfs_file_cache_refresh_interval_s corresponding to the preset refresh interval to the minute level. Since the number of tables in the refresh list AutoRefreshList is generally much less than the number of tables cached by Catalogd, such as less than 1% of the total number of tables cached by the query engine. Although the periodic collection of expired partitions during refresh also requires access to the HDFS NN, the load pressure caused is much smaller than the target table for periodically detecting whether there is expired file metadata in the table cached by the query engine. Nevertheless, for offline development scenarios, the typical data output is in the form of T+1, where T represents the day, which means that there is only one data output per day, that is, only one valid metadata refresh operation. Therefore, if periodic refresh is performed at the minute level, the refresh efficiency will be very low. In order to improve the refresh efficiency, in this embodiment, only expired partitions can be refreshed instead of refreshing the file metadata of the entire table. Specifically, before refreshing, detect which partitions of the table in the refresh list are expired, and only refresh the file metadata of expired partitions. Considering the interaction with HDFS NN, each detection only needs to obtain the latest modification time of the partition directory, which is a relatively lightweight operation. Only after the expired partition is discovered, the file metadata of the expired partition is reloaded from HMS and HDFS NN.

[0108] Considering the historical data output time of the table, we can collect the historical data output time of each partition of the table, summarize it, and find out at which time points in the day the table has historically produced data, and divide it into several possible data output time intervals, that is, determine the first time interval. Figure 5 , a higher frequency of expired partition collection and necessary refresh can be performed within the first time interval.

[0109] Taking into account the time corresponding to the query engine's query behavior on the table, the time distribution of the query engine's query of the table can be collected, and the time interval of the first query of the table by the query engine every day within a period of time (such as a week) can be analyzed. For example, a higher frequency of expired partition collection and necessary refresh can be performed within the [Atime-n, Btime] time interval (that is, the second time interval).

[0110] Optionally, the table metadata cache refresh method provided by the embodiment of the present disclosure may also include: when the query engine loads a table, obtaining the attribute value of the cache refresh attribute of the loaded table, and the attribute value is configurable; if the attribute value is to enable refresh, then adding the table with the attribute value to enable refresh to the refresh list; if the attribute value is to disable refresh, then not adding the table with the attribute value to disable refresh to the refresh list.

[0111] It can be understood that the sources of the tables in the refresh list include the target tables with expired file metadata determined by automatic detection and the tables whose attribute value of the user-configured cache refresh attribute (such as represented by enable_hdfs_file_cache_refresh) is ON. Figure 7 A schematic diagram of a source of a table in a refresh list provided by an embodiment of the present disclosure, such as Figure 7 As shown, the sources of the tables in the refresh list include the target table determined by automatic detection and the table whose attribute value of the cache refresh attribute configured by the user is to enable refresh. Among them, through the cache refresh attribute of the table, the table can be refreshed independently, and the user can control whether the table is added to the refresh list AutoRefreshList by setting the cache refresh attribute enable_hdfs_file_cache_refresh of the table. Exemplarily, taking the query engine as Impala as an example, Catalogd can determine whether the table has the enable_hdfs_file_cache_refresh attribute and its attribute value when loading the table from HMS. If the attribute value is to enable refresh (ON), it is determined whether the table with the attribute value of enabling refresh is already in the refresh list. If not, the table with the attribute value of enabling refresh is added to the refresh list. If the attribute value is to close refresh (OFF), it is determined whether the table with the attribute value of closing refresh is already in the refresh list. If so, the table is removed from the refresh list, that is, the table with the attribute value of closing refresh is not added to the refresh list at any time. If the table is not configured with cache refresh attributes, it can be determined whether the table needs to be added to the refresh list by automatic detection. It can be understood that users can set which tables need to perform periodic file metadata detection and refresh through table properties, which is a supplement to the existing metadata automatic synchronization solution.

[0112] Optionally, the table metadata cache refresh method provided by the embodiment of the present disclosure may further include: deleting the table in the refresh list that meets the first condition from the refresh list, the first condition being that the attribute value of the cache refresh attribute of the table is not to enable refresh, and there are no expired partitions in the partitions of the table within a preset time length; deleting the table in the refresh list that meets the second condition from the refresh list, the second condition being that the attribute value of the cache refresh attribute of the table is changed from enabling refresh to disabling refresh.

[0113] Exemplarily, the preset duration can be controlled by the configuration item auto_refresh_table_eliminate_days, and the preset duration is, for example, the last 7 consecutive days. For the tables in the refresh list AutoRefreshList, if the attribute value of the cache refresh attribute of the table is not to enable refresh, and there are no expired partitions in the partitions of the table within the preset duration, the table can be deleted from the refresh list. If the attribute value of the cache refresh attribute of the table is changed from enabling refresh to disabling refresh, when the table is reloaded from HMS through the metadata automatic synchronization mechanism based on the HMS change log, it is determined that the attribute value of the cache refresh attribute of the table is to disable refresh, and the table is deleted from the refresh list.

[0114] Optionally, the table metadata cache refresh method provided by the embodiment of the present disclosure may also include: outputting relevant information of the table in the refresh list, the relevant information including expired file metadata of the table in the refresh list; and / or, outputting a refresh report for refreshing the file metadata in the partitions of the table in the refresh list.

[0115] Exemplarily, taking the query engine as Impala, during the refresh of the file metadata in the partition of the table in the refresh list, the relevant information of the table in the refresh list can be output, and the relevant information includes the expired file metadata of the table in the refresh list. A refresh report for refreshing the file metadata in the partition of the table in the refresh list can also be output, and the refresh report can be displayed through the WebUI of Catalogd, so that the user can intuitively understand the refresh status of the file metadata. Table 3 shows the specific information of the refresh report.

[0116] Table 3

[0117]

[0118] Based on the above embodiments, the table metadata cache refresh method provided by the embodiment of the present disclosure can timely detect the target tables with expired file metadata in the tables cached by the query engine, and these tables cannot be refreshed by the metadata automatic synchronization mechanism based on the HMS change log; it can automatically refresh the target table to avoid the query anomaly problem of the table caused by the use of expired file metadata. In addition, users can set the tables that need to perform periodic file metadata detection and refresh through table properties, which is a supplement to the existing metadata automatic synchronization solution.

[0119] Exemplary Devices

[0120] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 8The table metadata cache refresh device according to the exemplary embodiment of the present disclosure is described. The device according to the exemplary embodiment of the present disclosure can implement each process in the aforementioned table metadata cache refresh method embodiment and achieve the same function and effect.

[0121] Figure 8 A schematic diagram of the structure of a table metadata cache refresh device provided by an embodiment of the present disclosure, such as Figure 8 As shown, the table metadata cache refresh device 800 of the embodiment of the present disclosure includes: an acquisition module 801, a determination module 802 and a refresh module 803. Among them:

[0122] The acquisition module 801 is used to acquire the first modification time of the partition directory of the table in the distributed file system.

[0123] The determination module 802 is used to determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine.

[0124] The refresh module 803 is used to refresh the file metadata in the partition of the target table if yes.

[0125] In a possible implementation, the determination module 802 may be specifically used to: determine the target partition directory based on the first modification time and the second modification time of the partition directory of the table cached by the query engine, the first modification time of the target partition directory being different from the second modification time; if the time difference between the first modification time of the target partition directory and the current time is greater than the duration threshold, determine that there is a target table with expired file metadata in the table cached by the query engine, and the target table includes the target partition directory; if the second modification time of the partition directory of the table cached by the query engine is the same as the first modification time of the corresponding partition directory in the distributed file system, or the time difference between the first modification time of the target partition directory and the current time is less than or equal to the duration threshold, determine that there is no target table with expired file metadata in the table cached by the query engine.

[0126] In a possible implementation, the table includes multiple partition directories. When the determination module 802 is used to determine the target partition directory according to the first modification time and the second modification time of the partition directory of the table cached by the query engine, it can be specifically used to: sort the multiple partition directories of the table in reverse order according to the modification time to obtain sorted partition directories; detect whether there is a partition directory in the sorted partition directory whose first modification time is different from the second modification time; if so, determine that the target partition directory is a partition directory in the sorted partition directory whose first modification time is different from the second modification time.

[0127] In a possible implementation, the refresh module 803 can be specifically used to: obtain an expired partition in the partition of the target table, the first modification time of the partition directory of the expired partition is different from the second modification time, and the time difference between the first modification time of the partition directory of the expired partition and the current time is greater than a duration threshold; reload the file metadata of the expired partition.

[0128] In one possible implementation, the determination module 802 can be specifically used to: determine the table to be detected from the table cached by the query engine based on the loading time of the table cached by the query engine, the preset detection interval duration, and the current time; and determine whether there is a target table with expired file metadata in the table cached by the query engine based on the first modification time and the second modification time of the partition directory of the table to be detected.

[0129] In a possible implementation, the acquisition module 801 can be specifically used for at least one of the following: periodically acquiring the first modification time of the partition directory of the table in the distributed file system according to a preset detection interval; acquiring the first modification time of the partition directory of the table in the distributed file system within a first time interval, and the first time interval is determined based on the historical data production time of the table; acquiring the first modification time of the partition directory of the table in the distributed file system within a second time interval, and the second time interval is determined based on the time corresponding to the query behavior of the query engine on the table.

[0130] In a possible implementation, the refresh module 803 may be specifically configured to: add the target table to the refresh list; and refresh the file metadata in the partitions of the table in the refresh list.

[0131] In one possible implementation, when the refresh module 803 is used to refresh the file metadata in the partitions of the table in the refresh list, it can be specifically used for at least one of the following: periodically refreshing the file metadata in the partitions of the table in the refresh list according to a preset refresh interval; refreshing the file metadata in the partitions of the table in the refresh list within a first time interval; refreshing the file metadata in the partitions of the table in the refresh list within a second time interval.

[0132] In a possible implementation, the refresh module 803 can also be used to: when the query engine loads a table, obtain the attribute value of the cache refresh attribute of the loaded table, and the attribute value is configurable; if the attribute value is to enable refresh, then add the table with the attribute value to enable refresh to the refresh list; if the attribute value is to disable refresh, then do not add the table with the attribute value to disable refresh to the refresh list.

[0133] In a possible implementation, the refresh module 803 can also be used to: delete from the refresh list a table that meets a first condition, wherein the first condition is that the attribute value of the cache refresh attribute of the table is not to enable refresh, and there are no expired partitions in the partitions of the table within a preset time period; delete from the refresh list a table that meets a second condition, wherein the second condition is that the attribute value of the cache refresh attribute of the table is changed from enabling refresh to disabling refresh.

[0134] In a possible implementation, the refresh module 803 may also be used to: output relevant information about the tables in the refresh list, the relevant information including expired file metadata of the tables in the refresh list; and / or output a refresh report for refreshing the file metadata in the partitions of the tables in the refresh list.

[0135] In one possible implementation, the determination module 802 can be specifically used to: lock the table cached by the query engine to obtain the locked table; obtain the second modification time of the partition directory of the locked table; unlock the locked table; and determine whether there is a target table with expired file metadata in the table cached by the query engine based on the first modification time and the second modification time of the partition directory of the locked table.

[0136] The device of the embodiment of the present disclosure can be used to execute the solution of the table metadata cache refresh method in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0137] Fig. 9 A schematic diagram of the structure of a table metadata cache refresh device provided by another embodiment of the present disclosure is shown in FIG. Fig. 9 As shown, the table metadata cache refresh device 900 of the embodiment of the present disclosure includes: a file metadata health detection module 901 and a file metadata refresh module 902. Among them:

[0138] The file metadata health detection module 901 is used to periodically detect the file metadata of the table cached by the query engine to determine whether there is a table with expired file metadata in the table cached by the query engine.

[0139] The file metadata refreshing module 902 is used to periodically detect tables that may have expired file metadata, and when expired file metadata is found, refresh the file metadata.

[0140] For example, in a possible implementation, secondary development can be performed based on Impala, using Impala as a carrier to transform the Catalog component, and by adding a new HDFS file metadata refresh function on the Catalog to solve the problem of outdated file metadata, it can actively identify outdated file metadata caches and efficiently perform partition-level refreshes based on the historical output time of data files. Among them, this function is a supplement to the existing metadata automatic synchronization function, and the solution includes the file metadata health detection module and the file metadata refresh module of the embodiment of the present disclosure.

[0141] The device of the embodiment of the present disclosure can be used to execute the solution of the table metadata cache refresh method in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0142] Exemplary Media

[0143] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig.10 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0144] Fig.10 A schematic diagram of a storage medium provided by an embodiment of the present disclosure. Fig.10 As shown, the storage medium 1000 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0145] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium.

[0147] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0148] Exemplary Computing Devices

[0149] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig.11 A computing device according to an exemplary embodiment of the present disclosure is described.

[0150] Fig.11 The computing device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] Fig.11 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the computing device 1100 is in the form of a general computing device. The components of the computing device 1100 may include, but are not limited to: the at least one processing unit 1101, the at least one storage unit 1102, and a bus 1103 connecting different system components (including the processing unit 1101 and the storage unit 1102). Exemplarily, the processing unit 1101 may be specifically a processor, the storage unit 1102 stores computer-executable instructions, and the processing unit 1101 executes the computer-executable instructions stored in the storage unit 1102 to implement the above-mentioned table metadata cache refresh method.

[0152] The bus 1103 includes a data bus, a control bus, and an address bus.

[0153] The storage unit 1102 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 11021 and / or a cache memory 11022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 11023 .

[0154] The storage unit 1102 may also include a program / utility 11025 having a set (at least one) of program modules 11024, such program modules 11024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0155] The computing device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 1105. Furthermore, the computing device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1106. Fig.11 As shown, the network adapter 1106 communicates with other modules of the computing device 1100 via the bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] It should be noted that although several units / modules or sub-units / modules of the table metadata cache refresh device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be concretized by multiple units / modules.

[0157] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0158] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A table metadata cache refresh method, comprising: Get the first modification time of the partition directory of the table in the distributed file system; Determining whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine; If so, the file metadata in the partition of the target table is refreshed.

2. The table metadata cache refresh method according to claim 1, wherein determining whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine comprises: Determine a target partition directory according to the first modification time and a second modification time of the partition directory of the table cached by the query engine, wherein the first modification time and the second modification time of the target partition directory are different; If the time difference between the first modification time of the target partition directory and the current time is greater than the duration threshold, it is determined that there is a target table with expired file metadata in the table cached by the query engine, and the target table includes the target partition directory; If the second modification time of the partition directory of the table cached by the query engine is the same as the first modification time of the corresponding partition directory in the distributed file system, or the time difference between the first modification time of the target partition directory and the current time is less than or equal to the duration threshold, it is determined that there is no target table with expired file metadata in the table cached by the query engine.

3. The table metadata cache refresh method according to claim 2, wherein the table includes multiple partition directories, and the determining the target partition directory according to the first modification time and the second modification time of the partition directory of the table cached by the query engine comprises: Sorting the multiple partition directories of the table in reverse order according to the modification time to obtain sorted partition directories; Detect whether there is a partition directory with a first modification time different from a second modification time in the sorted partition directories; If so, it is determined that the target partition directory is a partition directory in the sorted partition directories whose first modification time and second modification time are different.

4. According to the table metadata cache refresh method of claim 2, refreshing the file metadata in the partition of the target table comprises: Acquire an expired partition in the partitions of the target table, wherein the first modification time and the second modification time of the partition directory of the expired partition are different, and the time difference between the first modification time and the current time of the partition directory of the expired partition is greater than the duration threshold; Reload the file metadata of the expired partition.

5. The table metadata cache refresh method according to any one of claims 1 to 4, wherein determining whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and the second modification time of the partition directory of the table cached by the query engine comprises: Determine a table to be detected from the table cached by the query engine according to the loading time of the table cached by the query engine, the preset detection interval duration, and the current time; According to the first modification time and the second modification time of the partition directory of the table to be detected, it is determined whether there is a target table with expired file metadata in the table cached by the query engine.

6. According to the table metadata cache refresh method of claim 5, the step of obtaining the first modification time of the partition directory of the table in the distributed file system comprises at least one of the following: Periodically acquiring a first modification time of a partition directory of a table in the distributed file system according to the preset detection interval; Acquire a first modification time of a partition directory of a table in the distributed file system within a first time interval, where the first time interval is determined based on a historical data generation time of the table; A first modification time of a partition directory of a table in the distributed file system is obtained within a second time interval, where the second time interval is determined based on a time corresponding to a query behavior of the query engine on the table.

7. A table metadata cache refresh device, comprising: An acquisition module, used to acquire the first modification time of a partition directory of a table in a distributed file system; A determination module, configured to determine whether there is a target table with expired file metadata in the table cached by the query engine according to the first modification time and a second modification time of the partition directory of the table cached by the query engine; A refresh module is used to refresh the file metadata in the partition of the target table if yes.

8. A computing device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the table metadata cache refresh method according to any one of claims 1 to 6.

9. A storage medium storing computer program instructions, wherein the computer program instructions, when executed, implement the table metadata cache refresh method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed, the table metadata cache refresh method according to any one of claims 1 to 6 is implemented.

Images