Metadata information acquisition method and related equipment

Abstract

The invention discloses a metadata information acquisition method and related equipment. The method comprises the steps that historical execution statements are analyzed, information to be counted is obtained, the historical execution statements comprise execution statements for an original data table, and the information to be counted at least comprises column information of a target data table; performing aggregation processing on the to-be-counted information to generate a metadata statistical task; and executing the metadata statistics task to obtain metadata information. According to the scheme, the statistical efficiency and accuracy of the metadata information can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and related equipment for acquiring metadata information. Background Technology

[0002] In the era of big data, the amount of data continues to surge, and it is usually necessary to collect metadata information from data tables. The metadata information of data tables can be used to support various data-related operations such as data management, data governance, and data analysis.

[0003] Currently, the acquisition of metadata information often relies on user configuration or triggering of statistical execution, which is prone to statistical errors. In addition, when the data volume is large, the computational workload of metadata information statistics is also large. Therefore, there are problems such as low efficiency and low accuracy of metadata information statistics. Summary of the Invention

[0004] The main technical problem addressed in this application is to provide a method and related equipment for acquiring metadata information, which can improve the statistical efficiency and accuracy of metadata information.

[0005] The first aspect of this application provides a method for obtaining metadata information. The method includes: parsing historical execution statements to obtain information to be statistically analyzed, wherein the historical execution statements include execution statements on the original data table, and the information to be statistically analyzed includes at least column information of the target data table; aggregating the information to be statistically analyzed to generate a metadata statistical task; and executing the metadata statistical task to obtain metadata information.

[0006] A second aspect of this application provides a computer device including a memory and a processor coupled to each other, the memory storing program data and the processor executing the program data to implement any step of the above-described method for obtaining metadata information.

[0007] A third aspect of this application provides a computer-readable storage medium storing program data executable by a processor, the program data being used to implement any step of the method for obtaining the aforementioned metadata information.

[0008] The above solution parses historical execution statements to obtain statistical information. These historical execution statements include statements executed on the original data table, and the statistical information must at least contain column information from the target data table. This statistical information is then aggregated to generate a metadata statistical task. Executing the metadata statistical task yields metadata information. By parsing the actual historical execution statements, the solution identifies the columns requiring metadata statistics. Optimizing the statistical logic through aggregation processing and generating corresponding metadata statistical tasks allows for more accurate metadata statistics. Unnecessary columns are excluded from the statistics, reducing the amount of metadata collected and improving the accuracy and efficiency of metadata information acquisition.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them: Figure 1 This is a flowchart illustrating an embodiment of the method for obtaining metadata information in this application; Figure 2 This application Figure 1 A flowchart illustrating an embodiment of step S12; Figure 3 This application Figure 1 A flowchart illustrating an embodiment of step S13; Figure 4 This is a schematic diagram of an embodiment of the metadata information acquisition device of this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device of this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0013] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0015] This application provides the following embodiments, and each embodiment will be described in detail below.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a method for obtaining metadata information according to this application. The method may include the following steps: S11: Parse the historical execution statements to obtain the statistical information to be collected. The historical execution statements include the execution statements on the original data table, and the statistical information to be collected includes at least the column information of the target data table.

[0017] Optionally, the method for obtaining metadata information in this application can be executed by computer equipment (such as the data warehouse service Hive), and this application does not impose any restrictions on this. Hive is a data warehouse architecture component and also a data warehouse based on OLAP (On-Line Analysis Processing). It can perform some ETL (Extraction Transformation Loading) tasks on massive amounts of data, such as extraction, loading, and transformation, for analysis and querying, and use the query results for decision analysis.

[0018] It can retrieve historical execution statements, which may include statements executed on the original data table. These statements can represent processing of the column information of the original data table, such as querying, modifying, deleting, or adding data. This application does not restrict the specific execution statements. For example, historical execution statements are statements submitted by users or clients to the current service at a historical time. The current service can be the current execution engine, such as Hive or Spark SQL, and the execution statement can be the corresponding SQL statement in MySQL. The execution statement can be any statement that can be executed. For example, whenever Hive or Spark SQL successfully executes an SQL statement, the following steps are taken to parse that SQL statement.

[0019] Then, the historical execution statements are parsed to obtain the statistical information to be collected. This information must at least include the column information of the target data table, i.e., the column information of the metadata that needs to be statistically analyzed. Optionally, the execution engine can use an asynchronous parsing method to parse the historical execution statements to obtain the statistical information. For example, the asynchronous parsing method includes executing the SQL statement at a first time and parsing the SQL statement at a second time, where the first time and the second time are different times.

[0020] In some implementations, the information to be statistically analyzed can be stored in a first statistical table (such as table A in MySQL). The information to be statistically analyzed includes at least one of the following: column information of the target data table (such as column names), the table name of the target data table, whether preset metadata needs to be analyzed, execution time, etc. Optionally, the preset metadata can be a histogram. The histogram is metadata describing the distribution of data in a certain column, mainly including a set number of bins (specified by user parameters) and a bin height (hight). Each bin contains an upper bound (bin.hi), a lower bound (bin.lo), and ndv (the number of unique values, i.e., the number of distinct values). A bin represents that in this column of data, there are a total of hight data values ​​greater than the lower bound bin.lo and less than the upper bound bin.hi, of which there are ndv distinct values.

[0021] In some implementations, historical execution statements can be parsed to determine preset clauses; based on the column information corresponding to the preset clauses, the information to be statistically analyzed can be obtained. Optionally, the preset clauses may include at least one of the following: filtering clauses (such as WHERE clauses), join clauses (such as JOIN clauses), query clauses (such as SELECT clauses), grouping clauses (such as GROUP BY clauses), etc. This application does not limit the preset clauses. Optionally, the column information corresponding to the preset clauses, such as the condition values, query values, and key values ​​(i.e., key values) corresponding to the preset clauses, can be used to obtain the information to be statistically analyzed from the column information of the target data table. For example, the condition values ​​in the WHERE clause, the key values ​​in the JOIN clause, the query values ​​in the SELECT clause, and the key values ​​in the GROUP BY clause of the SQL statement can be obtained to obtain the column information of the target data table to be statistically analyzed.

[0022] In some implementations, historical execution statements can be divided into multiple layers, and each layer can be parsed separately to obtain the column information of the target data table to be statistically analyzed for each layer, thereby obtaining the statistical information for each layer. Specifically, historical execution statements can be parsed layer by layer to determine the preset clauses for each layer. Based on the column information corresponding to the preset clauses for each layer, the statistical information for each layer can be obtained. Then, the statistical information for each layer can be combined to obtain the final statistical information. The method of parsing the preset clauses for each layer to obtain the statistical information can refer to the above method, and will not be repeated here.

[0023] For example, taking SQL statements as an example, the historical execution statements are as follows: SELECT i_item_id item_id, sum( ss_ext_sales_price ) AS ss_item_rev FROM store_sales JOIN item ON store_sales.ss_item_sk = item.i_item_sk JOIN date_dim ON store_sales.ss_sold_date_sk = date_dim.d_date_sk JOIN ( SELECT d1.d_date FROM date_dim d1 JOIN date_dim d2 ON d1.d_week_seq = d2.d_week_seq WHERE d2.d_date = '1998-08-04' ) sub ON date_dim.d_date = sub.d_date The above SQL statement is parsed in layers to obtain the statistical information to be collected at each layer.

[0024] For example, the above SQL statement can be divided into two layers: the first layer is the outer query, and the second layer is the subquery.

[0025] For example, the SQL statement for the first-level query is as follows: SELECT i_item_id item_id, sum( ss_ext_sales_price ) AS ss_item_rev FROM store_sales JOIN item ON store_sales.ss_item_sk = item.i_item_sk JOIN date_dim ON store_sales.ss_sold_date_sk = date_dim.d_date_sk JOIN (second-level query) sub ON date_dim.d_date = sub.d_date GROUP BY i_item_id The first-level query can be parsed to determine the column information corresponding to its preset clauses, thus obtaining the statistical information to be collected in the first-level query. For example, in the first-level query, the keys of the JOIN clause include: ss_item_sk, i_item_sk, ss_sold_date_sk, d_date_sk, and d_date, which represent the column information of the target data table to be statistically analyzed in the first-level query. Each clause can represent the processing of column information in the original data table. For example, the JOIN clause "JOIN item ON store_sales.ss_item_sk = item.i_item_sk" indicates a JOIN operation between the column information ss_item_sk of the original data table store_sales and the column information i_item_sk of the original data table item.

[0026] For example, the SQL statement for the second-level query is as follows: SELECT d1.d_date FROM date_dim d1 JOIN date_dim d2 ON d1.d_week_seq = d2.d_week_seq WHERE d2.d_date = '1998-08-04' The second-level query can be parsed to determine the column information corresponding to its preset clauses, thus obtaining the statistical information to be collected in the second-level query. For example, in the second-level query, the column in the WHERE filter clause is d_date, the key in the JOIN clause is d_week_seq, and the column in the SELECT clause is d_date, which means obtaining the column information of the target data table to be collected in the second-level query.

[0027] By combining the column information from each layer to be analyzed, the final column information of the target data table to be analyzed is obtained. Specifically, the columns to be analyzed for this historical execution statement include: ss_item_sk, i_item_sk, d_date, ss_sold_date_sk, d_date_sk, and d_week_seq. These columns can be queried (e.g., based on preset clauses) to obtain the original data table to which each column belongs, thus retrieving the corresponding target data table. For example, it can be found that the ss_item_sk and ss_sold_date_sk columns belong to the store_sales table, the d_date_sk, d_week_seq, and d_date columns belong to the date_dim table, and the i_item_sk column belongs to the item table, thus obtaining the target data table. Through the above method, the column information (column names) and the table name of the target data table can be obtained.

[0028] In some embodiments, after obtaining the statistical information based on the column information corresponding to the preset clause, the statistical information can be deduplicated to obtain the deduplicated statistical information. The column names and table names of the target data table can be deduplicated to reduce the amount of computation.

[0029] In some embodiments, the information to be statistically analyzed may also include whether the column information of the target data table needs to be statistically analyzed using preset metadata. After the above steps, it can also be determined whether the column information needs to be statistically analyzed using preset metadata, that is, whether it is necessary to statistically analyze a histogram.

[0030] Whether column information needs to be included in the preset metadata can be determined based on the preset clause to which it belongs. Optionally, if the column information belongs only to the first preset clause, it is determined that the column information does not need to be included in the histogram; otherwise, it is determined that the histogram needs to be included. For example, if the first preset clause is a query clause, and the column information belongs to a column in the query clause (SELECT clause), it is determined that the column information does not need to be included in the histogram. If the column information belongs to both the first preset clause and other clauses, it is determined that the histogram needs to be included. Optionally, if the column information belongs to the second preset clause, it is determined that the column information needs to be included in the histogram; otherwise, it is determined that the histogram does not need to be included. The second preset clause is a filter clause, a join clause, or a grouping clause.

[0031] For example, column information ss_item_sk belongs to the key of the JOIN clause, so it is determined that its histogram needs to be calculated; column information d_date belongs to both the SELECT clause and the JOIN clause, so it is determined that its histogram needs to be calculated, etc.

[0032] Optionally, the above-mentioned information to be statistically analyzed can be encapsulated into individual data entries and stored in the first data table A. The information to be statistically analyzed includes: column information of the target data table (such as column names), the table name of the target data table, whether preset metadata needs to be statistically analyzed, execution time, etc. Among them, the preset metadata is a histogram, and the execution time is the execution time of the SQL statement or preset clause.

[0033] For example, the data format of the first data table A is as follows:

[0034] Table 1 First Data Table A S12: Aggregate the statistical information to generate metadata statistics tasks.

[0035] The data in the first data table A, which is the information to be statistically analyzed, can be aggregated to generate metadata statistical tasks, which can then be stored in a second data table (such as table B in MySQL). For example, the column information that needs to be statistically analyzed can be extracted from the first data table A, and then aggregated according to the table name and column name to assemble a metadata statistical task.

[0036] In some implementations, the information to be statistically analyzed also includes the execution time corresponding to the column information of the target data table. A first thread is started, for example, a first thread T1 can be started in the Hive service. Then, the first thread T1 uses the execution time in the information to be statistically analyzed to aggregate the various pieces of information, obtaining aggregated statistics. Based on the aggregated statistics, a metadata statistics task is generated.

[0037] In some embodiments, please refer to Figure 2This embodiment can be further extended to step S12 of the above embodiment. This embodiment may include the following steps: S121: Using the execution time in the information to be statistically analyzed, the information to be statistically analyzed in at least two preset execution ranges is filtered to obtain the filtered statistical information in each preset execution range.

[0038] Based on the execution time corresponding to each column of information in the information to be statistically analyzed, the information to be statistically analyzed within at least two preset execution ranges can be filtered. This yields the information to be statistically analyzed within each preset execution range, which serves as the filtered statistical information. The at least two preset execution ranges correspond to different durations, and each preset execution range corresponds to a priority level.

[0039] S122: Aggregate the various screening statistics to obtain the aggregated statistics corresponding to each preset execution range.

[0040] For each preset execution range, the filtered statistical information is aggregated to obtain the aggregated statistical information corresponding to each preset execution range. Optionally, the aggregation process can use the table names and column names in the filtered statistical information to remove duplicates, and then perform aggregation. The filtered statistical information is aggregated according to the table names and column names. During the aggregation process, the same aggregated statistical information can be aggregated. For example, the number of times the same table name and the same column name appear can be counted, and the number of times the same histogram needs to be calculated can be counted. This application does not limit the aggregation processing method.

[0041] In some implementations, the aggregated statistical information includes at least one of the following: the number of times the filtered statistical information appears within a preset execution range, the number of times preset metadata needs to be statistically analyzed, and the priority, etc., wherein the priority corresponds to the preset execution range, and the closer the time range corresponding to the preset execution range is to the current time, the higher the priority. For example, the preset time range includes within 1 month, within 1 week, and within 1 hour, etc., and the priority corresponding to within 1 hour is higher than the priority corresponding to within 1 week and within 1 month.

[0042] For example, the following embodiments use two preset execution ranges as examples to illustrate steps S121 to S122 above. Other situations can be referred to the following embodiments.

[0043] The first thread, T1, first filters out data from the first data table A (i.e., the data to be statistically analyzed) whose execution time falls within the first execution range, thus obtaining the first filtered statistical information. The first execution range includes the period from the current time to the first priority threshold, low_priority_interval. For example, the first priority threshold, low_priority_interval, can be configured to be one month. This allows filtering out the data corresponding to SQL statements executed within one month, which constitutes the first filtered statistical information.

[0044] Then, the first set of filtered statistics is deduplicated and aggregated. The number of times each column of information appears within the first execution range and the number of times a histogram needs to be calculated are counted are recorded. These data are assigned a priority of 1, resulting in the first aggregated statistics. For example, the first aggregated statistics can be represented as follows:

[0045] Table 2 First Aggregated Statistical Information Table Next, the first thread T1 is used to filter out data from the first data table A (i.e., the data to be statistically analyzed) whose execution time falls within the second execution range, thus obtaining the second filtered statistical information. The second execution range includes the period from the current time to the second priority threshold `high_priority_interval`. The second priority threshold is lower than the first priority threshold. For example, the second priority threshold `high_priority_interval` can be configured to be 1 hour, thereby filtering out the data corresponding to SQL statements executed within one hour, which constitutes the second filtered statistical information.

[0046] Then, the second set of filtering statistics is deduplicated and aggregated. The number of times each column of information appears within the second execution range and the number of times a histogram needs to be calculated are counted are recorded. These data are assigned a priority of 2, resulting in the second aggregated statistics. For example, the second aggregated statistics can be represented as follows:

[0047] Table 3 Second Aggregated Statistical Information Table Finally, by combining the aggregated statistical information within each preset execution range, the final aggregated statistical information can be obtained and stored in the second data table B.

[0048] Alternatively, after the above steps, the following steps may also be performed: S123: Generate metadata statistics tasks for each column of information in the aggregated statistics according to the preset sorting method.

[0049] By aggregating statistical information from various preset execution ranges, the final aggregated statistical information is obtained. Then, according to a preset sorting method, the aggregated statistical information is sorted, and metadata statistical tasks can be generated for each column of information in the aggregated statistical information or for each target data table. For example, one target data table corresponds to one metadata statistical task. The metadata statistical tasks can be stored in the second data table B.

[0050] In some implementations, a preset number of second threads T2 can be started to execute the metadata statistics tasks. Optionally, the preset number is related to the number of column information, or the preset number is related to the number of metadata statistics tasks.

[0051] In some implementations, the preset sorting method can be a descending order of priority and frequency of occurrence. For example, the data in the second data table can be sorted in descending order of priority and frequency of occurrence, with the first column (or aggregated statistics) corresponding to the highest priority and most frequent occurrence. A metadata statistics task can be generated from the column with the highest priority and most frequent occurrence, and the execution module can be called to execute this task. After the metadata statistics task is completed, the data corresponding to that task is deleted from the second data table B. The above steps can be repeated to generate and execute metadata statistics tasks for each column until there is no data in the second data table B. Then, steps S121 or S11 and subsequent steps are repeated to obtain a new second data table B, and the subsequent steps are continued.

[0052] The above solution can prioritize the collection of metadata information based on the execution time, frequency of occurrence, and priority of columns that require metadata statistics. This ensures that frequently used metadata information is collected first, providing timely metadata updates to the execution engine. Furthermore, by generating two priority statistical tasks based on the frequency of occurrence and execution time, the most frequently occurring metadata is collected first, further guaranteeing the timeliness of the metadata information.

[0053] Optionally, the specific implementation process for performing metadata tasks can refer to the following steps.

[0054] Continue reading Figure 1 Following step S12 above, the following steps are also included: S13: Perform metadata statistics task to obtain metadata information.

[0055] Metadata statistics tasks can be read from the second data table B, executed, and metadata information obtained. After completion, the metadata information is written to the target storage space. Optionally, the target storage space can be the Hive Metastore, a metadata storage service. The Hive Metastore is a service within the Hive component for storing Hive table information, including table names, column names, file storage locations, and statistical information about the table and columns. This application does not impose restrictions on the target storage space. Optionally, a second thread can be started to execute the metadata statistics task and obtain the metadata information.

[0056] Optionally, metadata refers to data that describes data, and can refer to statistical information used to describe the characteristics of data columns. For example, metadata information includes at least one of the following: the number of rows in the table, file size, maximum value, minimum value, average length, maximum length of the column, number of null values ​​(such as null values), number of distinct values, histogram of data distribution, etc. This application does not impose any restrictions on metadata information. A data column is the basic unit that constitutes a table in structured data, mainly referring to columns in Hive tables. Each data column has a specific data type (such as int, string, date, etc.) and business meaning (such as username, age, etc.). This application does not impose any restrictions on the columns in the tables of this application.

[0057] For example, the ANALYZE statement can be used to perform metadata statistics tasks and obtain metadata information. The ANALYZE statement is a command used in execution engines such as Spark SQL Service and Hive Service to trigger metadata statistics. For example, the Spark SQL Service's ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNScolumn1, column2 statement can calculate statistics for specified columns, obtain metadata information, and store it in HiveMetasotre. In some implementations, based on the metadata statistics task in the second data table B, the target data table is obtained from a preset storage, such as column name or table name, and then the metadata statistics task is performed on the target data table to obtain the corresponding metadata information.

[0058] The above solution parses historical execution statements to obtain statistical information. These historical execution statements include statements executed on the original data table, and the statistical information must at least contain column information from the target data table. This statistical information is then aggregated to generate a metadata statistical task. Executing the metadata statistical task yields metadata information. By parsing the actual historical execution statements, the solution identifies the columns requiring metadata statistics. Optimizing the statistical logic through aggregation processing and generating corresponding metadata statistical tasks allows for more accurate metadata statistics. Unnecessary columns are excluded from the statistics, reducing the amount of metadata collected and improving the accuracy and efficiency of metadata information acquisition.

[0059] In some embodiments, please refer to Figure 3 This embodiment can further extend step S13 of the above embodiment. Performing a metadata statistics task to obtain metadata information may include the following steps: S131: Based on metadata statistics tasks, retrieve the target data table from the preset storage.

[0060] A second thread is started to execute metadata tasks. This thread can retrieve the target data table from a preset storage based on metadata statistics tasks. The preset storage can be either the target storage space or the metadata storage HiveMetastore.

[0061] Optionally, based on a metadata statistics task, relevant information about the target data table can be retrieved from a preset storage. This relevant information includes at least one of the following: the storage location of the target data table, its type information, the last statistical time, and the last update time. The type information can include the table's file type, column type, etc. The last statistical time (lastAnalyzeTime) indicates the time when metadata statistics were last performed on the target data table. The last update time (lastDDLTime) indicates the time when the target data table was last updated. The update can be an execution of DDL (Data Definition Language), which in database management is a series of commands used to define or modify the database structure.

[0062] In some implementations, it is determined whether the target data table meets preset update conditions. These preset update conditions include: the last statistical time is not greater than the last update time. Specifically, it can be determined whether the last statistical time (lastAnalyzeTime) is greater than the last update time (lastDDLTime). If the last statistical time (lastAnalyzeTime) is greater than the last update time (lastDDLTime), the target data table is determined to meet the preset update conditions. This means that the target data table has not been updated since the last metadata statistics were completed, i.e., it has not changed, and therefore does not need to be re-statistically analyzed. The metadata statistics task ends here. Otherwise, the metadata statistics task continues. This method can further reduce the workload of metadata statistics. Data update refers to the addition, modification, or deletion of data in the original data table of the data source (such as Hive tables, Parquet files, etc.). Data updates may cause metadata statistics to become invalid, requiring re-statistical analysis of the metadata.

[0063] Optionally, in response to the target data table meeting the preset update conditions, it is further determined whether the current environment information meets the preset resource conditions.

[0064] S132: Determine whether the current environmental information meets the preset resource conditions.

[0065] Optionally, the current environment information includes at least one of the following: execution configuration information and current resource information. The execution configuration information refers to whether the configuration parameter `analyzeOnYarn` is true or false, meaning the `analyzeOnYarn` parameter indicates whether to submit a metadata statistics task to the resource manager. The resource manager can be Yarn, meaning whether to submit a Yarn task to collect metadata statistics. The current resource information refers to the current cluster resources of the resource manager Yarn. Optionally, the current environment information meeting preset resource conditions includes at least one of the following: the execution configuration information including the configuration parameter `analyzeOnYarn` being true, and the current cluster resources meeting execution conditions. Execution conditions include the current cluster resources being greater than a preset resource threshold, meaning the current cluster resources are sufficient to execute the metadata statistics task.

[0066] In response to at least one of the following: the configuration parameter analyzeOnYarn is true, or the current cluster resources meet the execution conditions, the current environment information is determined to meet the preset resource conditions; otherwise, the current environment information is determined not to meet the preset resource conditions.

[0067] Optionally, in response to the current environmental information meeting the preset resource conditions, step S133 can be executed. Optionally, in response to the current environmental information not meeting the preset resource conditions, step S134 can be executed.

[0068] S133: Submit the metadata statistics task to the first service so that the first service can perform the metadata statistics task on the target data table and obtain metadata information.

[0069] In response to the current environment information meeting the preset resource conditions, the metadata statistics task is submitted to the first service, so that the first service can perform the metadata statistics task on the target data table to obtain metadata information.

[0070] Optionally, the first service can be Spark SQL, a structured data processing module in the Apache Spark ecosystem. Spark SQL supports processing structured data using SQL statements or DataFrame / Dataset APIs, and can be integrated with data sources such as Hive to provide efficient distributed computing capabilities. The current service (e.g., Hive) submits metadata statistics tasks to the Spark SQL service (e.g., the Spark SQL engine) for processing. The Spark SQL service can submit Yarn tasks, request cluster resources from the Yarn resource manager, and execute metadata statistics tasks on the target data table within containers managed by Yarn to obtain metadata information. Finally, the metadata information is stored in the Hive Metastore metadata store.

[0071] Optionally, the metadata information includes at least one of the following: the number of rows in the table, the file size, the maximum value, minimum value, average length, maximum length of the column, the number of null values ​​(such as null values), the number of distinct values, histograms, etc. Each metadata information can be obtained using a preset statistical algorithm, and this application does not limit the preset statistical algorithm.

[0072] In some implementations, the number of rows in the target data table is obtained; in response to a number of rows exceeding a preset number, a preset sampling statistical method is used to perform metadata statistical tasks on the target data table to obtain metadata information. Optionally, the preset number of rows corresponding to the first service is greater than or equal to the preset number of rows corresponding to the second service. Optionally, when using a preset sampling statistical method, the target data table can be sampled using a preset sampling method to obtain statistical data to be collected, and then metadata statistics are performed on the statistical data to be collected to obtain metadata information. Sampling can be performed on column information or row information; column information refers to the column information for which metadata needs to be collected. The above method can dynamically determine whether to sample and calculate statistical metadata information based on the size of the number of rows, ensuring statistical efficiency and relative accuracy under large data volumes.

[0073] In some implementations, the preset metadata is a histogram. When it is necessary to calculate the histogram, in response to the need to calculate the histogram, a first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table, and a second preset algorithm is used to obtain the unique values ​​of two adjacent quantiles. The histogram is generated based on the unique values.

[0074] In some implementations, performing metadata statistics on the target data table using a first service to obtain metadata information may include the following steps: A1: Spark SQL service retrieves the number of rows in the target data table.

[0075] For example, the count(*) operator is applied to the target data table to calculate the rowCount of the target data table.

[0076] A2: Based on the number of rows, obtain the statistical data to be collected from the target data table.

[0077] For the column information of the target data table, based on the number of rows, the data of each row of the target data table is obtained to obtain the statistical data to be collected.

[0078] Optionally, determine whether the row count (rowCount) is greater than a preset row count. If the row count (rowCount) is greater than the preset row count (e.g., 5 million rows), perform a metadata statistics task on the target data table using a preset sampling statistical method to obtain metadata information; otherwise, perform a full statistical method on the target data table to obtain metadata information.

[0079] Optionally, in response to rowCount exceeding 5 million rows, the sampling parameter isSample is configured to true to perform metadata statistics using a preset sampling method; otherwise, the sampling parameter isSample is configured to false to perform metadata statistics using a full-scale statistical method.

[0080] Optionally, under a preset sampling statistical method, a sampling ratio 'a' is obtained, where the ratio of a preset quantity threshold to the number of rows can be obtained. For example, the preset quantity threshold can be a preset number of rows, and the sampling ratio 'a' = 5,000,000 / rowCount. Then, the sampling ratio is used to sample each row of the target data table to obtain the statistical data to be obtained. For example, a sampling operator (such as the sample(a) operator) can be generated using the sampling ratio 'a', and the sampling operator can be used to sample the target data table to obtain the statistical data to be obtained. For instance, if the target data table contains 100 columns of information, and the sampling ratio 'a' is 50, then data corresponding to 50 columns of information can be sampled from the 100 columns as the statistical data to be obtained.

[0081] Optionally, in the case of full statistics, data from each row of the target data table can be obtained to obtain the statistical data to be collected.

[0082] A3: Use a preset statistical algorithm to perform statistical analysis on the data to be analyzed and obtain metadata information.

[0083] Optionally, for the statistical data to be collected, a preset statistical algorithm is used, such as the min, max, average, maxlength, count(is null), approx_count_distinct algorithms of Spark SQL service, to calculate the maximum value, minimum value, average length, number of null values ​​(such as null values), number of distinct values, etc. of the column information.

[0084] Optionally, if a histogram needs to be calculated, step A4 is executed to obtain the histogram. Otherwise, step A5 is executed.

[0085] A4: In response to the need to generate a histogram, the first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table, and the second preset algorithm is used to obtain the unique values ​​of two adjacent quantiles, and a histogram is generated based on the unique values.

[0086] In response to the need to calculate a histogram, a first preset algorithm can be used to obtain the quantiles corresponding to the columns in the target data table. For example, the first preset algorithm can be the udf percentile_approx algorithm from Spark SQL Services. For each column of information in the target data table, the quantiles required for the histogram of each column are calculated using the udf percentile_approx algorithm from Spark SQL Services.

[0087] Optionally, the number of quantiles is determined by the number of bins in the histogram. For example, if the number of quantiles is determined by 1 / bins, then quantiles of 0*1 / bins, 1*1 / bins, 2*1 / bins, ..., bins*1 / bins need to be calculated. For instance, if the number of bins is 10, then 11 quantiles need to be calculated (0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1); if the number of bins is 100, then 101 quantiles need to be calculated (0, 0.01, 0.02, 0.03, 0.04, ..., 0.98, 0.99, 1).

[0088] Then, a second preset algorithm is used to obtain the unique values ​​between two adjacent quantiles, and a histogram is generated based on these unique values. For example, the second preset algorithm can be the approx_count_distinct udf algorithm from Spark SQL Service. For the aforementioned quantiles, the approx_count_distinct udf algorithm from Spark SQL Service is used to calculate the unique value between every two adjacent quantiles, obtaining the distinct count.

[0089] Then, a histogram can be generated based on the unique value (distinct number). For example, two adjacent quantiles are used as the upper and lower bounds of a bucket in the histogram, and the distinct number between adjacent quantiles is used as the distinct number of the bucket. A total of bins are generated as histogram information, thus generating the histogram.

[0090] A5: Write the metadata information obtained above to the metadata storage Hive Metastore.

[0091] The metadata information obtained in the above steps can be written to the metadata storage Hive Metastore, thus completing the metadata statistics task.

[0092] S134: Use the second service to perform metadata statistics on the target data table and obtain metadata information.

[0093] If the current environment information does not meet the preset resource conditions, a second service is used to perform a metadata statistics task on the target data table to obtain metadata information. This second service can be the current service, for example, the Hive service. The local service of the Hive service is used to perform the metadata statistics task on the target data table, obtain metadata information, and store the metadata information in the Hive Metastore.

[0094] In some implementations, the number of rows in the target data table is obtained; if the number of rows is greater than a preset number, a metadata statistical task is performed on the target data table using a preset sampling statistical method to obtain metadata information. Otherwise, a full statistical method is used to perform a metadata statistical task on the target data table to obtain metadata information.

[0095] In some implementations, the preset metadata is a histogram. In response to the need to calculate the histogram, a first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table, and a second preset algorithm is used to obtain the unique values ​​of two adjacent quantiles. The histogram is generated based on the unique values.

[0096] In some implementations, using a second service to perform metadata statistics on the target data table to obtain metadata information may include the following steps: B1: The Hive service retrieves the number of rows in the target data table.

[0097] Perform metadata statistics tasks in the Hive service. Based on the target data table file type, use a preset row count algorithm to obtain the number of rows in the target data table. Optionally, if the target data table file type is a columnar storage file such as ORC or Parquet, read the file footer of the target data table to obtain the row count of all files under the target data table, and use this as the number of rows in the target data table. If the target data table file type is a non-columnar storage file, do not perform metadata statistics, and end the metadata statistics task. Since in big data scenarios, most data is stored using Parquet or ORC files with high query performance, and other formats are not commonly used due to performance issues, metadata information can be omitted. Optionally, a preset interface can be called to count the size of all files as the file size of the target data table.

[0098] B2: Based on the number of rows, retrieve the statistical data to be collected from the target data table.

[0099] For the column information of the target data table, based on the number of rows, the data of each row of the target data table is obtained to obtain the statistical data to be collected.

[0100] Optionally, determine whether the row count (rowCount) is greater than the preset row count. Considering the performance of the statistical task, if the row count (rowCount) is greater than the preset row count (e.g., 500,000 rows), use the preset sampling statistical method to perform a metadata statistical task on the target data table to obtain metadata information; otherwise, use the full statistical method to perform a metadata statistical task on the target data table to obtain metadata information.

[0101] Optionally, in response to rowCount exceeding 500,000 rows, the sampling parameter isSample is configured to true to perform metadata statistics using a preset sampling method; otherwise, the sampling parameter isSample is configured to false to perform metadata statistics using a full-volume statistical method.

[0102] Optionally, in the case of full statistics, data from each row of the target data table can be obtained to obtain the statistical data to be collected.

[0103] Optionally, under a preset sampling statistical method, a sampling ratio 'a' is obtained, where the ratio of a preset quantity threshold to the number of rows can be obtained. For example, the preset quantity threshold can be a preset number of rows, and the sampling ratio 'a' = 5000000 / rowCount. Then, the sampling ratio is used to sample each row of the target data table to obtain the statistical data to be collected. For example, a sampling operator (such as the sample(a) operator) can be generated using the sampling ratio 'a'. The sampling operator can be the Bernoulli random function 'sample', which outputs 0 or 1, where the number of 1:0 pairs is approximately equal to 'a'. The sampling operator is then used to sample the target data table to obtain the statistical data to be collected.

[0104] Optionally, a column data reader is generated to read a column of data from the target data table according to the column information described above. If the sample function returns 1 under the preset sampling statistical method, it indicates that the column information is selected as the data to be collected, and the following steps are continued. Otherwise, this step is repeated to select the data to be collected. Under the preset sampling statistical method, the data corresponding to each column of the target data table can be used as the data to be collected, and the following steps are executed.

[0105] B3: Use a preset statistical algorithm to perform statistics on the data to be analyzed and obtain metadata information.

[0106] For the data to be collected in each column, a preset statistical algorithm is used to perform statistical analysis on the data to obtain metadata information.

[0107] Optionally, for each column of data read in the above steps, determine whether the value of each row in the column data is a null value. If it is a null value, increment the null_count to obtain the number of null values ​​in that column. Alternatively, if it is not a null value, perform either of the following steps: Optionally, further determine whether the value is greater than the maximum value (max) or less than the minimum value (min). If the value is greater than the maximum value (max), change the maximum value to that value; or if the value is less than the minimum value (min), change the minimum value to that value. If they are the same, do not modify the maximum value (max) or the minimum value (min). This is how the maximum value (max) or minimum value (min) of the column is obtained. Optionally, add the length of the data for that value to the total length of the column (sum_length) to obtain the length of the column. Optionally, input this value into the streaming algorithm HLL (HyperLogLog) for processing to obtain the distinct count. Optionally, increment the count of rows in the target data table (sum_analyze_count) to obtain the number of rows. Repeat the above steps until all the data in each column has been read, and finally obtain the metadata information.

[0108] B4: In response to the need to generate a histogram, the first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table, and the second preset algorithm is used to obtain the unique values ​​of two adjacent quantiles, and a histogram is generated based on the unique values.

[0109] If a histogram needs to be calculated, the information of each column for which the histogram needs to be calculated is input into a first preset algorithm for streaming quantile calculation. The first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table. For example, the first preset algorithm can be the KLL (Karnin-Liberty-Lin, an approximate quantile calculation method). Using the KLL algorithm, the required quantiles for the histogram of that column are calculated based on the data from the KLL algorithm. The quantiles are determined by 1 / bins (the number of buckets in the histogram), and the method for obtaining quantiles is the same as in the above embodiment, so it will not be repeated here. After processing one column of data from the target data table using the above method and obtaining the corresponding quantiles, repeat the above steps to generate a column data reader again. This reader then reads one column of data from the next column. Under the preset sampling statistical method, if the `sample` function returns 1, it indicates that the column is selected as the data to be analyzed, and the above steps are executed. Otherwise, the step is repeated to select the data to be analyzed, thus continuing with the following steps.

[0110] The second preset algorithm is used to obtain the unique value of two adjacent quantiles. For example, the second preset algorithm is the HLL algorithm (HyperLogLog, the logarithmic algorithm of superlogarithms), which obtains the unique value (distinct number) of two adjacent quantiles, thus obtaining the distinct number.

[0111] Then, a histogram is generated based on the unique value (distinct number). For example, two adjacent quantiles are used as the upper and lower bounds of a bucket in the histogram, and the distinct number between adjacent quantiles is used as the distinct number of the bucket. A total of bins are generated as histogram information, which is used to generate the histogram.

[0112] For example, the HLL algorithm can also be used to calculate the number of distinct values ​​(distinct count) across the entire target data table, denoted as distinct_num. The above calculation of distinct values ​​and histograms utilizes streaming algorithms HLL and KLL, significantly saving memory and time.

[0113] B5: Write the metadata information obtained above to the metadata storage Hive Metastore.

[0114] The metadata information obtained in the above steps can be written to the Hive Metastore. The values ​​max, min, sum_length / sum_analyze_count, null_count, distinct_num, and histogram are used as the maximum, minimum, average length, number of null values, number of distinct values, and histogram information for this column. The rowCount and totalsize are used as the number of rows in the table and the file size, respectively, and written to the Hive Metastore. This completes the metadata statistics tasks described above.

[0115] In some embodiments, there are multiple second threads T2 (such as second thread T21, second thread T22, etc.). Multiple second threads T2 can be used to execute multiple metadata statistics tasks in parallel, obtain the corresponding metadata information respectively, and then store the metadata information in Hive Metasotre respectively.

[0116] The above solution can dynamically determine whether to submit to SparkSQL service or use Hive service to collect metadata based on whether the current environment information meets the preset resource conditions, making metadata statistics more flexible and improving the efficiency of metadata information collection.

[0117] The above-mentioned application provides a technical solution for automatically identifying columns to be statistically analyzed, automatically triggering the generation and execution of statistical tasks, so as to improve the comprehensiveness, efficiency and timeliness of metadata statistics, thereby improving the execution efficiency of big data execution engines such as Spark SQL and Hive.

[0118] It is understood that in the above method of specific implementation, the order in which each step is written does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0119] In some embodiments, this application also provides a metadata information acquisition device for implementing the metadata information acquisition method of any of the above embodiments.

[0120] Please see Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the metadata information acquisition device of this application. The metadata information acquisition device 20 includes a statement parsing module 21, a task generation module 22, and a task execution module 23. The modules are interconnected.

[0121] The statement parsing module 21 is used to parse the historical execution statements to obtain the statistical information to be obtained. The historical execution statements include the execution statements on the original data table, and the statistical information to be obtained includes at least the column information of the target data table.

[0122] The task generation module 22 is used to aggregate the statistical information to generate metadata statistical tasks.

[0123] The task execution module 23 is used to perform metadata statistics tasks and obtain metadata information.

[0124] It should be noted that the metadata information acquisition device provided in the above embodiments and the metadata information acquisition method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the metadata information acquisition device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This application does not impose any limitations on this.

[0125] It is understood that the method for obtaining metadata information in this application can be executed by a computer device, which can be any device with processing capabilities, such as a mobile device, computer, server, etc., and this application does not impose any restrictions on it. In some possible implementations, the method for obtaining metadata information can be implemented by the processor calling program data stored in memory.

[0126] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. The computer device 30 includes a memory 31 and a processor 32 coupled to each other. The memory 31 stores program data, and the processor 32 executes the program data to implement the steps of any embodiment of the above-described method for obtaining metadata information. In a specific implementation scenario, the computer device 30 may include, but is not limited to, a microcomputer or a server. In addition, the computer device 30 may also include mobile devices such as laptops and tablets, which are not limited here.

[0127] In this embodiment, processor 32 can also be referred to as a CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor. Furthermore, processor 32 can be implemented using integrated circuit chips.

[0128] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a computer-readable storage medium. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 41 that can be executed by a processor. The program data 41 can be executed by the processor to implement the steps of any embodiment of the method for obtaining metadata information described above.

[0129] The computer-readable storage medium 40 in this embodiment includes: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program data 41. Alternatively, it can be a server storing the program data 41. The server can send the stored program data 41 to other devices for execution, or it can run the stored program data 41 itself.

[0130] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments. For the sake of brevity, this application will not repeat the details here.

[0131] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to. For the sake of brevity, the present application will not repeat them here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0136] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, and thus stored in a computer-readable storage medium for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this application is not limited to any particular hardware and software combination.

[0137] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for obtaining metadata information, characterized in that, include: The historical execution statements are parsed to obtain the statistical information to be obtained. The historical execution statements include the execution statements on the original data table, and the statistical information to be obtained includes at least the column information of the target data table. The information to be statistically analyzed is aggregated to generate a metadata statistical task; Perform the metadata statistics task to obtain metadata information.

2. The method according to claim 1, characterized in that, The parsing of historical executed statements yields statistical information, including: Parse the historical executed statements to determine the preset clauses; Based on the column information corresponding to the preset clause, the information to be statistically analyzed is obtained.

3. The method according to claim 2, characterized in that, The preset clauses include at least one of the following: filter clauses, join clauses, query clauses, and grouping clauses; And / or, the process of parsing historically executed statements to determine preset clauses; Based on the column information corresponding to the preset clause, the information to be statistically analyzed is obtained, including: The historical execution statements are parsed hierarchically to determine the preset clauses for each layer; Based on the column information corresponding to the preset clauses of each layer, the statistical information to be obtained for each layer is obtained; By combining the information to be statistically analyzed from each layer, the final information to be statistically analyzed is obtained. And / or, after obtaining the information to be statistically analyzed based on the column information corresponding to the preset clause, the process includes: The information to be statistically analyzed is deduplicated to obtain the deduplicated information to be statistically analyzed. And / or, the information to be statistically analyzed may also include whether the column information of the target data table needs to be statistically analyzed based on preset metadata; it may also include: Based on the preset clause to which the column information belongs, determine whether the column information needs to be statistically analyzed using preset metadata.

4. The method according to claim 1, characterized in that, The information to be statistically analyzed also includes the execution time corresponding to the column information of the target data table; the aggregation processing of the information to be statistically analyzed to generate metadata statistics tasks includes: By using the execution time in the information to be statistically analyzed, the information to be statistically analyzed is aggregated to obtain aggregated statistical information. Based on the aggregated statistical information, a metadata statistical task is generated.

5. The method according to claim 4, characterized in that, The process of aggregating the execution times from the information to be statistically analyzed to obtain aggregated statistical information includes: Using the execution time in the information to be statistically analyzed, the information to be statistically analyzed in at least two preset execution ranges is filtered to obtain the filtered statistical information in each preset execution range. The various filtering statistics are aggregated to obtain aggregated statistics corresponding to each preset execution range; wherein, the aggregated statistics include at least one of the following: the number of times the filtering statistics appear in the preset execution range, the number of times preset metadata needs to be statistically analyzed, and the priority, wherein the priority corresponds to the preset execution range; And / or, the task of generating metadata statistics based on the aggregated statistical information includes: According to the preset sorting method, metadata statistics tasks are generated for each column of information in the aggregated statistics.

6. The method according to claim 1, characterized in that, The execution of the metadata statistics task to obtain metadata information includes: Based on the metadata statistics task, the target data table is retrieved from the preset storage; If the current environment information meets the preset resource conditions, the metadata statistics task is submitted to the first service so that the first service can be used to perform the metadata statistics task on the target data table to obtain the metadata information; otherwise, the second service can be used to perform the metadata statistics task on the target data table to obtain the metadata information.

7. The method according to claim 6, characterized in that, The step of retrieving the target data table from a preset storage based on the metadata statistics task further includes: Based on the metadata statistics task, relevant information of the target data table is obtained from the preset storage; wherein, the relevant information includes at least one of the following: the storage location, type information, last statistics time, and last update time of the target data table; In response to the target data table meeting the preset update conditions, it is determined whether the current service meets the preset resource conditions; wherein, the preset update conditions include: the last statistical time is not greater than the last update time; And / or, the execution of the metadata statistics task on the target data table to obtain the metadata information includes: Obtain the number of rows in the target data table; In response to the number of rows being greater than the preset number of rows, a preset sampling statistical method is used to perform the metadata statistical task on the target data table to obtain the metadata information.

8. The method according to claim 6, characterized in that, The default metadata is a histogram; The step of performing the metadata statistics task on the target data table to obtain the metadata information further includes: In response to the need to generate a histogram, a first preset algorithm is used to obtain the quantiles corresponding to the columns in the target data table, and a second preset algorithm is used to obtain the unique values ​​of two adjacent quantiles, and a histogram is generated based on the unique values.

9. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, wherein the memory stores program data and the processor executes the program data to implement the method for obtaining metadata information as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The system stores program data that can be executed by a processor, the program data being used to implement the method for obtaining metadata information as described in any one of claims 1 to 8.

Images