Method, device and equipment for limiting access to metadata server

Abstract

The present application discloses a method, device and equipment for limiting access to a metadata server. The method is applied to a data lake metadata server, and receives a metadata access request sent by a client. The request is used to obtain metadata stored by a storage server for describing target data, and the target data is the data in a target table. The metadata service interface used by the data lake metadata server is determined according to the metadata access request. When the metadata service interface meets the preset conditions, the number of application tokens corresponding to the metadata access request is determined according to the number of table partitions and the number of partition levels of the obtained target table. That is, the present application adaptively determines the number of application tokens corresponding to the metadata access request through the metadata service interface and the request parameters of the metadata access request. Furthermore, judging whether to limit the metadata access request based on the number of application tokens can improve the service stability of the data lake metadata server and reduce the load pressure on the storage server.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for limiting access to a metadata server. Background Art

[0002] With the development of Internet technology, the amount of data stored in big data is increasing. In order to extract more value from massive data, data analysis tasks are becoming increasingly complex, posing a challenge to the stability of big data services.

[0003] The Data Lake Metadata Service is a widely used online big data service in big data scenarios, such as the Hive Metadata Service. The Data Lake Metadata Service can be used to obtain metadata. Complex data analysis tasks often require multiple access requests to the Data Lake Metadata Service. This places a significant load on metadata storage.

[0004] Based on this, there is an urgent need for an access flow limiting method to effectively restrict some accesses in order to improve the stability of the data lake metadata service and reduce the load pressure of metadata storage. Summary of the Invention

[0005] In view of this, the present application provides a method, apparatus, and device for limiting access to a metadata server, which are used to limit the access requests to metadata from a data lake metadata server that provides metadata services.

[0006] To solve the above problems, the technical solutions provided by this application are as follows:

[0007] In a first aspect, the present application provides a method for limiting access to a metadata server, which is applied to a data lake metadata server and includes:

[0008] Receive a metadata access request sent by a client, wherein the metadata access request is used to obtain metadata stored in a storage server for describing target data; the target data is data in a target table;

[0009] Determining a metadata service interface used by the data lake metadata server according to the metadata access request;

[0010] When the metadata service interface meets the preset conditions, obtaining the request parameters of the metadata access request; the request parameters include the number of table partitions and the number of partition levels of the target table;

[0011] Obtaining the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels;

[0012] When the number of applied tokens is greater than the number of remaining tokens corresponding to the client, the metadata access request is subjected to flow limiting processing.

[0013] In a second aspect, the present application provides an access flow limiting device for a metadata server, which is applied to a data lake metadata server and includes:

[0014] A receiving unit, configured to receive a metadata access request sent by a client, wherein the metadata access request is used to obtain metadata stored in a storage server and used to describe target data; the target data is data in a target table;

[0015] A first determining unit is configured to determine a metadata service interface used by the data lake metadata server according to the metadata access request;

[0016] A first acquiring unit is configured to acquire request parameters of the metadata access request when the metadata service interface satisfies a preset condition; the request parameters include the number of table partitions and the number of partition levels of the target table;

[0017] A second acquiring unit is configured to acquire the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels;

[0018] The processing unit is configured to perform flow limiting processing on the metadata access request when the number of applied tokens is greater than the number of remaining tokens corresponding to the client.

[0019] In a third aspect, the present application provides an electronic device, comprising:

[0020] one or more processors;

[0021] a storage device having one or more programs stored thereon,

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the metadata server access limiting methods.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned methods for limiting access to a metadata server.

[0024] It can be seen that this application has the following beneficial effects:

[0025] The present application provides a method, device and equipment for limiting access to a metadata server, which is applied to a data lake metadata server that provides metadata services. The data lake metadata server receives a metadata access request sent by a client, which is used to obtain metadata stored by a storage server for describing target data, and the target data is the data in a target table. After receiving the access request, the flow of limiting the flow of the metadata access request is started. Specifically, the metadata service interface used by the data lake metadata server is first determined based on the obtained metadata access request. When the metadata service interface meets the preset conditions, the request parameters of the metadata access request are obtained, that is, the number of table partitions and the number of partition levels of the target table. The number of table partitions and the number of partition levels of the target table are used to implement the flow limiting processing judgment of the metadata access request. That is, the number of application tokens corresponding to the metadata access request is first determined based on the number of table partitions and the number of partition levels of the target table. When the number of application tokens corresponding to the metadata access request is greater than the number of remaining tokens corresponding to the client, the metadata access request is subjected to flow limiting processing.

[0026] It can be seen that the request parameters corresponding to different tables in the metadata service interface that meets the preset conditions, or the same table at different times, may all be different, and the corresponding number of application tokens will be different. Therefore, this application can adaptively determine the number of application tokens corresponding to the metadata access request through the metadata service interface and the request parameters of the metadata access request, and determine whether to limit the metadata access request based on the number of application tokens. This can improve the service stability of the data lake metadata server and reduce the load pressure on the storage server. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of an exemplary application scenario provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of a method for limiting access to a metadata server provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of another exemplary application scenario provided in an embodiment of the present application;

[0031] Figure 5 A flowchart of another method for limiting access to a metadata server provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of an access flow limiting device for a metadata server provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] In order to facilitate understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0036] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0037] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0038] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] With the development of internet technology, the amount of data stored in big data lakes is growing. To extract as much valuable data as possible from this massive data volume, data analysis tasks are becoming increasingly complex, posing challenges to the stability of big data services. Data lake metadata services, such as the Hive metadata service, are widely used online big data services in big data scenarios. Metadata, also known as intermediate data or relay data, is data that describes the data. Data analysis tasks often involve multiple accesses to multiple databases, tables, and partitions. This means that a single data analysis task may initiate multiple accesses to the Data Lake Metadata Service. Complex data analysis tasks can lead to poor stability of the Data Lake Metadata Service and increased load on metadata storage. For example, storing metadata in a MySQL database can place significant load on the database.

[0041] Currently, the Data Lake Metadata Service can be throttled based on the number of requests per second (QPS). Specifically, when the metadata service's QPS exceeds the service system's upper limit, access requests are throttled. However, different access requests place varying loads on the metadata storage, and throttling access requests based on QPS can be ineffective. For example, if the Data Lake Metadata Service receives 100 access requests per second (possibly initiated by different clients), the QPS is 100. When the number of access requests exceeds 100, throttling is performed, rejecting subsequent access requests. However, different access requests place varying loads on the service. For example, if the load on the metadata storage is 1 for access request A and 100 for access request B, then accepting access request A 100 times would reach the Data Lake Metadata Service's access limit, while accepting access request B once would reach the Data Lake Metadata Service's access limit. If access request a is still accepted to access the Data Lake Metadata Service 100 times, the Data Lake Metadata Service may crash and the load pressure on the metadata storage will be extremely high.

[0042] Therefore, there is an urgent need for a metadata server access flow control method to effectively limit some access and improve service stability. Based on this, an embodiment of the present application provides a metadata server access flow control method, device, and equipment, which are applied to a data lake metadata server that provides metadata services. The data lake metadata server receives a metadata access request sent by a client, which is used to obtain metadata stored by a storage server that describes target data, where the target data is the data in a target table. After receiving the access request, the flow control processing flow for the metadata access request is initiated. Specifically, the metadata service interface used by the data lake metadata server is first determined based on the obtained metadata access request. When the metadata service interface meets the preset conditions, the request parameters of the metadata access request are obtained, namely, the number of table partitions and the number of partition levels of the target table. The number of table partitions and the number of partition levels of the target table are used to implement the flow control processing judgment of the metadata access request. That is, the number of application tokens corresponding to the metadata access request is first determined based on the number of table partitions and the number of partition levels of the target table. When the number of application tokens corresponding to the metadata access request is greater than the number of remaining tokens corresponding to the client, the metadata access request is flow controlled.

[0043] It can be seen that the request parameters corresponding to different tables in the metadata service interface that meets the preset conditions, or the same table at different times, may all be different, and the corresponding number of application tokens will be different. Therefore, this application can adaptively determine the number of application tokens corresponding to the metadata access request through the metadata service interface and the request parameters of the metadata access request, and determine whether to limit the metadata access request based on the number of application tokens. This can improve the service stability of the data lake metadata server and reduce the load pressure on the storage server.

[0044] It is understandable that the defects in the above solutions are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application below should be the applicant's contribution to the embodiments of the present application during the application process.

[0045] In order to facilitate understanding of the access flow limiting method of the metadata server provided in the embodiment of this application, the application scenario of the data lake metadata service and the metadata acquisition process are first introduced below. Figure 1 , Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application is provided. Figure 1 This article takes the Hive metadata service and metadata storage in the MySQL database as an example.

[0046] The Hive library is a data warehouse infrastructure built on Hadoop. It maps structured data files to database tables (or data tables). The data in the Hive database tables is stored in the Hadoop Distributed File System (HDFS). The metadata for the Hive database tables is stored in a MySQL database, an application running on a storage server. This metadata is data describing the database tables, such as the HDFS storage path for the table data, the Hive library name, the database table name, and the creation time of the database table.

[0047] In practice, to access data in database tables stored in HDFS, you must first obtain the corresponding metadata. Metadata for database tables in the Hive database can be obtained using the Hive Metadata Service. The Hive Metadata Service (HiveMetastore) is provided by the metadata server, which is an application running on the metadata server.

[0048] like Figure 1 As shown, the client is installed with SQL engines (such as Presto), SparkJar batch and stream processing engines, FlinkJar batch and stream processing engines, and a business intelligence (BI) analysis platform. Taking the Presto engine as an example, the front-end user of the client enters a query statement, such as an SQL statement, in the client. This query statement can be considered part of a data analysis task. For example, the query statement is select target data from table1. Table1 is a database table in the Hive database.

[0049] The Presto engine parses the query statement and retrieves the metadata corresponding to the target data in the query statement. Metadata describes the target data in the query statement, for example, including the target data's storage path in HDFS. Specifically, the Hive metadata service is called to retrieve the metadata corresponding to the target data. In practice, the Hive metadata service interface is called to access the MySQL database and retrieve the metadata corresponding to the target data from the MySQL database. After obtaining the metadata corresponding to the target data, the Presto engine determines the target data's storage path based on the metadata. It then reads the target data from HDFS based on this storage path and returns the read target data to the client frontend.

[0050] In practical applications, such as Figure 1As shown, a data analysis task on a client may initiate multiple accesses to the Hive Metadata Service. Each access corresponds to a Hive Metadata Service. Consequently, a data analysis task may be associated with multiple different Hive Metadata Services (i.e., multiple Hive Metadata Service instances), accessing the same MySQL database. Alternatively, multiple clients may initiate accesses to different Hive Metadata Services, resulting in multiple Hive Metadata Services accessing the same MySQL database. Each Hive Metadata Service can run on a different metadata server. In this case, the access pressure on the same MySQL database comes from each Hive Metadata Service.

[0051] like Figure 1 As shown, there is only one MySQL database. In actual applications, in order to alleviate the access pressure of the database, multiple MySQL databases can be deployed, and the storage servers where the multiple MySQL databases are located can be the same or multiple.

[0052] Understandably, complex data analysis tasks can lead to higher service load pressure. Service load pressure primarily refers to the load on the server side. Both the Hive Metadata Service and the MySQL database are server-side components. Service stability primarily depends on the storage layer, such as the MySQL database. The greater the number of accesses to the Hive Metadata Service, the greater the load on the MySQL database, impacting the stability of the Hive Metadata Service.

[0053] Based on the above content, the following Figure 2 The example scenario shown illustrates the access flow limiting method of the metadata server provided in the embodiment of the present application. Figure 2 As shown in FIG, this figure is a schematic diagram of the framework of an exemplary application scenario provided by an embodiment of the present application. The method is applied to a data lake metadata server running a metadata service.

[0054] The data lake metadata server receives metadata access requests from clients. These requests are used to obtain metadata describing target data stored on the storage server. The target data is the table data in the target table. The target table might be a data table in a Hive database. If the target data is stored in HDFS, the metadata service might be the Hive metadata service. The storage server might run a MySQL database.

[0055] After receiving an access request, the Data Lake metadata server initiates a rate limiting check (also known as a preCost rate limiting check) on the metadata access request. Specifically, it first determines whether the metadata service interface meets the preset conditions. If so, it uses subsequent rate limiting steps to determine whether the metadata access request should be rate limited. The subsequent rate limiting steps specifically obtain the metadata access request parameters, namely the number of table partitions (denoted as part_num) and the number of partition levels (denoted as part_level) of the target table in the metadata access request. Based on the number of table partitions and the number of partition levels, it determines the requested token count (denoted as preCost) corresponding to the metadata access request and requests the corresponding number of tokens from the token server. The token server's token bucket stores the maximum number of tokens (denoted as Quota) for each client. If the requested token count for the metadata access request exceeds the client's remaining token count, the metadata access request is rejected and rate limiting is performed. The front-end then displays a message indicating that the access request has exited abnormally. When a client first sends a metadata access request, the client's remaining token count equals the client's maximum token count.

[0056] It is understandable that if the current limiting check passes, normal logic processing is followed to obtain metadata corresponding to the target data, and the metadata is used to obtain the target data.

[0057] Those skilled in the art will understand that Figure 2 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0058] To facilitate understanding of the present application, an access flow limiting method for a metadata server provided in an embodiment of the present application is described below with reference to the accompanying drawings.

[0059] See also Figure 3 As shown, this figure is a flow chart of a metadata server access flow limiting method provided by an embodiment of the present application, which can be applied to a data lake metadata server running a data lake metadata service. Figure 3 As shown, the method may include S301-S305:

[0060] S301: Receive a metadata access request sent by a client, where the metadata access request is used to obtain metadata stored in a storage server for describing target data; the target data is data in a target table.

[0061] Users can construct data analysis tasks on the client front end. During these data analysis tasks, they may need to retrieve target data from a target table. This target data is specifically obtained through a user-constructed query statement. A query statement is first constructed, and the client engine parses it to determine the metadata corresponding to the target data. As an optional example, metadata corresponding to the target data can be obtained by running a data lake metadata server (referred to as the metadata service in this embodiment of the application). Therefore, the client sends a metadata access request to the data lake metadata server.

[0062] The data lake metadata server receives metadata access requests from clients. This metadata access request is used to obtain metadata describing the target data. The metadata describing the target data is stored on the storage server. As an optional example, if the storage server runs a target database, such as a SQL database, metadata describing the target data needs to be obtained by accessing the target database. If the target database is a SQL database, the query statements described in the above embodiments are specifically SQL query statements.

[0063] S302: Determine the metadata service interface used by the data lake metadata server according to the metadata access request.

[0064] The metadata access request corresponds to a metadata service interface. Based on the metadata access request, the metadata service interface corresponding to the metadata access request used by the data lake metadata server can be determined.

[0065] It's understandable that the Data Lake Metadata Server has multiple metadata service interfaces, each with different access frequencies. Different metadata service interfaces represent different access types. Different service interfaces and different access types exert varying levels of access pressure on the Data Lake Metadata Server and storage servers.

[0066] As an alternative example, the metadata service interface could be get_partitions_by_filter. This interface is one of the most frequently used interfaces on the Data Lake Metadata Server and exerts the greatest pressure on metadata storage load. It is designed to filter all partitions under a specified table that satisfy a query statement. This interface is complex. Another alternative example is get_database, a simple interface. Simple and complex interfaces exert different pressures on the Data Lake Metadata Server and storage server. Therefore, different service interfaces have different rules for obtaining metadata and corresponding throttling rules.

[0067] S303: When the metadata service interface meets the preset conditions, obtain the request parameters of the metadata access request; the request parameters include the number of table partitions and the number of partition levels of the target table.

[0068] It's understandable that even when calling the same metadata service interface, different request parameters in the metadata access request can result in different load pressures on the metadata storage. Request parameters include the number of table partitions and the number of partition levels in the target table. It's understandable that a larger number of table partitions and a larger number of partition levels in the request parameters may result in a greater load pressure on the metadata storage.

[0069] Therefore, in this step, we first determine whether the metadata service interface meets the preset conditions. If so, this indicates that the metadata service interface access type involves table partitions and table partition-level access. At this point, we obtain the request parameters for the metadata access request. Based on these request parameters, we determine the metadata storage load pressure and determine whether to throttle the metadata access request.

[0070] The preset conditions may be a pre-given list of metadata service interfaces, all of which involve access to table partitions and the table partition hierarchy. For example, a metadata service interface that meets the preset conditions may be the aforementioned get_partitions_by_filter interface or the drop_table interface. The drop_table interface is used to delete a data table. It is understood that the load pressure on metadata storage varies depending on the capacity of the deleted data table (e.g., the larger the number of table partitions, the larger the capacity).

[0071] Table partitioning levels are used to represent the dimensions of data stored in a data table. For example, a data table (such as a data table in a Hive library) can be partitioned by time. For example, if partitioning is performed by "date," such as January 1st, January 2nd, and so on, then the partition corresponding to January 1st will only store data from January 1st. Furthermore, the data table can also be partitioned by hour, such as 10:00 to 11:00, 11:00 to 12:00, and so on, then the partition corresponding to 10:00 to 11:00 will only store data between 10:00 and 11:00. If partitioning is performed only by "date," the table partitioning level includes the "date" level, and the number of partition levels is 1. If the data table is also partitioned by hour, the table partitioning level includes both the "date" level and the "hour" level, and the number of partition levels is 2. It is understood that each partition corresponds to a partition data storage path, which indicates the storage address of the data in the partition. For example, when all the data in a data table is stored in HDFS, the storage path of the above-mentioned data in the table in HDFS is the partition data storage path of the partition where the data in the table is located.

[0072] The number of table partitions is the number of partitions used by the data stored in the data table. When the table partitioning hierarchy is multi-level, the number of table partitions is simply the number of partitions at the finest level. For example, if the table partitioning hierarchy includes "date" and "hour" levels, the number of table partitions is the number of "hour"-level partitions used to store the data in the data table.

[0073] Based on the above, the metadata access request parameters in this step specifically include the number of table partitions and the number of partition levels of the target table. The target table's table partition level represents the dimensions of the data stored in the target table. The number of table partitions is the number of partitions occupied by the data within the target table. The number of table partitions can be represented by part_num, and the number of partition levels can be represented by part_level.

[0074] It's understood that the target database stores metadata for each partition of a data table, as well as the corresponding partition data storage path. In actual applications, the target database on the storage server is scanned to obtain the number of partitions in the target table. However, some large tables have many partitions, so scanning the target database in real time to obtain the partition count is time-consuming and increases the load on the target database.

[0075] Based on this, in one possible implementation, the present application provides a specific implementation method for obtaining request parameters for a metadata access request, including:

[0076] Based on the target table name, obtain the number of table partitions and partition levels of the target table from the memory of the data lake metadata server;

[0077] Among them, the number of table partitions of the target table is stored in the storage server. When the data lake metadata server is started, the number of table partitions of the target table is loaded into the memory of the data lake metadata server; the number of partition levels of the target table is loaded into the memory of the data lake metadata server according to the preset period.

[0078] It's understandable that to reduce the load on the target database and accelerate the process of obtaining the number of table partitions, the number of table partitions for a data table is stored in memory. Specifically, the Data Lake Metadata Server's memory stores the correspondence between the table name of the data table and the number of table partitions for the target table. Based on the target table name, the number of table partitions for the target table can be obtained from the Data Lake Metadata Server's memory. In actual applications, the Data Lake Metadata Server records the number of table partitions for each data table (for example, each data table in a Hive database). When the number of partitions in a data table changes (for example, when it is created or deleted), the recorded number of table partitions is updated accordingly. When the Data Lake Metadata Server starts, the recorded number of table partitions for the data table is loaded into the Data Lake Metadata Server's memory so that the number of table partitions can be obtained from memory. The target table is similar and will not be further described here.

[0079] In addition, memory can also be used to store the number of partition levels of the data table. It is understandable that the memory for storing the number of table partitions and the number of partition levels can be the same or different. Specifically, the memory stores the correspondence between the table name of the data table and the number of partition levels of the target table. Then, the number of partition levels of the target table can be obtained from the memory of the data lake metadata server according to the table name of the target table. In actual applications, the number of partition levels of each data table will be periodically loaded into the memory. For example, loading is performed according to a preset cycle. In actual applications, the data lake metadata server can first construct a partition level query instruction and send the partition level query instruction to the target database. The target database stores the table partition level of the data table, and the number of partition levels of the data table can be calculated based on the partition level query instruction.

[0080] Based on the above content, the number of table partitions and partition levels of the target table can be obtained from the memory of the data lake metadata server, which can speed up the calculation of request parameters and subsequent process processing, enabling the metadata service to achieve low latency and high availability.

[0081] S304: Obtain the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels.

[0082] It should be understood that even if the same metadata service interface is called, if the data to be retrieved in the metadata access request belongs to different data tables, or the same data table at different times, the number of application tokens obtained will be different due to the different number of table partitions part_num and the number of partition levels part_level of the data table. Even for the same data table, as business develops, the partitions in the data table may change, causing the number of table partitions part_num to change as well, resulting in a different number of application tokens corresponding to the metadata access request. Therefore, the embodiment of the present application determines the number of application tokens corresponding to the metadata access request based on the request parameters in the actual metadata access request (i.e., the number of table partitions and the number of partition levels of the target table).

[0083] Taking the get_partitions_by_filter interface as an example, a client user might send the query "get_partitions_by_filter(s:"olap_dev", s1:"hms_monitor", s2:"p_date>=2"" in the client. Here, olap_dev is the library name (e.g., a Hive library), hms_monitor is the data table name (e.g., the target table name), and p_date>=2 indicates data with a date greater than or equal to 2 (e.g., the target data is data in hms_monitor with a date greater than or equal to 2), representing the data to be retrieved. Because the get_partitions_by_filter interface is used to filter all partitions in the specified library or table that satisfy the query, the query results from this query include metadata for all partitions in the hms_monitor table with a date greater than or equal to 2. It should be understood that the get_partitions_by_filter interface is an interface of the Hive metadata service. In this example, after the user sends the query, the engine in the client parses the query, and the Hive metadata service processes the metadata access request, retrieving metadata for the data that satisfies the query. The final result may be two table partitions that satisfy the query statement (for example, table partition 1 and table partition 2). The data in table partitions 1 and 2 will contain data with a date greater than or equal to 2. The query result includes metadata such as the partition data storage path, partition creation time, partition creator, partition name, and table name for the two partitions. The client engine retrieves this metadata and then reads the table data for table partitions 1 and 2 from HDFS based on their partition data storage paths, obtaining the table data with a date greater than or equal to 2.

[0084] As can be seen from the get_partitions_by_filter API example above, as the number of table partitions and partition levels in the target table increases, the process of obtaining partitions that satisfy the query statement from each table partition becomes more complex and the processing cost increases, resulting in a larger number of request tokens required. The number of request tokens is used to represent the processing cost of metadata access requests.

[0085] Based on this, in one possible implementation, the present application embodiment provides a specific implementation method for obtaining the number of application tokens corresponding to a metadata access request based on the number of table partitions and the number of partition levels, including:

[0086] Calculate the logarithm of the number of table partitions and obtain the first value;

[0087] Calculate the product of the first constant, the number of partition levels, and the first value to obtain a second value;

[0088] Calculate the sum of the second value and the second constant to obtain the third value;

[0089] The product of the third value and the third constant is calculated as the number of application tokens for the metadata access request; wherein the first constant, the second constant, and the third constant are related to the configuration of the storage server.

[0090] Based on this specific implementation, when the number of table partitions is represented by part_num, the number of partition levels is represented by part_level, the first constant, the second constant, and the third constant are represented by x, y, and z respectively, and the number of application tokens is represented by pre_cost, the calculation formula for the number of application tokens can be expressed as: pre_cost=(x*part_level*log(part_num)+y)*z.

[0091] The first value calculated is log(part_num), the second value is x*part_level*log(part_num), and the third value is (x*part_level*log(part_num)+y). Log(part_num) is the logarithm of part_num with base 10.

[0092] As an optional example, the metadata service interface that meets the preset conditions may be the get_partitions_by_filter interface. The get_partitions_by_filter interface is one of the most frequently used interfaces online and has the largest change in metadata storage load pressure. Stress testing can be performed through the get_partitions_by_filter interface, and concurrent requests can be initiated to perform stress testing on data tables with different partition levels and different numbers of partitions online to obtain stress testing data. The calculation formula for the number of application tokens corresponding to the metadata access request is fitted based on the stress testing data, mainly to obtain the fitted first constant, second constant and third constant. Among them, the stress testing data includes the number of table partitions, the number of partition levels and the number of application tokens corresponding to the metadata access request of each data table.

[0093] It is understandable that the calculation formula for the number of application tokens corresponding to the above metadata access request can be applied to any metadata service interface that involves the number of table partitions, the table partition level and meets the preset conditions.

[0094] It can also be understood that the first constant x, the second constant y, and the third constant z can be collectively referred to as formula parameters, and the formula parameters are related to the configuration of the storage server. If the configuration of the storage server is different, the formula parameters fitted according to the stress test data will be different. For example, if the storage server uses a 20-core virtual machine, the calculation formula for the number of application tokens obtained can be specifically pre_cost=0.5227*part_level*log(part_num)+1.3747, where the first constant x is 0.5227, the second constant y is 0, and the third constant z is 1.3747. If the storage server adopts other configurations, the formula parameters will change accordingly.

[0095] S305: When the number of applied tokens is greater than the number of remaining tokens corresponding to the client, the metadata access request is subjected to flow control.

[0096] In practice, the token server's token bucket stores tokens corresponding to each client, and the number of tokens allocated to each client may vary. Before a client requests a token, the maximum number of tokens stored in the token server is the client's. When a client requests a token from the token server, if the request is approved, the corresponding number of tokens is removed from the token bucket. The remaining tokens in the token bucket are used by the client for subsequent requests.

[0097] After obtaining the number of applied tokens (pre_cost) corresponding to the metadata access request, it is possible to determine whether to trigger the current limiting process based on pre_cost before the request is executed. Specifically, apply for the required tokens from the token server, and send the number of applied tokens corresponding to the metadata access request to the token server. A token bucket is provided in the token server, and tokens are stored in the token bucket. As an optional example, the token server may be a Redis server. The script logic set by the token server compares the number of applied tokens with the number of remaining tokens corresponding to the stored client. If the number of applied tokens is greater than the number of remaining tokens corresponding to the client, it means that the number of remaining tokens in the token bucket cannot meet the requirements. The token server returns a failure result to the data lake metadata server. At this point, the current limiting check fails, the access request is unsuccessful, and the data lake metadata server rejects the metadata access request. In this way, while ensuring high concurrency and low latency of the metadata service, abnormal access requests can also be intercepted.

[0098] It is understandable that, since new tokens will continue to be generated in the token bucket, the metadata access request can be resent after being limited.

[0099] If the number of tokens requested is less than or equal to the client's remaining token count, the remaining tokens in the token bucket are sufficient. At this point, the token server deducts an equal number of tokens from the requested token count. The token server then returns a successful request to the Data Lake Metadata Server. At this point, the throttling check passes, the access request succeeds, and the Data Lake Metadata Server executes the metadata access request.

[0100] Based on the relevant contents of S301-S305 above, it can be seen that the present application provides a method for limiting access to a metadata server, which is applied to a data lake metadata server that provides metadata services. The data lake metadata server receives a metadata access request sent by a client, which is used to obtain metadata stored by a storage server for describing target data, where the target data is the data in a target table. After receiving the access request, the flow of limiting the flow of the metadata access request is started. Specifically, the metadata service interface used by the data lake metadata server is first determined based on the obtained metadata access request. When the metadata service interface meets the preset conditions, the request parameters of the metadata access request are obtained, namely, the number of table partitions and the number of partition levels of the target table. The number of table partitions and the number of partition levels of the target table are used to implement the flow limiting processing judgment of the metadata access request. That is, the number of application tokens corresponding to the metadata access request is first determined based on the number of table partitions and the number of partition levels of the target table. When the number of application tokens corresponding to the metadata access request is greater than the number of remaining tokens corresponding to the client, the metadata access request is subjected to flow limiting processing.

[0101] It can be seen that the request parameters corresponding to different tables in the metadata service interface that meets the preset conditions, or the same table at different times, may all be different, and the corresponding number of application tokens will be different. Therefore, this application can adaptively determine the number of application tokens corresponding to the metadata access request through the metadata service interface and the request parameters of the metadata access request, and determine whether to limit the metadata access request based on the number of application tokens. This can improve the service stability of the data lake metadata server and reduce the load pressure on the storage server.

[0102] It can be seen that when the client has not consumed the token in the token server, the number of tokens corresponding to the client stored in the token server is the maximum number of tokens. It should be understood that the number of tokens corresponding to the client / maximum number of tokens described in the embodiments of the present application are all for the data lake metadata server. The maximum number of tokens corresponding to the client can be represented by Quota, which can also be understood as a quota. Quota represents the total number of tokens that the client is allowed to execute requests on the data lake metadata server within 1 minute. Since the request sent by the client to the data lake metadata server is used to access the target database in the storage server, the number of tokens can also be understood as the maximum number of tokens for the client to access the target database. In actual applications, there are multiple data lake metadata server instances corresponding to one target database instance, then the total number of tokens required to access a single target database is related to the number of tokens required to be consumed by each data lake metadata server instance. It can be understood that the number of tokens corresponding to each client stored in the token server may be different.

[0103] Based on this, in one possible implementation, the embodiment of the present application provides a process for obtaining the maximum number of tokens corresponding to the client, including:

[0104] A1: Determine the address of the corresponding data lake metadata server based on the library name of the target table, and then determine the number of storage servers corresponding to the data lake metadata server based on the address of the data lake metadata server.

[0105] Because different data warehouses (such as Hive libraries) may use the same or different Data Lake Metadata Servers (such as Hive libraries), there is a routing relationship / correspondence between the data warehouse and the address of the Data Lake Metadata Server used. Based on this, there is also a routing relationship between the library to which the target table belongs and the address of the Data Lake Metadata Server. Therefore, the address of the corresponding Data Lake Metadata Server can be determined based on the library name of the target table.

[0106] To alleviate database access pressure, a single data lake metadata server can be deployed with multiple storage servers. If a storage server runs a target database, it can correspond to multiple target databases. After determining the address of the data lake metadata server, the number of storage servers corresponding to the data lake metadata server can be determined based on the address.

[0107] A2: Calculate the maximum number of tokens corresponding to the client based on the number of storage servers corresponding to the data lake metadata server and the number of tokens for each storage server. The number of tokens for a single storage server is related to the number of tokens consumed by online access requests to the storage server per unit time and the hardware utilization rate of the storage server.

[0108] The token count for a single storage server is the upper limit on the number of tokens a client can consume on a single storage server. The token counts for each storage server can be the same or different. As an alternative example, when the token counts for each storage server corresponding to a Data Lake Metadata Server are the same, the maximum token count for a client is the product of the number of storage servers corresponding to the Data Lake Metadata Server and the token count for each storage server. As another alternative example, when the token counts for each storage server corresponding to a Data Lake Metadata Server are different, the maximum token count for a client is the sum of the token counts for each storage server corresponding to the Data Lake Metadata Server.

[0109] The number of tokens per storage server is related to the number of tokens consumed by online access requests to the storage server per unit time and the storage server's hardware utilization rate. The number of tokens consumed by online access requests to the storage server per unit time is obtained through experiments. Hardware utilization rate includes CPU utilization and / or memory utilization.

[0110] During the actual test process, the mean or peak value of the number of tokens consumed by the metadata access requests received by the online storage server within a unit time (such as 1 minute) is recorded by the buried point statistics. The mean or peak value is the token number statistic. For example, the mean value is 1500 tokens. In addition, the CPU utilization rate at this time is recorded, for example, 10%. A CPU utilization rate of 10% corresponds to a mean value of 1500. As an optional example, the number of tokens of a single storage server is the number of tokens corresponding to the allowed CPU utilization rate of the storage server. For example, the allowed CPU utilization rate is 20%, and the storage server can still work normally at this utilization rate. Then, according to the proportional relationship, the number of tokens corresponding to the 20% CPU utilization rate is twice the above-mentioned token number statistic, that is, 3000 tokens.

[0111] It is understandable that the above test process is related to the configuration of the storage server (eg, the configuration of the target database). When the configuration of the storage server is changed, the number of tokens of the corresponding single storage server will also change.

[0112] Before the Data Lake Metadata Server's rate limiting feature is implemented, it's necessary to estimate the maximum number of tokens corresponding to each client. This is called the estimated maximum number of tokens. This allows the Data Lake Metadata Server's rate limiting feature to function properly. Therefore, in actual applications, it's necessary to obtain the estimated maximum number of tokens corresponding to each client.

[0113] As an optional example, the estimated maximum number of tokens corresponding to a client can be calculated based on stress testing data from a single, most frequently used online data table with the largest capacity. For example, during stress testing, a single storage server instance processed 445 requests per minute. At this point, the storage server's CPU utilization reached 40%. Furthermore, assuming a typical actual number of requests (i.e., the number of results returned by a request) of 72 (most online requests return less than 100 results), the actual number of tokens used is calculated to be 42 (see below for the calculation method). Therefore, at a CPU utilization of 40%, the estimated maximum number of tokens is 445 * 42 = 18,690. If the CPU utilization is 10%, the estimated maximum number of tokens corresponding to a single storage server is 18,690 / 4 = 4,672. It is understood that if there are multiple storage servers, the estimated maximum number of tokens corresponding to each storage server can be determined in this manner. It is understood that the token corresponding to a client is used to access a storage server, meaning that the estimated maximum number of tokens corresponding to a storage server is equal to the estimated maximum number of tokens corresponding to the client.

[0114] It can be seen that there are still some metadata service interfaces that do not meet the preset conditions. In this case, the access flow limiting method of the metadata server provided in the embodiment of the present application may also include the following steps:

[0115] When the metadata service interface does not meet the preset conditions, it is determined whether to perform current limiting processing on the metadata access request according to the current limiting rules of the metadata service interface.

[0116] S303-S305 described in the above embodiment can be understood as the rate limiting rules for the metadata service interface when the preset conditions are met. However, when the metadata service interface does not meet the preset conditions, this indicates that the metadata service interface access type does not involve accessing table partitions. In this case, the metadata service interface that does not meet the preset conditions has its own rate limiting rules, and the access request rate limiting process described in S303-S305 is not necessary.

[0117] The throttling rules for metadata service interfaces that don't meet preset conditions are related to the metadata acquisition method used by the metadata service interface. Some metadata service interfaces (such as the get_table interface) obtain metadata through indexes, eliminating the need to scan the target database to obtain metadata corresponding to target data that meets the query conditions. For example, the client records the correspondence between indexes and target data. When determining the target data to be acquired, the target data's index is determined based on this correspondence. The target data's index indicates the target data's storage location in HDFS. Determining the target data's storage location through indexes eliminates the need for extensive scans of the target database on the storage server, resulting in a low load on the target database, typically at a constant level. In this case, throttling can be implemented using QPS. This is provided here as an example only; specific throttling rules can be set based on the actual metadata service interface.

[0118] For example, some other metadata service interfaces do not obtain information at the table partition level, but obtain information at the table level. In this case, current limiting is required according to the current limiting rules of the metadata service interface itself, which will not be detailed here.

[0119] See also Figure 4 , Figure 4 This is a schematic diagram of another exemplary application scenario provided by the embodiment of the present application. Figure 4 As shown in the figure, after normal logic processing (i.e., executing metadata access request without current limiting), the actual number of tokens used will be obtained. The actual number of tokens used can be represented by postCost. Further, the number of tokens used for metadata access request is corrected (i.e., Figure 4 (See postCost correction for details.) Before the correction process, the client first obtains the actual number of table partitions returned by the metadata access request, represented by result_num. The client then calculates the postCost using part_num, part_level, and result_num. The client then requests tokens for the postCost number from the token server to complete the postCost correction and return the tokens to the client.

[0120] See also Figure 5 , Figure 5 This is a flowchart of another metadata server access flow limiting method provided in an embodiment of the present application. The method describes a process of correcting the number of tokens used for metadata access requests. The method includes S501-S504:

[0121] S501: When the number of applied tokens is less than or equal to the number of remaining tokens corresponding to the client, a metadata access request is executed.

[0122] If the number of tokens requested is less than or equal to the number of remaining tokens corresponding to the client, no throttling is performed and a metadata access request is executed, accessing the storage server to obtain metadata describing the target data. This metadata includes the name of the database and table to which the selected partition belongs, the creation time, and storage information. The storage information includes the HDFS path of the partition to which the selected target data belongs.

[0123] It can be understood that the number of remaining tokens corresponding to the client is the number of remaining tokens corresponding to the client stored in the token server when the current limiting logic judgment is performed on the metadata access request.

[0124] S502: Obtain the actual number of table partitions returned after the metadata access request is executed.

[0125] After executing a metadata access request, the number of results that actually satisfy the query statement (also known as the number of records returned by the query) is returned. This number is the partition number of the partition to which the target data in the query statement belongs, that is, the actual number of table partitions. For example, in the above example, the target data in the query statement is data with a date greater than or equal to 2, and the table partitions that satisfy the query statement are table partition 1 and table partition 2. Therefore, the actual number of table partitions is 2, which means the number of results is 2. Each result includes metadata for one table partition, such as metadata for table partition 1 and metadata for table partition 2. Since the data in table partition 1 and table partition 2 is the target data, the metadata for table partition 1 and table partition 2 describes the target data.

[0126] S503: Based on the number of table partitions, the number of partition levels, and the actual number of table partitions in the request parameters, obtain the actual number of tokens used in the metadata access request.

[0127] In actual applications, stress testing has determined that the number of tokens used in metadata access requests is related to the number of table partitions (part_num), the number of partition levels (part_level), and the actual number of table partitions (result_num). The actual number of table partitions is only available after the metadata access request is executed. Therefore, the impact of the actual number of table partitions (result_num) on the number of tokens used in metadata access requests can only be calculated after the metadata access request is executed.

[0128] It is understandable that if part_num and part_level are the same but result_num is different for the same data table (i.e., the data table to which the target data belongs, such as a Hive table), this means that the number of tokens used in metadata access requests may also be different.

[0129] Specifically, based on the number of table partitions, the number of partition levels, and the actual number of table partitions in the request parameters, the actual number of tokens used in the metadata access request is obtained.

[0130] In one possible implementation, an embodiment of the present application provides a specific implementation method for obtaining the actual number of tokens used in a metadata access request based on the number of table partitions, the number of partition levels, and the actual number of table partitions in the request parameters, including:

[0131] Calculate the logarithm of the number of table partitions and obtain the first value;

[0132] Calculate the logarithm of the actual number of table partitions;

[0133] The product of a fourth constant, the number of partition levels, the first value, and the logarithm of the actual number of table partitions is calculated as the number of tokens actually used in the metadata access request; the fourth constant is related to the configuration of the storage server.

[0134] Based on this specific implementation, when the number of table partitions is represented by part_num, the number of partition levels is represented by part_level, the actual number of table partitions is represented by result_num, the fourth constant is represented by a, and the actual number of tokens used is represented by post_cost, the calculation formula for the actual number of tokens used can be expressed as: post_cost=a*part_level*log(part_num)*log(result_num).

[0135] Among them, the first value obtained by calculation is log(part_num); the logarithm of the actual number of table partitions is log(result_num), which is the logarithm of result_num with base 10. The calculation formula for the actual number of tokens used is similar to the calculation formula for the number of tokens applied for, and both are obtained by stress testing and fitting. The fourth constant a is obtained in a similar way to the first, second, and third constants, and the fourth constant is also a formula parameter. For example: if the storage server uses a 20-core virtual machine, the calculation formula for the actual number of tokens used is specifically: post_cost=0.8214*part_level*log(part_num)*log(result_num), and the fourth constant a is 0.8214.

[0136] S504: When the number of tokens actually used is greater than the number of tokens applied for, the difference between the number of tokens actually used and the number of tokens applied for is calculated, and a number of tokens equal to the difference is applied for from the token server.

[0137] If the number of tokens actually used exceeds the number of tokens requested, the precalculated number of tokens requested is too low and needs to be corrected. In this case, the client should contact the token server again and request (actual tokens used - requested tokens). Furthermore, the token server will recalculate the remaining tokens for the client and reflect this in the next request limit.

[0138] It is understandable that when the application fails, the subsequent processing flow can be directly entered. The subsequent processing flow is the flow of limiting the next metadata access request.

[0139] Based on the relevant content of S501-S504, it can be known that in order to perform flow control processing of metadata access requests, the number of application tokens is first calculated based on the metadata service interface and the request parameters of the metadata access request (the number of table partitions and the number of table partition levels). When the number of application tokens meets the requirements, the metadata access request is executed. After the metadata access request is executed, the actual number of table partitions returned after the execution of the metadata access request is obtained, and then the number of tokens used for the metadata access request is corrected based on the actual number of table partitions, so that the total number of tokens applied by the client to the token server is the actual number of tokens used. In this way, the number of remaining tokens of the client stored in the token server can be made more accurate, so that the flow control processing result of the metadata access request of the client next time can be more accurate.

[0140] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0141] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0142] Based on the metadata server access limiting method provided in the above-mentioned method embodiment, the present application also provides a metadata server access limiting device. The metadata server access limiting device is described below with reference to the accompanying drawings. Because the principles of the device in the disclosed embodiment are similar to those of the metadata server access limiting method in the above-mentioned embodiment, the implementation of the device can be referenced to the implementation of the method, and any repetitions will not be repeated.

[0143] See also Figure 6 As shown in FIG, this figure is a schematic diagram of the structure of an access flow limiting device of a metadata server provided in an embodiment of the present application. Figure 6 As shown, the access flow limiting device of the metadata server includes:

[0144] The receiving unit 601 is configured to receive a metadata access request sent by a client, wherein the metadata access request is used to obtain metadata stored in a storage server and used to describe target data; the target data is data in a target table;

[0145] A first determining unit 602 is configured to determine a metadata service interface used by the data lake metadata server according to the metadata access request;

[0146] A first acquiring unit 603 is configured to acquire request parameters of the metadata access request when the metadata service interface meets a preset condition; the request parameters include the number of table partitions and the number of partition levels of the target table;

[0147] A second obtaining unit 604 is configured to obtain the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels;

[0148] The processing unit 605 is configured to perform flow limiting processing on the metadata access request when the number of applied tokens is greater than the number of remaining tokens corresponding to the client.

[0149] In a possible implementation, the second obtaining unit 604 includes:

[0150] A first calculation subunit, configured to calculate the logarithm of the number of table partitions to obtain a first value;

[0151] A second calculation subunit is configured to calculate a product of a first constant, the partition level number, and the first value to obtain a second value;

[0152] a third calculating subunit, configured to calculate the sum of the second value and a second constant to obtain a third value;

[0153] The fourth calculation subunit is used to calculate the product of the third value and the third constant as the application token number of the metadata access request; wherein the first constant, the second constant, and the third constant are related to the configuration of the storage server.

[0154] In a possible implementation, the apparatus further includes:

[0155] an execution unit, configured to execute the metadata access request when the number of applied tokens is less than or equal to the number of remaining tokens corresponding to the client;

[0156] A third obtaining unit is used to obtain the actual number of table partitions returned after the metadata access request is executed;

[0157] A fourth acquiring unit, configured to acquire the number of tokens actually used in the metadata access request based on the number of table partitions, the number of partition levels, and the actual number of table partitions in the request parameters;

[0158] The application unit is used to calculate the difference between the actual number of used tokens and the applied number of tokens when the actual number of used tokens is greater than the applied number of tokens, and apply to the token server for a number of tokens equal to the difference.

[0159] In a possible implementation, the fourth acquiring unit includes:

[0160] a fifth calculating subunit, configured to calculate the logarithm of the number of table partitions to obtain a first value;

[0161] a sixth calculation subunit, configured to calculate the logarithm of the actual number of table partitions;

[0162] A seventh computing subunit is configured to calculate the product of a fourth constant, the number of partition levels, the first value, and the logarithm of the actual number of table partitions as the number of tokens actually used for the metadata access request; the fourth constant is related to the configuration of the storage server.

[0163] In a possible implementation, the first acquiring unit 603 is specifically configured to:

[0164] According to the table name of the target table, obtain the number of table partitions and the number of partition levels of the target table from the memory of the data lake metadata server;

[0165] The number of table partitions of the target table is stored in the storage server. When the data lake metadata server is started, the number of table partitions of the target table is loaded into the memory of the data lake metadata server. The number of partition levels of the target table is loaded into the memory of the data lake metadata server according to a preset period.

[0166] In one possible implementation, the client corresponds to a maximum number of tokens; the process of obtaining the maximum number of tokens corresponding to the client includes: determining the address of the corresponding data lake metadata server according to the library name of the library to which the target table belongs, and determining the number of storage servers corresponding to the data lake metadata server according to the address of the data lake metadata server; calculating the maximum number of tokens corresponding to the client according to the number of storage servers corresponding to the data lake metadata server and the number of tokens of a single storage server; wherein the number of tokens of a single storage server is related to the statistical number of tokens consumed by online access requests to the storage server per unit time and the hardware utilization rate of the storage server.

[0167] In a possible implementation, the apparatus further includes:

[0168] The second determining unit is configured to determine whether to perform flow limiting processing on the metadata access request according to the flow limiting rule of the metadata service interface when the metadata service interface does not meet the preset condition.

[0169] It should be noted that the specific implementation of each unit in this embodiment can refer to the relevant description in the above method embodiment. The division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiment, the processing unit and the sending unit can be the same unit or different units. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0170] Based on the access flow limiting method of a metadata server provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the access flow limiting method of the metadata server described in any of the above embodiments.

[0171] Reference below Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing an embodiment of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable Android devices), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs (televisions) and desktop computers. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0172] like Figure 7 As shown, electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 are also stored in RAM 703. Processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0173] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0174] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0175] The electronic device provided in the embodiment of the present application and the access flow limiting method of the metadata server provided in the above embodiment belong to the same inventive concept. For technical details not described in detail in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0176] Based on the access flow limiting method of a metadata server provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the access flow limiting method of the metadata server as described in any of the above embodiments is implemented.

[0177] It should be noted that the computer-readable medium referred to in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0178] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0179] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0180] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the access flow limiting method of the metadata server.

[0181] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0183] The units involved in the embodiments described in this application may be implemented by software or hardware. In some cases, the name of a unit / module does not constitute a limitation of the unit itself. For example, a voice data acquisition module may also be described as a "data acquisition module."

[0184] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0185] In the context of this application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0187] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0188] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0189] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0190] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for limiting access to a metadata server, characterized in that: The method is applied to a data lake metadata server and includes: Receive a metadata access request sent by a client, wherein the metadata access request is used to obtain metadata stored in a storage server for describing target data; the target data is data in a target table; Determining a metadata service interface used by the data lake metadata server according to the metadata access request; When the metadata service interface meets the preset conditions, obtaining the request parameters of the metadata access request; the request parameters include the number of table partitions and the number of partition levels of the target table; Obtaining the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels; When the number of applied tokens is greater than the number of remaining tokens corresponding to the client, the metadata access request is subjected to flow limiting processing.

2. The method according to claim 1, characterized in that The acquiring, according to the number of table partitions and the number of partition levels, the number of application tokens corresponding to the metadata access request includes: Calculate the logarithm of the number of partitions of the table to obtain a first value; Calculate the product of a first constant, the partition level number, and the first value to obtain a second value; Calculate the sum of the second value and the second constant to obtain a third value; The product of the third value and the third constant is calculated as the application token number of the metadata access request; wherein the first constant, the second constant, and the third constant are related to the configuration of the storage server.

3. The method according to claim 1 or 2, characterized in that The method further comprises: When the number of applied tokens is less than or equal to the number of remaining tokens corresponding to the client, executing the metadata access request; Obtaining the actual number of table partitions returned after the metadata access request is executed; Based on the number of table partitions, the number of partition levels and the actual number of table partitions in the request parameters, obtaining the actual number of tokens used in the metadata access request; When the number of tokens actually used is greater than the number of tokens applied for, the difference between the number of tokens actually used and the number of tokens applied for is calculated, and a number of tokens equal to the difference is applied for from the token server.

4. The method according to claim 3, characterized in that The acquiring, based on the table partition number, the partition level number, and the actual table partition number in the request parameters, the number of tokens actually used in the metadata access request includes: Calculate the logarithm of the number of partitions of the table to obtain a first value; Calculating the logarithm of the actual number of table partitions; The product of a fourth constant, the partition level number, the first value, and the logarithm of the actual table partition number is calculated as the actual number of tokens used for the metadata access request; the fourth constant is related to the configuration of the storage server.

5. The method according to claim 1, characterized in that The request parameters for obtaining the metadata access request include: According to the table name of the target table, obtain the number of table partitions and the number of partition levels of the target table from the memory of the data lake metadata server; The number of table partitions of the target table is stored in the storage server. When the data lake metadata server is started, the number of table partitions of the target table is loaded into the memory of the data lake metadata server. The number of partition levels of the target table is loaded into the memory of the data lake metadata server according to a preset period.

6. The method according to claim 1, characterized in that The client corresponds to a maximum number of tokens; and the process of obtaining the maximum number of tokens corresponding to the client includes: Determine the address of the corresponding data lake metadata server according to the library name of the library to which the target table belongs, and determine the number of storage servers corresponding to the data lake metadata server according to the address of the data lake metadata server; Calculate the maximum number of tokens corresponding to the client according to the number of storage servers corresponding to the data lake metadata server and the number of tokens of a single storage server; The number of tokens of a single storage server is related to a statistical amount of tokens consumed by online access requests to the storage server within a unit time and the hardware usage rate of the storage server.

7. The method according to claim 1, characterized in that The method further comprises: When the metadata service interface does not meet the preset condition, it is determined whether to perform current limiting processing on the metadata access request according to the current limiting rule of the metadata service interface.

8. An access current limiting device for a metadata server, characterized in that: The device is applied to a data lake metadata server and includes: A receiving unit, configured to receive a metadata access request sent by a client, wherein the metadata access request is used to obtain metadata stored in a storage server and used to describe target data; the target data is data in a target table; A first determining unit is configured to determine a metadata service interface used by the data lake metadata server according to the metadata access request; A first acquiring unit is configured to acquire request parameters of the metadata access request when the metadata service interface satisfies a preset condition; the request parameters include the number of table partitions and the number of partition levels of the target table; A second acquiring unit is configured to acquire the number of application tokens corresponding to the metadata access request according to the number of table partitions and the number of partition levels; The processing unit is configured to perform flow limiting processing on the metadata access request when the number of applied tokens is greater than the number of remaining tokens corresponding to the client.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the access flow limiting method for the metadata server as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the access flow limiting method of the metadata server as described in any one of claims 1 to 7 is implemented.

Images