A permission control method based on HiveMetaStore

Abstract

The present invention relates to the technical field of permission control, and in particular to a permission control method based on HiveMetaStore, comprising the following steps: submitting metadata information in HiveMetaStore to Hiveserver2 to output optimized data, inputting the optimized data into a permission verification plug-in to match and verify with a policy file; returning the corresponding metadata information to an input end when the matching verification passes; obtaining the number of verification passes of permission verification within a number of verification cycles; judging the comprehensiveness of permission control based on the variance of the number of verification passes of permission verification; if it is determined that the comprehensiveness does not meet the requirements, adjusting the number of concurrent synchronous connections of the permission verification plug-in; if it is determined that the stability does not meet the requirements, adjusting the cache refresh frequency of the policy file; and performing secondary adjustment on the amount of data processed by the permission verification plug-in in a single batch based on the average throughput of Hiveserver2. The present invention achieves an improvement in the comprehensiveness of permission control.

Description

Technical Field

[0001] The present invention relates to the technical field of permission control, and in particular to a permission control method based on HiveMetaStore. Background Art

[0002] In the prior art, Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism that can query, analyze and store large-scale data in Hadoop. HiveMetaStore and HiveServer2 are two different components in Hive. They play different roles in the Hive architecture, but they have a certain relationship. HiveMetaStore is the metadata storage and management component of Hive. HiveMetaStore is responsible for managing metadata information such as the properties of tables, partitions and columns in Hive. HiveServer2 is the service interface of Hive, which is used to provide remote clients with access to Hive. It allows clients to use different programming languages ​​(such as Java and Python) to communicate with Hive through the Thrift interface. In addition, there are two clients, hive-cli and beeline. Hive-cli accesses metadata through Metastore, and beeLine submits SQL to HiveServer2, which is compiled by HiveServer2, and then accesses Metastore to submit jobs.

[0003] Currently, permission control technologies for Hive, including Ranger and Sentry, verify user permissions on the HiveServer2 side to achieve the purpose of permission control. However, if users use hive-cli to submit jobs, this method does not go through HiveServer2 and directly interacts with Metastore. In this way, there is no way to control user permissions.

[0004] Chinese Patent Publication No.: CN111046447A discloses a permission control method and permission control system, including: obtaining corresponding display position information of each signal source on the touch input and display screen, each signal source can receive the touch input from the touch input and the corresponding display position on the display screen; receiving the touch input and the first touch input on the display screen, generating a control instruction according to the first touch input, and sending the control instruction to one or more corresponding signal sources, so that the one or more corresponding signal sources stop responding or restart responding to the subsequent second touch input from the touch input and the corresponding display position of the signal source on the display screen, wherein the one or more corresponding signal sources to which the control instruction is to be sent are determined according to the first touch input and the corresponding display position information of each signal source on the touch input and the display screen. It can be seen from this that the permission control method and the permission control system have the problem that the data synchronization is delayed and the data synchronization is incomplete due to the large amount of data that needs to be synchronized in a certain time period, which leads to errors in the configuration of permissions, resulting in a reduction in the number of permissions passed, thereby causing a decrease in the comprehensiveness of permission control. Summary of the invention

[0005] To this end, the present invention provides a permission control method based on HiveMetaStore, which is used to overcome the problem in the prior art that due to the large amount of data that needs to be synchronized in a certain period of time, data synchronization is delayed, resulting in incomplete data synchronization, and then leading to errors in permission configuration, resulting in a reduction in the number of permissions passed, thereby causing a decrease in the comprehensiveness of permission control.

[0006] To achieve the above-mentioned purpose, the present invention provides a permission control method based on HiveMetaStore, comprising the following steps: submitting metadata information in HiveMetaStore to Hiveserver2 to output optimized data, inputting the optimized data into a permission verification plug-in to match and verify with a policy file; when the matching verification passes, returning the corresponding metadata information to the input end; obtaining the number of verification passes of the permission verification within a number of verification cycles; judging the comprehensiveness of the permission control based on the variance of the number of verification passes of the permission verification; if it is determined that the comprehensiveness does not meet the requirements, adjusting the number of concurrent synchronous connections of the permission verification plug-in or executing a determination process for the stability of the policy file matching; if it is determined that the stability does not meet the requirements, adjusting the cache refresh frequency of the policy file or initially adjusting the data volume of a single batch processing of the permission verification plug-in based on the average permission verification time; obtaining the throughput of Hiveserver2 within a number of submission cycles after batch processing of the optimized data in a single cycle based on the data volume of the single batch processing after the initial adjustment; and performing a secondary adjustment on the data volume of a single batch processing of the permission verification plug-in based on the average throughput of Hiveserver2.

[0007] Further, the step of adjusting the number of concurrent synchronous connections of the permission verification plug-in includes:

[0008] The comprehensiveness of the permission control is determined based on the variance of the number of times the permission verification is passed;

[0009] If the variance of the number of times the permission verification is passed satisfies the first variance condition or the second variance condition, it is determined that the comprehensiveness of the permission control does not meet the requirements;

[0010] If the variance of the number of times the permission verification passes only satisfies the second variance condition, the number of concurrent synchronous connections of the permission verification plug-in is increased;

[0011] Among them, the first variance condition is that the variance of the number of verification passes of the authority verification is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the number of verification passes of the authority verification is greater than the preset second variance.

[0012] Furthermore, the process of determining the stability of the policy file matching includes: when the variance of the number of verification passes of the permission verification only satisfies the first variance condition, it is preliminarily determined that the stability of the policy file matching does not meet the requirements, and the stability of the policy file matching is secondary determined based on the average pass count of the verification permission.

[0013] Furthermore, the increase in the number of concurrent synchronous connections of the permission verification plug-in is determined by the difference between the variance of the number of verification passes of the permission verification and a preset second variance.

[0014] Furthermore, the step of adjusting the cache refresh frequency of the policy file includes:

[0015] Obtain the number of authentication permissions passed during several authentication cycles, and calculate the average number of authentication permissions passed;

[0016] Comparing the average throughput of the verification authority with a preset first throughput and a preset second throughput respectively;

[0017] If the average pass rate of the verification authority is greater than the preset first pass rate, the stability of the secondary determination policy file matching does not meet the requirements;

[0018] If the average throughput of the verification authority is greater than the preset first throughput and less than or equal to the preset second throughput, the cache refresh frequency of the policy file is increased;

[0019] If the average pass amount of the verification authority is greater than the preset second pass amount, it is preliminarily determined that the storage validity of the optimized data does not meet the requirements, and a secondary determination is made on the storage validity of the optimized data according to the average authority verification time;

[0020] The increase range of the cache refresh frequency of the policy file is determined by the difference between the average throughput of the authentication authority and the preset first throughput.

[0021] Furthermore, the calculation formula for the average throughput of the verification authority is:

[0022]

[0023] Among them, Z is the average number of verification permissions passed, X a is the number of users who pass the permission verification in the ath permission verification cycle, n is the number of permission verification cycles, and n is a natural number greater than or equal to 1.

[0024] Further, the step of initially adjusting the amount of data processed in a single batch by the permission verification plug-in includes:

[0025] Secondary determination of storage validity of optimization data based on the average permission verification duration;

[0026] If the average permission verification time is longer than the preset verification time, the storage validity of the optimization data is secondarily determined to be unsatisfactory, and the amount of data processed in a single batch by the permission verification plug-in is reduced.

[0027] Furthermore, the reduction in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average permission verification time and the preset verification time.

[0028] Furthermore, the step of secondarily adjusting the amount of data processed in a single batch by the permission verification plug-in includes:

[0029] Get the throughput of Hiveserver2 in several submission cycles and calculate the average throughput of Hiveserver2;

[0030] Compare the average throughput of the Hiveserver2 with the preset throughput;

[0031] If the average throughput of Hiveserver2 is greater than the preset throughput, it is determined that the processing efficiency of the metadata information does not meet the requirements, and the amount of data processed in a single batch by the permission verification plug-in is increased;

[0032] The increase in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average throughput of Hiveserver2 and the preset throughput.

[0033] Furthermore, the data volume of a single batch processing by the permission verification plug-in means the data volume of the optimization data batch processed by the permission verification plug-in at a single time.

[0034] Compared with the prior art, the beneficial effect of the present invention lies in that the method of the present invention adjusts the number of concurrent synchronous connections of the permission verification plug-in according to the variance of the number of verification passes of the permission verification. Since the amount of data that needs to be synchronized in a certain time period is relatively large, data synchronization is delayed, resulting in incomplete data synchronization, which in turn leads to errors in the configuration of permissions, resulting in a reduction in the number of permissions passed. By increasing the number of concurrent synchronous connections of the permission verification plug-in, more data can be processed at the same time, reducing synchronization delays, thereby improving the speed and integrity of data synchronization. By adjusting the cache refresh frequency of the policy file according to the average pass volume of the verified permissions, due to network failures encountered during the data refresh process, the refresh process is abnormal, resulting in an imperfect cache refresh mechanism, which in turn leads to inconsistent data cache, which causes some users to be unable to pass during the matching process. By increasing the cache refresh frequency, timely data updates are ensured, and the impact of access authorization problems caused by network failures is reduced. The data volume of a single batch processing of the permission verification plug-in is initially adjusted. Since the data volume of the permission access information that needs to be processed is too large, a large number of processing operations are required, resulting in a decrease in the performance of the storage space, which in turn leads to a decrease in the processing speed of permission access, which in turn leads to an increase in the duration of permission access. By reducing the data volume of a single batch processing of the permission verification plug-in, the duration of a single processing operation is reduced, thereby reducing the load and overhead of the storage space and improving the processing efficiency. The data volume of a single batch processing of the permission verification plug-in is secondarily adjusted according to the average throughput of Hiveserver2. Since the data volume of a single batch processing is too small, Hiveserver2 frequently makes access requests to data when accessing HiveMetaStore to obtain metadata information, increasing the load on HiveMetaStore and causing the performance of Hiveserver2 to degrade. By increasing the data volume of a single batch processing of the permission verification plug-in, the processing efficiency of access information is improved, and the comprehensiveness of permission control is improved.

[0035] Furthermore, the method of the present invention adjusts the number of concurrent synchronous connections of the permission verification plug-in by setting a preset first variance and a preset second variance. Since the amount of data that needs to be synchronized in a certain time period is relatively large, data synchronization is delayed, resulting in incomplete data synchronization, and then causing errors in the configuration of permissions, thereby reducing the number of permissions passed. By increasing the number of concurrent synchronous connections of the permission verification plug-in, more data can be processed at the same time, reducing synchronization delays, thereby improving the speed and integrity of data synchronization, and further achieving a comprehensive improvement in permission control.

[0036] Furthermore, the method of the present invention adjusts the cache refresh frequency of the policy file by setting a preset first throughput and a preset second throughput. Due to a network failure encountered during the data refresh process, the refresh process becomes abnormal, resulting in an imperfect cache refresh mechanism, which in turn leads to inconsistent data cache, thereby causing some users to be unable to pass during the matching process. By increasing the cache refresh frequency, timely data updating is ensured, reducing the impact of access authorization problems caused by network failures, and further achieving a comprehensive improvement in permission control.

[0037] Furthermore, the method of the present invention performs an initial adjustment on the amount of data that is processed in a single batch by the permission verification plug-in by setting a preset verification time. Since the amount of permission access information that needs to be processed is too large, a large number of processing operations are required, which leads to a decrease in the performance of the storage space, and then the processing speed of permission access is reduced, which leads to an increase in the duration of permission access. By reducing the amount of data that is processed in a single batch by the permission verification plug-in, the duration of a single processing operation is reduced, thereby reducing the load and overhead of the storage space, improving processing efficiency, and further achieving an improvement in the comprehensiveness of permission control.

[0038] Furthermore, the method of the present invention performs secondary adjustment on the amount of data processed in a single batch by the permission verification plug-in by setting a preset throughput. Since the amount of data processed in a single batch is too small, Hiveserver2 frequently makes access requests to data when accessing HiveMetaStore to obtain metadata information, which increases the load of HiveMetaStore and causes the performance of Hiveserver2 to degrade. By increasing the amount of data processed in a single batch by the permission verification plug-in, the processing efficiency of access information is improved, and the comprehensiveness of permission control is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is an overall flow chart of the permission control method based on HiveMetaStore in an embodiment of the present invention;

[0040] Figure 2 It is a logic flow chart of the permission control method based on HiveMetaStore in an embodiment of the present invention;

[0041] Figure 3 A specific flow chart of the process of initially adjusting the data volume of a single batch processing of a permission verification plug-in in the permission control method based on HiveMetaStore according to an embodiment of the present invention;

[0042] Figure 4 The present invention is a specific flow chart of the process of adjusting the data volume of a single batch processing of a permission verification plug-in in a secondary adjustment of the permission control method based on HiveMetaStore in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0045] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0046] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively an overall flowchart of the permission control method based on HiveMetaStore in an embodiment of the present invention, a logical flowchart, a specific flowchart of the process of initially adjusting the data volume of a single batch of the permission verification plug-in, and a specific flowchart of the process of secondary adjusting the data volume of a single batch of the permission verification plug-in. A permission control method based on HiveMetaStore of the present invention comprises:

[0048] Step S1, submitting metadata information in HiveMetaStore to Hiveserver2 to output optimized data, and inputting the optimized data into the permission verification plug-in to match and verify with the policy file;

[0049] Step S2, when the matching verification is passed, the corresponding metadata information is returned to the input end;

[0050] Step S3, obtaining the number of times the permission verification is passed within a number of verification cycles;

[0051] Step S4, judging the comprehensiveness of the authority control based on the variance of the number of times the authority verification is passed;

[0052] Step S5, if it is determined that the comprehensiveness does not meet the requirements, the number of concurrent synchronous connections of the permission verification plug-in is adjusted or a process of determining the stability of the policy file matching is performed;

[0053] Step S6: if it is determined that the stability does not meet the requirement, the cache refresh frequency of the policy file is adjusted or the amount of data processed by the permission verification plug-in in a single batch is initially adjusted based on the average permission verification time;

[0054] Step S7, performing batch processing of the optimized data in a single cycle based on the data volume of the single batch processing after the initial adjustment, and obtaining the throughput of Hiveserver2 in several submission cycles;

[0055] Step S8, based on the average throughput of Hiveserver2, a secondary adjustment is made to the data volume of a single batch processing of the permission verification plug-in.

[0056] Specifically, the metadata information submission methods include beeline method, jdbc method, and hive-cli method.

[0057] Specifically, when using the hive-cli method, the metadata information is directly submitted to the permission verification plug-in, and the permission verification plug-in matches the metadata information according to the policy file to verify the permission. After the permission is verified, the corresponding metadata information is returned to the hive-cli.

[0058] Specifically, the hive-cli method directly accesses metadata through the Metastore.

[0059] Specifically, the process of matching and verifying the optimization data with the policy file is as follows: the optimization data and the corresponding metadata information are submitted to the permission verification plug-in. The permission verification plug-in checks whether it has the permission to access the corresponding data based on the policy file match. If it has the corresponding permission, the permission verification plug-in interacts with the Metastore and returns the corresponding metadata information; if it does not have the corresponding permission, access is denied.

[0060] Specifically, the number of concurrent synchronous connections of the permission verification plug-in means the number of services that need to interact with the permission verification plug-in at the same time.

[0061] Specifically, permissions include read permission, write permission, and delete permission.

[0062] Specifically, permission verification includes data query permission verification, data update permission verification, and data insertion permission verification.

[0063] Specifically, the cache refresh frequency of the policy file means the frequency at which the cache version of the policy file in the storage space is refreshed.

[0064] Specifically, HiveMetaStore is the metadata storage and management component of Hive, which is responsible for managing metadata information such as tables, partitions, columns, and table attributes in Hive. HiveServer2 is the service interface of Hive, which is used to provide remote clients with access to Hive. It is a service based on the Thrift protocol, allowing clients to use different programming languages ​​(such as Java and Python) to communicate with Hive through the Thrift interface. Hive is a data warehouse tool based on Hadoop for data extraction, transformation, and loading.

[0065] Specifically, the beeline method is that beeline submits the SQL to HiveServer2, which compiles it and then accesses the Metastore to submit the job.

[0066] Specifically, the jdbc method is that jdbc submits SQL to HiveServer2, which compiles it and then accesses the Metastore to submit the job.

[0067] Specifically, the metadata information includes the creation time of the database, the name of the data table, and the type of the data table.

[0068] Specifically, the optimization data includes query execution time, number of IO operations, and data transmission volume of metadata information transmitted to Hiveserver2 per unit time.

[0069] Specifically, the policy file includes permission types, permission allocation, and policy updates.

[0070] Specifically, the process by which the permission verification plug-in verifies the optimization data is as follows:

[0071] The Ranger HiveMetastore Store Plugin is implemented based on the Ranger system. The permission verification plug-in actively pulls policies from RangerAdmin. If the policies change, the new policies are pulled to update the authentication engine in memory. At the same time, a backup file is saved locally. When RangerAdmin crashes and the component is restarted, the local backup can be used to continue authentication.

[0072] The plug-in is loaded when Metastore is started. When Hiveserver2 or hive-cli communicates with Metastore, it needs to carry the user information of the submitted SQL. The Ranger plug-in parses the user information and metadata information such as the library table columns to be accessed from the thrift link, and then performs permission verification based on the policy information.

[0073] It can be understood that Ranger HiveMetastore Store Plugin is a Ranger Hive metastore plug-in, RangerAdmin is a permission manager, Metastore is metastore, thrift is an interface description language and a binary communication protocol. Ranger HiveMetastore Store Plugin, RangerAdmin, Metastore and thrift are professional terms well known to those skilled in the art, so the specific meanings of the above terms are not repeated here.

[0074] In implementation, the method of the present invention adjusts the number of concurrent synchronous connections of the permission verification plug-in according to the variance of the number of times the permission verification is passed. Since the amount of data that needs to be synchronized in a certain time period is relatively large, data synchronization is delayed, resulting in incomplete data synchronization, which in turn leads to errors in the configuration of permissions, resulting in a reduction in the number of permissions passed. By increasing the number of concurrent synchronous connections of the permission verification plug-in, more data can be processed at the same time, reducing synchronization delays, thereby improving the speed and integrity of data synchronization. By adjusting the cache refresh frequency of the policy file according to the average pass volume of the verified permissions, due to network failures encountered during the data refresh process, the refresh process is abnormal, resulting in an imperfect cache refresh mechanism, which in turn leads to inconsistent data cache, resulting in some users being unable to pass during the matching process. By increasing the cache refresh frequency, timely data updates are ensured, and the impact of access authorization problems caused by network failures is reduced. By adjusting the permission verification plug-in according to the average permission verification time, The amount of data processed in a single batch of the permission verification plug-in is initially adjusted. Since the amount of permission access information that needs to be processed is too large, a large number of processing operations are required, which leads to a decrease in the performance of the storage space, and then the processing speed of permission access is reduced, which leads to an increase in the duration of permission access. By reducing the amount of data processed in a single batch by the permission verification plug-in, the duration of a single processing operation is reduced, thereby reducing the load and overhead of the storage space and improving processing efficiency. The amount of data processed in a single batch of the permission verification plug-in is secondarily adjusted according to the average throughput of Hiveserver2. Since the amount of data processed in a single batch is too small, Hiveserver2 frequently makes access requests to data when accessing HiveMetaStore to obtain metadata information, which increases the load on HiveMetaStore and causes the performance of Hiveserver2 to decrease. By increasing the amount of data processed in a single batch by the permission verification plug-in, the processing efficiency of access information is improved, and the comprehensiveness of permission control is improved.

[0075] Specifically, the step of adjusting the number of concurrent synchronous connections of the permission verification plug-in includes:

[0076] The comprehensiveness of the permission control is determined based on the variance of the number of times the permission verification is passed;

[0077] If the variance of the number of times the permission verification is passed satisfies the first variance condition or the second variance condition, it is determined that the comprehensiveness of the permission control does not meet the requirements;

[0078] If the variance of the number of times the permission verification passes only satisfies the second variance condition, the number of concurrent synchronous connections of the permission verification plug-in is increased;

[0079] Among them, the first variance condition is that the variance of the number of verification passes of the authority verification is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the number of verification passes of the authority verification is greater than the preset second variance.

[0080] It can be understood that the three intervals corresponding to the preset first variance and the preset second variance correspond to three situations respectively. The first situation is that the variance of the number of verification passes of permission verification is less than or equal to the preset first variance, and it is determined that the comprehensiveness of permission control meets the requirements; the second situation is that the variance of the number of verification passes of permission verification is greater than the preset first variance and less than or equal to the preset second variance, and it is preliminarily determined that the stability of policy file matching does not meet the requirements, and a secondary judgment is made on the stability of policy file matching based on the average number of verification permissions; the third situation is that the variance of the number of verification passes of permission verification is greater than the preset second variance, and the number of concurrent synchronous connections of the permission verification plug-in is increased. In practice, the preset first variance is generally selected in the range of [140 times 2 , 160 times 2 ], the preset second variance is generally selected in the range of [190 times 2 , 210 times 2 ].

[0081] Preferably, the first variance Q1 is preset to be 150 times 2 , preset the second variance Q2=200 times 2 .

[0082] Specifically, the variance of the number of times the permission verification is passed is recorded as Q.

[0083] Specifically, the variance of the number of times the authority verification is passed is the variance of the number of times the authority verification is passed within several verification cycles. The calculation method for the variance of the number of times the authority verification is passed is a conventional technical means well known to those skilled in the art, and therefore the calculation process of the variance of the number of times the authority verification is passed will not be repeated here.

[0084] In implementation, the method of the present invention determines the comprehensiveness of authority control by setting a preset first variance and a preset second variance, thereby reducing the impact of decreased accuracy of authority control due to inaccurate determination of the comprehensiveness of authority control, and further improving the comprehensiveness of authority control.

[0085] Specifically, the process of determining the stability of the policy file matching includes: when the variance of the number of verification passes of the permission verification only satisfies the first variance condition, it is preliminarily determined that the stability of the policy file matching does not meet the requirements, and the stability of the policy file matching is secondary determined based on the average pass count of the verification permission.

[0086] Specifically, the increase in the number of concurrent synchronous connections of the permission verification plug-in is determined by the difference between the variance of the number of verification passes of the permission verification and a preset second variance.

[0087] Specifically, the calculation formula for the number of concurrent synchronous connections of the increased permission verification plug-in is:

[0088]

[0089] Among them, V' is the increased number of concurrent synchronous connections of the permission verification plug-in, V is the current number of concurrent synchronous connections of the permission verification plug-in, A is the minimum unit variance for adjusting the number of concurrent synchronous connections of the permission verification plug-in, and v is the number of concurrent synchronous connections of the permission verification plug-in when the variance of the number of verification passes of the permission verification is the minimum unit variance.

[0090] In practice, the value of A is usually 200 times 2 ~230 times 2 , preferably, the preferred embodiment of A is 220 times 2 .

[0091] In implementation, the optional value range of v is [18, 22]. Preferably, when the number of concurrent synchronous connections of the permission verification plug-in is less than 20, synchronization delays may occur. Therefore, the preferred embodiment of v is 20.

[0092] Specifically, when the value of V' is a decimal, the value of V' is automatically rounded up to an integer.

[0093] In implementation, the method of the present invention adjusts the number of concurrent synchronous connections of the permission verification plug-in by setting a preset first variance and a preset second variance. Since the amount of data that needs to be synchronized in a certain time period is relatively large, data synchronization is delayed, resulting in incomplete data synchronization, which in turn leads to errors in the configuration of permissions, thereby reducing the number of permissions passed. By increasing the number of concurrent synchronous connections of the permission verification plug-in, more data can be processed at the same time, reducing synchronization delays, thereby improving the speed and integrity of data synchronization, and further achieving a comprehensive improvement in permission control.

[0094] Specifically, the step of adjusting the cache refresh frequency of the policy file includes:

[0095] Obtain the number of authentication permissions passed during several authentication cycles, and calculate the average number of authentication permissions passed;

[0096] Comparing the average throughput of the verification authority with a preset first throughput and a preset second throughput respectively;

[0097] If the average pass rate of the verification authority is less than or equal to the preset first pass rate, the stability of the secondary determination policy file matching meets the requirements;

[0098] If the average throughput of the verification authority is greater than the preset first throughput and less than or equal to the preset second throughput, the cache refresh frequency of the policy file is increased;

[0099] If the average throughput of the verification authority is greater than the preset second throughput, it is preliminarily determined that the storage validity of the optimized data does not meet the requirements, and a secondary determination is made on the storage validity of the optimized data based on the average authority verification time.

[0100] It can be understood that the three intervals corresponding to the preset first throughput and the preset second throughput correspond to three situations respectively. The first situation is that the average throughput of the verification authority is less than or equal to the preset first throughput, and the stability of the policy file matching is determined to meet the requirements; the second situation is that the average throughput of the verification authority is greater than the preset first throughput and less than or equal to the preset second throughput, and the cache refresh frequency of the policy file is increased; the third situation is that the average throughput of the verification authority is greater than the preset second throughput, and the storage validity of the optimized data is preliminarily determined to be unsatisfactory, and the storage validity of the optimized data is determined to be secondary based on the average authority verification time. In practice, the preset first throughput is generally selected in the range of [140 / minute, 160 / minute], and the preset second throughput is generally selected in the range of [190 / minute, 210 / minute].

[0101] Preferably, the first throughput P1 is preset to be 150 pieces / minute, and the second throughput P2 is preset to be 200 pieces / minute.

[0102] Specifically, the average throughput of verification authority is denoted as P.

[0103] In implementation, the method of the present invention performs a secondary judgment on the stability of policy file matching by setting a preset first pass and a preset second pass, thereby reducing the impact of the reduced comprehensiveness of permission control due to inaccurate secondary judgment on the stability of policy file matching, and further achieving an improvement in the comprehensiveness of permission control.

[0104] Specifically, the increase range of the cache refresh frequency of the policy file is determined by the difference between the average throughput of the verification authority and the preset first throughput.

[0105] Specifically, the calculation formula for the cache refresh frequency of the increased policy file is:

[0106]

[0107] Among them, H' is the increased cache refresh frequency of the policy file, H is the current cache refresh frequency of the policy file, B is the minimum unit throughput for adjusting the cache refresh frequency of the policy file, and h is the cache refresh frequency of the policy file when the average throughput of the verification authority is the minimum unit throughput.

[0108] In practice, the value of B is generally 220 to 240 pieces / minute. Preferably, the preferred embodiment of B is 230 pieces / minute.

[0109] In implementation, the optional value range of h is [3min, 6min]. Preferably, when the cache refresh frequency of the policy file is greater than 5min, data may be updated untimely, so the preferred embodiment of h is 5min.

[0110] Specifically, when the value of H' is a decimal, the value of H' is automatically rounded up to an integer.

[0111] In implementation, the method of the present invention adjusts the cache refresh frequency of the policy file by setting a preset first throughput and a preset second throughput. Due to a network failure during the data refresh process, the refresh process becomes abnormal, resulting in an imperfect cache refresh mechanism, which in turn leads to inconsistent data cache, thereby causing some users to be unable to pass during the matching process. By increasing the cache refresh frequency, timely data updates are ensured, reducing the impact of access authorization problems caused by network failures, and further achieving a comprehensive improvement in permission control.

[0112] Specifically, the calculation formula for the average throughput of the verification authority is:

[0113]

[0114] Among them, Z is the average number of verification permissions passed, X a is the number of users who pass the permission verification in the ath permission verification cycle, n is the number of permission verification cycles, and n is a natural number greater than or equal to 1.

[0115] Specifically, the steps of initially adjusting the amount of data processed in a single batch by the permission verification plug-in include:

[0116] Secondary determination of storage validity of optimization data based on the average permission verification duration;

[0117] If the average permission verification time is longer than the preset verification time, the storage validity of the optimization data is secondarily determined to be unsatisfactory, and the amount of data processed in a single batch by the permission verification plug-in is reduced.

[0118] It can be understood that the two intervals corresponding to the preset verification time correspond to two situations. The first situation is that the average permission verification time is less than or equal to the preset verification time, and the storage validity of the secondary judgment optimization data meets the requirements; the second situation is that the average permission verification time is greater than the preset verification time, and the amount of data processed by the permission verification plug-in in a single batch is reduced. In practice, the preset verification time is generally selected in the range of [1s, 3s].

[0119] Preferably, the preset verification time length Y0=2s.

[0120] Specifically, the average permission verification time is recorded as Y.

[0121] Specifically, the calculation formula for the average permission verification time is:

[0122]

[0123] Among them, Y is the average permission verification time, T k is the verification time of the kth permission verification, s is the number of permission verifications, and s is a natural number greater than or equal to 1.

[0124] In implementation, the method of the present invention performs a secondary determination on the storage validity of the optimized data by setting a preset verification time, thereby reducing the impact of the reduced comprehensiveness of permission control due to inaccurate secondary determination on the storage validity of the optimized data, and further improving the comprehensiveness of permission control.

[0125] Specifically, the reduction in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average permission verification time and the preset verification time.

[0126] Specifically, the calculation formula for the reduced amount of data processed in a single batch by the permission verification plug-in is:

[0127]

[0128] Among them, L' is the reduced amount of data processed in a single batch by the permission verification plug-in, L is the amount of data processed in a single batch by the current permission verification plug-in, C is the minimum unit verification time for adjusting the amount of data processed in a single batch by the permission verification plug-in, and l is the amount of data processed in a single batch by the permission verification plug-in when the average permission verification time is the minimum unit verification time.

[0129] In practice, the value of C is generally 2s to 4s, and preferably, the preferred embodiment of C is 3s.

[0130] In implementation, the optional value range of l is [200Byte, 400Byte]. Preferably, when the amount of data processed by the permission verification plug-in in a single batch is greater than 250Byte, insufficient storage space may be caused. Therefore, the preferred embodiment of l is 250Byte.

[0131] Specifically, when the value of L' is a decimal, the value of L' is automatically rounded up to an integer.

[0132] In implementation, the method of the present invention performs an initial adjustment on the amount of data that is processed in a single batch by the permission verification plug-in by setting a preset verification time. Since the amount of permission access information that needs to be processed is too large, a large number of processing operations are required, which leads to a decrease in the performance of the storage space, and then the processing speed of permission access is reduced, which leads to an increase in the duration of permission access. By reducing the amount of data that is processed in a single batch by the permission verification plug-in, the duration of a single processing operation is reduced, thereby reducing the load and overhead of the storage space, improving processing efficiency, and further achieving an improvement in the comprehensiveness of permission control.

[0133] Specifically, the step of secondarily adjusting the amount of data processed in a single batch by the permission verification plug-in includes:

[0134] Get the throughput of Hiveserver2 in several submission cycles and calculate the average throughput of Hiveserver2;

[0135] Compare the average throughput of the Hiveserver2 with the preset throughput;

[0136] If the average throughput of the Hiveserver2 is greater than the preset throughput, it is determined that the processing efficiency of the metadata information does not meet the requirements, and the amount of data processed in a single batch by the permission verification plug-in is increased.

[0137] It can be understood that the two intervals corresponding to the preset throughput correspond to two situations. The first situation is that the average throughput of Hiveserver2 is less than or equal to the preset throughput, and the processing efficiency of metadata information meets the requirements; the second situation is that the average throughput of Hiveserver2 is greater than the preset throughput, and the amount of data processed in a single batch by the permission verification plug-in is increased. In practice, the preset throughput is generally selected in the range of [480MB / s, 520MB / s].

[0138] Preferably, the preset throughput R0=500MB / s.

[0139] Specifically, the average throughput of Hiveserver2 is denoted as R.

[0140] Specifically, the calculation formula for the average throughput of Hiveserver2 is:

[0141]

[0142] Where R is the average throughput of Hiveserver2, E f is the throughput of Hiveserver2 in the fth submission cycle, w is the number of submission cycles, and w is a natural number greater than or equal to 1.

[0143] In implementation, the method of the present invention determines the processing efficiency of metadata information by setting a preset throughput, thereby reducing the impact of reduced comprehensiveness of permission control due to inaccurate determination of the processing efficiency of metadata information, and further improving the comprehensiveness of permission control.

[0144] Specifically, the increase in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average throughput of Hiveserver2 and the preset throughput.

[0145] Specifically, the calculation formula for the amount of data processed in a single batch by the increased permission verification plug-in is:

[0146]

[0147] Wherein, L" is the amount of data processed in a single batch by the permission verification plug-in after the increase, D is the minimum unit throughput for secondary adjustment of the amount of data processed in a single batch by the permission verification plug-in, and l' is the amount of data processed in a single batch by the permission verification plug-in when the average throughput of Hiveserver2 is the minimum unit throughput.

[0148] In practice, the value of D is generally 520MB / s to 560MB / s. Preferably, the preferred embodiment of D is 550MB / s.

[0149] In implementation, the optional range of l' is [100Byte, 200Byte]. Preferably, when the amount of data processed by the permission verification plug-in in a single batch is less than 150Byte, the processing efficiency may be reduced. Therefore, the preferred embodiment of l' is 150Byte.

[0150] Specifically, when the value of L" is a decimal, the value of L" is automatically rounded up to an integer.

[0151] In implementation, the method of the present invention performs secondary adjustment on the amount of data processed in a single batch by the permission verification plug-in by setting a preset throughput. Since the amount of data processed in a single batch is too small, Hiveserver2 frequently makes access requests to data when accessing HiveMetaStore to obtain metadata information, which increases the load of HiveMetaStore and causes the performance of Hiveserver2 to degrade. By increasing the amount of data processed in a single batch by the permission verification plug-in, the processing efficiency of access information is improved, and the comprehensiveness of permission control is further improved.

[0152] Specifically, the amount of data processed in a single batch by the permission verification plug-in means the amount of data that the permission verification plug-in processes in a single batch on the optimization data. Example

[0153] This embodiment 1 uses a permission control method based on HiveMetaStore to control user permissions, and adjusts the number of concurrent synchronous connections of the permission verification plug-in according to the difference between the variance of the number of verification passes of the permission verification and the preset second variance, wherein the minimum unit variance for adjusting the number of concurrent synchronous connections of the permission verification plug-in is recorded as A, and the number of concurrent synchronous connections of the permission verification plug-in when the variance of the number of verification passes of the permission verification is the minimum unit variance is recorded as v, wherein A=220 times 2 , v=20, V=80, Q=250 times 2 , Q2=200 times 2 .

[0154] Calculate V'= ≈85. Example

[0155] This embodiment 2 uses a permission control method based on HiveMetaStore to control user permissions, and adjusts the cache refresh frequency of the policy file according to the difference between the average throughput of the verified permissions and the preset first throughput, wherein the minimum unit throughput for adjusting the cache refresh frequency of the policy file is denoted by B, and the cache refresh frequency of the policy file when the average throughput of the verified permissions is the minimum unit throughput is denoted by h, wherein B=230 / minute, h=5min, H=30min, P=250 / minute, and P1=150 / minute.

[0156] Calculate H'= ≈33min. Example

[0157] This embodiment 3 uses a permission control method based on HiveMetaStore to control user permissions, and reduces the amount of data processed in a single batch by the permission verification plug-in according to the difference between the average permission verification time and the preset verification time. The minimum unit verification time for adjusting the amount of data processed in a single batch by the permission verification plug-in is recorded as C, and the amount of data processed in a single batch by the permission verification plug-in when the average permission verification time is the minimum unit verification time is recorded as l, wherein C=3s, l=250Byte, L=600Byte, Y=5s, and Y0=2s.

[0158] Calculate L'= =350Byte. Example

[0159] This embodiment 4 uses a permission control method based on HiveMetaStore to control user permissions, and increases the amount of data processed in a single batch by the permission verification plug-in according to the difference between the average throughput of Hiveserver2 and the preset throughput. The minimum unit throughput for secondary adjustment of the amount of data processed in a single batch by the permission verification plug-in is recorded as D, and the amount of data processed in a single batch by the permission verification plug-in when the average throughput of Hiveserver2 is the minimum unit throughput is recorded as l', wherein D=550MB / s, l'=150Byte, L'=350Byte, R=600MB / s, and R0=500MB / s.

[0160] Calculate L” = ≈378Byte.

[0161] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A permission control method based on HiveMetaStore, characterized in that: The following steps are involved: Submit the metadata information in HiveMetaStore to Hiveserver2 to output optimized data, and input the optimized data and corresponding metadata information into the permission verification plug-in for matching and verification with the policy file; When the matching verification passes, the corresponding metadata information is returned to the hive-cli client; Get the number of times permission verification has been passed within a certain verification cycle; The comprehensiveness of permission control is determined based on the variance of the number of times permission verification is passed; If it is determined that the comprehensiveness does not meet the requirements, the number of concurrent synchronous connections of the permission verification plug-in is adjusted or a process for determining the stability of the policy file matching is performed; If it is determined that the stability does not meet the requirements, the cache refresh frequency of the policy file is adjusted or the amount of data processed in a single batch by the permission verification plug-in is initially adjusted based on the average permission verification time; Based on the data volume of a single batch after the initial adjustment, the optimized data is batch processed in a single cycle to obtain the throughput of Hiveserver2 in several submission cycles; Based on the average throughput of Hiveserver2, the amount of data processed in a single batch by the permission verification plug-in is adjusted twice; The optimization data includes query execution time, number of IO operations, and data transmission volume of metadata information transmitted to Hiveserver2 per unit time; The policy file includes permission types, permission allocation, and policy updates; The steps of adjusting the number of concurrent synchronous connections of the permission verification plug-in include: The comprehensiveness of permission control is determined based on the variance of the number of times permission verification is passed; If the variance of the number of times the permission verification is passed satisfies the first variance condition or the second variance condition, it is determined that the comprehensiveness of the permission control does not meet the requirements; If the variance of the number of times the permission verification passes only satisfies the second variance condition, increasing the number of concurrent synchronous connections of the permission verification plug-in; The first variance condition is that the variance of the number of times the authority verification is passed is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the number of times the authority verification is passed is greater than the preset second variance; The process of determining the stability of the policy file matching includes: when the variance of the number of verification passes of the permission verification only satisfies the first variance condition, it is preliminarily determined that the stability of the policy file matching does not meet the requirements, and the stability of the policy file matching is secondary determined based on the average pass count of the verification permission.

2. According to the HiveMetaStore-based permission control method of claim 1, it is characterized in that: The increase in the number of concurrent synchronous connections of the permission verification plug-in is determined by the difference between the variance of the number of verification passes of the permission verification and a preset second variance.

3. The permission control method based on HiveMetaStore according to claim 2 is characterized in that: The step of adjusting the cache refresh frequency of the policy file includes: Obtain the number of authentication permissions passed during several authentication cycles, and calculate the average number of authentication permissions passed; Comparing the average throughput of the verification authority with a preset first throughput and a preset second throughput respectively; If the average pass rate of the verification authority is less than or equal to the preset first pass rate, the stability of the secondary determination policy file matching meets the requirements; If the average throughput of the verification authority is greater than the preset first throughput and less than or equal to the preset second throughput, the cache refresh frequency of the policy file is increased; If the average pass amount of the verification authority is greater than the preset second pass amount, it is preliminarily determined that the storage validity of the optimized data does not meet the requirements, and a secondary determination is made on the storage validity of the optimized data according to the average authority verification time; The increase range of the cache refresh frequency of the policy file is determined by the difference between the average throughput of the authentication authority and the preset first throughput.

4. The permission control method based on HiveMetaStore according to claim 3 is characterized in that: The calculation formula for the average throughput of the verification authority is: Among them, Z is the average number of verification permissions passed, X a is the number of users who pass the permission verification in the ath permission verification cycle, n is the number of permission verification cycles, and n is a natural number greater than or equal to 1.

5. The permission control method based on HiveMetaStore according to claim 4 is characterized in that: The steps of initially adjusting the amount of data processed in a single batch by the permission verification plug-in include: Secondarily determine the storage validity of the optimization data based on the average permission verification time; If the average permission verification time is longer than the preset verification time, the storage validity of the optimization data is secondarily determined to be unsatisfactory, and the amount of data processed in a single batch by the permission verification plug-in is reduced.

6. The permission control method based on HiveMetaStore according to claim 5 is characterized in that: The reduction in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average permission verification time and the preset verification time.

7. The permission control method based on HiveMetaStore according to claim 6 is characterized in that: The step of secondarily adjusting the amount of data processed in a single batch by the permission verification plug-in includes: Get the throughput of Hiveserver2 in several submission cycles and calculate the average throughput of Hiveserver2; Compare the average throughput of the Hiveserver2 with the preset throughput; If the average throughput of Hiveserver2 is greater than the preset throughput, it is determined that the processing efficiency of the metadata information does not meet the requirements, and the amount of data processed in a single batch by the permission verification plug-in is increased; The increase in the amount of data processed in a single batch by the permission verification plug-in is determined by the difference between the average throughput of Hiveserver2 and the preset throughput.

8. The permission control method based on HiveMetaStore according to claim 7 is characterized in that: The data volume of a single batch processing by the permission verification plug-in means the data volume of the optimization data that is batch processed by the permission verification plug-in at a single time.

Images