A method for implementing a data warehouse function security engine

Abstract

The present invention discloses a method for implementing a data warehouse function security engine, and belongs to the technical field of security engines. In the present invention, by secondary encapsulation of UDF functions, the function authentication module is expanded by combining microservices and distributed configuration systems, so that high throughput, high concurrency and high availability are supported in massive data computing scenarios. The UDF function expands the pre-computation data role acquisition and function authority authentication modules, so that the calculation and authentication are combined into one. At the same time, by implementing the data role module of the UDF function based on a distributed configuration center, its local cache service can support high throughput under large-scale data computing. The UDF function authentication module adopts a microservice architecture, and its configuration center and registration center ensure high availability and data throughput of the service, thereby effectively improving the security call processing of the data warehouse function.

Description

Technical Field

[0001] The present invention belongs to the technical field of security engines, and in particular relates to a method for implementing a data warehouse function security engine. Background Art

[0002] With the popularization of information systems and the development of big data technology, the amount of data has shown a geometric growth. Therefore, ensuring data security has become a top priority. Information security is no longer limited to the security of the system itself, but more about data security.

[0003] Data security is the first problem that the data center must solve. Data security needs to start with data encryption, which usually refers to symmetric and asymmetric encryption. It means that encryption and decryption use the same key, and the same encryption algorithm is used for the key; for asymmetric encryption, it means that the keys used for encryption and decryption are different, and are asymmetric. One is a private key and the other is a public key. They are used in pairs, otherwise the encrypted file cannot be opened.

[0004] The importance of data encryption for data security is self-evident, but the encryption / decryption function cannot guarantee that the function itself will not be leaked, and multi-tenant management authorization is difficult. At the same time, the performance cannot be guaranteed to support PB-level data. The account keys authorized to users cannot be regularly hot-loaded and updated. In the process of data processing, how to solve the technical difficulties in the big data field such as how to connect the account keys authorized by functions with computing big data engines: Hive, Spark, Impala, etc. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that data encryption is of self-evident importance for data security, but the encryption / decryption function cannot guarantee that the function itself will not be leaked, and multi-tenant management authorization is difficult, and the performance cannot be well guaranteed to support PB-level data. The account keys authorized to users cannot be regularly hot-loaded and updated. In the process of data processing, how to solve the technical difficulties in the field of big data such as the account keys for function authorization and the computing big data engines: Hive, Spark, Impala, etc., and a method for implementing a data warehouse function security engine is proposed.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for implementing a data warehouse function security engine comprises the following steps:

[0008] S101. Build a data role management platform and define the relationship between "users - resources - permissions";

[0009] S102. Build a distributed configuration management system to provide services for account passwords;

[0010] S103. Build a function permission management platform and authentication microservices;

[0011] S 104. UDF function development, obtain management authority.

[0012] As a further description of the above technical solution:

[0013] The construction of the data role management platform specifically includes the management and processing of users, data roles and permissions.

[0014] As a further description of the above technical solution:

[0015] The distributed configuration management system includes providing account and password services for the UDF module. After the data role management platform defines the user, resource and permission information, it initializes the password information corresponding to the data role and synchronizes the data role and password information to the distributed configuration management system.

[0016] As a further description of the above technical solution:

[0017] The local client in the distributed configuration management system will cache the last configuration information and can support high-concurrency requests.

[0018] As a further description of the above technical solution:

[0019] Modifications to account passwords in the distributed configuration management system will also be synchronized to the client in a timely manner to ensure timely data updates and prevent single point failures.

[0020] As a further description of the above technical solution:

[0021] The UDF function development method includes:

[0022] S201. In the UDF function initialization phase, a user acquisition module is implemented. The module adopts a plug-in mechanism and acquires Linux local users, i.e., data roles, by default.

[0023] S202, in the UDF function initialization phase, scan the code package function name to obtain the function name of this call;

[0024] S203, obtaining the password information of the data role through the configuration center address;

[0025] S204: In the UDF function calculation engine calling phase, an application is initiated to the authentication service through "data role / password" and "UDF function".

[0026] As a further description of the above technical solution:

[0027] The function authority management platform includes the following service modules: a UDF function registration module including information on function name, type, calling syntax and return type, and a UDF function audit module including information on access records, application records and access frequency.

[0028] As a further description of the above technical solution:

[0029] The authentication microservice needs to pass in the user executing the UDF, i.e. the Linux local user, data role, password information and function information, and returns whether it has permission.

[0030] As a further description of the above technical solution:

[0031] The authentication microservice deploys multiple service nodes by adopting a microservice architecture system, publishes address information to a service registration center, and obtains the service address from the service registration center before the client module calls it, and then initiates the call.

[0032] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0033] In the present invention, through secondary encapsulation of UDF functions, the function authentication module is expanded by combining microservices and distributed configuration systems, so that high throughput, high concurrency and high availability capabilities are supported in massive data computing scenarios. The UDF function expands the pre-computation data role acquisition and function permission authentication modules, so that calculation and authentication are combined into one. At the same time, by implementing the data role module of the UDF function based on a distributed configuration center, its local cache service can support high throughput under large-scale data computing. The UDF function authentication module adopts a microservice architecture, and its configuration center and registration center ensure high availability and data throughput of the service, thereby effectively improving the secure call processing of data warehouse functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A structural diagram of a method for implementing a data warehouse function security engine proposed by the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Please refer to 1. The present invention provides a technical solution: a method for implementing a data warehouse function security engine, specifically comprising the following steps:

[0037] S101, build a data role management platform and define the relationship between "user-resource-authority". The building of the data role management platform specifically includes the management and processing of users, data roles and authorities;

[0038] Among them, user: is expressed by User or Group. User represents the user who accesses resources, and Group represents the user group to which the user belongs.

[0039] Resources: Different components correspond to different business resources, such as FilePath of HDFS, Table, Column-family, Column of HBase, Database, Table, Column of Hive, and Queue of Yarn. In this solution, the "resource" is the Linux local user running the UDF, not the real data staff.

[0040] Permissions: expressed by (Allow ACL, Deny ACL), similar to the whitelist and blacklist mechanism. Allow ACL is used to describe situations where access is allowed, and Deny ACL is used to describe situations where access is denied. Different components have different corresponding permissions. In this solution, permissions are the user's usage permissions as a local user of the server Linux. For example, the data worker user "Zhang San" has the permission to use the "work" data role of the server Linux. The data role management platform mainly implements the management of users, data roles (Linux local users), and permissions.

[0041] Among them, UDF is generally used in large-scale data calculation scenarios, so it is necessary to ensure the high availability and high throughput of the authentication module. The entire system uses microservices to provide function authentication services and a distributed configuration system to provide account and password services. The machine where UDF runs uses Linux local user authentication by default. In Simple mode, the client identity is determined by the host operating system. For example, in Unix-like systems, the user name is the output of the whoami command, which is consistent with the default simple authorization mode of the file system.

[0042] S102, build a distributed configuration management system to provide services for accounts and passwords, the distributed configuration management system includes providing account and password services for UDF modules, after the data role management platform defines the user, resource and permission information, it initializes the password information corresponding to the data role, and synchronizes the data role and password information to the distributed configuration management system, the local client in the distributed configuration management system will cache the last configuration information, can support high concurrent requests, the modification of the account and password in the distributed configuration management system will also be synchronized to the client in time, to ensure timely data update, and can prevent single point failure;

[0043] S103, build a function permission management platform and an authentication microservice, the function permission management platform includes the following service modules: UDF function registration module includes function name, type, calling syntax and return type information, UDF function audit module: includes access records, application records and access frequency information, the authentication microservice needs to pass in the user who executes the UDF, that is, the Linux local user, the data role, password information and function information, and returns whether it has permission. The authentication microservice deploys multiple service nodes by adopting a microservice architecture system, and publishes the address information to the service registration center. Before the client module calls, it first obtains the service address from the service registration center and then initiates the call;

[0044] S104: UDF function development, obtaining management authority, the UDF function development method includes:

[0045] In the UDF function initialization phase, the user acquisition module is implemented. This module adopts a plug-in mechanism and obtains Linux local users, that is, data roles, by default.

[0046] During the UDF function initialization phase, scan the function name of the code package to obtain the function name of this call;

[0047] Get the password information of the data role through the configuration center address;

[0048] During the UDF function calculation engine calling phase, an application is initiated to the authentication service through "data role / password" and "UDF function". The present invention solves the scenario of using functions in data processing. There are some sensitive functions, such as processing personal privacy data, which require unified permission management without affecting the performance of function use.

[0049] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for implementing a data warehouse function security engine, characterized in that: The specific steps include: S101. Build a data role management platform and define the relationship between "users - resources - permissions"; S102. Build a distributed configuration management system to provide services for account passwords; S103. Build a function permission management platform and authentication microservices; S104, UDF function development, obtain management authority; The UDF function development method includes: S201. In the UDF function initialization phase, a user acquisition module is implemented. The module adopts a plug-in mechanism and acquires Linux local users, i.e., data roles, by default. S202, in the UDF function initialization phase, scan the code package function name to obtain the function name of this call; S203, obtaining the password information of the data role through the configuration center address; S204: In the UDF function calculation engine call phase, an application is initiated to the authentication service through "data role / password" and "UDF function"; The function rights management platform includes the following service modules: UDF function registration module including information on function name, type, calling syntax and return type, UDF function audit module including information on access records, application records and access frequency; The authentication microservice needs to pass in the user executing the UDF, i.e. the Linux local user, data role, password information, and function information, and returns whether it has permission; The authentication microservice deploys multiple service nodes by adopting a microservice architecture system, publishes address information to a service registration center, and obtains the service address from the service registration center before the client module calls it, and then initiates the call.

2. A method for implementing a data warehouse function security engine according to claim 1, characterized in that: The construction of the data role management platform specifically includes the management and processing of users, data roles and permissions.

3. A method for implementing a data warehouse function security engine according to claim 1, characterized in that: The distributed configuration management system includes providing account and password services for the UDF module. After the data role management platform defines the user, resource and permission information, it initializes the password information corresponding to the data role and synchronizes the data role and password information to the distributed configuration management system.

4. A method for implementing a data warehouse function security engine according to claim 3, characterized in that: The local client in the distributed configuration management system will cache the last configuration information and can support high-concurrency requests.

5. The method for implementing a data warehouse function security engine according to claim 1, characterized in that: Modifications to account passwords in the distributed configuration management system will also be synchronized to the client in a timely manner to ensure timely data updates and prevent single point failures.

Images