Ranger-based data lake operation and maintenance permission control system and method

Abstract

The application provides a Ranger-based data lake operation and maintenance permission control system and method, and relates to the technical field of big data. The system comprises a data capture subsystem, a data logic strategy detection subsystem and a data processing subsystem; the data capture subsystem is used to call a timing scanning program to scan a big data cluster node installed with Ranger, so as to obtain a newly added permission strategy and write it into a graph database; the data logic strategy detection subsystem is used to call an artificial intelligence scout program to detect whether there is an abnormal permission strategy in the graph database; and the data processing subsystem is used to process the abnormal permission strategy when it exists. Through the connection of Ranger and the graph database, the application realizes the centralized storage and management of the permission strategy, and combines the artificial intelligence technology to detect whether there is an abnormality and process the abnormal permission strategy in time, so as to improve the security and reliability of the Ranger-based data lake operation and maintenance permission control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data lake operation and maintenance permission control system and method based on Ranger. BACKGROUND

[0002] With the rapid development of information technology, big data technology has become the core tool for data processing and analysis in various industries. As a data-intensive industry, large commercial banks face the demand for storage, processing and analysis of massive data. In order to effectively manage and utilize these data, it is inevitable to build a big data center. In the process of building a big data center, the security of data and the control of access permissions are crucial links. Apache Ranger, as an open source big data component, is widely used in the permission control of big data clusters. Ranger provides a centralized security management framework that can perform fine-grained permission control on multiple components in the Hadoop ecosystem. However, as the number of users increases and the complexity of permission policies increases, the number of Ranger entries also increases, making it more difficult to control permissions for Ranger. Changes in permission policies may have unpredictable effects on the normal operation of the system, and even cause logical contradictions and security risks.

[0003] Therefore, it is urgent to provide an effective Ranger-based data lake operation and maintenance permission control scheme to improve the security and reliability of Ranger-based data lake operation and maintenance permission control. SUMMARY

[0004] The present application provides a Ranger-based data lake operation and maintenance permission control system and method to improve the security and reliability of Ranger-based data lake operation and maintenance permission control.

[0005] In a first aspect, the present application provides a Ranger-based data lake operation and maintenance permission control system, comprising a data crawling subsystem, a data logic policy detection subsystem and a data processing subsystem; wherein:

[0006] The data crawling subsystem is configured to call a timing scanning program to scan a big data cluster node installed with Ranger to obtain newly added permission policies, and write the newly added permission policies into a graph database.

[0007] The data logic policy detection subsystem is configured to call an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, and the artificial intelligence scout program is set based on artificial intelligence technology.

[0008] The data processing subsystem is configured to process the abnormal permission policy when the abnormal permission policy exists in the graph database.

[0009] In a second aspect, the application provides a Ranger-based data lake operation and maintenance permission control method, applied to the Ranger-based data lake operation and maintenance permission control system as described in any one of the first aspect, the Ranger-based data lake operation and maintenance permission control method comprising:

[0010] invoking a timing scanning program to scan a big data cluster node installed with Ranger to obtain a newly added permission policy;

[0011] writing the newly added permission policy into a graph database;

[0012] invoking an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, the artificial intelligence scout program being set based on artificial intelligence technology;

[0013] when there is an abnormal permission policy in the graph database, processing the abnormal permission policy.

[0014] In a third aspect, the application provides a Ranger-based data lake operation and maintenance permission control device, applied to the Ranger-based data lake operation and maintenance permission control system as described in any one of the first aspect, the Ranger-based data lake operation and maintenance permission control device comprising:

[0015] a data capture module, configured to invoke a timing scanning program to scan a big data cluster node installed with Ranger to obtain a newly added permission policy, and write the newly added permission policy into a graph database;

[0016] a data logic policy detection module, configured to invoke an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database;

[0017] a data processing module, configured to, when there is an abnormal permission policy in the graph database, process the abnormal permission policy.

[0018] In a fourth aspect, the application provides an electronic device, comprising a processor and a memory in communication connection with the processor;

[0019] the memory is configured to store computer execution instructions;

[0020] the processor is configured to execute the computer execution instructions stored in the memory to implement the method of the second aspect.

[0021] In a fifth aspect, the application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, the computer execution instructions being executed to implement the method of the second aspect.

[0022] In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed, implements the method of the second aspect.

[0023] The system provided by the present application comprises a data capturing subsystem, a data logic strategy detection subsystem and a data processing subsystem. The data capturing subsystem is configured to call a timing scanning program to scan a big data cluster node installed with the Ranger, so as to obtain a newly added permission strategy, and write the newly added permission strategy into a graph database. The data logic strategy detection subsystem is configured to call an artificial intelligence scout program to detect whether there is an abnormal permission strategy in the graph database. The artificial intelligence scout program is set based on artificial intelligence technology. The data processing subsystem is configured to process the abnormal permission strategy when the abnormal permission strategy exists in the graph database. The data capturing subsystem is configured to scan the big data cluster node at a timing, automatically obtain the newly added permission strategy and write the newly added permission strategy into the graph database. The data logic strategy detection subsystem is configured to call the artificial intelligence scout program to intelligently analyze and detect the permission strategy in the graph database, so as to ensure that the potential abnormal permission strategy can be quickly detected. When the abnormal permission strategy is detected, the data processing subsystem is configured to quickly process the abnormal permission strategy, so as to ensure the normal operation of the system, reduce the logical contradiction and security risks caused by the change of the permission strategy, and improve the security and reliability of the permission control of the data lake operation and maintenance based on the Ranger. In addition, the system can store and manage the permission strategy in the graph database, so as to facilitate the query, analysis and maintenance of the permission strategy, improve the efficiency and maintainability of the permission control, and systematize and standardize the control process by using the automatic processing mode, thereby reducing the workload and error rate of manual work. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the present application.

[0025] Figure 1 A structural schematic diagram of the Ranger-based data lake operation and maintenance permission control system provided by the exemplary embodiments of the present application;

[0026] Figure 2 A flowchart of the Ranger-based data lake operation and maintenance permission control method provided by the exemplary embodiments of the present application;

[0027] Figure 3 Another flowchart of the Ranger-based data lake operation and maintenance permission control method provided by the exemplary embodiments of the present application;

[0028] Figure 4Another flow diagram of the method for controlling operation and maintenance permissions of a data lake based on a Ranger according to an example embodiment of the present application is shown in FIG. 6.

[0029] Figure 5 A structure diagram of the device for controlling operation and maintenance permissions of a data lake based on a Ranger according to an example embodiment of the present application is shown in FIG. 7.

[0030] Figure 6 A structure diagram of an electronic device according to an example embodiment of the present application is shown in FIG. 8.

[0031] The specific embodiments of the present application have been shown by the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated, in different drawings. The embodiments described in the following example embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0033] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and do not necessarily indicate a particular order or sequence. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present application described herein can be carried out in other than the order shown or described herein. Furthermore, the terms "comprise" and "have", and any variations thereof, are intended to cover non-exclusive inclusion, for example, processes, systems, products, or devices that include a series of steps or units not necessarily limited to those clearly listed, but can include other steps or units not clearly listed or inherent to such processes, products, or devices.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards, and provide corresponding operation portals for users to choose authorization or rejection.

[0035] First, the terms involved in the present application are explained:

[0036] Ranger: An open-source big data access management component, mainly used to provide fine-grained access control. Fine-grained access control refers to more detailed access management below the database level, such as table-level permissions and object storage directory permissions. This control allows more precise definition of user access to data resources.

[0037] Data lake: Generally refers to an enterprise-level data lake, which is a unified scheduling and management platform system for enterprise big data processing and operation. It is used to handle big data business scenarios in the data lifecycle dimension.

[0038] Graph database: A data storage and management system based on graph structure. Its data model consists of nodes (points) and edges, where nodes represent entities and edges represent relationships between entities. This type of database is particularly suitable for handling complex relational data, supporting efficient relationship queries and analysis.

[0039] In the related art, manual identification and processing of abnormal permission policies are usually relied on. However, with the increase in the number of users and the complexity of permission policies, the number of Ranger permission entries has significantly increased. This growth makes permission management more complex, increasing the difficulty of manually identifying and processing abnormal permissions. In addition, manually managing a large number of permission policies is not only time-consuming but also prone to errors. The system may not be able to timely detect potential abnormal permission policies, thereby affecting the security and reliability of data lake operation permission control.

[0040] To solve the above problems, the inventors found in the research process that some studies have proposed the application of graph databases to permission control, but only describe an application method and direction of graph databases. Whether the permission control in the graph database is reasonable still depends on manual judgment. Moreover, there is no clear guidance for how to apply it to the big data field. Specifically, how Ranger interfaces, how to identify abnormal permissions, whether to add enterprise internal permission policies, and how to detect abnormal permission policies are not involved. Therefore, managing permission policies through graph databases still lacks feasibility and practicality, and the function is simple, which cannot meet the current diverse permission policy needs.

[0041] To solve the above problems, the embodiment of the application provides a data lake operation and maintenance permission control scheme based on Ranger. By connecting Ranger with a graph database, centralized storage and management of permission policies are realized. In combination with artificial intelligence technology, an artificial intelligence scout program is called to detect whether there is an abnormal permission policy in the graph database. When there is an abnormal permission policy, the abnormal permission policy is processed in time to ensure the normal operation of the system, reduce logical contradictions and security risks caused by permission policy changes, and thus improve the security and reliability of the data lake operation and maintenance permission control based on Ranger. In addition, through the graph database, the system can uniformly store and manage permission policies, facilitating the query, analysis and maintenance of the permission policies, thereby improving the efficiency and maintainability of the permission control. In addition, the automatic processing mode can systematize and standardize the control process, reducing the workload and error rate of manual control.

[0042] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0043] Figure 1 A structural schematic diagram of a data lake operation and maintenance permission control system based on Ranger provided by an exemplary embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the data lake operation and maintenance permission control system 10 provided by the exemplary embodiment of the application includes a data grabbing subsystem 11, a data logic policy detection subsystem 12 and a data processing subsystem 13. Figure 1

[0044] The data grabbing subsystem 11 is configured to call a timing scanning program to scan a big data cluster node installed with Ranger to obtain a newly added permission policy, and write the newly added permission policy into a graph database.

[0045] ​The big data cluster node installed with the Ranger refers to a cluster node deployed with the Ranger, and the node uses the Ranger to manage and control access permissions to data. For example, an operator can set the scanning frequency of the timing scanner through a configuration file or a management interface, such as setting the scanning frequency to scan at midnight every day, and maintain a list containing all big data cluster nodes that need to be scanned to ensure that the timing scanner can traverse all relevant nodes. Correspondingly, the data crawling subsystem 11 invokes the timing scanner, which automatically executes at a set time interval, connects to each big data cluster node, invokes the Ranger API or directly accesses the Ranger to obtain the current permission policy, and identifies the newly added or changed permission policy since the last scan. Further, the obtained permission policy data is converted into a format recognizable by the graph database, such as converting the permission policy data into a language recognizable by the graph database, and converting it into graph database logic, which specifically includes mapping the permission policy to nodes and edges in the graph database to represent, for example, the relationship between users, resources, and permissions, and writing the converted permission policy data into the graph database to update the existing permission relationship graph.

[0046] It should be noted that the operator can modify the scanning frequency of the timing scanner according to the business scenario requirements, and the specific value of the scanning frequency is not limited here.

[0047] The data logic policy detection subsystem 12 is configured to invoke the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, and the artificial intelligence scout program is set based on artificial intelligence technology.

[0048] The artificial intelligence scout program uses artificial intelligence technology for monitoring and detection, aiming to identify and respond to abnormal situations or potential threats. For example, after the graph database is updated, the data logic policy detection subsystem 12 invokes the artificial intelligence scout program, detects whether there is an abnormal permission policy in the graph database according to the set detection rules, and sends the related information of the abnormal permission policy to the data processing subsystem 13 when an abnormal permission policy is detected.

[0049] The data processing subsystem 13 is configured to process the abnormal permission policy when the abnormal permission policy exists in the graph database.

[0050] Correspondingly, the data processing subsystem 13 is configured to trigger an automatic processing mechanism for the abnormal permission policy when the abnormal permission policy exists in the graph database, such as automatically triggering an alarm mechanism to timely notify relevant operators or security teams. The alarm can be sent through email, SMS, or other instant messaging tools, or according to a predefined policy, the system can automatically take measures such as temporarily disabling the abnormal permission policy or limiting the access permissions of related users, etc.

[0051] It should be noted that the Ranger-based data lake operation and maintenance permission control system 10 can be an application software deployed on a server, or the Ranger-based data lake operation and maintenance permission control system 10 can also be a server or a server cluster. Among them, the server can be replaced by a server cluster or other computing devices with certain computing power; in addition, the data crawling subsystem 11, the data logic policy detection subsystem 12 and the data processing subsystem 13 in the Ranger-based data lake operation and maintenance permission control system 10 can be deployed on a server, or independently deployed in a server cluster or other computing devices with certain computing power, which is not limited here.

[0052] The Ranger-based data lake operation and maintenance permission control system provided by the embodiments of the present application can automatically obtain and write the newly added permission policy into the graph database through the data crawling subsystem which scans the big data cluster node regularly; and through the data logic policy detection subsystem calling the artificial intelligence scout program, the permission policy in the graph database is intelligently analyzed and detected to ensure that the potential abnormal permission policy can be quickly detected; and when the abnormal permission policy is detected, the data processing subsystem is used to quickly process the abnormal permission policy to ensure the normal operation of the system, reduce the logical contradiction and security risks caused by the change of the permission policy, and improve the security and reliability of the Ranger-based data lake operation and maintenance permission control. In addition, through the graph database, the system can uniformly store and manage the permission policy, which is convenient for the query, analysis and maintenance of the permission policy, thereby improving the efficiency and maintainability of the permission control; and the automatic processing mode can systematize and standardize the control process, thereby reducing the workload and error rate of manual work.

[0053] In some embodiments, the artificial intelligence scout program includes a scout program and an artificial intelligence analysis program, and calling the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database includes: calling the scout program to traverse the rule library table and the permission library table in the graph database to detect whether there is an abnormal permission policy in the rule library table and the permission library table; and calling the artificial intelligence analysis program to use a pre-trained machine learning algorithm model to perform cross-cluster dimension analysis on the graph database to screen the abnormal permission policy in the graph database.

[0054] The permission library table is used to store all permission policies, and the rule library table is used to store related rules of the permission policies. Accordingly, after the graph database is updated, the data logic policy detection subsystem calls the scout program to traverse the rule library table and the permission library table in the graph database. In the traversal process, the scout program checks each rule and permission policy, identifies obvious abnormalities or inconsistencies, such as repeated permissions, conflicting rules, or unexpected permission levels, and so on. In order to avoid a large number of Cartesian product data generated after the rule library table and the permission library table are combined, which affects the system, a strategy of sacrificing space to extend time is usually adopted to ensure the stability of the system. When the contradictions between the rule library table and the permission library table are identified, the permission policies with contradictions or abnormalities are marked as exceptions in a timely manner. In addition, the data logic policy detection subsystem also calls an artificial intelligence analysis program to perform cross-cluster dimension analysis on the graph database by using a pre-trained machine learning algorithm model. This analysis considers the permission policy relationship between different clusters and can be used to identify potential abnormal patterns, such as unusual permission propagation or permission inconsistency between clusters. Further, the pre-trained machine learning algorithm model is used to perform in-depth analysis on the preliminary marked abnormal policies, and through pattern recognition and anomaly detection algorithms, the real abnormal permission policies are screened out, and the false positive abnormal permission policies are filtered out.

[0055] For example, the machine learning algorithm model is obtained in the following manner:

[0056] (1) Data preprocessing and feature extraction: an iterative self-organizing data analysis algorithm is used to detect outliers in the original data, so as to construct a mapping relationship between the characteristics of the outliers and the abnormal permission policies by an artificial neural network (ANN) in the subsequent step. A data cleaning algorithm is used to clean the detected outliers, so as to reduce the influence of noise data on the subsequent model analysis. Feature analysis and correlation analysis are performed on the original data, and features with low absolute value of Pearson correlation coefficient are removed, and a new feature index library is generated. The data cleaning algorithm includes at least one of abnormal value detection, missing value processing, smoothing processing, and filtering processing, but is not limited thereto.

[0057] (2) Analysis based on historical data: mathematical models and operation and maintenance experience are used to analyze abnormal permission policies, identify and label abnormal permission policies, and these labeled data are used as labels for model training. In this way, rich training data are provided for the machine learning algorithm model.

[0058] (3) Establishing a machine learning algorithm model: selecting a suitable machine learning algorithm model, such as a Radial Basis Function (RBF) for classification, or an ANN for fitting complex relationships, etc.; using labeled historical data for initial training to build the mapping relationship between outlier characteristics and abnormal permission policies; inputting the cleaned data into the model for training, for example, for RBF, the model classifies data points into different categories, for example, for ANN, the model fits the multivariate function relationship between each statistical indicator and the abnormal policy probability, etc.; during the training process, the model continuously adjusts internal parameters (such as weights and biases) to minimize prediction errors; and using optimization algorithms such as gradient descent, stochastic gradient descent, or Adam optimizer to adjust model parameters to improve the accuracy and generalization ability of the model; through cross-validation and hyperparameter tuning (such as adjusting learning rate and regularization parameters, etc.) to further optimize the performance of the model; further, the analysis results of new historical data are continuously used for incremental training of the model to ensure that the model can adapt to changes in permission policies and new abnormal patterns, and by regularly updating the model, its sensitivity and accuracy to new data can be ensured.

[0059] The embodiments of the present application realize multi-level detection of permission policies in a graph database through an artificial intelligence scout program, which can quickly identify abnormal permission policies and reduce potential security risks, thereby providing strong support for data lake operation and maintenance permission control based on Ranger. In addition, the machine learning algorithm model is effectively trained and optimized in the artificial intelligence scout program, further enhancing the ability of the artificial intelligence scout program to identify and detect abnormal permission policies.

[0060] In some embodiments, the data processing subsystem includes a permission control platform that processes abnormal permission policies, including: if the abnormal permission policy is an illegal permission policy, the permission control platform starts the scout program to shield the abnormal permission policy and / or generates an alarm information for the abnormal permission policy; if the execution result of the abnormal permission policy does not match the set expected result, the permission control platform executes the set policy correction program.

[0061] For example, when the abnormal permission policy is an illegal permission policy, the permission management platform initiates a scout procedure. Specifically, in one implementation, the scout procedure immediately shields the abnormal permission policy to prevent the abnormal permission policy from being executed, and the shielding operation may be temporary, for example, and whether to continue shielding or unshield may be determined in combination with the instruction of manual review; in another implementation, the scout procedure generates alarm information for the abnormal permission policy and sends it to the relevant operation and maintenance personnel; in yet another implementation, the scout procedure immediately shields the abnormal permission policy to prevent the abnormal permission policy from being executed, and simultaneously generates alarm information for the abnormal permission policy and sends it to the relevant operation and maintenance personnel. The alarm information may include, but is not limited to, a detailed description of the abnormal permission policy, the time of detection, the involved user or system component, etc.; the illegal permission policy may include, but is not limited to, at least one of the following: the permission grant exceeds the normal range of the user role, the permission policy conflicts with the enterprise security policy, and the execution frequency of the permission policy is abnormal.

[0062] Correspondingly, when the execution result of the abnormal permission policy does not match the set expected result, the permission management platform automatically triggers a policy correction procedure, which analyzes the difference between the execution result and the expected result and attempts to automatically adjust the permission policy to correct the deviation; all correction operations and analysis results are recorded in the system log, and a detailed report is generated for the operation and maintenance personnel to review, which may include, but is not limited to, analysis of abnormal behavior, description of correction measures, and subsequent suggestions, etc.; further, after the automated processing, the operation and maintenance personnel can manually review the abnormal permission policy, and determine whether further manual adjustment or additional security measures are needed according to the report and suggestions provided by the system.

[0063] The embodiments of the present application, through the automatic monitoring and abnormal processing mechanism, the system can effectively prevent potential security vulnerabilities and unauthorized access, and such a proactive defense mechanism helps to protect sensitive data and critical system resources; in addition, the automatic adjustment and processing mechanism reduces the need for human intervention, improves the response speed and accuracy of the system, and also reduces the complexity of manual configuration and monitoring, thereby reducing the possibility of human error and ensuring the accurate execution of the permission policy.

[0064] On the basis of the above-mentioned embodiments, in some embodiments, the data processing subsystem further includes a front-end page, and the front-end page is configured to display the alarm information.

[0065] By way of example, the permission management platform in the data processing subsystem continuously monitors system activities, and when an abnormal permission policy or other security event is detected, the permission management platform generates corresponding alarm information and transmits the generated alarm information to the front-end page in real time for display. The front-end page is usually built using modern World Wide Web (Web) technologies such as HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and JavaScript, and the page design can include but is not limited to elements such as a dashboard, an alarm list, filtering and search functions, etc. The dashboard is used to display all currently active alarm information and provides an intuitive overview of the system status. The alarm list can list all alarm information in a table format, allowing users or operation and maintenance personnel to click to view detailed information. The filtering and search functions allow users to filter and search alarm information according to time, severity, or type.

[0066] Correspondingly, the front-end page communicates with the permission management platform through an Application Programming Interface (API), and the specific steps can include: the front-end page sends a request to the back-end API periodically or upon user operation to obtain the latest alarm information; the back-end API responds to the request and returns the alarm data, and the front-end page dynamically updates the display content according to the returned data; for example, through WebSocket or other real-time communication technologies, the back-end API can actively push alarm updates to the front-end page to ensure that the user sees the latest alarm information.

[0067] The embodiments of the present application display alarm information in real time on the front-end page, allowing users to immediately understand the system status and potential problems, not only improving the real-time and visualization capabilities of the system, but also improving user experience and operation efficiency, and further providing strong support for efficient management and safe operation of the system.

[0068] In some embodiments, the permission management platform is also used to set alarm thresholds in the alarm policy library, different alarm thresholds corresponding to different alarm levels; the front-end page is also used to determine whether to display alarm information according to the alarm threshold; and when it is determined to display alarm information, the alarm level corresponding to the alarm information is displayed according to the alarm threshold.

[0069] For example, an alarm policy library is maintained by the permission management platform, wherein the alarm policy library contains alarm thresholds of multiple alarm types, each alarm type can set different alarm thresholds according to its importance and influence, and multiple alarm levels are defined according to the alarm thresholds, such as low, medium, high and urgent, and each alarm level can correspond to different response strategies. Correspondingly, when a system activity triggers a certain alarm threshold, the permission management platform generates corresponding alarm information; the front-end page determines whether to display the alarm information according to the alarm information and threshold settings provided by the permission management platform; and when it is determined to display the alarm information, the front-end page displays the alarm information with different visual effects (such as color and icon) according to the alarm level, so that the user can quickly identify its importance; further, the user can view detailed alarm information and historical records through the front-end page. In addition, different notification strategies can also be configured according to the alarm level, for example, alarm information of low and medium alarm levels usually does not send notifications; for high-level alarm information, relevant personnel are notified by email; for urgent-level alarm information, relevant personnel are notified by short message or instant message, etc.; such hierarchical notification strategy can ensure that important information can be timely delivered to relevant personnel. It should be noted that users or operation and maintenance personnel with corresponding permissions can also dynamically adjust the alarm threshold according to actual needs and application scenarios to ensure the flexibility and adaptability of the system.

[0070] In the embodiments of the present application, different alarm thresholds are set in the alarm policy library, and it is determined whether to display the alarm information according to the alarm threshold, so that the system can effectively filter out unnecessary alarms, reduce the risk of false positives and information overload, thereby ensuring that users focus on truly important events; in addition, the front-end page can also display alarm information with different visual effects according to the alarm level, which helps users quickly identify and prioritize important alarms, improves user operation efficiency and experience, and further improves the security and reliability of the data lake operation and maintenance permission control based on Ranger.

[0071] In the related art, no duplicate permission policies are allowed in each big data cluster node installed with Ranger, and the conditional filtering function is not supported to query the permission policy, so when an operation and maintenance personnel tries to add a permission policy, if informed that the permission policy already exists, the operation and maintenance personnel needs to check the content of each permission policy and make a manual judgment. In this case, it is very difficult for the operation and maintenance personnel to filter out the existing permission policy from a large number of permission policy entries. To solve this problem, the embodiments of the present application extend the functions based on Ranger, simplify the permission policy control process by integrating enhanced functions on the front-end page, which can include but are not limited to intelligent filtering and prompting, conditional query, etc., aiming to significantly reduce the work difficulty of the operation and maintenance personnel and improve their work efficiency.

[0072] Therefore, in some embodiments, the front-end page is further configured to display an operation result corresponding to the operation instruction in response to the operation instruction to the graph database, the operation instruction comprising at least one of adding, deleting, modifying, and querying.

[0073] For example, the intelligent filtering function is integrated on the front-end page, allowing the operation and maintenance personnel to automatically detect and prompt possible duplicate policies when adding or querying permission policies. This function can help operation and maintenance personnel quickly identify existing policies through condition matching and similarity analysis. In addition, the condition filtering function is provided on the front-end page, allowing operation and maintenance personnel to quickly query and filter permission policies according to specific conditions (such as policy name, resource type, permission level, etc.). In addition, the front-end page can display the operation result in real time in response to the operation instruction to the graph database. Through the real-time feedback mechanism, operation and maintenance personnel can quickly understand the success or failure of the operation without relying on manual checking of permission policy content and manual judgment, greatly reducing the workload of manual judgment, thereby significantly improving work efficiency and reducing the difficulty of Ranger-based data lake operation and maintenance permission control.

[0074] In related technologies, when changing permission policies in a production environment, even if the permission policies to be changed have been fully tested and verified in a test environment, due to the differences between the production environment and the test environment, potential delays or logical conflicts may still not be identified. Such differences may adversely affect the stable operation of the current system after adding permission policies.

[0075] Therefore, in some embodiments, the front-end page includes a pre-operation window; the data extraction subsystem is further configured to obtain a simulated permission policy in the pre-operation window, and write the simulated permission policy into the graph database; the data logic policy detection subsystem is further configured to call the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy; the data processing subsystem is further configured to display alarm information corresponding to the simulated permission policy when the simulated permission policy is an abnormal permission policy, and delete the simulated permission policy in the graph database; and further configured to display a prompt information that the simulated permission policy is a normal permission policy when the simulated permission policy is a normal permission policy, and delete the simulated permission policy in the graph database.

[0076] For example, when the operation and maintenance personnel are uncertain whether the added permission policy will have adverse effects, the operation and maintenance personnel input a simulation permission policy through a pre-operation window on a front-end page; the data grabbing subsystem is responsible for obtaining the simulation permission policy in the pre-operation window and writing the simulation permission policy into a graph database, and this process can ensure that the simulation operation does not directly affect Ranger or the production environment; further, the data logic policy detection subsystem calls an artificial intelligence scout program to detect whether the simulation permission policy is an abnormal permission policy, and through intelligent analysis, the data logic policy detection subsystem identifies potential policy conflicts or logic errors in advance; correspondingly, when the simulation permission policy is detected as an abnormal permission policy, the front-end page displays corresponding alarm information, and the simulation permission policy is deleted from the graph database, so as to ensure that the production environment is not affected; when the simulation permission policy is detected as a normal policy, the front-end page displays prompt information that the simulation permission policy is a normal permission policy, and the simulation permission policy is also deleted from the graph database.

[0077] According to the embodiments of the present application, the operation and maintenance personnel can simulate the addition of a permission policy in the production environment through a pre-operation window on a front-end page, so as to obtain a real and reliable verification result, which effectively reduces the difficulty of identifying potential delays or logic conflicts, thereby significantly reducing the potential risk to the production environment.

[0078] In some embodiments, the front-end page further includes an actual operation window; and the data grabbing subsystem is further configured to obtain a target permission policy in the actual operation window, and write the target permission policy into a target node in a big data cluster node in which the Ranger is installed; call a timing scanning program to scan the target node to obtain the target permission policy; and write the target permission policy into the graph database.

[0079] For example, after confirming the safety of the permission policy, the operation and maintenance personnel can input a target permission policy in the actual operation window to make a formal policy change; correspondingly, the data grabbing subsystem is responsible for obtaining the target permission policy from the actual operation window and writing the target permission policy into a target node in a big data cluster node in which the Ranger is installed; further, the data grabbing subsystem calls a timing scanning program to periodically scan the target node to obtain the target permission policy and write the target permission policy into the graph database.

[0080] According to the embodiments of the present application, the target permission policy can be directly applied to the production environment through the actual operation window, which reduces the manual operation steps and simplifies the operation and maintenance process, thereby improving the work efficiency and reducing the possibility of human errors.

[0081] In summary, the present application has at least the following advantages:

[0082] I. The data crawling subsystem scans the big data cluster nodes regularly, automatically obtains and writes the newly added permission policies into the graph database. Through the data logic policy detection subsystem, the artificial intelligence scout program is called to intelligently analyze and detect the permission policies in the graph database, ensuring that potential abnormal permission policies can be quickly detected. When abnormal permission policies are detected, the data processing subsystem quickly processes the abnormal permission policies to ensure the normal operation of the system, reducing the logical contradictions and security risks caused by changes in permission policies, thereby improving the security and reliability of the data lake operation and maintenance permission control based on Ranger. In addition, through the graph database, the system can uniformly store and manage the permission policies, facilitating the query, analysis and maintenance of the permission policies, thereby improving the efficiency and maintainability of the permission control. In addition, the automated processing mode can systematize and standardize the control process, reducing the workload and error rate of manual work.

[0083] II. The artificial intelligence scout program realizes multi-level detection of the permission policies in the graph database, which can quickly identify abnormal permission policies and reduce potential security risks, thereby providing strong support for the data lake operation and maintenance permission control based on Ranger. In addition, the machine learning algorithm model is effectively trained and optimized in the artificial intelligence scout program, further enhancing the ability of the artificial intelligence scout program to identify and detect abnormal permission policies.

[0084] III. Through the interaction between Ranger and the front-end page, the front-end page can display the operation results of the operation instructions for the graph database in real time. Through the real-time feedback mechanism, the operation personnel can quickly understand the success or failure of the operation without relying on manual checking of the permission policy content and manual judgment, greatly reducing the workload of manual judgment, thereby significantly improving the work efficiency and reducing the difficulty of the data lake operation and maintenance permission control based on Ranger. In addition, the front-end page can also display alarm information in real time, so that users can immediately understand the system status and potential problems, not only improving the real-time performance and visualization ability of the system, but also enhancing the user experience and operation efficiency, further providing strong support for the efficient management and safe operation of the system.

[0085] IV. Through the pre-operation window on the front-end page, the operation personnel are allowed to simulate the addition of permission policies in the production environment, thereby obtaining real and reliable verification results. This mechanism effectively reduces the difficulty of identifying potential delays or logical conflicts, thereby significantly reducing the potential risks to the production environment.

[0086] The above embodiments introduce the implementation method of the data lake operation and maintenance permission control system based on Ranger. Next, the application of the data lake operation and maintenance permission control system based on Ranger will be introduced through specific embodiments.

[0087] Exemplarily, Figure 2 A flowchart of a method for controlling permissions of data lake operation and maintenance based on Ranger is provided for an exemplary embodiment of the present application. The method for controlling permissions of data lake operation and maintenance based on Ranger provided by the embodiment of the present application is applied to the system for controlling permissions of data lake operation and maintenance based on Ranger described in any of the above embodiments. As shown in the figure, Figure 2 the method for controlling permissions of data lake operation and maintenance based on Ranger includes the following steps:

[0088] S201, calling a timing scanning program to scan a big data cluster node installed with Ranger to obtain newly added permission policies.

[0089] Exemplarily, the timing scanning program is automatically executed at a set time interval, automatically connects to each big data cluster node, obtains the current permission policies by calling Ranger API or directly accessing Ranger, and identifies the newly added or changed permission policies since the last scan.

[0090] S202, writing the newly added permission policies into a graph database.

[0091] Exemplarily, the newly added permission policies are converted into a language recognizable by the graph database, and are converted into graph database logic, specifically including mapping the newly added permission policies into nodes and edges in the graph database to represent the relationship between, for example, users, resources and permissions, and writing the converted permission policy data into the graph database to update the existing permission relationship graph.

[0092] S203, calling an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, the artificial intelligence scout program being set based on artificial intelligence technology.

[0093] Exemplarily, the scout program is called to traverse the rule library table and the permission library table in the graph database to detect whether there is an abnormal permission policy in the rule library table and the permission library table; and an artificial intelligence analysis program is called to perform cross-cluster dimension analysis on the graph database using a pre-trained machine learning algorithm model to screen the abnormal permission policy in the graph database.

[0094] S204, processing the abnormal permission policy when there is an abnormal permission policy in the graph database.

[0095] Exemplarily, if the abnormal permission policy is an irregular permission policy, the scout program is started to shield the abnormal permission policy, alarm information is generated for the abnormal permission policy, and the front-end page is controlled to display the related alarm information; if the execution result of the abnormal permission policy does not match the set expected result, a set policy correction program is executed, the difference between the execution result and the expected result is analyzed through the policy correction program, and the permission policy is automatically adjusted to correct the deviation.

[0096] Figure 3 Another flowchart of the Ranger-based data lake operation and maintenance permission control method provided for the exemplary embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the Ranger-based data lake operation and maintenance permission control method includes the following steps: Figure 3

[0097] S301, obtaining a simulation permission policy in a pre-operation window.

[0098] S302, writing the simulation permission policy into a graph database.

[0099] S303, calling an artificial intelligence scout program to detect whether the simulation permission policy is an abnormal permission policy.

[0100] If yes, S304 is performed;

[0101] If no, S305 is performed.

[0102] S304, displaying alarm information corresponding to the simulation permission policy on a front-end page, and deleting the simulation permission policy in the graph database.

[0103] S305, displaying prompt information that the simulation permission policy is a normal permission policy, and deleting the simulation permission policy in the graph database.

[0104] In the embodiments of the present application, the simulation permission policy in the pre-operation window is obtained, and the simulation permission policy is simulated and verified in the production environment, so that the verification result obtained is real and reliable, effectively reducing the identification difficulty of potential delay or logical conflict and other problems, thereby significantly reducing the potential risk to the production environment.

[0105] Figure 4 Another flowchart of the Ranger-based data lake operation and maintenance permission control method provided for the exemplary embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the Ranger-based data lake operation and maintenance permission control method includes the following steps: Figure 4

[0106] S401, obtaining a target permission policy in a real operation window.

[0107] S402, writing the target permission policy into a target node in a big data cluster node installed with Ranger.

[0108] It should be noted that the target node can be one or multiple, and the number of the target node in the big data cluster node installed with Ranger is not limited herein.

[0109] S403, calling a timing scanning program to scan the target node to obtain the target permission policy.​​

[0110] S404, write the target permission policy into the graph database.

[0111] S405, invoke the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database.

[0112] If yes, perform S406;

[0113] If no, perform S407.

[0114] S406, trigger an automatic processing mechanism for the abnormal permission policy.

[0115] For example, start the scout program to shield the abnormal permission policy; or generate an alarm information for the abnormal permission policy, and display the related alarm information on the front-end page.

[0116] S407, determine that the target permission policy is a normal permission policy.

[0117] Correspondingly, by displaying the prompt information that the target permission policy is a normal permission policy through the front-end page, the operation and maintenance personnel can be timely notified that the policy has been successfully applied and no abnormality has been detected.

[0118] In the embodiments of the present application, the target permission policy in the real operation window is acquired, and the target permission policy is directly applied to the production environment, thereby reducing the manual operation steps and simplifying the operation and maintenance process, so as to improve the work efficiency and reduce the possibility of human error.

[0119] Figure 5 A structural schematic diagram of a data lake operation and maintenance permission control device based on Ranger provided by an exemplary embodiment of the present application is shown in FIG. 1. Figure 5 As shown in the figure, the data lake operation and maintenance permission control device 50 based on Ranger includes a data grabbing module 51, a data logic policy detection module 52, and a data processing module 53, wherein:

[0120] The data grabbing module 51 is configured to invoke a timing scanning program to scan a big data cluster node installed with Ranger to acquire a newly added permission policy; and write the newly added permission policy into a graph database.

[0121] The data logic policy detection module 52 is configured to invoke an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database.

[0122] The data processing module 53 is configured to process the abnormal permission policy when the abnormal permission policy exists in the graph database.

[0123] The device for controlling the operation and maintenance permissions of the data lake based on the Ranger provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments for controlling the operation and maintenance permissions of the data lake based on the Ranger, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0124] It should be noted that the division of each module of the above device is only a logical division of functions, and all or part of the modules can be integrated into one physical entity or physically separated when actually implemented. The modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the data processing module can be a separately established processing element, or can be integrated into a chip of the above device, and in addition, the data processing module can be stored in the form of program code in the memory of the above device, and the function of the data processing module can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of the modules can be integrated together or independently implemented. The processing element herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of the hardware or the instruction of the software in the processing element.

[0125] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of program code called by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, the modules can be integrated together to implement in the form of system on a chip (SOC).

[0126] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as digital versatile disc (Digital Video Disc, DVD)) or semiconductor media (such as solid state disk (solid state disk, SSD)) and the like.

[0127] Figure 6 The structural schematic diagram of the electronic device provided in the exemplary embodiments of the present application is shown in the figure. As shown in the figure, the electronic device 60 of the present embodiment includes: Figure 6

[0128] at least one processor 61; and a memory 62 connected in communication with the at least one processor;

[0129] The memory 62 stores instructions executable by the at least one processor 61, and the instructions are executed by the at least one processor 61 to enable the electronic device to perform the method according to any one of the above embodiments.

[0130] Optionally, the memory 62 can be independent or integrated with the processor 61.

[0131] The memory 62 can include a high-speed random access memory (Random Access Memory, RAM) and can also include a non-volatile memory such as at least one disk memory.

[0132] ​The processor 61 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to perform the operations of the embodiments of the present application. Specifically, when implementing the method of controlling the operation and maintenance permissions of the Ranger-based data lake described in the foregoing method embodiments, the electronic device can be a server or other electronic device having a processing function.

[0133] Optionally, the electronic device can further include a communication interface 63. In a specific implementation, if the communication interface 63, the memory 62, and the processor 61 are implemented independently, the communication interface 63, the memory 62, and the processor 61 can be connected to each other through a bus and complete communication between each other. The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0134] Optionally, in a specific implementation, if the communication interface 63, the memory 62, and the processor 61 are integrated on a chip, the communication interface 63, the memory 62, and the processor 61 can complete communication through an internal interface.

[0135] The implementation principle and technical effects of the electronic device provided by the embodiments of the present application can be referred to the foregoing embodiments, which will not be described here.

[0136] The embodiments of the present application also provide a computer readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed, the computer execution instructions are used to implement the method steps in the method embodiments described above. The specific implementation and technical effects are similar, which will not be described here.

[0137] The computer readable storage medium described above can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0138] An exemplary readable storage medium is coupled to the processor, thereby enabling the processor to read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be part of the processor. The processor and the readable storage medium can be located in a special integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in the Ranger-based data lake operation permission control device.

[0139] The embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed, the method steps in the above method embodiments are implemented. The specific implementation manners and technical effects are similar, and will not be described here.

[0140] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0141] It should be understood that many of the materials and devices exemplified in this disclosure are articles of manufacture (i.e., articles of manufacture) according to this disclosure. The articles of manufacture can be employed as such to provide advantageous results. The articles of manufacture can also be employed in the practice or testing of the present disclosure. The articles of manufacture can also be employed in the practice or testing of other examples of the present disclosure. The articles of manufacture employed as such for realizing the advantageous results described herein, for practicing the instant disclosure, and for practicing other examples of the instant disclosure will vary depending on the specific context.

Claims

1. A data lake operation and maintenance access control system based on Ranger, characterized in that, It includes a data acquisition subsystem, a data logic strategy detection subsystem, and a data processing subsystem; among which: The data capture subsystem is used to call a timed scanning program to scan big data cluster nodes with Ranger installed in order to obtain new permission policies and write the new permission policies into the graph database. The data logic policy detection subsystem is used to call an artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database. The artificial intelligence scout program is set based on artificial intelligence technology. The data processing subsystem is used to process abnormal permission policies when abnormal permission policies exist in the graph database.

2. The system according to claim 1, characterized in that, The AI ​​scout program includes a scout program and an AI analysis program. The step of calling the AI ​​scout program to detect whether there are abnormal permission policies in the graph database includes: The scout program is invoked to traverse the rule base table and permission base table in the graph database to detect whether there are any abnormal permission policies in the rule base table and the permission base table. In addition, the artificial intelligence analysis program is invoked to perform cross-cluster dimension analysis on the graph database using a pre-trained machine learning algorithm model in order to filter out abnormal permission policies in the graph database.

3. The system according to claim 2, characterized in that, The data processing subsystem includes an access control platform, and the processing of the abnormal access policies includes: If the abnormal permission policy is a non-compliant permission policy, the permission management platform will activate the scout program to block the abnormal permission policy and / or generate alarm information for the abnormal permission policy. If the execution result of the abnormal permission policy does not match the set expected result, the permission management platform will execute the set policy correction procedure.

4. The system according to claim 3, characterized in that, The data processing subsystem also includes a front-end page, which is used to display the alarm information.

5. The system according to claim 4, characterized in that, The access control platform is also used to set alarm thresholds in the alarm policy library, with different alarm thresholds corresponding to different alarm levels; The front-end page is also used to determine whether to display the alarm information based on the alarm threshold; and when it is determined to display the alarm information, to display the alarm level corresponding to the alarm information based on the alarm threshold.

6. The system according to claim 4 or 5, characterized in that, The front-end page is also used to: display the operation result corresponding to the operation instruction when responding to the operation instruction for the graph database, wherein the operation instruction includes at least one of adding, deleting, modifying and querying.

7. The system according to claim 4 or 5, characterized in that, The front-end page includes a pre-operation window; the data capture subsystem is also used to obtain the simulated permission policy in the pre-operation window and write the simulated permission policy into the graph database; The data logic policy detection subsystem is also used to call the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy; The data processing subsystem is further configured to display alarm information corresponding to the simulated permission policy and delete the simulated permission policy from the graph database when the simulated permission policy is an abnormal permission policy; and to display a prompt message indicating that the simulated permission policy is a normal permission policy and delete the simulated permission policy from the graph database when the simulated permission policy is a normal permission policy.

8. The system according to claim 4 or 5, characterized in that, The front-end page also includes a real-operation window; The data capture subsystem is also used to obtain the target permission policy in the actual operation window, and to write the target permission policy into the target node in the big data cluster node with Ranger installed; to call a timed scanning program to scan the target node to obtain the target permission policy; and to write the target permission policy into the graph database.

9. A data lake operation and maintenance access control method based on Ranger, characterized in that, The Ranger-based data lake operation and maintenance access control method, applied to any one of claims 1 to 8, comprises: Call the scheduled scanning program to scan the big data cluster nodes with Ranger installed in order to obtain the newly added permission policies; Write the newly added permission policy into the graph database; An AI scout program is invoked to detect whether there are any abnormal permission policies in the graph database. The AI ​​scout program is set up based on artificial intelligence technology. When an abnormal permission policy exists in the graph database, the abnormal permission policy is processed.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory is used to store computer-executed instructions; The processor is configured to execute the computer execution instructions to implement the method as described in claim 9.

Images