Ranger-based data lake operation and maintenance right control system and method

Abstract

The invention provides a Ranger-based data lake operation and maintenance right control system and method, and relates to the technical field of big data. The system comprises a data capture subsystem, a data logic strategy detection subsystem and a data processing subsystem, the data capture subsystem is used for calling a timed scanning program to scan big data cluster nodes provided with Rangers so as to obtain a newly added authority strategy and write the newly added authority strategy into a graph database; the data logic strategy detection subsystem is used for calling an artificial intelligence repulsion program to detect whether an abnormal permission strategy exists in a graph database or not; and the data processing subsystem is used for processing the abnormal permission strategy when the abnormal permission strategy exists. According to the application, the Ranger is in butt joint with the graph database, centralized storage and management of authority strategies are realized, and meanwhile, whether abnormity exists or not is detected in combination with an artificial intelligence technology, and the abnormal authority strategies are processed in time, so that the security and reliability of Ranger-based data lake operation and maintenance authority control are improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a Ranger-based data lake operation and maintenance authority control system and method. Background Art

[0002] With the rapid development of information technology, big data technology has become a core tool for data processing and analysis in various industries. As a data-intensive industry, large commercial banks are faced with the need to store, process and analyze massive amounts of data. In order to effectively manage, process and utilize these data, building a big data center has become an inevitable choice. In the process of building a big data center, data security and access control are crucial links. As an open source big data component, Apache Ranger is widely used in the permission control of big data clusters. Ranger provides a centralized security management framework that can perform fine-grained permission control on multiple components in the Hadoop ecosystem. However, with the increase in the number of users and the complexity of permission policies, the number of Ranger entries has also increased, making it more difficult to control permissions for Ranger. Changes in permission policies may have unforeseen effects on the normal operation of the system, and even cause logical contradictions and security risks.

[0003] Therefore, there is an urgent need to provide an effective Ranger-based data lake operation and maintenance permission control solution to improve the security and reliability of Ranger-based data lake operation and maintenance permission control. Summary of the invention

[0004] The present application provides a Ranger-based data lake operation and maintenance permission control system and method to improve the security and reliability of Ranger-based data lake operation and maintenance permission control.

[0005] In a first aspect, the present application provides a data lake operation and maintenance authority control system based on Ranger, including a data capture subsystem, a data logic strategy detection subsystem and a data processing subsystem; wherein:

[0006] The data crawling subsystem is used to call the scheduled scanning program to scan the big data cluster nodes where Ranger is installed to obtain the newly added permission policies and write the newly added permission policies into the graph database;

[0007] The data logic policy detection subsystem is used to call the artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database. The artificial intelligence scout program is set based on artificial intelligence technology;

[0008] The data processing subsystem is used to process abnormal permission policies when there are abnormal permission policies in the graph database.

[0009] In a second aspect, the present application provides a Ranger-based data lake operation and maintenance authority control method, which is applied to the Ranger-based data lake operation and maintenance authority control system as described in any one of the first aspects. The Ranger-based data lake operation and maintenance authority control method includes:

[0010] Call the scheduled scanning program to scan the big data cluster nodes where Ranger is installed to obtain the newly added permission policies;

[0011] Write the newly added permission policy into the graph database;

[0012] Call the AI ​​scout program to detect whether there are abnormal permission policies in the graph database. The AI ​​scout program is set based on AI technology;

[0013] When there are abnormal permission policies in the graph database, the abnormal permission policies are processed.

[0014] In a third aspect, the present application provides a Ranger-based data lake operation and maintenance authority control device, which is applied to the Ranger-based data lake operation and maintenance authority control system as described in any one of the first aspects, and the Ranger-based data lake operation and maintenance authority control device includes:

[0015] The data capture module is used to call the scheduled scanning program to scan the big data cluster nodes installed with Ranger to obtain the newly added permission policies; and write the newly added permission policies into the graph database;

[0016] The data logic policy detection module is used to call the artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database;

[0017] The data processing module is used to process abnormal permission policies when there are abnormal permission policies in the graph database.

[0018] In a fourth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0019] Memory for storing computer-executable instructions;

[0020] A processor is used to execute computer-executable instructions stored in the memory to implement the method described in the second aspect.

[0021] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed, they are used to implement the method described in the second aspect.

[0022] In a sixth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the second aspect when executed.

[0023] The present application provides a Ranger-based data lake operation and maintenance permission control system and method, wherein the system includes a data capture subsystem, a data logic policy detection subsystem and a data processing subsystem; the data capture subsystem is used to call a scheduled scanning program to scan the big data cluster nodes installed with Ranger to obtain the newly added permission policy, and write the newly added permission policy into the graph database; the data logic policy detection subsystem is used to call an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, and the artificial intelligence scout program is set based on artificial intelligence technology; the data processing subsystem is used to process the abnormal permission policy when there is an abnormal permission policy in the graph database. The present application uses a data capture subsystem to regularly scan the big data cluster nodes, automatically obtain and write the newly added permission policy into the graph database; and calls the artificial intelligence scout program through the data logic policy detection subsystem to intelligently analyze and detect the permission policy in the graph database to ensure that potential abnormal permission policies can be quickly detected; and when an abnormal permission policy is detected, the abnormal permission policy is quickly processed by the data processing subsystem to ensure the normal operation of the system, reducing the logical contradictions and security risks caused by the change of permission policies, thereby improving the security and reliability of the data lake operation and maintenance permission control based on Ranger. In addition, through the graph database, the system can uniformly store and manage permission policies, which facilitates the query, analysis and maintenance of permission policies, thereby improving the efficiency and maintainability of permission control; and the automated processing method can systematize and standardize the control process, reducing manual workload and error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] Figure 1 A schematic diagram of a structure of a data lake operation and maintenance authority control system based on Ranger provided for an exemplary embodiment of the present application;

[0026] Figure 2 A flow chart of a Ranger-based data lake operation and maintenance authority control method provided for an exemplary embodiment of the present application;

[0027] Figure 3 Another flowchart of a Ranger-based data lake operation and maintenance authority control method provided for an exemplary embodiment of the present application;

[0028] Figure 4Another flow chart of a Ranger-based data lake operation and maintenance authority control method provided for an exemplary embodiment of the present application;

[0029] Figure 5 A schematic diagram of a structure of a Ranger-based data lake operation and maintenance authority control device provided for an exemplary embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application.

[0031] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products or equipment.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] First, the terms involved in this application are explained:

[0036] Ranger: An open source big data permission management component that is mainly used to provide fine-grained permission control. Fine-grained permission control refers to more sophisticated permission management below the database level, such as table-level permissions and directory permissions for object storage. This control method allows users to more accurately define access rights to data resources.

[0037] Data lake: usually refers to an enterprise-level data lake, which is a unified scheduling and management platform system for enterprises to process and operate big data, and is used to process big data business scenarios in the data life cycle dimension.

[0038] Graph database: a data storage and management system based on graph structure. Its data model consists of nodes (points) and edges, where nodes represent entities and edges represent the relationship between entities. This type of database is particularly suitable for processing complex relational data and supports efficient relational query and analysis.

[0039] In related technologies, abnormal permission policies are usually identified and processed manually. However, with the increase in the number of users and the complexity of permission policies, the number of Ranger permission items has increased significantly. This growth has made permission management more complicated and increased the difficulty of manually identifying and processing abnormal permissions. Manually managing a large number of permission policies is not only time-consuming but also prone to errors. The system may not be able to detect potential abnormal permission policies in a timely manner, which in turn affects the security and reliability of data lake operation and maintenance permission control.

[0040] In response to the above problems, the inventors found during the research process that some studies proposed applying graph databases to permission control, but they only described one application method and application direction of graph databases. Whether the permission control in the graph database is reasonable still depends on manual judgment; and there is no clear guidance on how to apply it to the big data field. Specifically, how Ranger is connected, how to identify abnormal permissions, whether the internal permission policies of the enterprise can be attached, and how to detect abnormal permission policies are not involved. Therefore, managing permission policies through graph databases is still not feasible and implementable, and the functions are simple and cannot meet the current diverse permission policy needs.

[0041] In order to solve the above problems, the embodiment of the present application provides a data lake operation and maintenance permission control solution based on Ranger, which realizes the centralized storage and management of permission policies by connecting Ranger with the graph database; and combines artificial intelligence technology to call the artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database, and when there are abnormal permission policies, the abnormal permission policies are processed in time to ensure the normal operation of the system, reduce the logical contradictions and security risks caused by permission policy changes, and thus improve the security and reliability of data lake operation and maintenance permission control based on Ranger. In addition, through the graph database, the system can uniformly store and manage permission policies, which is convenient for querying, analyzing and maintaining permission policies, thereby improving the efficiency and maintainability of permission control; and the automated processing method can systematize and standardize the control process, reducing the workload and error rate of manual control.

[0042] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0043] Figure 1 A schematic diagram of a data lake operation and maintenance authority control system based on Ranger provided in an exemplary embodiment of the present application. Figure 1 As shown, the Ranger-based data lake operation and maintenance authority control system 10 provided by the exemplary embodiment of the present application includes a data capture subsystem 11, a data logic strategy detection subsystem 12 and a data processing subsystem 13; wherein:

[0044] The data capture subsystem 11 is used to call the scheduled scanning program to scan the big data cluster nodes installed with Ranger to obtain the newly added permission policies and write the newly added permission policies into the graph database.

[0045] Among them, the big data cluster node with Ranger installed refers to the cluster node with Ranger deployed, which uses Ranger to manage and control access rights to data. For example, the operation and maintenance personnel can set the scanning frequency of the scheduled scanning program through the configuration file or management interface, such as setting the scanning frequency to scan at midnight every day; and maintain a list of all big data cluster nodes that need to be scanned to ensure that the scheduled scanning program can traverse all relevant nodes. Accordingly, the data capture subsystem 11 calls the scheduled scanning program, which is automatically executed within the set time interval, connects to each big data cluster node, calls the Ranger API or directly accesses Ranger to obtain the current permission policy, and identifies the newly added or changed permission policies since the last scan; further, the obtained permission policy data is converted into a format recognizable by the graph database, such as converting the permission policy data into a language recognizable by the graph database, and converting it into the graph database logic, specifically including mapping the permission policy to nodes and edges in the graph database to represent, for example, the relationship between users, resources and permissions, and writing the converted permission policy data into the graph database to update the existing permission relationship graph.

[0046] It should be noted that the operation and maintenance personnel can modify the scanning frequency of the scheduled scanning program according to the business scenario requirements, and the specific value of the scanning frequency is not limited here.

[0047] The data logic policy detection subsystem 12 is used to call the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database. The artificial intelligence scout program is set based on artificial intelligence technology.

[0048] Among them, the artificial intelligence scout program uses artificial intelligence technology to monitor and detect, aiming to identify and respond to abnormal situations or potential threats. For example, after the graph database is updated, the data logic policy detection subsystem 12 calls the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database according to the set detection rules, and when an abnormal permission policy is detected, the relevant information of the abnormal permission policy is sent to the data processing subsystem 13.

[0049] The data processing subsystem 13 is used to process abnormal permission policies when there are abnormal permission policies in the graph database.

[0050] Correspondingly, the data processing subsystem 13 is used to trigger an automatic processing mechanism for abnormal permission policies when there are abnormal permission policies in the graph database, such as automatically triggering an alarm mechanism to promptly notify relevant operation and maintenance personnel or security teams. The alarm can be sent via email, SMS or other instant messaging tools; or, according to predefined policies, the system can automatically take measures, such as temporarily disabling abnormal permission policies or limiting the access rights of relevant users.

[0051] It should be noted that the Ranger-based data lake operation and maintenance authority control system 10 can be an application software deployed on a server; or the Ranger-based data lake operation and maintenance authority control system 10 can also be a server or a server cluster. Among them, the server can be replaced by a server cluster or other computing devices with a certain computing power; in addition, the data capture subsystem 11, data logic strategy detection subsystem 12 and data processing subsystem 13 in the Ranger-based data lake operation and maintenance authority control system 10 can be deployed on a server, or independently deployed in a server cluster or other computing devices with a certain computing power, which is not limited here.

[0052] The Ranger-based data lake operation and maintenance permission control system provided in the embodiment of the present application periodically scans the big data cluster nodes through the data capture subsystem, automatically obtains and writes the newly added permission policies into the graph database; and calls the artificial intelligence scout program through the data logic policy detection subsystem to intelligently analyze and detect the permission policies in the graph database to ensure that potential abnormal permission policies can be quickly detected; and when abnormal permission policies are detected, the abnormal permission policies are quickly processed through the data processing subsystem to ensure the normal operation of the system, reduce the logical contradictions and security risks caused by the change of permission policies, thereby improving the security and reliability of the Ranger-based data lake operation and maintenance permission control. In addition, through the graph database, the system can uniformly store and manage permission policies, facilitate the query, analysis and maintenance of permission policies, thereby improving the efficiency and maintainability of permission control; and the automated processing method can systematize and standardize the control process, reducing the manual workload and error rate.

[0053] In some embodiments, the artificial intelligence scout program includes a scout program and an artificial intelligence analysis program. Calling the artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database includes: calling the scout program to traverse the rule library table and the permission library table in the graph database to detect whether there are abnormal permission policies in the rule library table and the permission library table; and calling the artificial intelligence analysis program to perform cross-cluster dimension analysis on the graph database using a pre-trained machine learning algorithm model to screen out abnormal permission policies in the graph database.

[0054] The permission library table is used to store all permission policies; the rule library table is used to store the relevant rules of the permission policies. Correspondingly, after the graph database is updated, the data logic policy detection subsystem calls the scout program to traverse the rule library table and the permission library table in the graph database. During the traversal process, the scout program checks each rule and permission policy to identify obvious anomalies or inconsistencies, such as duplicate permissions, conflicting rules, or permission levels that do not meet expectations. In order to avoid the large amount of Cartesian product data generated by the combination of the rule library table and the permission library table to affect the system, a strategy of sacrificing space and extending time is usually adopted to ensure system stability. When the contradiction between the rule library table and the permission library table is identified, the contradictory or abnormal permission policy is marked as abnormal in time; in addition, the data logic policy detection subsystem also calls the artificial intelligence analysis program, and uses the pre-trained machine learning algorithm model to perform cross-cluster dimension analysis on the graph database. This analysis takes into account the permission policy relationship between different clusters and can be used to identify potential abnormal patterns, such as unusual permission propagation or inconsistent permissions between clusters; further, the pre-trained machine learning algorithm model is used to conduct in-depth analysis of the initially marked abnormal policies, and the truly abnormal permission policies are screened out through pattern recognition and anomaly detection algorithms, and the abnormal permission policies that are falsely reported are filtered out.

[0055] For example, the above machine learning algorithm model is obtained in the following way:

[0056] (1) Data preprocessing and feature extraction: Use an iterative self-organizing data analysis algorithm to detect outliers in the original data, so as to prepare for the subsequent construction of an artificial neural network (ANN) mapping relationship between outlier characteristics and abnormal authority policies; use a data cleaning algorithm to clean the detected outliers to reduce the impact of noise data on subsequent model analysis; and perform feature analysis and correlation analysis on the original data to eliminate features with low absolute values ​​of the Pearson correlation coefficient and generate a new feature index library. The data cleaning algorithm includes but is not limited to at least one of outlier detection, missing value processing, smoothing processing, and filtering processing.

[0057] (2) Analysis based on historical data: Analyze abnormal permission policies using mathematical models and operation and maintenance experience, identify and annotate abnormal permission policies. These annotated data will be used as labels for model training. In this way, rich training data is provided for machine learning algorithm models.

[0058] (3) Establishing a machine learning algorithm model: Select an appropriate machine learning algorithm model, such as the radial basis function (RBF) for classification, or ANN for fitting complex relationships; use annotated historical data for initial training to construct a mapping relationship between outlier characteristics and abnormal permission policies; input the cleaned data into the model for training, for example, for RBF, the model classifies data points into different categories, for example, for ANN, the model fits the multivariate function relationship between various statistical indicators and abnormal policy probabilities, etc.; during the training process, the model continuously adjusts internal parameters (such as weights and biases) to minimize prediction errors; and uses optimization algorithms such as gradient descent, stochastic gradient descent, or adaptive moment estimation (Adam) optimizer to adjust model parameters to improve the accuracy and generalization ability of the model; further optimize model performance through cross-validation and hyperparameter tuning (such as adjusting learning rate and regularization parameters); further, continuously use the analysis results of new historical data for incremental training of the model to ensure that the model can adapt to changes in permission policies and emerging abnormal patterns. By regularly updating the model, its sensitivity and accuracy to new data can be ensured.

[0059] In the embodiment of the present application, the AI ​​scout program is used to implement multi-level detection of permission policies in the graph database, which can quickly identify abnormal permission policies and reduce potential security risks, thereby providing strong support for Ranger-based data lake operation and maintenance permission control. In addition, the machine learning algorithm model is effectively trained and optimized in the AI ​​scout program, further enhancing the AI ​​scout program's ability to identify and detect abnormal permission policies.

[0060] In some embodiments, the data processing subsystem includes a permission management and control platform, which processes abnormal permission policies, including: if the abnormal permission policy is a non-compliant permission policy, the permission management and control platform initiates a scout program to shield the abnormal permission policy and / or generates alarm information for the abnormal permission policy; if the execution result of the abnormal permission policy does not match the set expected result, the permission management and control platform executes the set policy correction program.

[0061] For example, when the abnormal permission policy is a non-compliant permission policy, the permission management and control platform starts the scout program. Specifically, in one implementation, the scout program immediately blocks the abnormal permission policy to prevent the abnormal permission policy from being executed. The blocking operation can be temporary, for example, and can be combined with the instructions of manual review to determine whether to continue blocking or unblocking; in another implementation, the scout program generates an alarm message for the abnormal permission policy and sends it to the relevant operation and maintenance personnel; in another implementation, the scout program immediately blocks the abnormal permission policy to prevent the abnormal permission policy from being executed, and generates an alarm message for the abnormal permission policy and sends it to the relevant operation and maintenance personnel. Among them, the alarm information may include but is not limited to a detailed description of the abnormal permission policy, the time of detection, the users or system components involved, etc.; the non-compliant permission policy may include but is not limited to at least one of the following: the permission grant exceeds the normal range of the user role, the permission policy conflicts with the enterprise security policy, and the abnormal execution frequency of the permission policy.

[0062] Accordingly, when the execution result of the abnormal permission policy does not match the set expected result, the permission management and control platform automatically triggers the policy correction program, which analyzes the difference between the execution result and the expected result, and attempts to automatically adjust the permission policy to correct the deviation; all correction operations and analysis results are recorded in the system log, and a detailed report is generated for review by the operation and maintenance personnel. The report may include but is not limited to analysis of abnormal behavior, description of corrective measures, and subsequent suggestions; further, after automated processing, the operation and maintenance personnel can manually review the abnormal permission policy and determine whether further manual adjustments or additional security measures are needed based on the reports and suggestions provided by the system.

[0063] In the embodiments of the present application, through the automated monitoring and exception handling mechanism, the system can effectively prevent potential security vulnerabilities and unauthorized access. This active defense mechanism helps to protect sensitive data and critical system resources. In addition, the automated adjustment and processing mechanism reduces the need for manual intervention, improves the response speed and accuracy of the system, and also reduces the complexity of manual configuration and monitoring, thereby reducing the possibility of human error, thereby ensuring the accurate execution of permission policies.

[0064] Based on the above embodiments, in some embodiments, the data processing subsystem further includes a front-end page, and the front-end page is used to display alarm information.

[0065] For example, the permission control platform in the data processing subsystem continuously monitors system activities. When abnormal permission policies or other security events are detected, the permission control platform will generate corresponding alarm information and transmit the generated alarm information to the front-end page in real time for display. Among them, the front-end page is usually built with modern World Wide Web (Web) technology, such as HyperText Markup Language (HTML), Cascading Style Sheets (CSS) and JavaScript, etc. The page design may include but is not limited to elements such as dashboards, alarm lists, filtering and search functions. Among them, the dashboard is used to display all currently active alarm information and provide an intuitive overview of the system status; the alarm list can list all alarm information in a table, which is convenient for users or operation and maintenance personnel to click to view detailed information; the filtering and search functions allow users to filter and search alarm information according to time, severity or type.

[0066] Correspondingly, the front-end page communicates with the permission management platform through the Application Programming Interface (API). The specific steps may include: the front-end page sends requests to the back-end API regularly or when the user operates to obtain the latest alarm information; the back-end API responds to the request and returns the alarm data, and the front-end page dynamically updates the display content according to the returned data; for example, through WebSocket or other real-time communication technologies, the back-end API can actively push alarm updates to the front-end page to ensure that the user sees the latest alarm information.

[0067] The embodiment of the present application displays alarm information in real time on the front-end page, allowing users to instantly understand the system status and potential problems, which not only improves the real-time and visualization capabilities of the system, but also improves user experience and operational efficiency, further providing strong support for efficient management and safe operation of the system.

[0068] In some embodiments, the authority management platform is also used to set alarm thresholds in the alarm policy library, and different alarm thresholds correspond to different alarm levels; the front-end page is also used to determine whether to display alarm information based on the alarm threshold; and, when determining to display the alarm information, the alarm level corresponding to the alarm information is displayed according to the alarm threshold.

[0069] For example, an alarm policy library is maintained through the authority management and control platform, wherein the alarm policy library contains alarm thresholds for multiple alarm types, and each alarm type can set different alarm thresholds according to its importance and impact, and define multiple alarm levels according to the alarm threshold, such as low, medium, high and emergency, and each alarm level can correspond to a different response strategy. Correspondingly, when a system activity triggers a certain alarm threshold, the authority management and control platform generates corresponding alarm information; the front-end page decides whether to display the alarm information based on the alarm information and threshold settings provided by the authority management and control platform; and when it is determined to display the alarm information, the front-end page displays the alarm information with different visual effects (such as color and icon) according to the alarm level, so that users can quickly identify its importance; further, users can view detailed alarm information and historical records through the front-end page. In addition, different notification strategies can be configured according to the alarm level. For example, for low and medium alarm levels, no notification is usually sent; for high-level alarms, relevant personnel are notified via email; for emergency alarms, relevant personnel are notified via SMS or instant messaging, etc. This hierarchical notification strategy can ensure that important information can be delivered to relevant personnel in a timely manner. It should be noted that users or operation and maintenance personnel with corresponding permissions can also dynamically adjust the alarm threshold according to actual needs and application scenarios to ensure the flexibility and adaptability of the system.

[0070] In the embodiment of the present application, by setting different alarm thresholds in the alarm policy library and deciding whether to display the alarm information according to the alarm threshold, the system can effectively filter out unnecessary alarms and reduce the risk of false alarms and information overload, thereby ensuring that users focus on truly important events; in addition, the front-end page can also display the alarm information with different visual effects according to the alarm level, which helps users to quickly identify and prioritize important alarms, improves the user's operating efficiency and experience, and further improves the security and reliability of Ranger-based data lake operation and maintenance authority control.

[0071] Considering that in the related art, duplicate permission policies are not allowed in each big data cluster node installed with Ranger, and the conditional filtering function is not supported to query permission policies, therefore, when the operation and maintenance personnel try to add a permission policy, if they are told that the permission policy already exists, they need to check the permission policy content one by one and make manual judgments. In this case, filtering out existing permission policies from a large number of permission policy entries brings great operation and maintenance difficulties to the operation and maintenance personnel. In response to this problem, the embodiment of the present application expands the functions based on Ranger, and simplifies the permission policy control process by integrating enhanced functions on the front-end page. These functions may include but are not limited to intelligent filtering and prompts, conditional queries, etc., aiming to significantly reduce the difficulty of the work of the operation and maintenance personnel and improve their work efficiency.

[0072] Therefore, in some embodiments, the front-end page is also used to: when responding to an operation instruction for the graph database, display the operation result corresponding to the operation instruction, and the operation instruction includes at least one of adding, deleting, modifying and querying.

[0073] For example, an intelligent filtering function is integrated on the front-end page, allowing operation and maintenance personnel to automatically detect and prompt possible duplicate policies when adding or querying permission policies. This function can help operation and maintenance personnel quickly identify existing policies through condition matching and similarity analysis; and provide a conditional filtering function on the front-end page, allowing operation and maintenance personnel to quickly query and filter permission policies according to specific conditions (such as policy name, resource type, permission level, etc.). In addition, when responding to operation instructions for the graph database, the front-end page can display the operation results in real time. The real-time feedback mechanism can help operation and maintenance personnel quickly understand the success or failure of the operation without relying on manual review of the permission policy content one by one and manual judgment, which greatly reduces the workload of manual judgment, thereby significantly improving work efficiency and reducing the difficulty of Ranger-based data lake operation and maintenance permission control.

[0074] In the related art, when changing the permission policy in the production environment, even if the permission policy to be changed has been fully tested and verified in the test environment, due to the possible differences between the production environment and the test environment, potential delays or logical conflicts may not be identified. Such differences may cause adverse effects on the stable operation of the current system after adding the permission policy.

[0075] Therefore, in some embodiments, the front-end page includes a pre-operation window; the data capture subsystem is also used to obtain the simulated permission policy in the pre-operation window, and write the simulated permission policy into the graph database; the data logic policy detection subsystem is also used to call the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy; the data processing subsystem is also used to display the alarm information corresponding to the simulated permission policy when the simulated permission policy is an abnormal permission policy, and delete the simulated permission policy from the graph database; and, it is also used to display the prompt information that the simulated permission policy is a normal permission policy when the simulated permission policy is a normal permission policy, and delete the simulated permission policy from the graph database.

[0076] For example, when the operation and maintenance personnel are unsure whether the added permission policy will have adverse effects, they enter the simulated permission policy through the pre-operation window on the front-end page; the data capture subsystem is responsible for obtaining the simulated permission policy in the pre-operation window and writing it to the graph database. This process can ensure that the simulated operation does not directly affect Ranger or the production environment; further, the data logic policy detection subsystem calls the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy, and through intelligent analysis, identifies potential policy conflicts or logical errors in advance; correspondingly, when the simulated permission policy is detected to be an abnormal permission policy, the corresponding alarm information is displayed on the front-end page, and the simulated permission policy is deleted from the graph database to ensure that there is no impact on the production environment; when the simulated permission policy is detected to be a normal policy, a prompt message indicating that the simulated permission policy is a normal permission policy is displayed on the front-end page, and the simulated permission policy is also deleted from the graph database.

[0077] The embodiment of the present application allows operation and maintenance personnel to simulate adding permission policies in the production environment through the pre-operation window on the front-end page, so as to obtain real and reliable verification results. This mechanism effectively reduces the difficulty of identifying potential delays or logical conflicts, thereby significantly reducing potential risks to the production environment.

[0078] In some embodiments, the front-end page also includes an actual operation window; the data capture subsystem is also used to obtain the target permission policy in the actual operation window, and write the target permission policy into the target node in the big data cluster node where Ranger is installed; call the scheduled scanning program to scan the target node to obtain the target permission policy; and write the target permission policy into the graph database.

[0079] For example, after confirming the security of the permission policy, the operation and maintenance personnel can enter the target permission policy in the actual operation window to make a formal policy change; correspondingly, the data capture subsystem is responsible for obtaining the target permission policy from the actual operation window and writing it to the target node in the big data cluster node where Ranger is installed; further, the data capture subsystem calls the scheduled scanning program to regularly scan the target node to obtain the target permission policy, and writes the target permission policy into the graph database.

[0080] In the embodiment of the present application, the target permission policy can be directly applied to the production environment through the actual operation window, which reduces the manual operation steps and simplifies the operation and maintenance process, thereby improving work efficiency and reducing the possibility of human error.

[0081] In summary, this application has at least the following advantages:

[0082] 1. The data capture subsystem periodically scans the big data cluster nodes, automatically obtains and writes the newly added permission policies into the graph database; and the data logic policy detection subsystem calls the artificial intelligence scout program to intelligently analyze and detect the permission policies in the graph database to ensure that potential abnormal permission policies can be quickly detected; and when abnormal permission policies are detected, the data processing subsystem quickly processes the abnormal permission policies to ensure the normal operation of the system, reducing the logical contradictions and security risks caused by permission policy changes, thereby improving the security and reliability of Ranger-based data lake operation and maintenance permission control. In addition, through the graph database, the system can uniformly store and manage permission policies, facilitate the query, analysis and maintenance of permission policies, thereby improving the efficiency and maintainability of permission control; and the automated processing method can systematize and standardize the control process, reducing manual workload and error rate.

[0083] Second, the AI ​​scout program can realize multi-level detection of permission policies in the graph database, which can quickly identify abnormal permission policies and reduce potential security risks, thus providing strong support for Ranger-based data lake operation and maintenance permission control. In addition, the machine learning algorithm model can be effectively trained and optimized in the AI ​​scout program, further enhancing the AI ​​scout program's ability to identify and detect abnormal permission policies.

[0084] 3. Through the interaction between Ranger and the front-end page, the front-end page can display the operation results of the operation instructions for the graph database in real time. The real-time feedback mechanism can help operation and maintenance personnel quickly understand the success or failure of the operation without relying on manual review of the permission policy content and manual judgment, which greatly reduces the workload of manual judgment, thereby significantly improving work efficiency and reducing the difficulty of Ranger-based data lake operation and maintenance permission control. In addition, the front-end page can also display alarm information in real time, allowing users to immediately understand the system status and potential problems, which not only improves the real-time and visualization capabilities of the system, but also enhances user experience and operation efficiency, and further provides strong support for efficient management and safe operation of the system.

[0085] 4. Through the pre-operation window on the front-end page, operation and maintenance personnel are allowed to simulate adding permission policies in the production environment to obtain real and reliable verification results. This mechanism effectively reduces the difficulty of identifying potential delays or logical conflicts, thereby significantly reducing potential risks to the production environment.

[0086] The above embodiment introduces the implementation method of the data lake operation and maintenance authority control system based on Ranger. Next, the application of the data lake operation and maintenance authority control system based on Ranger is introduced through specific embodiments.

[0087] For example, Figure 2 A flow chart of a Ranger-based data lake operation and maintenance authority control method provided in an exemplary embodiment of the present application. The Ranger-based data lake operation and maintenance authority control method provided in an embodiment of the present application is applied to any of the Ranger-based data lake operation and maintenance authority control systems in the above embodiments. Figure 2 As shown, the Ranger-based data lake operation and maintenance authority control method includes the following steps:

[0088] S201. Call a scheduled scanning program to scan the big data cluster nodes where Ranger is installed to obtain newly added permission policies.

[0089] For example, the scheduled scanning program is automatically executed at a set time interval, automatically connects to each big data cluster node, obtains the current permission policy by calling the Ranger API or directly accessing Ranger, and identifies the new or changed permission policies since the last scan.

[0090] S202. Write the newly added permission policy into the graph database.

[0091] For example, the newly added permission policy is converted into a language recognizable by the graph database and converted into graph database logic, specifically including mapping the newly added permission policy into nodes and edges in the graph database to represent the relationship between users, resources and permissions, for example, and writing the converted permission policy data into the graph database to update the existing permission relationship graph.

[0092] S203. Calling an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, the artificial intelligence scout program is set based on artificial intelligence technology.

[0093] For example, a scout program is called to traverse the rule base table and permission base table in the graph database to detect whether there are abnormal permission policies in the rule base table and permission base table; and an artificial intelligence analysis program is called to use a pre-trained machine learning algorithm model to perform a cross-cluster dimension analysis on the graph database to screen out abnormal permission policies in the graph database.

[0094] S204: When an abnormal permission policy exists in the graph database, the abnormal permission policy is processed.

[0095] For example, if the abnormal permission policy is a non-compliant permission policy, the scout program is started to block the abnormal permission policy, an alarm message is generated for the abnormal permission policy, and the front-end page is controlled to display the relevant alarm message; if the execution result of the abnormal permission policy does not match the set expected result, the set policy correction program is executed, the difference between the execution result and the expected result is analyzed through the policy correction program, and the permission policy is automatically adjusted to correct the deviation.

[0096] Figure 3 Another flow chart of the Ranger-based data lake operation and maintenance authority control method provided in the exemplary embodiment of the present application. Figure 3 As shown, the Ranger-based data lake operation and maintenance authority control method includes the following steps:

[0097] S301: Obtain the simulation permission policy in the pre-operation window.

[0098] S302. Write the simulation permission policy into the graph database.

[0099] S303. Call the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy.

[0100] If yes, execute S304;

[0101] If not, execute S305.

[0102] S304: Display the warning information corresponding to the simulation authority policy on the front-end page, and delete the simulation authority policy in the graph database.

[0103] S305: Display a prompt message that the simulated permission policy is a normal permission policy, and delete the simulated permission policy in the graph database.

[0104] The embodiment of the present application obtains the simulated permission policy in the pre-operation window and performs simulation verification on the simulated permission policy in the production environment, so that the obtained verification result is true and reliable, and the difficulty of identifying potential delays or logical conflicts and other problems is effectively reduced, thereby significantly reducing the potential risks to the production environment.

[0105] Figure 4 Another flow chart of the Ranger-based data lake operation and maintenance authority control method provided for the exemplary embodiment of the present application. Figure 4 As shown, the Ranger-based data lake operation and maintenance authority control method also includes the following steps:

[0106] S401: Obtain the target permission policy in the actual operation window.

[0107] S402: Write the target permission policy into the target node in the big data cluster node where Ranger is installed.

[0108] It should be noted that the target node may be one or more, and the number of target nodes in the big data cluster node where Ranger is installed is not limited.

[0109] S403: Call the scheduled scanning program to scan the target node and obtain the target permission policy.

[0110] S404: Write the target permission policy into the graph database.

[0111] S405. Call the artificial intelligence scout program to detect whether there are abnormal permission policies in the graph database.

[0112] If yes, execute S406;

[0113] If not, execute S407.

[0114] S406: triggering an automatic processing mechanism for abnormal permission policies.

[0115] For example, a scout program is started to shield abnormal permission policies; or an alarm message is generated for abnormal permission policies, and the relevant alarm message is displayed on the front-end page.

[0116] S407: Determine that the target permission policy is a normal permission policy.

[0117] Accordingly, by displaying a prompt message on the front-end page that the target permission policy is a normal permission policy, the operation and maintenance personnel can be promptly notified that the policy has been successfully applied and no abnormalities have been detected.

[0118] The embodiment of the present application reduces manual operation steps and simplifies the operation and maintenance process by obtaining the target permission policy in the actual operation window and directly applying the target permission policy to the production environment, thereby improving work efficiency and reducing the possibility of human errors.

[0119] Figure 5 A schematic diagram of a data lake operation and maintenance authority control device based on Ranger provided in an exemplary embodiment of the present application. Figure 5 As shown, the Ranger-based data lake operation and maintenance authority control device 50 includes a data capture module 51, a data logic strategy detection module 52 and a data processing module 53, wherein:

[0120] The data capture module 51 is used to call the scheduled scanning program to scan the big data cluster nodes installed with Ranger to obtain the newly added permission policies; and write the newly added permission policies into the graph database;

[0121] A data logic policy detection module 52, used to call an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database;

[0122] The data processing module 53 is used to process the abnormal permission policy when there is an abnormal permission policy in the graph database.

[0123] The Ranger-based data lake operation and maintenance authority control device provided in the embodiment of the present application can execute the technical solution shown in the above-mentioned Ranger-based data lake operation and maintenance authority control method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0124] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0125] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, referred to as DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, Digital Video Discs (DVDs)), or semiconductor media (eg, solid state disks (SSDs)).

[0127] Figure 6 The schematic diagram of the structure of the electronic device provided by the exemplary embodiment of the present application. Figure 6 As shown, the electronic device 60 of this embodiment includes:

[0128] At least one processor 61; and a memory 62 communicatively connected to the at least one processor;

[0129] The memory 62 stores instructions that can be executed by the at least one processor 61, and the instructions are executed by the at least one processor 61 to enable the electronic device to execute the method described in any of the above embodiments.

[0130] Optionally, the memory 62 may be independent or integrated with the processor 61 .

[0131] The memory 62 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0132] The processor 61 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. Specifically, when implementing the Ranger-based data lake operation and maintenance authority control method described in the aforementioned method embodiment, the electronic device may be, for example, an electronic device with processing capabilities such as a server.

[0133] Optionally, the electronic device may further include a communication interface 63. In a specific implementation, if the communication interface 63, the memory 62 and the processor 61 are implemented independently, the communication interface 63, the memory 62 and the processor 61 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0134] Optionally, in a specific implementation, if the communication interface 63, the memory 62 and the processor 61 are integrated on a chip, the communication interface 63, the memory 62 and the processor 61 can communicate through an internal interface.

[0135] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0136] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed, they are used to implement the method steps in the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0137] The above-mentioned computer readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0138] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in a Ranger-based data lake operation and maintenance authority control device.

[0139] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed, the method steps in the above method embodiment are implemented. The specific implementation method and technical effect are similar and will not be repeated here.

[0140] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0141] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data lake operation and maintenance authority control system based on Ranger, characterized in that: It includes data capture subsystem, data logic strategy detection subsystem and data processing subsystem; among which: The data capture subsystem is used to call a scheduled scanning program to scan the big data cluster nodes installed with Ranger to obtain the newly added permission policy, and write the newly added permission policy into the graph database; The data logic strategy detection subsystem is used to call the artificial intelligence scout program to detect whether there is an abnormal permission strategy in the graph database, and the artificial intelligence scout program is set based on artificial intelligence technology; The data processing subsystem is used to process the abnormal permission policy when there is an abnormal permission policy in the graph database.

2. The system according to claim 1, characterized in that The artificial intelligence scout program includes a scout program and an artificial intelligence analysis program, and calling the artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database includes: Calling the scout program to traverse the rule library table and the permission library table in the graph database to detect whether there are abnormal permission policies in the rule library table and the permission library table; Furthermore, the artificial intelligence analysis program is called to use a pre-trained machine learning algorithm model to perform a cross-cluster dimension analysis on the graph database to screen out abnormal permission policies in the graph database.

3. The system according to claim 2, characterized in that The data processing subsystem includes a permission management and control platform, and the processing of the abnormal permission policy includes: If the abnormal permission policy is a non-compliant permission policy, the permission management and control platform starts the scout program to shield the abnormal permission policy and / or generates an alarm message for the abnormal permission policy; If the execution result of the abnormal permission policy does not match the set expected result, the permission management and control platform executes the set policy correction procedure.

4. The system according to claim 3, characterized in that The data processing subsystem also includes a front-end page, and the front-end page is used to display the alarm information.

5. The system according to claim 4, characterized in that The authority management and control platform is also used to set alarm thresholds in the alarm strategy library, and different alarm thresholds correspond to different alarm levels; The front-end page is further used to determine whether to display the alarm information according to the alarm threshold; and when it is determined to display the alarm information, the alarm level corresponding to the alarm information is displayed according to the alarm threshold.

6. The system according to claim 4 or 5, characterized in that: The front-end page is also used to: when responding to an operation instruction for the graph database, display the operation result corresponding to the operation instruction, and the operation instruction includes at least one of adding, deleting, modifying and querying.

7. The system according to claim 4 or 5, characterized in that: The front-end page includes a pre-operation window; the data capture subsystem is also used to obtain the simulation authority policy in the pre-operation window, and write the simulation authority policy into the graph database; The data logic policy detection subsystem is further used to call the artificial intelligence scout program to detect whether the simulated permission policy is an abnormal permission policy; The data processing subsystem is further used to display the alarm information corresponding to the simulated permission policy when the simulated permission policy is an abnormal permission policy, and delete the simulated permission policy from the graph database; and is also used to display the prompt information that the simulated permission policy is a normal permission policy when the simulated permission policy is a normal permission policy, and delete the simulated permission policy from the graph database.

8. The system according to claim 4 or 5, characterized in that: The front-end page also includes a real operation window; The data capture subsystem is also used to obtain the target permission policy in the actual operation window, and write the target permission policy into the target node in the big data cluster node where Ranger is installed; call the scheduled scanning program to scan the target node to obtain the target permission policy; and write the target permission policy into the graph database.

9. A Ranger-based data lake operation and maintenance authority control method, characterized in that: Applied to the Ranger-based data lake operation and maintenance authority control system according to any one of claims 1 to 8, the Ranger-based data lake operation and maintenance authority control method comprises: Call the scheduled scanning program to scan the big data cluster nodes where Ranger is installed to obtain the newly added permission policies; Writing the newly added permission policy into the graph database; Calling an artificial intelligence scout program to detect whether there is an abnormal permission policy in the graph database, wherein the artificial intelligence scout program is set based on artificial intelligence technology; When an abnormal authority policy exists in the graph database, the abnormal authority policy is processed.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to claim 9.

Images