Data authority management method, system and equipment for big data platform, and storage medium

Abstract

The invention provides a data authority management method, system and device for a big data platform and a storage medium, and the data authority management method for the big data platform comprises the steps: according to a target task, obtaining data information having a high correlation degree with the target task in the big data platform, taking the corresponding data information as a key attribute of the target task; performing permission configuration on the key attribute and the applicable object set, and constructing a permission configuration database according to a configuration result, so that the applicable object set has corresponding operation permission for data information corresponding to the key attribute; receiving an access request of the user for the data information corresponding to the key attribute, performing identity verification on the user, judging whether the user passing the identity verification is an object in the applicable object set, and if so, responding to the access request of the user and a corresponding operation request; the method has the beneficial effects of performing flexible data authority management according to key attributes and adapting to complex scenes; the method is suitable for the field of data authority management.

Description

Technical Field

[0001] The present application relates to the technical field of data permission configuration, and specifically to a data permission management method, system, device, and storage medium for a big data platform. Background Art

[0002] At present, data permission management on big data platforms mainly relies on the traditional role-based permission control method, that is, roles are defined in advance and then associated with data permissions. Users obtain corresponding data access rights by obtaining roles.

[0003] The traditional role-based permission control method is applicable to certain scenarios where the data volume is small and the access requirements are relatively simple. However, with the rapid development of big data technology, the amount of data has increased dramatically, and the sources and types of data have become increasingly complex. Therefore, the traditional role-based permission control method has gradually exposed the following problems: 1. The permission management is not flexible. It is impossible to perform flexible permission management according to the source and type of data, and cannot meet the permission management requirements of complex scenarios; 2. The control granularity is too coarse to meet the refined requirements and security requirements of data rights management; 3. Data permissions need to be frequently allocated and reclaimed, resulting in low management efficiency and difficulty in meeting real-time requirements. Summary of the invention

[0004] In order to solve one of the above-mentioned technical defects, the present application provides a data permission management method, system and device, and storage medium for a big data platform.

[0005] According to a first aspect of the present application, a data rights management method for a big data platform is provided, comprising: According to the target task, obtain the data information with high correlation with the target task in the big data platform, and use the corresponding data information as the key attribute of the target task; Configure the relevant permissions for the key attributes and the preset applicable object set, and build a permission configuration database based on the configuration results, so that the applicable object set has the corresponding operation permissions for the data information corresponding to the key attributes; Receive a user's access request for data information corresponding to a key attribute, authenticate the user, and determine whether the authenticated user is an object in the applicable object set. If so, respond to the user's access request and corresponding operation request.

[0006] Preferably, according to the target task, obtaining data information in the big data platform that is highly correlated with the target task, and using the corresponding data information as the key attribute of the target task, specifically includes: Obtain data information related to the target task in the big data platform; Extract the target variable from the data information according to the target task, and use the data information other than the target variable as candidate attributes; Select and train candidate attributes to obtain an attribute selection model, and evaluate the performance of the attribute selection model; dynamically adjust the selection of candidate attributes according to the performance evaluation results and / or the feedback information obtained, and retrain the attribute selection model until the attribute selection model with the best performance evaluation result is obtained; Perform data cleaning on the data information corresponding to the candidate attributes in the optimal attribute selection model; Extract key attributes from the candidate attributes after data cleaning, and perform model training and evaluation on the extracted key attributes.

[0007] Preferably, the key attributes are configured with relevant permissions for a preset applicable object set, and a permission configuration database is constructed according to the configuration result, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, specifically including: respond to access requests from management personnel; Authenticate the administrators and execute the following operations on the authenticated administrators: Output the permission management interface, which includes policy creation, policy editing, and policy deletion options; Respond to a policy creation, policy editing or policy deletion request from an administrator and obtain corresponding permission configuration operation information; wherein the permission configuration operation information includes an applicable object set, key attributes and permission levels; The legality and validity of the permission configuration operation information are verified based on the pre-built matching condition library and permission configuration database. If the verification passes, the permission configuration operation information is updated to the permission configuration database to update the permission level of the applicable object set for the key attributes, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, and outputs a prompt message of successful update, and updates the permission management interface at the same time.

[0008] Preferably, the data cleaning of the data information corresponding to the candidate attributes in the optimal attribute selection model specifically includes: Processing the missing data values ​​specifically includes: determining whether the number of missing data values ​​is higher than a third threshold, if so, eliminating the candidate attribute, otherwise, filling the missing data values; Identify data outliers and process the data outliers, specifically including: determine whether the number of data outliers is lower than a fourth threshold, if so, delete the data outliers, otherwise, replace the data outliers with replacement values, and save the replacement values ​​as new data categories; Processing of noisy data; The data information corresponding to each candidate attribute is unified, standardized and normalized.

[0009] Preferably, the method for constructing the matching condition library includes: Presetting the applicable object set specifically includes: dividing objects into adapted applicable object sets based on business requirements; Divide the permission levels into: read-only permission, read-write permission, and full control permission; Assign corresponding applicable object sets, key attributes and permission levels to managers, and build a matching condition library based on the assignment results.

[0010] Preferably, extracting key attributes from candidate attributes after data cleaning specifically includes: Transform and combine the data information corresponding to the candidate attributes after data cleaning to obtain multiple combined features; Select and train multiple combined features to obtain a feature selection model, evaluate the performance of the feature selection model, and dynamically adjust the selection of combined features according to the performance evaluation results until a feature selection model with the best performance evaluation results is obtained; Extract key attributes from candidate attributes corresponding to multiple combined features in the optimal feature selection model.

[0011] More preferably, the candidate attributes are screened before the step of selecting and training the candidate attributes to obtain the attribute selection model, specifically including: analyzing the correlation between each candidate attribute and the target variable, and eliminating candidate attributes with correlation values ​​lower than a first threshold; analyzing the correlation between multiple candidate attributes, and eliminating redundant candidate attributes with correlation values ​​higher than a second threshold.

[0012] According to a second aspect of the present application, a data rights management system for a big data platform is provided, comprising a module for implementing a data rights management method for a big data platform as described in any one of the above items.

[0013] According to a third aspect of the present application, a device is provided, comprising: Memory; Processor; and Computer programs; Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the data authority management method for a big data platform as described in any one of the above items.

[0014] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement a data authority management method for a big data platform as described in any one of the above items.

[0015] In this application, data information with a high correlation with the target task in the big data platform is extracted, and the corresponding data information is used as the key attribute of the target task, which is convenient for the subsequent flexible data permission management of the key attributes, and is suitable for the permission management requirements of complex scenarios. The data information with a high correlation with the target task in the big data platform is obtained, wherein the highly correlated data information may include the source, type, content, database table, etc. of the data, which can realize the permission configuration of the data at the level of in-depth data items or data fields, and solves the problem that the prior art can only go deep into the data set level, and the data permission configuration granularity is too coarse, and cannot meet the security requirements of refined data permission configuration. With the deepening of the granularity of key attributes, by configuring different key attributes and permission levels for different applicable object sets, it is possible to make a clear division of business, and the key attributes and permission levels that are only applicable to the department can be configured for different applicable object sets, which solves the problem that the control granularity is too coarse in the prior art and cannot meet the refined requirements and security requirements of data permission management. When users access and operate, corresponding permissions and responsibilities can be defined for users to ensure the security and stability of permission management. By authenticating the user and determining whether the user is an object in the applicable object set, the permission granting process can be further optimized, significantly improving the efficiency, security and real-time performance of permission management.

[0016] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the contents indicated in the written description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of an implementation method of a data rights management method for a big data platform provided in an embodiment of the present application; Figure 2 for Figure 1 A flowchart of an implementation method of obtaining data information highly associated with the target task in the big data platform according to the target task and using the corresponding data information as a key attribute of the target task in the embodiment; Figure 3 for Figure 2 A flowchart of an implementation method of extracting key attributes from candidate attributes after data cleaning in the embodiment; Figure 4 for Figure 1 A flowchart of an implementation method of performing relevant permission configuration on key attributes and preset applicable object sets in the embodiment, and constructing a permission configuration database according to the configuration results; Figure 5 A functional structure diagram of the modules of the data rights management method for a big data platform provided in this application; Figure 6 A schematic diagram of the functional structure of the acquisition module provided in this application; Figure 7 A schematic diagram of the functional structure of the cleaning unit provided in this application; Figure 8 A schematic diagram of the functional structure of the extraction unit provided in this application; Fig. 9 A functional structure diagram of the permission configuration module provided for this application; In the figure: 10 is an acquisition module, 20 is a permission configuration module, 30 is a verification module, 40 is a response module, 101 is a data information acquisition unit, 102 is a data information division unit, 103 is an optimal attribute selection model training unit, 104 is a cleaning unit, 105 is an extraction unit, 106 is a training evaluation unit, 201 is an access response unit, 202 is an identity authentication unit, 203 is an output unit, 204 is a policy response unit, 205 is a verification unit, 206 is an update unit, 1041 is a missing value processing unit, 1042 is an outlier processing unit, 1043 is a noise data processing unit, 1044 is a unification processing unit, 1045 is a standardization processing unit, 1046 is a normalization processing unit, 1051 is a transformation combination unit, 1052 is an optimal feature selection model training unit, and 1053 is a key attribute extraction unit. DETAILED DESCRIPTION

[0018] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0019] Figure 1 A flowchart of an implementation method of a data rights management method for a big data platform provided in an embodiment of the present application is shown in FIG. Figure 1As shown, in order to solve the above problems, the present application provides a data rights management method for a big data platform in an embodiment, including the following steps: According to the target task, obtain the data information with high correlation with the target task in the big data platform, and use the corresponding data information as the key attribute of the target task; Configure the relevant permissions for the key attributes and the preset applicable object set, and build a permission configuration database based on the configuration results, so that the applicable object set has the corresponding operation permissions for the data information corresponding to the key attributes; Receive a user's access request for data information corresponding to a key attribute, authenticate the user, and determine whether the authenticated user is an object in the applicable object set. If so, respond to the user's access request and corresponding operation request.

[0020] In this application, data information with a high correlation with the target task in the big data platform is extracted, and the corresponding data information is used as the key attribute of the target task, which is convenient for the subsequent flexible data permission management of the key attributes, and is suitable for the permission management requirements of complex scenarios. The data information with a high correlation with the target task in the big data platform is obtained, wherein the highly correlated data information may include the source, type, content, database table, etc. of the data, which can realize the permission configuration of the data at the level of in-depth data items or data fields, and solves the problem that the prior art can only go deep into the data set level, and the data permission configuration granularity is too coarse, and cannot meet the security requirements of refined data permission configuration. With the deepening of the granularity of key attributes, by configuring different key attributes and permission levels for different applicable object sets, it is possible to make a clear division of business, and the key attributes and permission levels that are only applicable to the department can be configured for different applicable object sets, which solves the problem that the control granularity is too coarse in the prior art and cannot meet the refined requirements and security requirements of data permission management. When users access and operate, corresponding permissions and responsibilities can be defined for users to ensure the security and stability of permission management. By authenticating the user and determining whether the user is an object in the applicable object set, the permission granting process can be further optimized, significantly improving the efficiency, security and real-time performance of permission management.

[0021] Specifically, user identity authentication includes: one or more identity authentication methods of user name and password authentication, digital certificate authentication, and biometric authentication.

[0022] Figure 2 for Figure 1 The embodiment of the present invention is a flowchart of an implementation method of obtaining data information with a high correlation with the target task in the big data platform according to the target task, and using the corresponding data information as a key attribute of the target task, as shown in FIG. Figure 2As shown, further, according to the target task, data information with a high correlation with the target task in the big data platform is obtained, and the corresponding data information is used as the key attribute of the target task, specifically including: Obtain data information related to the target task in the big data platform; specifically, the data information in the big data platform is obtained by collecting questionnaires, experimental data, existing data, etc.; Extract the target variable from the data information according to the target task, and use the data information other than the target variable as candidate attributes; Select and train candidate attributes to obtain an attribute selection model, and evaluate the performance of the attribute selection model; dynamically adjust the selection of candidate attributes according to the performance evaluation results of the attribute selection model and / or the feedback information obtained, and retrain the attribute selection model until the attribute selection model with the best performance evaluation result is obtained; Perform data cleaning on the data information corresponding to the candidate attributes in the optimal attribute selection model; Extract key attributes from the candidate attributes after data cleaning, and perform model training and evaluation on the extracted key attributes.

[0023] In this application, the performance of the attribute selection model is evaluated. If the performance evaluation result is not good, it means that the relationship between some candidate attributes and the target variable in the attribute selection model is weak, or there is redundancy between the candidate attributes, and they need to be dynamically adjusted.

[0024] Furthermore, the step of selecting and training the candidate attributes to obtain the attribute selection model includes screening the candidate attributes, which specifically includes: Analyze the correlation between each candidate attribute and the target variable, and eliminate the candidate attributes whose correlation values ​​are lower than the first threshold value; in the specific implementation process, when the target variable is a discrete, categorical or nominal variable, a classification method can be used to analyze the correlation between each candidate attribute and the target variable, wherein the classification method can be a chi-square test, information gain, Gini index and other methods; when the target variable is a continuous numerical type, a regression method can be used to analyze the correlation between each candidate attribute and the target variable, wherein the regression method used can be a Pearson correlation coefficient, a Spearman rank correlation coefficient and other methods; Analyze the correlation between multiple candidate attributes, and eliminate redundant candidate attributes whose correlation values ​​are higher than a second threshold. In a specific implementation, a correlation matrix may be used to analyze the correlation between multiple candidate attributes, or a heat map may be used to analyze the correlation between multiple candidate attributes. In the present application, the correlation between each candidate attribute and the target variable is analyzed, and the candidate attributes with correlation values ​​lower than the first threshold are eliminated, which can preliminarily screen out the candidate attributes with a weak relationship with the target variable; the correlation between multiple candidate attributes is analyzed, and redundant candidate attributes with correlation values ​​higher than the second threshold are eliminated, which can prevent overfitting caused by highly correlated candidate attributes and improve the efficiency of subsequent selection and training of candidate attributes.

[0025] Specifically, after screening the candidate attributes, statistical test methods such as t-test, analysis of variance (ANOVA), chi-square test, etc. can be used to evaluate the significance between the screened candidate attributes and the target variable, and determine whether they are consistent with the expected values; the distribution of the data corresponding to the screened candidate attributes can be evaluated by calculating statistics such as mean, median, mode, standard deviation, etc.

[0026] In the specific implementation process, attribute selection strategies and machine learning models can be used to select and train candidate attributes to obtain attribute selection models. Among them, the attribute selection strategy can be a packaging method or an embedded method; the machine learning model can be a model training method such as linear regression, decision tree, neural network, etc. Among them, a packaging method and a machine learning model are used to select and train candidate attributes to obtain an attribute selection model, specifically including: embedding the packaging method into the model machine learning model training process, evaluating the quality of candidate attributes in the attribute selection model through the performance of the attribute selection model, and outputting the candidate attributes in the optimal attribute selection model in the form of a list or a Boolean mask, and the Boolean mask can indicate whether each feature subset is selected.

[0027] Specifically, wrapper methods include: forward search, backward search or recursive feature elimination methods.

[0028] During forward search, the attribute subset is initialized to an empty set; the following steps are executed in a loop, one candidate attribute is selected from the screened candidate attributes and added to the current attribute subset, and then the attribute selection model is obtained by training the machine learning model and its performance is evaluated; until the stopping criteria are met, such as the number of loops reaches the predetermined number of candidate attributes, or all the screened candidate attributes are added to the attribute subset, or the performance is no longer significantly improved.

[0029] During the backward search, the attribute subset is initialized as the set of all filtered candidate attributes; the following steps are executed repeatedly to remove a candidate attribute from the attribute subset, and then the attribute selection model is obtained by training the machine learning model and its performance is evaluated; until the stopping criteria are met, such as the number of cycles reaches the predetermined number of candidate attributes, or all filtered candidate attributes are removed from the attribute subset, or the performance is no longer significantly improved.

[0030] The recursive feature elimination method uses logistic regression or support vector machine to perform multiple rounds of training on the screened candidate attributes to obtain an attribute selection model. In each round of training, the least important candidate attributes are removed according to the feature importance output by the model, such as the coefficient (coef) or the feature importance value (feature_importances); until the stopping criteria are met, such as the number of cycles reaches the predetermined number of candidate attributes, or all the screened candidate attributes are removed from the attribute subset.

[0031] Among them, the embedded method and machine learning model are used to select and train candidate attributes to obtain an attribute selection model, and the candidate attributes are automatically selected during the model training process, specifically including: Preprocess the data to make it meet the requirements of the attribute selection model; Train the attribute selection model to automatically adjust the attribute coefficients of candidate attributes through algorithms such as decision trees and random forests; Eliminate candidate attributes with attribute coefficients of 0.

[0032] Specifically, the feedback information may be the detection results of data drift or concept drift; it may also be the detection results of outliers or prediction errors in the output values ​​of the attribute selection model; it may also be external interaction information, where the external interaction information may be feedback information such as clicks, purchases, and ratings; it may also be feedback information required by industry standards and regulations; and it may also be feedback information based on market changes or strategy adjustments.

[0033] Furthermore, the data information corresponding to the candidate attributes in the optimal attribute selection model is cleaned, specifically including: Processing missing data values, specifically including: determining whether the number of missing data values ​​is higher than a third threshold, if so, eliminating the candidate attribute, otherwise, filling the missing data values; in specific implementation, constants may be used to fill missing data values, such as 0, mean, median, etc.; special markers may also be used to fill missing data values, such as non-numeric values ​​(NaN), null values ​​(None), etc.; Identify data outliers and process them, specifically including: determine whether the number of data outliers is lower than a fourth threshold, if so, delete the data outliers, otherwise, replace the data outliers with replacement values, and save the replacement values ​​as new data categories; in the specific implementation process, a visualization method can be used to identify data outliers, and the visualization method can be a histogram, a line chart, a pie chart, a box plot, a scatter plot, etc.; a statistical method can also be used to identify data outliers; the statistical method can be the interquartile range rule (IQR rule), the standard score method (Z-score method), etc., and the replacement value can be the median, the mean, etc.; Process the noise data; in the specific implementation process, the noise data can be processed by smoothing, binning or clustering algorithm; among them, smoothing is to process the noise data by using the average value, median and other data, and smoothing is more effective for the noise data of continuous data; binning is to divide the data into multiple intervals by equal width binning or equal frequency binning, and then replace all the data in the interval with the median or average of each interval; clustering algorithm is to cluster similar data together to form clusters, and then replace the data in each cluster with the center point or representative value of the cluster; The data information corresponding to each candidate attribute is unified, standardized and normalized; in the specific implementation process, the unified processing may include the unification of data formats and units, for example, the formats and units of specific data corresponding to dates, currencies, etc. are unified respectively. For text data, the unified processing may also include the unification of uppercase and lowercase, the removal of punctuation marks, the removal of stop words, etc.; the standardized processing may be to scale the data using the mean and standard deviation. After the standardized processing, the mean of the data is 0 and the standard deviation is 1, so that the influence of the dimension is eliminated and the data is placed on the same scale for comparison; the normalized processing is to scale the data to a preset range, which may be [0,1] or [-1,1]. The normalized processing helps algorithms such as neural networks that are sensitive to numerical ranges to perform subsequent processing on the data.

[0034] In this application, the above-mentioned data cleaning method is used to clean the data information corresponding to the candidate attributes in the optimal attribute selection model, which further improves the quality and accuracy of the data, provides reliable guarantee for the subsequent extraction of key attributes, and reduces errors and deviations.

[0035] Specifically, after data cleaning of the data information corresponding to the candidate attributes in the optimal attribute selection model, the quality and accuracy of the cleaned data can be analyzed using descriptive statistics, correlation analysis, visualization tools, etc.

[0036] Figure 3 for Figure 2 A flowchart of an implementation method of extracting key attributes from candidate attributes after data cleaning in the embodiment is shown in FIG. Figure 3 As shown, further, extracting key attributes from the candidate attributes after data cleaning specifically includes: Analyze the relationship between candidate attributes and target variables after data cleaning; Transform and combine the data information corresponding to the candidate attributes after data cleaning to obtain multiple combined features; Select and train multiple combined features to obtain a feature selection model, and evaluate the performance of the feature selection model. Dynamically adjust the selection of combined features according to the performance evaluation results until a feature selection model with optimal performance evaluation results is obtained; extract key attributes from candidate attributes corresponding to multiple combined features in the optimal feature selection model.

[0037] In the specific implementation process, the data information can be transformed by using a feature scaling method or a feature encoding method.

[0038] Among them, the feature coding method is used to transform the data information, specifically including: Convert categorical variables in candidate attributes into numerical values. Categorical variables are usually represented by text or symbols, such as unordered categorical variables: color, gender, etc., and ordered categorical variables: credit rating, rating, etc.; For unordered categorical variables or categorical variables with a small number of categories, one-hot encoding is used for feature encoding, creating a new binary column for each category in the categorical variable; for ordered categorical variables or categorical variables with a large number of categories, label encoding is used for feature encoding, converting each category in the categorical variable into an integer; when it is necessary to capture the relationship between the categorical variable and the target variable, target encoding is used for feature encoding, replacing the data corresponding to each category in the categorical variable with the average value or other statistics of the target variable related to the category.

[0039] In the specific implementation process, the feature construction method can be used to combine the data information, wherein the feature construction method may include: direct construction, polynomial construction, text construction, time construction, etc.; specifically, direct construction is to directly combine two or more candidate attributes, for example, in house price prediction, the number of bedrooms and the size of bedrooms can be combined to create the feature combination "total bedroom area"; polynomial construction is to create a new feature combination by polynomial expansion, for example, for candidate attributes x and y, new feature combinations such as x^2, x*y, y^2 can be created; text construction is to extract features from text data, such as Bag of Words, TF-IDF, Word2Vec, etc.; time features are to extract features from date and time data, such as year, month, day of the week, hour, season, etc.

[0040] In the specific implementation process, feature selection strategies and machine learning models can be used to select and train from multiple combined features to obtain a feature selection model, wherein the feature selection strategy can be the same as the attribute selection strategy or different, and can be a packaged method or an embedded method.

[0041] Specifically, before selecting and training multiple combined features to obtain a feature selection model, a filtering method can be used to filter the obtained multiple combined features. In the specific implementation process, the filtering method can be a classification method, a regression method, a correlation matrix or a heat map to filter multiple feature combinations.

[0042] In the specific implementation process, a feature selection algorithm can be used to extract key attributes from candidate attributes corresponding to multiple combined features in the optimal feature selection model, or a dimensionality reduction algorithm can be used to extract key attributes from candidate attributes corresponding to multiple combined features in the optimal feature selection model.

[0043] Among them, the feature selection algorithm is used to extract key attributes from the candidate attributes corresponding to multiple combined features in the optimal feature selection model, including: Preprocess the data and convert it into the format to be extracted; Using forward search, backward search or recursive feature elimination to cyclically construct feature subsets; Evaluate the performance of the current feature subset; you can use methods such as accuracy, recall, F1 score, information gain, and Gini index for evaluation; The candidate attribute with the best performance in the current feature subset is selected as the key attribute.

[0044] Among them, the dimension reduction algorithm is used to extract key attributes from the candidate attributes corresponding to multiple combined features in the optimal feature selection model, including: Preprocess the data and convert it into the format to be extracted; Select a dimensionality reduction algorithm based on the target task, including principal component analysis (PCA), linear discriminant analysis (LDA), t-distributed nexus embedding (t-SNE) and other algorithms; Use the selected dimensionality reduction algorithm to reduce the dimension of the candidate attributes to form a low-dimensional feature space; An evaluation method adapted to the dimensionality reduction algorithm is used to evaluate the importance of features in the low-dimensional feature space, and the important features or a certain number of important features in the evaluation results that exceed the fifth threshold are used as key attributes; in the specific implementation process, the evaluation method adapted for the principal component analysis algorithm is to calculate the principal component variance explanation rate, the evaluation method adapted for the linear discriminant analysis algorithm is to calculate the weight coefficient, and the evaluation method adapted for the t-distribution domain embedding algorithm is to calculate the feature importance.

[0045] The extraction of key attributes in this application has broad application prospects in fields such as business intelligence and decision support. For example, when the target task is performance evaluation and optimization, the extraction of key attributes can help enterprises obtain performance data of various departments, product lines, and regions, and then evaluate the performance of each business unit; when the target task is cost control and management, the cost data type is extracted as a key attribute, so that the enterprise can analyze the cost structure, find out the links with excessively high costs, and thus formulate cost control measures; when the target task is customer analysis and relationship management, the customer data type is extracted as a key attribute, so that the enterprise can understand the customer's purchasing habits, demand preferences, and satisfaction, and thus formulate more accurate marketing strategies.

[0046] Figure 4 for Figure 1 In the embodiment, the key attributes are configured with relevant permissions of the preset applicable object set, and a flowchart of an implementation method of building a permission configuration database according to the configuration results is shown in FIG. Figure 4 As shown, further, the key attributes are configured with relevant permissions for the preset applicable object set, and a permission configuration database is constructed according to the configuration result, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, specifically including: respond to access requests from management personnel; Authenticate the administrators and execute the following operations on the authenticated administrators: Output the permission management interface, which includes policy creation, policy editing, and policy deletion options; Respond to the administrator's policy creation, policy editing or policy deletion request and obtain the corresponding permission configuration operation information; wherein the permission configuration operation information includes the applicable object set, key attributes and permission level, and may also include the policy number, policy name, and policy description; wherein the applicable object set may include user groups, users, and roles; The legality and validity of the permission configuration operation information are verified based on the pre-built matching condition library and permission configuration database. If the verification passes, the permission configuration operation information is updated to the permission configuration database to update the permission level of the applicable object set for the key attributes, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, and outputs a prompt message of successful update, and updates the permission management interface at the same time.

[0047] In this application, the pre-built matching condition library can verify whether the management personnel have the right to configure the corresponding permissions for the applicable object set, and can also verify whether the applicable object set in the permission configuration operation information has the right to execute the corresponding permission level on the key attributes. At the same time, the permission configuration database can also verify whether the current policy creation, policy editing or policy deletion request conflicts with the existing policies in the permission configuration database. Through the above verification, the legality and validity of the permission configuration operation information can be verified, further improving the rationality, security and stability of permission management.

[0048] Specifically, the permission configuration database can be pre-created and then constructed through the policy creation request of the administrator, or it can be constructed directly according to the policy creation request of the administrator, and then further updated according to the subsequent policy creation, policy editing and policy deletion requests of the administrator.

[0049] Specifically, the pre-built matching condition library can limit or prohibit the use of sensitive application programming interfaces (APIs), which include privacy information such as cameras, microphones, and geographic locations, to improve security and prevent malicious websites from abusing sensitive APIs to collect personal privacy information. At the same time, it limits access rights to certain powerful but potentially risky APIs, such as inline frames (iframes), proxy servers (Service Workers), notifications, etc., to reduce network security threats such as malicious attacks and cross-site scripting.

[0050] Furthermore, the method for constructing the matching condition library includes: Preset applicable object set, specifically including: classifying objects into adapted applicable object sets based on business needs; in the specific implementation process, the preset applicable object set may include: administrators, department managers, and ordinary employees; Categorize key attributes into different types according to their nature, sensitivity and importance: public attributes, departmental attributes, sensitive attributes, etc. The permission levels are divided into: read-only permission, read-write permission and full control permission; the permission levels can also include: temporary permission, emergency permission, etc., to facilitate emergency situations; Assign corresponding applicable object sets, key attributes and permission levels to different managers, and build a matching condition library based on the assignment results.

[0051] In this application, management personnel, applicable object sets, key attributes and permission levels are divided and allocated. The allocation results in the matching condition library can be adjusted according to actual usage needs, thereby facilitating real-time permission allocation and recovery operations and improving management efficiency.

[0052] Specifically, the training of the model in the present application can adopt a machine learning model, including the training of an attribute selection model and a feature selection model, wherein the machine learning model can be a linear regression, a decision tree, a neural network or other methods.

[0053] More specifically, the model evaluation method may be a leave-out validation method, a cross-validation method, a bootstrap method, a confusion matrix method, or a receiver operating characteristic curve (ROC curve) method.

[0054] The holdout validation method specifically includes: dividing the original data set into a training set and a test set, usually using a 70%-30% or 80%-20% ratio; using the training set to train the model, and using an independent test set to evaluate the model's performance. The holdout validation method is simple and easy to operate, and can truly reflect the performance of the model on unknown data.

[0055] The cross-validation method specifically includes: dividing the original data into K non-overlapping subsets (for example, K=5 or K=10); using K-1 subsets as training sets each time, and the remaining subset as validation set, and performing K training and validation; finally, taking the average of the K validation results as the performance indicator of the model. The cross-validation method can reduce accidental errors and make the evaluation results more comprehensive.

[0056] Bootstrapping specifically includes: sampling from the original data with replacement to form a training set and a test set. Bootstrapping is particularly suitable for small data sets and can maximize the use of data resources.

[0057] Confusion Matrix specifically includes: For classification tasks, the confusion matrix is ​​an important tool for evaluating model performance. The confusion matrix provides detailed information on true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). From the confusion matrix, evaluation indicators such as accuracy, precision, recall, and F1 value can be calculated.

[0058] ROC curve method: The ROC curve is used to evaluate the classification performance of the model, showing the true positive rate and false positive rate at different thresholds. The AUC value represents the area under the ROC curve, which quantifies the overall performance of the model. The higher the AUC value, the better the model performance.

[0059] It should be noted that during the permission management process of this application, detailed records of each step of the execution and the data before and after the execution are required to facilitate subsequent analysis and maintenance.

[0060] Accordingly, the present application also provides a data rights management system for a big data platform, including a module for implementing the data rights management method for a big data platform as described above.

[0061] Figure 5 The functional structure diagram of the module of the data rights management method for a big data platform provided in this application is as follows: Figure 5 As shown, specifically, the modules for implementing the data rights management method for a big data platform specifically include: The acquisition module 10 is used to acquire data information highly associated with the target task in the big data platform according to the target task, and use the corresponding data information as the key attribute of the target task; The permission configuration module 20 is used to configure the relevant permissions of the key attributes and the preset applicable object set, and to build a permission configuration database according to the configuration results, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes; The verification module 30 is used to receive a user's access request to the data information corresponding to the key attribute, authenticate the user, and determine whether the authenticated user is an object in the applicable object set. If so, respond through the response module 40; The response module 40 is used to respond to the user's access request and the corresponding operation request.

[0062] Figure 6 The functional structure diagram of the acquisition module provided in this application is as follows: Figure 6 As shown, further, the acquisition module 10 includes: The data information acquisition unit 101 is used to acquire data information related to the target task in the big data platform; The data information partitioning unit 102 is used to extract the target variable in the data information according to the target task, and use the data information other than the target variable as candidate attributes; The optimal attribute selection model training unit 103 is used to select and train candidate attributes to obtain an attribute selection model, and evaluate the performance of the attribute selection model; dynamically adjust the selection of candidate attributes according to the performance evaluation results and / or the obtained feedback information, and retrain the attribute selection model until the attribute selection model with the optimal performance evaluation result is obtained; A cleaning unit 104, used for performing data cleaning on data information corresponding to candidate attributes in the optimal attribute selection model; An extraction unit 105, used to extract key attributes from candidate attributes after data cleaning; The training and evaluation unit 106 is used to perform model training and evaluation on the extracted key attributes.

[0063] Figure 7 The functional structure diagram of the cleaning unit provided in this application is as follows: Figure 7 As shown, further, the cleaning unit 104 includes: The missing value processing unit 1041 is used to process the missing data values, specifically including: determining whether the number of missing data values ​​is higher than a third threshold value, if so, eliminating the candidate attribute, otherwise, filling the missing data values; The outlier processing unit 1042 is used to identify data outliers and process the data outliers, specifically including: determining whether the number of data outliers is lower than a fourth threshold value, if so, deleting the data outliers, otherwise, replacing the data outliers with replacement values, and saving the replacement values ​​as new data categories; A noise data processing unit 1043, used for processing noise data; The unified processing unit 1044 is used to perform unified processing on the data information corresponding to each candidate attribute; A standardization processing unit 1045 is used to perform standardization processing on the data information corresponding to each candidate attribute; The normalization processing unit 1046 is used to perform normalization processing on the data information corresponding to each candidate attribute.

[0064] Figure 8 The functional structure diagram of the extraction unit provided in this application is as follows: Figure 8 As shown, specifically, the extraction unit 105 includes: The transformation and combination unit 1051 is used to transform and combine the data information corresponding to the candidate attributes after data cleaning to obtain multiple combined features; The optimal feature selection model training unit 1052 is used to select and train multiple combined features to obtain a feature selection model, and evaluate the performance of the feature selection model, and dynamically adjust the selection of the combined features according to the performance evaluation results until the feature selection model with the optimal performance evaluation results is obtained; The key attribute extraction unit 1053 is used to extract key attributes from the candidate attributes corresponding to the multiple combined features in the optimal feature selection model.

[0065] Fig. 9 The functional structure diagram of the permission configuration module provided for this application is as follows: Fig. 9 As shown, specifically, the permission configuration module 20 includes: An access response unit 201, used to respond to an access request from an administrator; An identity verification unit 202, used for verifying the identity of the administrator; The output unit 203 is used to execute, for the administrator who has passed the identity authentication: output the authority management interface, which includes the policy creation, policy editing and policy deletion options; The policy response unit 204 is used to respond to the administrator's policy creation, policy editing or policy deletion request and obtain the corresponding permission configuration operation information; wherein the permission configuration operation information includes the applicable object set, key attributes and permission level; Verification unit 205, used to verify the legitimacy and validity of the permission configuration operation information based on the pre-built matching condition library and permission configuration database; Update unit 206 is used to update the permission configuration operation information to the permission configuration database if the legality and validity verification is passed, so as to update the permission level of the applicable object set to the key attribute, so that the applicable object set has corresponding operation permissions to the data information corresponding to the key attribute, and output a prompt message of successful update, and update the permission management interface at the same time.

[0066] The present application also provides a device, comprising: Memory; Processor; and Computer programs; Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the data authority management method for a big data platform as described above.

[0067] The present application also provides a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor to implement the data authority management method for a big data platform as described above.

[0068] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The scheme in the embodiments of the present application may be implemented in various computer languages, for example, C language, VHDL language, Verilog language, object-oriented programming language Java, and literal scripting language JavaScript, etc.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0073] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data rights management method for a big data platform, characterized in that: include: According to the target task, obtain the data information with high correlation with the target task in the big data platform, and use the corresponding data information as the key attribute of the target task; Configure the relevant permissions for the key attributes and the preset applicable object set, and build a permission configuration database based on the configuration results, so that the applicable object set has the corresponding operation permissions for the data information corresponding to the key attributes; Receive a user's access request for data information corresponding to a key attribute, authenticate the user, and determine whether the authenticated user is an object in the applicable object set. If so, respond to the user's access request and corresponding operation request.

2. The data rights management method for a big data platform according to claim 1, characterized in that: According to the target task, data information with a high correlation with the target task is obtained from the big data platform, and the corresponding data information is used as the key attribute of the target task, specifically including: Obtain data information related to the target task in the big data platform; Extract the target variable from the data information according to the target task, and use the data information other than the target variable as candidate attributes; Select and train candidate attributes to obtain an attribute selection model, and evaluate the performance of the attribute selection model; dynamically adjust the selection of candidate attributes according to the performance evaluation results and / or the feedback information obtained, and retrain the attribute selection model until the attribute selection model with the best performance evaluation result is obtained; Perform data cleaning on the data information corresponding to the candidate attributes in the optimal attribute selection model; Extract key attributes from the candidate attributes after data cleaning, and perform model training and evaluation on the extracted key attributes.

3. The data rights management method for a big data platform according to claim 1, characterized in that: The key attributes are configured with relevant permissions for the preset applicable object set, and a permission configuration database is constructed according to the configuration results so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, specifically including: respond to access requests from management personnel; Authenticate the administrators and execute the following operations on the authenticated administrators: Output the permission management interface, which includes policy creation, policy editing, and policy deletion options; Respond to a policy creation, policy editing or policy deletion request from an administrator and obtain corresponding permission configuration operation information; wherein the permission configuration operation information includes an applicable object set, key attributes and permission levels; The legality and validity of the permission configuration operation information are verified based on the pre-built matching condition library and permission configuration database. If the verification passes, the permission configuration operation information is updated to the permission configuration database to update the permission level of the applicable object set for the key attributes, so that the applicable object set has corresponding operation permissions for the data information corresponding to the key attributes, and outputs a prompt message of successful update, and updates the permission management interface at the same time.

4. The data rights management method for a big data platform according to claim 2, characterized in that: The data cleaning of the data information corresponding to the candidate attributes in the optimal attribute selection model specifically includes: Processing the missing data values ​​specifically includes: determining whether the number of missing data values ​​is higher than a third threshold, if so, eliminating the candidate attribute, otherwise, filling the missing data values; Identify data outliers and process the data outliers, specifically including: determine whether the number of data outliers is lower than a fourth threshold, if so, delete the data outliers, otherwise, replace the data outliers with replacement values, and save the replacement values ​​as new data categories; Processing of noisy data; The data information corresponding to each candidate attribute is unified, standardized and normalized.

5. The data rights management method for a big data platform according to claim 3, characterized in that: The method for constructing the matching condition library includes: Presetting the applicable object set specifically includes: dividing objects into adapted applicable object sets based on business requirements; Divide the permission levels into: read-only permission, read-write permission, and full control permission; Assign corresponding applicable object sets, key attributes and permission levels to managers, and build a matching condition library based on the assignment results.

6. The data rights management method for a big data platform according to claim 2, characterized in that: The step of extracting key attributes from candidate attributes after data cleaning specifically includes: Transform and combine the data information corresponding to the candidate attributes after data cleaning to obtain multiple combined features; Select and train multiple combined features to obtain a feature selection model, evaluate the performance of the feature selection model, and dynamically adjust the selection of combined features according to the performance evaluation results until a feature selection model with the best performance evaluation results is obtained; Extract key attributes from candidate attributes corresponding to multiple combined features in the optimal feature selection model.

7. The data rights management method for a big data platform according to claim 6, characterized in that: The step of selecting and training the candidate attributes to obtain the attribute selection model includes screening the candidate attributes, specifically including: Analyze the correlation between each candidate attribute and the target variable, and eliminate candidate attributes whose correlation values ​​are lower than a first threshold; The correlations among the multiple candidate attributes are analyzed, and redundant candidate attributes with correlation values ​​higher than a second threshold are eliminated.

8. A data rights management system for a big data platform, characterized in that: It includes a module for implementing the data authority management method for a big data platform described in any one of claims 1-7.

9. A device, characterized in that: include: Memory; processor; as well as Computer programs; Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the data authority management method for a big data platform as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement a data authority management method for a big data platform as described in any one of claims 1-7.

Images