Dynamic authorization method based on reinforcement learning and policy optimization

Abstract

This invention discloses a dynamic authorization method based on reinforcement learning and policy optimization, belonging to the field of computer security access control technology. To address the security risks caused by non-compliant authorizations during authorization exploration in reinforcement learning, this invention generates action masks based on compliance rules and masks non-compliant actions. It performs risk assessment on the state to adaptively determine the upper limit of policy deviation, constrains the deviation of the target policy from the baseline policy within the compliance action space, and backs down in high-risk situations. At the same time, it adopts an offline reinforcement learning training strategy with masks, achieving the technical effect of dynamic authorization optimization under compliance premise and reducing the probability of high-risk decisions.

Description

Technical Field

[0001] This invention relates to the field of computer security access control, and in particular to a dynamic authorization method based on reinforcement learning and policy optimization. Background Technology

[0002] With the development of enterprise information systems, cloud computing, and microservice architectures, access control has evolved from early discretionary and mandatory access control to role-based and attribute-based access control. Building upon this, dynamic authorization mechanisms incorporating contextual information such as time, location, device status, and data hierarchy / domain have emerged. To improve the adaptability of authorization decisions to business changes, the industry has begun to introduce data-driven methods, such as log-based policy mining, machine learning-based risk assessment and anomaly detection, and reinforcement learning-based policy optimization for long-term benefits. These methods enable authorization policies to iteratively improve based on historical authorization results and subsequent impacts.

[0003] However, existing technologies still have the following shortcomings: 1. Reinforcement learning typically relies on trial and error. During online learning or policy updates, non-compliant or high-risk authorization actions may occur, which can easily lead to unacceptable consequences such as unauthorized access and data leakage, making it difficult to meet the hard constraints of security and compliance required by the production environment.

[0004] 2. In existing dynamic authorization schemes, compliance constraints are often superimposed on the policy output through rule verification or post-event interception. The inconsistent handling of the action space during the training and inference phases can easily lead to biases in the policy's valuation of non-compliant actions, thereby affecting the stability and security of the policy.

[0005] 3. Some strategy improvement methods use fixed strategy deviation limits to constrain the changes of the new strategy relative to the baseline strategy, but lack an adaptive mechanism linked to the risk level. This makes it difficult to fully optimize in low-risk scenarios and strictly converge in high-risk scenarios, thus failing to balance authorization effectiveness and security compliance requirements.

[0006] Therefore, a dynamic licensing method is needed to address the shortcomings of the existing technology. Summary of the Invention

[0007] One objective of this invention is to propose a dynamic authorization method based on reinforcement learning and policy optimization. Addressing the problems in existing technologies, such as potential non-compliant authorizations during reinforcement learning exploration, inconsistent constraints between the training and inference phases, and a lack of risk-adaptive constraints in policy improvement, this invention proposes a technical solution: generating action masks based on compliance rules and compliantly processing the action distributions of the baseline and target policies; adaptively determining the upper limit of policy deviation based on state risk scores; calculating and constraining the deviation of the target policy relative to the baseline policy within the compliant action space and backing up in high-risk situations; and employing offline reinforcement learning with action masks to train and suppress the value of non-compliant actions. This invention achieves the technical effects of continuously learning and optimizing authorization policies while ensuring compliance, reducing the probability of non-compliant authorizations and high-risk decisions, and improving the stability of authorization decisions.

[0008] This invention provides a dynamic authorization method based on reinforcement learning and policy optimization, comprising: S1, receiving a current authorization request and extracting features to obtain current state features; S2, matching the current state features with pre-stored compliance constraint rules to generate a current compliance action mask for a predefined set of authorization actions, used to indicate whether each authorization action in the set is a compliance action under the current state features; S3, performing a risk assessment on the current state features to obtain a current risk score and determining a current deviation upper limit, used to limit the policy deviation of the target policy relative to the baseline policy, and the current deviation upper limit decreases as the current risk score increases; S4, inputting the current state features into the baseline policy model to obtain baseline actions. The distribution of actions is calculated, and compliance processing is performed on the baseline action distribution based on the current compliance action mask to obtain the compliant baseline action distribution; S5, the current state features are input into the target policy model to obtain the target action distribution, and compliance processing is performed on the target action distribution based on the current compliance action mask to obtain the compliant target action distribution. The policy deviation between the target action distribution and the compliant baseline action distribution is calculated, and deviation constraint processing is performed on the compliant target action distribution under the current deviation upper limit constraint to obtain the constrained target action distribution that satisfies the current deviation upper limit. The target authorized action and the corresponding authorization decision are determined. The compliance processing includes masking the authorized actions that are indicated as non-compliant actions by the current compliance action mask so that the target authorized action is determined only from the compliant actions.

[0009] Optionally, S1 includes: The current authorization request is received from the access control interface. The current authorization request includes a subject identifier, a resource identifier, a request operation type, and environment information. The subject identifier, resource identifier, request operation type, and environmental information are parsed and normalized, and the normalized information is encoded to obtain the current state features, including subject features, resource features, operation features, and environmental features.

[0010] Optionally, S2 includes: Using the current state characteristics as input, the pre-stored compliance constraint rules are invoked, and compliance is determined for each authorized action in the predefined set of authorized actions one by one, to obtain a compliance mark corresponding to each authorized action. The compliance mark is used to indicate whether the corresponding authorized action meets the compliance constraint rules under the current state characteristics. The compliance tags corresponding to each authorized action are combined according to the action order of the predefined set of authorized actions to generate the current compliance action mask; the compliance constraint rules include at least one of the following: role-based access control rules, attribute-based access control rules, blacklist rules, whitelist rules, time window rules, geographic location rules, device security posture rules, data classification rules, data domain rules, and regulatory compliance library rules; when an update to the compliance constraint rules is detected, the current compliance action mask is regenerated for the same current state feature; Furthermore, while generating the current compliance action mask, compliance evidence information associated with the current compliance action mask is generated. The compliance evidence information includes the identifier of the triggered compliance constraint rule, the rule version number, and a summary of the judgment basis for giving compliance marks to each authorized action.

[0011] Optionally, S3 includes: Using the current state features as input, the current state features are input into the risk assessment model, and the current risk score is output. The current risk score is normalized to obtain a normalized current risk score; The normalized current risk score is compared with a pre-set set of risk thresholds. Based on the comparison result, the current deviation upper limit is determined from a pre-set set of deviation upper limits. The deviation upper limits in the deviation upper limit set correspond one-to-one with the risk thresholds in the risk threshold set, and the higher the risk threshold, the smaller the corresponding deviation upper limit. Furthermore, the risk assessment model further outputs an uncertainty metric value corresponding to the current risk score, and corrects the normalized current risk score based on the uncertainty metric value to obtain a corrected risk score. The current deviation upper limit is determined based on the corrected risk score, and the larger the uncertainty metric value, the smaller the current deviation upper limit. Furthermore, it also includes an out-of-distribution detection step: inputting the current state features into the state density evaluation model to obtain the state confidence score; when the state confidence score is less than the confidence score threshold, setting the current risk score to the risk score corresponding to the maximum risk threshold, or setting the current deviation upper limit to a preset minimum deviation upper limit.

[0012] Optionally, S4 includes: Using the current state features as input, the current state features are input into the baseline policy model, and the output is the baseline action probability distribution that corresponds one-to-one with the predefined set of authorized actions; Using the current compliant action mask as a constraint, the baseline action probability distribution is processed to make it compliant, so that the probability of the baseline action corresponding to the authorized action that is indicated as a non-compliant action by the current compliant action mask is zero. The baseline action probability distribution after compliance processing is normalized to obtain the compliant baseline action probability distribution.

[0013] Optionally, S5 includes: Using the current state features, current compliance action mask, current risk score, current deviation limit, and compliance baseline action probability distribution as inputs, the current state features are input into the target strategy model, and the output is the initial target action probability distribution that corresponds one-to-one with the predefined set of authorized actions. The initial target action probability distribution is compliant based on the current compliant action mask, so that the probability of the target action corresponding to the authorized action that is indicated as non-compliant by the current compliant action mask is zero. The initial target action probability distribution after the compliant action mask is then normalized to obtain the compliant target action probability distribution. Calculate the policy deviation between the compliance target action probability distribution and the compliance baseline action probability distribution, wherein the policy deviation is any one of the following: Kullback-Leibler divergence, Jensen-Shannon divergence, total variation distance, Wasserstein distance, first norm distance of the difference between the compliance target action probability distribution and the compliance baseline action probability distribution, and second norm distance of the difference between the compliance target action probability distribution and the compliance baseline action probability distribution; When the policy deviation is greater than the current deviation limit, the compliance target action probability distribution is subjected to deviation constraint processing to obtain a constrained target action probability distribution that satisfies the current deviation limit, and the constrained target action probability distribution is used as the target action probability distribution. The deviation constraint processing includes: determining a coefficient α with a value range of 0 to 1, such that the policy deviation between the constrained target action probability distribution obtained by multiplying the coefficient α by the compliance baseline action probability distribution and the compliance baseline action probability distribution is not greater than the current deviation limit. When the deviation of the strategy is not greater than the current deviation limit, the probability distribution of the compliance target action is used as the probability distribution of the target action; when the current risk score is not less than the risk rollback threshold, the compliance authorization action with the highest action probability is selected as the target authorization action based on the compliance baseline action probability distribution. When the current risk score is less than the risk rollback threshold, the compliant authorization action with the highest probability of action is selected as the target authorization action based on the target action probability distribution. Output the authorization decision corresponding to the target authorization action; wherein, the risk backoff threshold is the maximum risk threshold in a pre-set risk threshold set, and the risk threshold set is used to determine the current deviation upper limit in S3.

[0014] Optionally, the target strategy model is obtained through the following steps: acquiring historical authorization records, which include context information of authorization requests, historical authorization actions, and subsequent result information corresponding to the historical authorization actions; The historical authorization records were compiled into an offline training dataset; An offline training sample set is constructed based on the offline training dataset. Each offline training sample in the offline training sample set includes state features, authorized actions, and reward values. The state features are extracted from the context information, and the reward values ​​are determined by the subsequent result information. Based on the offline training sample set and the compliance constraint rules, a compliance action mask is generated for each state feature, and the compliance action mask is associated with the corresponding offline training sample to obtain a mask annotation sample set. A risk score is calculated for each state feature based on the masked annotation sample set, and a deviation upper limit is determined based on the risk score to obtain a risk-annotated sample set. The deviation upper limit decreases as the risk score increases. A baseline strategy model is constructed based on the risk-labeled sample set. The baseline strategy model is obtained by statistically estimating historical authorization actions or by training it through supervised learning with state features as input and historical authorization actions as supervision labels. The target policy model and the target value function model are trained using an offline reinforcement learning training method. The offline reinforcement learning training method includes a conservative value constraint term, and when calculating the conservative value constraint term, the authorized actions are limited to the set of compliant actions indicated by the compliant action mask. During the process of updating the target strategy model and the target value function model, non-compliant actions indicated by the compliant action mask are removed from the strategy output space, and the output value of the target value function model for non-compliant actions is made no greater than a preset lower bound value Qillegal, where Qillegal is a preset negative constant, so that non-compliant actions cannot be selected when selecting actions based on the target value function model. Based on the upper limit of deviation, a policy deviation constraint relative to the baseline policy model is imposed on the target policy model. The policy deviation constraint is bounded by the upper limit of deviation, and the upper limit of deviation decreases as the risk score increases, so that the policy deviation constraint corresponding to the state feature with the higher risk score is more stringent, thus obtaining the trained target policy model.

[0015] The beneficial effects of this invention are: 1. By using compliant action masks to process the action distribution of baseline and target policies in a compliant manner, and consistently eliminating non-compliant actions during the training and inference phases, we ensure that authorized actions are generated only from the set of compliant actions, thereby reducing the probability of non-compliant authorizations from the source and improving the compliance and controllability of the production environment.

[0016] 2. Risk scores are obtained through risk assessment, and the upper limit of policy deviation is adaptively determined based on these scores. The policy deviation of the target policy relative to the baseline policy is constrained, and a fallback to the baseline policy is triggered under high-risk conditions. This enables the policy optimization process to achieve differentiated convergence under different risk levels, reduces the probability of wrong authorization in high-risk scenarios, and improves decision stability.

[0017] 3. The target policy and value function are trained by offline reinforcement learning with compliance action masking. During the conservative value constraint calculation and policy update process, the actions are restricted to the compliance action space, and a lower bound is set for the value of non-compliant actions. This suppresses the overestimation of non-compliant actions, reduces dangerous policy output caused by offline data distribution bias, and improves the security and generalization ability of offline learning. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the dynamic authorization method based on reinforcement learning and policy optimization proposed in this invention; Figure 2 This is a flowchart illustrating the specific execution of step S5, the authorization decision, in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-2 A dynamic authorization method based on reinforcement learning and policy optimization is characterized by the following steps: S1. Receiving the current authorization request and extracting features to obtain the current state features; S2. Matching the current state features with pre-stored compliance constraint rules to generate a current compliance action mask for a predefined set of authorization actions, indicating whether each authorization action in the set is a compliant action under the current state features; S3. Performing a risk assessment on the current state features to obtain a current risk score and determining a current deviation upper limit to limit the policy deviation of the target policy relative to the baseline policy, wherein the current deviation upper limit decreases as the current risk score increases; S4. Inputting the current state features into the baseline policy model to obtain the baseline actions. The distribution of actions is calculated, and compliance processing is performed on the baseline action distribution based on the current compliance action mask to obtain the compliant baseline action distribution; S5, the current state features are input into the target policy model to obtain the target action distribution, and compliance processing is performed on the target action distribution based on the current compliance action mask to obtain the compliant target action distribution. The policy deviation between the target action distribution and the compliant baseline action distribution is calculated, and deviation constraint processing is performed on the compliant target action distribution under the current deviation upper limit constraint to obtain the constrained target action distribution that satisfies the current deviation upper limit. The target authorized action and the corresponding authorization decision are determined. The compliance processing includes masking the authorized actions that are indicated as non-compliant actions by the current compliance action mask so that the target authorized action is determined only from the compliant actions.

[0021] In this specific embodiment, S1 includes: The system receives a current authorization request from an access control interface, which is a network interface that sends and receives responses to the authorization service. The current authorization request is carried in a UTF-8 encoded structured message and includes a subject identifier. Resource Identifier Request operation type and environmental information The main logo A unique string identifier for an entity within the identity system, a resource identifier. A unique string identifier for the controlled resource in the resource directory; request operation type. For enumerated values ​​in a predefined set of operation types, environment information It consists of the request arrival timestamp, client network address, geolocation identifier, device identifier, and device security status field, and each field has a field name and a field value; For the main body identifier The resource identifier The request operation type and the environmental information The system performs parsing and normalization processes. The parsing includes matching field names and extracting field values ​​based on a preset field pattern. The normalization process includes removing leading and trailing whitespace and unifying case for string fields, setting fields that do not meet character set constraints to default identifiers, performing set member validation on enumeration fields and setting illegal enumerations to default enumerations, uniformly converting timestamp fields to UTC second-level time and deriving minute-level time bucket indexes, uniformly parsing network address fields to IPv4 or IPv6 and deriving network prefix indexes, uniformly parsing geolocation identifier fields to fixed-precision grid-encoded indexes, and generating integer status levels ranging from 0 to 4 for device security status fields based on four check items: patch status, system integrity, disk encryption, and malware protection, and further linearly scaling them to numerical features ranging from 0 to 1. When any environment field is missing, its corresponding index is set to 0 and its corresponding numerical feature is set to 0. The normalized information is encoded to obtain the current state features. The encoding process includes encoding the subject identifier. A 32-bit unsigned hash value was calculated using MurmurHash3, and the number of buckets was then used. The subject index is obtained by taking the modulus, and then the subject embedding matrix is ​​used. The main feature vector is obtained by looking up the table. And the embedding dimension is taken Resource Identifier Use the same hash and modulo method to count the number of buckets Obtain the resource index and use the resource embedding matrix Resource feature vectors are obtained by looking up the table. And the embedding dimension is taken Request operation type Mapping to operation indices and embedding the matrix through operations The operation feature vector is obtained by looking up the table. And the number of operation types is taken With Embedded Dimensions Environmental information The derived time bucket index, network prefix index, grid coding index, and device identifier index are used to look up several environmental sub-vectors through the corresponding environmental embedding matrix, and then concatenated with the situation level numerical features to form an environmental feature vector. And the total dimension of environmental features is taken The parameters of each embedding matrix are initialized during system initialization. The interval is uniformly distributed and assigned values, which are then used as trainable parameters in subsequent training. The embedding vector corresponding to the default index 0 is fixed as an all-zero vector to prevent missing fields from introducing additional bias. Main feature vector Resource feature vector , Operational feature vector and environmental feature vectors The current state feature vector is obtained by concatenating the components in a fixed order. Its form is: ; in Indicates the first The current state feature vector corresponding to this authorization request This represents the request sequence number, incremented in the order the requests arrived. Represents the main feature vector. Represents the resource feature vector. This represents the operation feature vector. Represents the environmental feature vector, symbol This represents the vector concatenation operation.

[0022] In this specific embodiment, S2 includes: With the current state feature vector Taking the normalized request context as input, it invokes pre-stored compliance constraint rules to apply a predefined set of authorization actions. Perform compliance checks one by one and generate a mask for the current compliance action. ; The compliance constraint rules are stored in the rule base and loaded and executed by the rule evaluation engine. The rule base maintains a rule identifier and rule version number for each rule and forms an immutable snapshot in memory to ensure the consistency of the judgment for the same authorization request. The rule evaluation engine executes role-based access control rules, attribute-based access control rules, blacklist rules, whitelist rules, time window rules, geolocation rules, device security posture rules, data classification rules, data domain rules, and regulatory compliance library rules in a fixed order. It outputs a corresponding compliance flag for each authorization action and adopts a short-circuit policy. If any rule determines that the authorization action is not satisfied, the authorization action is immediately judged as non-compliant and subsequent rule evaluation for the authorization action is stopped. At the same time, a default denial policy is adopted. If the authorization action is not explicitly allowed by the whitelist rule and does not meet the role-based access control rules and attribute-based access control rules, the authorization action is judged as non-compliant. In the set of authorized actions After completing the individual checks, the compliance tags of each authorized action are combined in the order of the actions to obtain the current compliant action mask. Each mask element is determined by the following formula: ; in Indicates the first The current state feature vector corresponding to this authorization request This represents the request sequence number, incremented according to the order in which requests arrive. This represents a predefined set of authorized actions. Indicates the first in the set of authorized actions One authorization action, Indicates the authorized action index and satisfies This represents the total number of actions in the authorized action set. This represents the current compliance action mask vector. This indicates the current compliance action mask vector and the authorized action. The corresponding mask element, This indicates that the state characteristics are determined by the rule evaluation engine based on a snapshot of the compliance constraint rules. With authorization action The boolean output obtained from the compliance determination is true, indicating compliance and allowing entry into the subsequent action distribution calculation, and false, indicating non-compliance and being blocked in subsequent steps. In generating the current compliance action mask Simultaneously, it generates associated compliance evidence information and binds it to the request identifier of this authorization request for storage. The compliance evidence information includes a set of compliance constraint rule identifiers triggered in this judgment, the rule version number corresponding to each rule in the rule identifier set, and a judgment basis summary that gives a compliance mark for each authorization action. The judgment basis summary is extracted by the rule evaluation engine from the normalized request context, including fields related to the triggering rule and their normalized values, and records the corresponding comparison relationship and judgment result. When it is detected that the global version number of the rule base changes between the start and end of this judgment, the rule evaluation engine performs a feature vector analysis on the same current state based on the updated rule base snapshot. With the same set of authorized actions Re-execute the compliance determination to regenerate the current compliance action mask. And simultaneously update the aforementioned compliance evidence information.

[0023] In this specific embodiment, S3 includes: With the current state feature vector The input is processed sequentially during the inference phase, consisting of two parts: out-of-distribution detection and risk assessment, as well as deviation upper limit determination. The out-of-distribution detection outputs state confidence through a state density evaluation model. The state density evaluation model is a Gaussian mixture model. And the number of mixed components is taken Each component employs a diagonal covariance matrix, and during the training phase, the parameters are optimized by performing 50 rounds of expectation maximization on the set of historical state feature vectors. Estimate and add covariance diagonal The regularization term is used to avoid singularities, and the state confidence is... To apply the Gaussian mixture model to The output log-likelihood is standardized by the mean and standard deviation of the log-likelihood of the training set and then mapped using a Sigmoid function, resulting in a value between 0 and 1. The confidence threshold is set to... ,when The current risk score is directly set to the risk score corresponding to the maximum risk threshold. And set the current deviation limit to the preset minimum deviation limit. To ensure that subsequent strategies deviate from the strictest constraints; when Time A risk assessment is performed, which outputs a raw risk score and an uncertainty measure value through a risk assessment model, wherein the risk assessment model is a multilayer perceptron with Dropout for uncertainty perception. Input dimensions and The dimensions are consistent and fixed at 104. The hidden layers are two fully connected layers with 128 and 64 neurons respectively. The activation function is ReLU. Dropout is placed at the output of each hidden layer with a fixed dropout probability. The output layer consists of a single neuron and is mapped via a Sigmoid function to obtain the risk probability output for a single forward pass. During the inference phase, Dropout remains enabled and is applied to the same... implement The second forward propagation yields eight risk probability outputs, whose arithmetic mean is used as the original risk score, and whose sample variance is used as the uncertainty measure. The original risk score is used as the normalized current risk score. To meet the normalization requirements of S3, and based on this, the risk is corrected according to uncertainty. The corrected risk score is then used as the current risk score. And determine it according to the following formula: ; in This represents the current risk score and is used for subsequent steps to determine the rollback threshold and upper limit of deviation. This represents the risk score corresponding to the maximum risk threshold in a predefined set of risk thresholds, and its value is... This represents the state confidence level output by the state density assessment model. This represents the confidence threshold and has a value of 0.2. This represents the normalized current risk score, with values ​​ranging from 0 to... Denotes the uncertainty penalty coefficient and takes the value of Indicates and The corresponding uncertainty measure, with a value ranging from 0 to 0.25; In obtaining Then, it is compared with a pre-set set of risk thresholds. Compare in ascending order and from a pre-set set of deviations from the upper limit. Determine the current deviation from the upper limit. , among which when season ,when season ,when season ,when season and output the current risk score. With respect to the current deviation upper limit .

[0024] In this specific embodiment, S4 includes: With the current state feature vector Input and invoke baseline policy model Output and a predefined set of authorized actions One-to-one baseline action probability distribution: ; Among them, the baseline strategy model Let be a multilayer perceptron classification model and its parameter set be denoted as . Input layer dimensions and The dimensions are consistent and fixed at 104, the hidden layers are two fully connected layers with 128 and 64 neurons respectively, the activation function is ReLU, and the output layer is... Each neuron outputs an action pair numerical vector. The baseline action probability distribution From the Softmax is performed to obtain and satisfy any All and satisfy ; Subsequently, based on the current compliance action mask For the baseline action probability distribution To make compliance processing possible The probability of an authorized action deemed non-compliant is zero, and the remaining probabilities are normalized to obtain the probability distribution of compliant baseline actions. The calculation is as follows: ; in This represents the probability distribution of compliance baseline actions. This indicates the mask for current compliance actions. Represents the baseline action probability distribution. Indication and Authorization Actions The corresponding mask element and This represents the baseline action probability distribution and the authorized action. The corresponding probability value, symbol This indicates element-wise multiplication. This represents the sum of probabilities corresponding to compliant actions. Represents a numerically stable term and takes the value of ; To ensure that the normalized denominator is strictly positive and to avoid division by zero due to all non-compliance, the authorized action set... Includes actions to deny authorization And its index is Compliance constraint rules Heng judged it to be compliant and thus met the requirements. and make ; The compliance baseline action probability distribution As a baseline input for subsequent steps to calculate policy deviation and execute high-risk rollbacks, it is associated with the request identifier of this authorization request and cached in memory.

[0025] In this specific embodiment, S5 includes: With the current state feature vector Current compliance action mask Current risk score Deviation from the current upper limit and the probability distribution of compliance baseline actions Generate the target authorization action for the input and output the authorization decision; Constructing the input vector of the target policy model The Depend on Scalar risk characteristics Scalar deviation from upper limit characteristic Vector features and vector features The data is obtained by splicing together in a fixed order and used to enable the target strategy model to perceive the current compliance space and risk constraint strength when outputting action distribution; Will Input target policy model Get and Authorize Action Set One-to-one initial target action pair numerical vector The target strategy model Let be a multilayer perceptron and its parameter set be denoted as . Input dimensions and The dimensions are consistent, the hidden layers are two fully connected layers with 256 and 128 neurons respectively, the activation function is ReLU, and the output layer is... 1 neuron and use Softmax to... Transform into the initial target action probability distribution And satisfy ; Subsequently, based on the current compliance action mask right Perform compliance processing and ensure all requirements are met. Authorization action The corresponding probability is set to zero, and the remaining probabilities are normalized by their sum to obtain the compliant target action probability distribution. Furthermore, a numerical stability term is added to the denominator during the normalization calculation. To avoid numerical underflow; In obtaining Then calculate its with Strategy deviation between The policy deviation Kullback-Leibler divergence was used and the calculation was performed using... and Truncate each element to its lower bound. To avoid undefined terms, if Then Determined as the probability distribution of the target action ,like Then for Performing deviation constraint processing yields the probability distribution of the constrained target action. and identified as The deviation constraint processing is performed through the interval Internal coefficients And construct a hybrid distribution to achieve the desired policy deviation that is no greater than [value missing]. The mixed distribution is calculated using the following formula: ; in This indicates that the current deviation from the upper limit is met. The probability distribution of the constrained target action, This indicates the deviation from the constraint coefficient and its value ranges from 0 to... This represents the probability distribution of compliance baseline actions. This represents the probability distribution of actions aimed at achieving compliance. Indicates the request sequence number in ascending order of arrival; The The algorithm uses a binary search to determine the location and the number of iterations is fixed at 20. The lower bound is initialized to 0 and the upper bound to 1. In each iteration, the algorithm is calculated using the current midpoint. And based on this, calculate and The policy deviation is used to update the upper and lower bounds to obtain a result that makes the policy deviation no greater than 1. and as close as possible of ; During the action selection phase, the risk fallback threshold is set as follows: And it is consistent with the maximum risk threshold in step S3, when Time based Select the compliant authorization action with the highest probability as the target authorization action. ,when Time based Select the compliant authorization action with the highest probability as the target authorization action. When multiple actions have the same maximum probability, the action with the smallest index is selected to ensure determinism. Final output and target authorization action The corresponding authorization decision is encoded as allow or deny, along with the corresponding operation type and constraints, where the set of authorization actions is... Includes actions to deny authorization Furthermore, it is always compliant in S2 to ensure that a definite authorization decision can be output in any state.

[0026] In this specific implementation, the target strategy model Obtained through offline training, with the training process executed on an offline training server; Historical authorization records are retrieved from the audit logs of the access control system and formed into a time-sorted sequence. Each historical authorization record includes context information, the historical authorization action, and subsequent result information corresponding to the historical authorization action. The context information includes a subject identifier. Resource Identifier Request operation type and environmental information The historical authorization actions are recorded as and The subsequent result information is a set of authorized actions consistent with online inference, and includes business execution result markers and security event markers; The historical authorization records were compiled into an offline training dataset. Following the parsing, normalization, and encoding process of S1, the context information of each historical authorization record is converted into a state feature vector. Simultaneously, a subsequent state feature vector is constructed based on the context information of the next historical authorization record. And organize each sample into Among them, the return value The determination is based on subsequent results and uses deterministic mapping rules. Specifically, when the business execution result is marked as successful and the security event is marked as no event, the mapping is set. When the business execution result is marked as failed or timed out and the security event is marked as no event, set When a security event is marked as any of the following: unauthorized access, data breach, or high-risk alert, set to... ; Subsequently, the offline training dataset was analyzed. Each state feature vector in Invoke compliance constraint rule snapshots and generate compliance action masks using the rule evaluation engine. And bind them with samples to form masked labeled samples, and the authorized action set is used to ensure consistency between training and inference. Includes actions to deny authorization And ensure that it remains compliant during rule assessments; After obtaining the masked annotation samples, for each Calling the state density evaluation model With risk assessment model Calculate the current risk score And based on this, according to the risk threshold set With deviation from the upper limit set Determine the deviation from the upper limit This yields a risk-labeled sample set; A baseline strategy model is trained based on the aforementioned risk-labeled sample set. The baseline strategy model Employing a multilayer perceptron structure with input as The output is an action distribution, and the training method is supervised learning based on historical authorized actions. As the supervised label minimizes the cross-entropy loss, the optimizer uses Adam and the learning rate is set to... The batch size is set to 256, the number of training rounds is set to 20, and the early stop threshold for action accuracy on the validation set is fixed at 3 consecutive rounds without improvement. In obtaining the baseline strategy model Subsequently, offline reinforcement learning with compliant action masks is used to simultaneously train the target policy model. With the target value function model The target strategy model Employing a multilayer perceptron structure and using For input and Depend on and by exist The compliance baseline action probability distribution obtained by outputting the above and then masking it for compliance is as follows. The model is constructed by piecing together components to form the target value function model. It is a multilayer perceptron and the input is The output is a scalar Its network structure consists of two hidden layers with 256 and 128 neurons respectively, and the activation function is ReLU. Offline reinforcement learning training uses discount factors A dual-Q network is adopted and target network parameters are introduced. With stable training and target network soft update coefficients taken Each iteration starts from The parameters are sampled and updated once with a batch size of 256, and the total number of iterations is 200,000. The value function model is updated using a loss function that includes a conservative value constraint term and a lower bound constraint term for non-compliant actions, and the lower bound constant for the value of non-compliant actions is set to... The value function loss is: ; in This represents the parameters used to update the target value function model. The loss of value, This indicates the offline training dataset. Calculate the expectation of the sample obtained from sampling. Indicates the first The state feature vector of each sample Indicates the first The historical authorization actions of the sample Indicates the first The return value of a sample Indicates the first The successor state feature vector of each sample, Indicates the first The compliance action mask vector of the sample. express In and authorized actions The corresponding mask element and The objective value function model represents the state. With action Value output, Indicates the time difference target value and is calculated as follows: ; And in the calculation exist The distribution of actions on To achieve compliance, obtain the probability of compliance target actions. The parameter is Target value network Indicates the weight of the conservative value constraint term and takes Represents the probability distribution of compliance baseline actions. In and action The corresponding probability value, Indicates the weight of the lower bound constraint term for non-compliant actions and takes This represents a lower bound constant for the value of non-compliant actions and takes... The term and the linear term together constitute a conservative value constraint and are expressed through coefficients. This makes it effective only within the set of compliant actions. This is used to punish non-compliant actions whose value is higher than the lower bound, thus making the non-compliant actions undesirable at the value level; In updating the target strategy model At that time, it will be by Non-compliant actions are removed from the policy output space, and the remaining actions are normalized before calculating the policy objective. Simultaneously, a policy model relative to the baseline is applied to the target policy. The strategy deviates from the constraint and the deviation strength is determined by Control, specifically for each Calculate the Kullback-Leibler divergence between the compliance target action distribution and the compliance baseline action distribution, and use... Perform a projection update as the upper limit, resulting in a higher risk score. Under this state, the policy deviation constraint is more stringent and consistent with the inference phase constraint of S5, and is solidified after training is completed. parameter And As the target policy model invoked in S5.

[0027] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0028] This invention employs a combined mechanism of "offline reinforcement learning, security policy improvement, and compliance action masking" to achieve closed-loop suppression of technical problems. First, a compliance action mask is generated based on compliance constraint rules, and during authorization inference, the action distributions of the baseline and target policies are simultaneously masked and normalized, ensuring that authorized actions can only be generated from the set of compliant actions. This eliminates non-compliant authorization outputs that may result from reinforcement learning trial and error. Second, a policy deviation constraint relative to the baseline policy is introduced within the compliant action space, and the upper limit of deviation is adaptively determined based on the risk score of the current state. This allows the target policy to deviate moderately in low-risk states to improve authorization benefits, while in high-risk states, deviation is strictly limited and can trigger a fallback to the baseline policy. This achieves dynamic learning and optimization of authorization policies without affecting production safety and compliance, reducing the probability of misauthorization and high-risk decisions and improving decision stability.

[0029] This invention addresses the aforementioned technical problems by making safety and compliance-oriented improvements to the algorithm structure: First, it constructs a compliance constraint layer consistent with training and inference, integrating action masks throughout the baseline policy output processing, target policy output processing, and action selection process, avoiding inconsistencies between policy output and constraints caused by post-event verification. Second, it proposes a mask-aware offline reinforcement learning training method, limiting actions to the set of compliant actions during conservative value constraint calculation and policy update, and setting a lower bound on the value of non-compliant actions, making non-compliant actions undesirable in both value estimation and policy selection, thereby reducing dangerous policies caused by offline data distribution bias. Third, it proposes a risk-adaptive SPIBB deviation radius mechanism, dynamically adjusting the policy improvement intensity according to the risk level, further enhancing safety and controllability in high-risk scenarios, and better achieving the technical effect of "safe exploration and constraint policy optimization".

Claims

1. A dynamic authorization method based on reinforcement learning and policy optimization, characterized in that, include: S1. Receive the current authorization request, perform feature extraction, and obtain the current state features; S2. Match the current state characteristics with the pre-stored compliance constraint rules to generate a current compliance action mask for the predefined set of authorized actions, which is used to indicate whether each authorized action in the set of authorized actions is a compliant action under the current state characteristics. S3. Conduct a risk assessment on the current state characteristics to obtain the current risk score and determine the current deviation limit, which is used to limit the deviation of the target strategy from the baseline strategy. The current deviation limit decreases as the current risk score increases. S4. Input the current state features into the baseline policy model to obtain the baseline action distribution, and perform compliance processing on the baseline action distribution based on the current compliant action mask to obtain the compliant baseline action distribution. S5. Input the current state features into the target policy model to obtain the target action distribution, and perform compliance processing on the target action distribution based on the current compliance action mask to obtain the compliant target action distribution. Calculate the policy deviation between the target action distribution and the compliance baseline action distribution, and perform deviation constraint processing on the compliant target action distribution under the current deviation upper limit constraint to obtain the constrained target action distribution that satisfies the current deviation upper limit. Determine the target authorized action and the corresponding authorization decision. Compliance processing includes masking authorized actions that are identified as non-compliant actions by the current compliance action mask, so that the target authorized action is determined only from compliant actions.

2. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, S1 includes: The current authorization request is received from the access control interface. The current authorization request includes a subject identifier, a resource identifier, a request operation type, and environment information. The subject identifier, resource identifier, request operation type, and environmental information are parsed and normalized, and the normalized information is encoded to obtain the current state features, including subject features, resource features, operation features, and environmental features.

3. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, S2 include: Using the current state characteristics as input, the pre-stored compliance constraint rules are invoked, and compliance is determined for each authorized action in the predefined set of authorized actions one by one, to obtain a compliance mark corresponding to each authorized action. The compliance mark is used to indicate whether the corresponding authorized action meets the compliance constraint rules under the current state characteristics. The compliance tags corresponding to each authorized action are combined according to the action order of the predefined set of authorized actions to generate the current compliance action mask; The compliance constraint rules include at least one of the following: role-based access control rules, attribute-based access control rules, blacklist rules, whitelist rules, time window rules, geolocation rules, device security posture rules, data classification rules, data domain rules, and regulatory compliance library rules. When an update to the compliance constraint rule is detected, a new compliance action mask is generated for the same current state feature.

4. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, S3 includes: Using the current state features as input, the current state features are input into the risk assessment model, and the current risk score is output. The current risk score is normalized to obtain a normalized current risk score; The normalized current risk score is compared with a pre-set set of risk thresholds. Based on the comparison result, the current deviation upper limit is determined from a pre-set set of deviation upper limits. The deviation upper limits in the deviation upper limit set correspond one-to-one with the risk thresholds in the risk threshold set, and the higher the risk threshold, the smaller the corresponding deviation upper limit.

5. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, S4 includes: Using the current state features as input, the current state features are input into the baseline policy model, and the output is the baseline action probability distribution that corresponds one-to-one with the predefined set of authorized actions; Using the current compliant action mask as a constraint, the baseline action probability distribution is processed to make it compliant, so that the probability of the baseline action corresponding to the authorized action that is indicated as a non-compliant action by the current compliant action mask is zero. The baseline action probability distribution after compliance processing is normalized to obtain the compliant baseline action probability distribution.

6. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, S5 include: Using the current state features, current compliance action mask, current risk score, current deviation limit, and compliance baseline action probability distribution as inputs, the current state features are input into the target strategy model, and the output is the initial target action probability distribution that corresponds one-to-one with the predefined set of authorized actions. The initial target action probability distribution is compliant based on the current compliant action mask, so that the probability of the target action corresponding to the authorized action that is indicated as non-compliant by the current compliant action mask is zero. The initial target action probability distribution after the compliant action mask is then normalized to obtain the compliant target action probability distribution. Calculate the policy deviation between the compliance target action probability distribution and the compliance baseline action probability distribution, wherein the policy deviation is any one of the following: Kullback-Leibler divergence, Jensen-Shannon divergence, total variation distance, Wasserstein distance, first norm distance of the difference between the compliance target action probability distribution and the compliance baseline action probability distribution, and second norm distance of the difference between the compliance target action probability distribution and the compliance baseline action probability distribution; When the policy deviation is greater than the current deviation limit, the compliance target action probability distribution is subjected to deviation constraint processing to obtain a constrained target action probability distribution that satisfies the current deviation limit, and the constrained target action probability distribution is used as the target action probability distribution. The deviation constraint processing includes: determining a coefficient α with a value range of 0 to 1, such that the policy deviation between the constrained target action probability distribution obtained by multiplying the coefficient α by the compliance baseline action probability distribution and the compliance baseline action probability distribution is not greater than the current deviation limit. When the deviation of the strategy is not greater than the current deviation limit, the probability distribution of the compliant target action is used as the probability distribution of the target action. When the current risk score is not less than the risk rollback threshold, the compliance authorization action with the highest probability of action is selected as the target authorization action based on the compliance baseline action probability distribution. When the current risk score is less than the risk rollback threshold, the compliant authorization action with the highest probability of action is selected as the target authorization action based on the target action probability distribution. Output the authorization decision corresponding to the target authorization action; The risk rollback threshold is the maximum risk threshold in a pre-set set of risk thresholds, and the set of risk thresholds is used in S3 to determine the current deviation upper limit.

7. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 1, characterized in that, The target policy model is obtained through the following steps: Obtain historical authorization records, which include context information of authorization requests, historical authorization actions, and subsequent result information corresponding to the historical authorization actions; The historical authorization records were compiled into an offline training dataset; An offline training sample set is constructed based on the offline training dataset. Each offline training sample in the offline training sample set includes state features, authorized actions, and reward values. The state features are extracted from the context information, and the reward values ​​are determined by the subsequent result information. Based on the offline training sample set and the compliance constraint rules, a compliance action mask is generated for each state feature, and the compliance action mask is associated with the corresponding offline training sample to obtain a mask annotation sample set. A risk score is calculated for each state feature based on the masked annotation sample set, and a deviation upper limit is determined based on the risk score to obtain a risk-annotated sample set. The deviation upper limit decreases as the risk score increases. A baseline strategy model is constructed based on the risk-labeled sample set. The baseline strategy model is obtained by statistically estimating historical authorization actions or by training it through supervised learning with state features as input and historical authorization actions as supervision labels. The target policy model and the target value function model are trained using an offline reinforcement learning training method. The offline reinforcement learning training method includes a conservative value constraint term, and when calculating the conservative value constraint term, the authorized actions are limited to the set of compliant actions indicated by the compliant action mask. During the process of updating the target strategy model and the target value function model, non-compliant actions indicated by the compliant action mask are removed from the strategy output space, and the output value of the target value function model for non-compliant actions is made no greater than a preset lower bound value Qillegal, where Qillegal is a preset negative constant, so that non-compliant actions cannot be selected when selecting actions based on the target value function model. Based on the upper limit of deviation, a policy deviation constraint relative to the baseline policy model is imposed on the target policy model. The policy deviation constraint is bounded by the upper limit of deviation, and the upper limit of deviation decreases as the risk score increases, so that the policy deviation constraint corresponding to the state feature with the higher risk score is more stringent, thus obtaining the trained target policy model.

8. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 3, characterized in that, While generating the current compliance action mask, compliance evidence information associated with the current compliance action mask is generated. The compliance evidence information includes the identifier of the triggered compliance constraint rule, the rule version number, and a summary of the judgment basis for giving compliance marks to each authorized action.

9. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 4, characterized in that, The risk assessment model further outputs an uncertainty metric corresponding to the current risk score, and corrects the normalized current risk score based on the uncertainty metric to obtain a corrected risk score. The current deviation upper limit is determined based on the corrected risk score, and the larger the uncertainty metric, the smaller the current deviation upper limit.

10. The dynamic authorization method based on reinforcement learning and policy optimization according to claim 4, characterized in that, The method further includes an out-of-distribution detection step: inputting the current state features into the state density evaluation model to obtain the state confidence score; when the state confidence score is less than the confidence threshold, setting the current risk score to the risk score corresponding to the maximum risk threshold, or setting the current deviation upper limit to a preset minimum deviation upper limit.

Images