Policy Gradient Method for Regulation Recommendation

Abstract

The present invention discloses a policy gradient method for regulation recommendation, comprising the following steps: obtaining situation data x, regulation y, and regulations matching x; selecting K regulations that do not match x but are most similar as negative samples to reject false matches; using rule-based one-hot encoding and classifier-encoded semantic vectors for the structured situation data to obtain intermediate features, which are then concatenated to obtain a situation vector; for regulation y, using an LSTM to extract feature vectors from the regulation text word vectors; using an MLP evaluator to learn the matching relationship between the two and determine whether the policy network has learned correctly; the policy network learns the matching results of the situation and regulations, and optimizes the policy network with the goal of maximizing the total reward obtained by the policy network on all sampled situation data; and outputting the matching results of the situation and regulation data. The present invention can automatically output the matching results of the situation and regulation data.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a policy gradient method for regulation recommendation. Background Art

[0002] Regulation data is a text-based data that specifies a specific measure to be taken in response to a specific situation. For example:

[0003] Regulation 1-1: When a vessel of type XX from country XX is discovered conducting XX activities in the XX area, XX type of equipment may be dispatched to implement XX measures. If the vessel exhibits XX behavior, XX measures may be taken. Situational data is information about the current state of the target and is structured data composed of multiple elements. Natural language analysis requires analyzing whether structured data exists within text data and recommending matching data, automatically filtering the required data from massive amounts of data. Existing research on matching regulatory data with situational data is insufficient. Summary of the Invention

[0004] In view of this, the present invention proposes a policy gradient method for regulation recommendation.

[0005] The policy gradient method for rule recommendation disclosed in the present invention, wherein the rule recommendation is to determine whether the situation and rule data match, includes the following steps:

[0006] Obtaining situational data x , Regulations y and with x Matching regulations;

[0007] Select and x The K most similar but unmatched rules are used as negative samples to reject false matches. For structured situation data, rule-based one-hot encoding and classifier encoding semantic vectors are used to obtain their intermediate features, and then the situation vector is obtained by splicing. ;

[0008] For regulation data y , use LSTM to extract feature vectors from regulation text word vectors ;

[0009] Use the MLP evaluator to learn the matching relationship between the two, and judge whether the policy network learning is correct based on the matching relationship label. If it is correct, a positive reward is given, otherwise a negative reward is given;

[0010] The policy network learns the matching results of situations and regulations, and optimizes the policy network with the goal of maximizing the total reward obtained by the policy network on all sampled situation data;

[0011] Output the matching results of situation and regulation data.

[0012] Furthermore, the strategy network includes two modules: an agent and an environment env. The agent initializes the state and takes actions according to a random strategy. The actions change the state in the environment. At the same time, the environment generates a feedback signal. The agent adjusts the strategy according to the feedback signal and takes actions according to the adjusted strategy. The agent maximizes the total reward by continuously interacting with the environment env.

[0013] Furthermore, the environment includes historical information, state space, action space, and state transition process, where the historical information is defined as follows:

[0014] Indicates a size of The historical information of the episode, For one of them to transfer;

[0015] For each situation, select the unmatched The most similar regulations are obtained candidate pairs, then define the state space as: ,

[0016] among them Expression and posture Matching regulations, Expression and posture No. Similar but not matching regulations ( );

[0017] Adjusting the strategy is divided into two steps: one is strategy evaluation, which obtains the rewards that can be obtained by taking each action in the current strategy; the other is strategy improvement, which selects the optimal action based on the results of strategy evaluation; these two steps are continuously iterated and updated until convergence to obtain the optimal strategy.

[0018] Furthermore, in the process of strategy improvement, randomness is introduced into the strategy for exploration, allowing the agent to Choose to skip the current state and explore to ensure the dynamic nature of the learning process. The specific method is as follows:

[0019] For samples with greater learning difficulty, skip them with a certain probability and use the following method to evaluate the samples Learning difficulty:

[0020] ;

[0021] in is a hyperparameter, for peace The most similar regulations, is a similarity evaluation method, or the cosine similarity of word vectors is used. hour, and The greater the difference, the more difficult it is to learn the sample; when On the contrary, for samples with greater learning difficulty, they should be skipped with a greater probability, that is, they should be sampled less frequently. The following method is used to adjust the t Episode No. k The probability of a sample being skipped is the randomness of the exploration strategy:

[0022] ;

[0023] Among them, the hyperparameters is the discount factor, hyperparameter is the minimum skip probability.

[0024] Further, Here’s how:

[0025] By using named entity recognition or manual annotation methods, the regulation data according to The schema is structured as ;

[0026] right and Use the situation data encoder to extract features ;

[0027] Evaluate the importance of each dimension feature and configure the weight vector according to the importance ;

[0028] calculate , yes V x The transpose of .

[0029] Furthermore, the policy network is updated using the following formula:

[0030] in, For strategy;

[0031] The present invention can automatically output the matching results of situation and regulation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Model architecture diagram of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0034] This method defines policy recommendation as a decision problem involving the matching of situations and policies. The process of continuously determining the matching of each situation with its candidate policies is defined as a sequential decision process. Therefore, finding the optimal strategy for sequential matching is a sequential decision problem. This method constructs a policy network to learn the matching results between situations and policies. The policy network is then optimized with the goal of maximizing the total reward obtained by the policy network across all (sampled) situation data, thereby solving the sequential decision problem.

[0035] Figure 1 The structure of the policy gradient method-based regulation recommendation model is shown. For the input situation data x, regulation y, and regulations matching x, this method selects the K regulations that do not match x but are most similar as negative samples to reject false matches. For structured situation data, we use rule-based one-hot encoding and classifiers such as textCNN to encode semantic vectors to obtain their intermediate features, and then splice them to obtain the situation vector. ; For the regulation text data y, this method uses LSTM to advance the feature vector from the regulation text word vector Finally, the MLP evaluator is used to learn the matching relationship between the two, and based on the matching relationship label, it is judged whether the strategy network learning is correct. If it is correct, a positive reward is given, otherwise a negative reward is given.

[0036] Situation data: The current status information of the target, which is structured data composed of multiple elements, such as name, type, faction, longitude, latitude, historical activity type and number, etc. Structured data defined as follows:

[0037] .

[0038] The input and output of the present invention are as follows:

[0039] 1) Input: situation-rule pair, encoding of situation data, and matching (recommended) label of situation-rule pair;

[0040] 2) Output: The matching (recommendation) result (yes or no) of the predicted situation-regulation pair.

[0041] The reinforcement learning process of the present invention is defined as follows:

[0042] Reinforcement learning primarily consists of two modules: the agent and the environment. The figure below illustrates the interaction between the agent and the environment: First, the agent initializes its state and takes actions according to a random policy. These actions cause the state of the environment to change, and the environment generates feedback signals. The agent then adjusts its policy based on these feedback signals and takes actions based on the adjusted policy. Through continuous interaction with the environment, the agent maximizes its total reward.

[0043] Agent module

[0044] For the regulation recommendation problem, define the regulation dataset , situation data set ,in Represents Matching regulations;

[0045] state : A state ,in Indicates a situation, Indicates an ordinance;

[0046] Status Label : ,for ,when When matching, ;otherwise ;

[0047] action :action ,when When , the strategy selected by the agent is: If it matches, otherwise evaluate for mismatched;

[0048] Strategy :Strategy It is a mapping from state to action, indicating the agent's response to the state middle Evaluation methods for matching problems;

[0049] award : , where positive integer Is a hyperparameter. When the action selected is correct (the action matches the label), the reward is 1 (positive feedback); when the action selected is wrong, the reward is (Negative feedback).

[0050] Environment Env module

[0051] The state space, action space, and state transition process are all defined by the environment. This method defines some elements of the environment as follows:

[0052] Historical Information: Indicates a size of The historical information of the episode (period), For one of them to transfer;

[0053] State space: For the above rule recommendation problem, select the unmatched The most similar regulations are obtained candidate pairs, then define the state space as: , among which Expression and posture Matching regulations, Expression and posture No. Similar but not matching regulations ( ) .

[0054] Stochastic Exploration Method: Model-based reinforcement learning consists of two steps: strategy evaluation, which determines the reward for taking each action in the current strategy; and strategy improvement, which selects the optimal action based on the strategy evaluation results. These two steps are iterated and updated continuously until convergence and the optimal strategy is obtained. In the process of strategy improvement, reinforcement learning often introduces randomness into the strategy for exploration, so that the agent is allowed to take actions with a small probability. The option to skip the current state for exploration ensures the dynamic nature of the learning process.

[0055] Exploration strategy: For samples with high learning difficulty, they are skipped with a certain probability. This method uses the following method to evaluate samples Learning difficulty:

[0056] ;

[0057] in is a hyperparameter, for peace The most similar regulations, is the similarity evaluation method described in Algorithm 1. You can also use evaluation indicators such as the cosine similarity of word vectors. From the above formula, we can see that when hour, The greater the difference, the more difficult it is to learn the sample; when For samples with greater learning difficulty, they should be skipped with a greater probability, that is, they should be sampled less frequently. The present invention adjusts the first tThe probability that the kth sample of an episode is skipped is the randomness of the exploration strategy:

[0058] ;

[0059] Among them, the hyperparameters is the discount factor, hyperparameter is the minimum skip probability.

[0060] Model construction of the present invention

[0061] The method of the present invention mainly includes two important parts: situation data encoder and exploration strategy.

[0062] Situation data encoder: For structured situation data (subject type, power, geographic location, etc.), different encoding methods are selected according to the characteristics of the data. By splicing the encoding of each dimension of data, the overall embedding of the situation data is formed. First, for data with abstract data features, data annotation can be performed first, and then a multi-classification model can be trained as a feature extractor, such as using textCNN to extract semantic features for activity types. For data with more specific features, rule reasoning can be used directly to obtain their one-hot encoding. For example, encode country X, country Y, country Z, and country J as ; Through geometric relationship calculation, the specific latitude and longitude values (geographic location) can be classified into pre-divided areas, thereby obtaining their one-hot encoding.

[0063] Exploration strategy: Since samples with high learning difficulty are prone to introduce more errors, this method uses the PRsim method to approximately evaluate the similarity of the regulation-state data pairs, and then evaluate the learning difficulty of the samples. Then, by sampling fewer samples with high learning difficulty, the introduction of bias is reduced.

[0064] Algorithm 1: Policy gradient-based regulation recommendation algorithm:

[0065] ;

[0066]

[0067] The following describes a method for generating a situation-rule pair dataset, which generates a certain number of candidate rule pairs for each situation data.

[0068] Algorithm 2: Situation-rule pair dataset acquisition method

[0069]

[0070] in The algorithm is defined as follows:

[0071] Algorithm 3: Situation-regulation matching evaluation method PRsim

[0072]

[0073] 1. Recommended steps for the regulations

[0074] 1) Situation-rule pair dataset construction: manually label each situation data to find the matching rule, initialize the parameter K, and obtain the situation-rule pair dataset according to Algorithm 2 , and at a ratio of 8:2 Divide into training set and validation set;

[0075] 2) Model construction and training of the present invention: Figure 1 Build a policy network, input the training set, and adjust the model hyperparameters Perform training and output the trained strategy network;

[0076] 3) Policy Recommendation: Traverse the policy dataset, input the situation data and policy data into the trained policy network, and output the action. If the action is 1, it means the policy is recommended, otherwise it is not recommended.

[0077] The present invention can automatically output the matching results of situation and regulation data.

[0078] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0079] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0080] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0081] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. The policy gradient method recommended by the regulations is characterized by: The regulation recommendation is to determine whether the situation and regulation data match, including the following steps: Obtaining situational data x , Regulations y and with x Matching regulations; Select and x The K most similar but unmatched rules are used as negative samples to reject false matches. For structured situation data, rule-based one-hot encoding and classifier encoding semantic vectors are used to obtain their intermediate features, and then the situation vector is obtained by splicing. ; For regulation data y , use LSTM to extract feature vectors from regulation text word vectors ; Use the MLP evaluator to learn the matching relationship between the two, and judge whether the policy network learning is correct based on the matching relationship label. If it is correct, a positive reward is given, otherwise a negative reward is given; The policy network learns the matching results of situations and regulations, and optimizes the policy network with the goal of maximizing the total reward obtained by the policy network on all sampled situation data; Traverse the regulation data set, input the situation data and regulation data into the trained policy network, and output the action. If the action is 1, it means that the regulation is recommended, otherwise it is not recommended.

2. The policy gradient method for rule recommendation according to claim 1, characterized in that: The strategy network includes two modules: the intelligent agent and the environment env. The intelligent agent initializes the state and takes actions according to the random strategy. The actions change the state in the environment. At the same time, the environment generates feedback signals. The intelligent agent adjusts the strategy according to the feedback signals and takes actions according to the adjusted strategy. The intelligent agent maximizes the total reward by continuously interacting with the environment env.

3. The policy gradient method for rule recommendation according to claim 2, characterized in that: The environment includes historical information, state space, action space, and state transition process, where historical information is defined as follows: Indicates a size of The historical information of the episode, For one of the transitions, s is the state, r is the reward, a is the action, and episode is the period; For each situation, select the unmatched The most similar regulations are obtained candidate pairs, then define the state space as: , in Expression and posture Matching regulations, Expression and posture No. Similar but not matching regulations ( ); Adjusting the strategy is divided into two steps: one is strategy evaluation, which obtains the rewards that can be obtained by taking each action in the current strategy; the other is strategy improvement, which selects the optimal action based on the results of strategy evaluation; these two steps are continuously iterated and updated until convergence to obtain the optimal strategy.

4. The policy gradient method for rule recommendation according to claim 3, characterized in that: In the process of strategy improvement, randomness is introduced into the strategy for exploration, allowing the agent to Choose to skip the current state to explore, ensuring the dynamic nature of the learning process, Adjust the following t Episode No. k The probability of a sample being skipped is the randomness of the exploration strategy: ； Among them, the hyperparameters is the discount factor, hyperparameter is the minimum skip probability, The learning difficulty is based on the situation and regulations The similarity between the situation and its most similar regulation The similarity is evaluated by the difference value.

5. The policy gradient method for rule recommendation according to claim 4, characterized in that: Update the policy network using the following formula: ； in, For the strategy, hyperparameters is the adjustment factor.

Images