A Relation Extraction Method Based on Reinforcement Learning

Abstract

The invention discloses a method for relation extraction based on reinforcement learning. The relation extraction model based on reinforcement learning comprises an instance selector based on reinforcement learning and a relation classifier based on a PCNN model. The instance selector based on reinforcement learning obtains a selection result through a reinforcement learning process, and the selection result is a sentence that best expresses a package label in a package composed of sentences of the same entity pair. The selection result is input into the relation classifier, and sentences expressing the same relationship are identified according to the input sentences, and the instance selector is fed back. A strategy function is updated according to the feedback, and a better relation classifier is trained with the selected sentences to optimize the instance selection and relation classification process. Finally, the sentence that best represents the relation label is selected as the training data of the relation extraction model based on reinforcement learning. The invention reduces the influence of noise and coarse-grained problems of a remote supervision corpus on relation extraction, and improves the performance of relation extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a relationship extraction method based on reinforcement learning. Background Art

[0002] Information extraction is a basic task in natural language processing. It extracts structured information from unstructured text as input for subsequent natural language processing tasks. Relation extraction is an important part of information extraction technology. It classifies the semantic relationship between two entities in a sentence, automatically identifies a pair of concepts and the semantic relationship between the two concepts, and forms a triple, which can be used to construct a knowledge graph. Traditional relationship extraction research methods are mainly divided into four categories: supervised, semi-supervised, weakly supervised, and unsupervised. Supervised relationship extraction algorithms are the mainstream direction of current research.

[0003] The research objects of relation extraction mainly include the following categories: 1) structured data obtained from the Internet; 2) semi-structured data crawled from various web pages on the Internet; 3) widely existing unstructured text data. Supervised relation extraction algorithms have achieved good performance, but this method is heavily dependent on NLP annotation tools such as part-of-speech tagging. The errors of these tools will be transferred to the relation extraction task, and annotating a large amount of corpus requires a lot of manpower, and the data scale in practical applications is often large.

[0004] The relationship extraction algorithm based on distant supervision solves the problem of requiring a large amount of labeled data. It can use the existing knowledge base to annotate text data and migrate existing supervised methods. However, the heuristic matching of distant supervision will generate a large amount of noise data and error propagation. Summary of the invention

[0005] The purpose of the present invention is to propose a relationship extraction method based on reinforcement learning to improve the performance of relationship extraction in order to solve the problems of noisy data and coarse-grained supervision signals in the remote supervision relationship extraction model in the above-mentioned prior art.

[0006] The technical solution to achieve the purpose of the present invention is as follows: A relationship extraction method based on reinforcement learning, comprising the following steps:

[0007] Step S1: constructing a relation extraction model framework based on reinforcement learning, wherein the relation extraction model framework includes an instance selector based on reinforcement learning and a relation classifier based on a PCNN model;

[0008] Step S2: the instance selector based on reinforcement learning obtains a selection result through a reinforcement learning process, and the selection result is a sentence that best expresses the package label in the package consisting of sentences of the same entity pair;

[0009] Step S3: inputting the selection result into the relation classifier based on the PCNN model, identifying sentences expressing the same relation according to the input sentences, and providing feedback to the instance selector based on reinforcement learning;

[0010] Step S4: the instance selector based on reinforcement learning selects sentences according to the feedback update strategy function, and uses the selected sentences to train the relation classifier based on the PCNN model to optimize the instance selection and relation classification process;

[0011] Step S5: Finally, the sentence that best represents the relationship label is selected as the training data of the relationship extraction model based on reinforcement learning, so as to reduce the influence of the noise and coarse-grained problem of the remote supervision corpus on the relationship extraction.

[0012] Compared with the prior art, the present invention has the following significant advantages: (1) a relation extraction model is formed by an instance selector based on reinforcement learning and a relation classifier based on a PCNN model. By jointly training the relation classifier and the instance selector, the instance selection and relation classification process are optimized; (2) the instance selector is promoted to select the sentence that best expresses the relationship through the reinforcement learning method, that is, the sentence that best expresses the package relationship label is selected from the package consisting of sentences containing the same entity pairs, thereby reducing the impact of noise data and coarse-grained supervision signal problems in the remote supervision relation extraction model on the performance of the relation extraction model, and improving the performance of the relation extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the main process of the reinforcement learning-based relationship extraction method of the present invention.

[0014] Figure 2 This is the model structure of the reinforcement learning-based relationship extraction method of the present invention.

[0015] Figure 3 It is a framework diagram of the instance selector based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0016] The present invention provides a method for extracting relations based on reinforcement learning, comprising the following steps:

[0017] Step S1: constructing a relation extraction model framework based on reinforcement learning, wherein the relation extraction model framework includes an instance selector based on reinforcement learning and a relation classifier based on a PCNN model;

[0018] Step S2: the instance selector based on reinforcement learning obtains a selection result through a reinforcement learning process, and the selection result is a sentence that best expresses the package label in the package consisting of sentences of the same entity pair;

[0019] Step S3: inputting the selection result into the relation classifier based on the PCNN model, identifying sentences expressing the same relation according to the input sentences, and providing feedback to the instance selector based on reinforcement learning;

[0020] Step S4: the instance selector based on reinforcement learning selects sentences according to the feedback update strategy function, and uses the selected sentences to train the relation classifier based on the PCNN model to optimize the instance selection and relation classification process;

[0021] Step S5: Finally, the sentence that best represents the relationship label is selected as the training data of the relationship extraction model based on reinforcement learning, so as to reduce the influence of the noise and coarse-grained problem of the remote supervision corpus on the relationship extraction.

[0022] Furthermore, in step S1, the instance selector based on reinforcement learning also includes state, behavior, and reward;

[0023] The state is used to express the semantics of the sentence and provide evidence for the agent to select the sentence;

[0024] The behavior refers to the actions taken by the agent in different environments;

[0025] The reward is used to help the agent choose the correct behavior.

[0026] Furthermore, in step S1, the relationship classifier based on the PCNN model uses S = {w1, w2, w3, ..., w n} to represent a sentence, where n represents the number of words, that is, the length of the sentence; w i Represents the i-th word in the sentence. The word encoding includes the word vector and the position vector. First, the sentence encoding is convolved, and then each filter is segmented and pooled. Use V S To represent the encoding of the sentence, given a sentence encoding S, define a classifier using Softmax: Where |r| is the total number of relations, k is the relation count number, Respectively represent the relationship r i 、r k Confidence level, and o = W r (S o D)+b r , W r is a transformation vector, representing the weight of the corresponding feature predicting a specific relationship, S o Encode a given sentence, D is the dropout layer representing the regularization, b r is the bias vector, p(r i |S,θ) means that statement S takes θ as a parameter for relation r iThe confidence level of , k is the sentence count number.

[0027] Furthermore, the state and behavior are represented as follows:

[0028] The state is represented by the formula S = [e1; e2; S pre ; p(r|S pre ;θ);S cur p(r|S cur ;θ)], where the symbol [;] represents the concatenation operation, e1, e2 represent the target entity pair, S pre represents the previous sentence vector, S cur represents the current sentence vector, p(r|S pre ; θ) represents the confidence of sentence S for all relations when θ is used as a parameter, r is the sentence relation, and θ is the weight parameter;

[0029] The behavior A={U,P}, where U represents the behavior controlled by the agent, U∈{0,1}, 0 represents keeping the previous relationship label, and 1 represents using the label of the current sentence; P∈{0,1} refers to whether to stop the search behavior in advance, 0 represents continuing the search, and 1 represents stopping the search in advance. Since the second behavior P of the reinforcement model is constructed, the search will be stopped in advance under corresponding circumstances. When the scale of the sentence package is large, the performance of the instance selector based on reinforcement learning can be improved. There are two situations in which the stopping operation occurs:

[0030] The first is that all sentences have been processed;

[0031] The second is that the agent has enough information to prove that it has obtained the sentence that best expresses the selected relationship;

[0032] The agent determines the optimal behavior through calculation. The calculated value is represented by Q(St,A), which represents the value of action A in state St. It is composed of the value of behavior U and behavior P in state St, namely Q(St,U) and Q(St,P). The final behavior A * Equal to argmax A Q(St,A); The Q function acts as a guide to guide the Agent to obtain higher cumulative rewards.

[0033] Furthermore, in step S2, the step of reinforcement learning specifically includes:

[0034] Step S21: For training packet B = {(e1, e2), (r1, ..., rl), {S1, ..., S n}}, e1, e2 represent the target entity pair, (r1,…,r l ) represents the relationship label expressed by the entity pair in the knowledge base, {S1,…,Sn} represents a sentence set, and defines that the Agent executes a series of actions according to a certain strategy until the end is an episode. Start n episodes, and each episode represents a relationship label, expressed as Ep1, Ep2, ..., Ep n , the initial state is where i∈{1,2,…,n}, The two 0s represent no selected semantic features and no confidence, and p(r|S1,θ) represents the confidence of each relation in sentence S1;

[0035] Step S22: The reinforcement learning agent will take the best action according to Q(St,A), and the ultimate goal is to select the sentence that best expresses the purpose relationship label;

[0036] Step S23: When multiple episodes select the current sentence as the selected sentence, the Agent selects the sentence with the highest Q(St,U=1) value and updates this sentence as the selected sentence. The remaining update processes will be prohibited.

[0037] Step S24: When the initial state of each package is the same, a heuristic initialization method is used. First, t sentences are found from the package. For the first sentence, the confidence of p(r|S1,θ) for each relation in the package is calculated, and the relation label with the highest confidence is set as the relation label of this sentence. Then, the sentence is assigned to the relation r with the highest confidence among the candidate relations. * ←argmax r p(r|S1,θ), and the initial state of the episode corresponding to this relationship label is modified to [e1; e2; S1; p(r|S1,θ); S l+1 ; p(r|S l+1 ,θ)];

[0038] Step S25: Process the second sentence according to the remaining relations, and repeat the operation until all the sentences in the bag are processed in this way.

[0039] Furthermore, in step S4, the training is the joint training of the instance selector based on reinforcement learning and the relationship classifier based on the PCNN model, and the input of the relationship classifier is the sentence that best expresses the package relationship label selected in step S2, thereby optimizing the instance selection process and the relationship classification process.

[0040] Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[0041] Combination Figure 1 to Figure 3 , a relation extraction model based on reinforcement learning, including the following steps:

[0042] Step S1: constructing a relation extraction model framework based on reinforcement learning; the relation extraction model framework includes an instance selector based on reinforcement learning and a relation classifier based on a PCNN model;

[0043] Step S2: the instance selector based on reinforcement learning obtains a selection result through a reinforcement learning process, and the selection result is a sentence that best expresses the package label in the package consisting of sentences of the same entity pair;

[0044] Step S3: inputting the selection result into the relationship classifier, identifying sentences expressing the same relationship according to the input sentences, and providing feedback to the instance selector;

[0045] Step S4: the instance selector selects sentences according to the feedback update strategy function, uses the selected sentences to train a better relation classifier, and optimizes the instance selection and relation classification process;

[0046] Step S5: Finally, the sentence that best represents the relationship label is selected as the training data of the relationship extraction model based on reinforcement learning, so as to reduce the influence of the noise and coarse-grained problem of the remote supervision corpus on the relationship extraction.

[0047] As a preferred example, in step S1, the instance selector based on reinforcement learning also includes state, behavior, and reward;

[0048] The state is used to express the semantics of the sentence and provide evidence for the agent to select the sentence;

[0049] The behavior refers to the actions taken by the agent in different environments;

[0050] The reward is used to help the agent choose the correct behavior.

[0051] As a preferred example, in step S1, the relationship classifier based on the PCNN model uses

[0052] S={w1,w2,w3,…,w n} to represent a sentence, where n represents the number of words, i.e. the length of the sentence, and w i Represents the i-th word in the sentence. The word encoding includes the word vector and the position vector.

[0053] First, perform a convolution operation on the sentence encoding, and then perform a segmented pooling operation on each filter;

[0054] Using V S To represent the encoding of the sentence, given a sentence encoding S, define a classifier using Softmax: where |r| is the total number of relations, o = W r (So D)+b r , W r is the transformation vector, representing the weight of the corresponding feature prediction specific relationship. D is the dropout layer representing the regularization, b r is the bias vector.

[0055] As a preferred example, the state, behavior, and feedback are expressed as follows:

[0056] The state is represented by the formula S = [e1; e2; S pre ; p(r|S pre ;θ);S cur p(r|S cur ;θ)], where the symbol represents the concatenation operation, e1, e2 represent the target entity pair, S pre represents the previous sentence vector, S cur represents the current sentence vector, p(r|S pre ; θ) represents the confidence of statement S for all relations when θ is used as a parameter;

[0057] The behavior A={U,P}, where U represents the behavior controlled by the agent and takes a value between 0 and 1, where 0 represents keeping the previous relationship label and 1 represents using the label of the current sentence; P refers to whether to stop the search behavior in advance, which can improve the performance of the instance selector based on reinforcement learning when the sentence package size is large. There are two situations in which the stopping operation occurs:

[0058] The first is that all sentences have been processed;

[0059] The second is that the agent has enough information to prove that it has obtained the sentence that best expresses the selected relationship;

[0060] The agent determines the optimal behavior through calculation. The calculated value is represented by Q(St,A), which includes two parts: Q(St,U) and Q(St,P). The final behavior A * Equal to argmax A Q(St,A); The Q function acts as a guide to guide the Agent to obtain higher cumulative rewards.

[0061] Use a multilayer perceptron to define the Q function,

[0062]

[0063] Among them, f() refers to the nonlinear function, W is the intermediate matrix that further encodes the state characteristics, and W u and W p is the mapping matrix from state to behavior, b u and b pis the bias vector.

[0064] The reward is to help the agent choose the right behavior. In order to solve the coarse-grained problem, the sentence level should be focused on, and the impact of the semantic expression of the sentence on the relationship should be put first;

[0065] When the sentences in the bag are continuously input, the agent determines the degree of deviation between the current sentence and the previously selected sentence from the target relationship. The goal of the agent in each episode is to maximize the overall reward. Finally, the rewards obtained by all episodes constitute the total overall reward.

[0066] The reward should be able to help the agent obtain the sentence that best represents the target relationship. The way to calculate the reward is different for different input sentences.

[0067] For example, when the agent keeps the current sentence, the reward value is equal to the difference between the relationship confidence of the current sentence and the previously selected sentence. However, when the episode stops and the agent selects the sentence currently being processed, if the relationship label with the highest confidence of the current sentence is not the target relationship, the reward is equal to the confidence of the target relationship minus the confidence of its own highest label. This will give the agent a negative feedback, which will make the agent tend to choose the correct sentence.

[0068] Based on the above analysis, define the Reward function

[0069]

[0070] Where T represents the termination state of the episode, and the relationship label selected by the episode is r t .

[0071] As a preferred example, in step S2, the reinforcement learning step specifically includes:

[0072] Step S21: For training packet B = {(e1, e2), (r1, ..., rl), {S1, ..., S n}}, start n episodes, each episode represents a relationship label, expressed as Ep1, Ep2, ..., Ep n , the initial state is where i∈1,2,…,n}, The two 0s represent no selected semantic features and no confidence, respectively;

[0073] Step S22: The reinforcement learning agent will take the best action according to Q(St,A), and the ultimate goal is to select the sentence that best expresses the purpose relationship label;

[0074] Step S23: When multiple episodes select the current sentence as the selected sentence, the Agent selects the sentence with the highest Q(St,U=1) value and updates this sentence as the selected sentence. The remaining update processes will be prohibited.

[0075] Step S24: When the initial state of each package is the same, a heuristic initialization method is used. First, t sentences are found from the package. For the first sentence, the confidence of p(r|S1,θ) for each relation in the package is calculated, and the relation label with the highest confidence is set as the relation label of this sentence. Then, the sentence is assigned to the relation r with the highest confidence among the candidate relations. * ←argmax r p(r|S1,θ), and the initial state of the episode corresponding to this relationship label is modified to [e1; e2; S1; p(r|S1,θ); S l+1 ; p(r|S l+1 ,θ)];

[0076] Step S25: Process the second sentence according to the remaining relations, and repeat the operation until all the sentences in the bag are processed in this way.

[0077] As a preferred example, in step S4, the training is the joint training of the instance selector based on reinforcement learning and the relationship classifier based on the PCNN model, so as to optimize the instance selection process and the relationship classification process.

[0078] The experimental verification and comparison are as follows:

[0079] In order to verify the effectiveness of the present invention, common methods of remote supervised relationship extraction are used for experimental comparison, including the feature-based Mintz method, which is the first model to realize large-scale remote supervised relationship extraction, the probabilistic graph model proposed by Hoffmann, and the MIML model. In addition, the neural network models PCNN and CNN+ATT are selected for comparison, and compared with the policy-based reinforcement learning model proposed by Feng. The reinforcement learning-based relationship extraction method of the present invention is called PCNN+RL in the experiment, which mainly solves the problems of data noise and coarse-grained signals.

[0080] Table 1 Comparison between the PCNN+RL model of the present invention and the neural network relationship extraction model

[0081]

[0082] Table 1 compares the PCNN+RL model with the prior neural network. The difference between these models lies in the way they handle the noise data for remote supervision relationship extraction. From the results, we can see that:

[0083] (1) The performance of PCNN+RL is better than the common benchmark methods, which shows that the relationship extraction method based on reinforcement learning proposed in this paper improves the performance of relationship extraction.

[0084] (2) Under the same recall rate, the PCNN model has the lowest performance because PCNN does not handle the noise problem in the dataset well.

[0085] (3) Compared with Feng's reinforcement learning method, PCNN+RL shows higher performance. Because the reward function of Feng's method is based on the package level, the reward function of PCNN+RL is based on the sentence level, which can generate fine-grained supervision signals to help the agent choose the correct action in the environment. In the end, the model will select the sentence that best expresses the package label, reducing the impact of noisy data and coarse-grained signals on the relationship extraction model.

[0086] Table 2 Comparison between the proposed model PCNN+RL and feature-based methods

[0087]

[0088] Table 2 shows the results of the comparison between PCNN+RL and feature-based methods. It is believed that feature-based methods (Mintz, Hoffmann, Surdeanu) use features extracted from NLP tools, which will inevitably suffer from error propagation and accumulation. Unlike traditional methods, PCNN+RL directly uses segmented convolutional neural networks to extract semantic features, which can achieve better performance.

[0089] In addition, P@N is used to evaluate the model. The specific method is to sort the predicted results in descending order according to the confidence, take the top N, and check their correctness. Table 3 shows the P@N scores of various methods. From the table, we can get:

[0090] (1) For P@100, P@200, P@300 and P@500, PCNN+RL achieves the highest average accuracy among them, indicating that PCNN+RL can select valid sentences as training instances in mixed data, thus improving the extractor’s ability to generate accurate prediction results.

[0091] (2) Feng performs better than the PCNN model, but weaker than PCNN+RL. This result shows that the coarse-grained reward function cannot guide the agent to make ideal behavior decisions, and ignoring the multi-label problem will affect the effect of the relationship extraction model.

[0092] Table 3 Relationship extraction P@N score

[0093]

[0094] In order to further verify the effectiveness of the model, the training examples are studied. Table 4 shows two multi-label examples;

[0095] Table 4 Examples of training sentences

[0096]

[0097] In the first example, Feng's model selects both S1 and S2 as training sentences for the relational labels " / location / contains" and " / country / capital". However, only S1 expresses " / country / capital", while S2 expresses " / location / contain". This is because there is no relational information in the state (State) designed by Feng's reinforcement learning, and the multi-label problem is not considered, resulting in incorrectly labeled data. Compared with Feng's model, with the help of heuristic initialization, the model of the present invention can select more ideal sentences.

[0098] In Example 2, Feng did not find that the factual relationship clearly expressed by S4 was: Jane died in Toronto. The reason for this phenomenon is that the model does not use confidence information as the basis for sentence selection, and sentence information is difficult to provide significant effective information when the sentence is long, because this part of the sentence contains too much information, diluting the original useful information.

[0099] In addition, the rough reward function cannot guide the agent to make accurate sentence-level behaviors. In contrast, the method of the present invention embeds confidence information and develops a fine-grained reward function to make up for these defects. Therefore, PCNN+RL can select S3 as the training sentence of the relation " / person / place_lived" and S4 as the training sentence of the relation " / decease_person / place_of_birth".

[0100] The present invention comprises an instance selector based on reinforcement learning and a relation classifier based on a PCNN model to form a relation extraction model. By jointly training the relation classifier and the instance selector, the instance selection and relation classification process are optimized. The instance selector is promoted to select the sentence that best expresses the relation by the reinforcement learning method, that is, the sentence that best expresses the package relation label is selected from the package consisting of sentences containing the same entity pairs, thereby reducing the influence of noise data and coarse-grained supervision signal problems in the remote supervision relation extraction model on the performance of the relation extraction model and improving the performance of the relation extraction.

Claims

1. A relationship extraction method based on reinforcement learning, characterized in that: The following steps are involved: Step S1: constructing a relation extraction model framework based on reinforcement learning, wherein the relation extraction model framework includes an instance selector based on reinforcement learning and a relation classifier based on a PCNN model; Step S2: the instance selector based on reinforcement learning obtains a selection result through a reinforcement learning process, and the selection result is a sentence that best expresses the package label in the package consisting of sentences of the same entity pair; Step S3: inputting the selection result into the relation classifier based on the PCNN model, identifying sentences expressing the same relation according to the input sentences, and providing feedback to the instance selector based on reinforcement learning; Step S4: the instance selector based on reinforcement learning selects sentences according to the feedback update strategy function, and uses the selected sentences to train the relation classifier based on the PCNN model to optimize the instance selection and relation classification process; Step S5: Finally, the sentence that best represents the relationship label is selected as the training data of the relationship extraction model based on reinforcement learning, so as to reduce the influence of the noise and coarse-grained problem of the remote supervision corpus on the relationship extraction.

2. The method for extracting relations based on reinforcement learning according to claim 1, characterized in that: In step S1, the instance selector based on reinforcement learning also includes state, behavior, and reward; The state is used to express the semantics of the sentence and provide evidence for the agent to select the sentence; The behavior refers to the actions taken by the agent in different environments; The reward is used to help the agent choose the correct behavior.

3. The method for extracting relations based on reinforcement learning according to claim 1, characterized in that: In step S1, the PCNN model-based relation classifier uses S={w i ,w2,w3,...,w n } to represent a sentence, where n represents the number of words, that is, the length of the sentence; w i Represents the i-th word in the sentence. The word encoding includes the word vector and the position vector. First, the sentence encoding is convolved, and then each filter is segmented and pooled. Use V S To represent the encoding of the sentence, given a sentence encoding S, define a classifier using Softmax: Where |r| is the total number of relations, k is the relation count number, Respectively represent the relationship r i 、r k Confidence level, and o = W r (S o D)+b r , W r is a transformation vector, representing the weight of the corresponding feature predicting a specific relationship, S o Encode a given sentence, D is the dropout layer representing the regularization, b r is the bias vector, p(r i |S, θ) means that the statement S is used for the relation r when θ is used as a parameter. i The confidence level of , k is the sentence count number.

4. The method for extracting relations based on reinforcement learning according to claim 2, characterized in that: The state and behavior are expressed as follows: The state is represented by the formula S = [e1; e2; S pre ; p(r|S pre ;θ);S cur p(r|S cur ;θ)], where the symbol [;] represents the concatenation operation, e1, e2 represent the target entity pair, S pre represents the previous sentence vector, S cur represents the current sentence vector, p(r|S pre ; θ) represents the confidence of sentence S for all relations when θ is used as a parameter, r is the sentence relation, and θ is the weight parameter; The behavior A = {U, P}, where U represents the behavior controlled by the agent, and U∈{0, 1}, where 0 represents keeping the previous relationship label, and 1 represents using the label of the current sentence; P∈{0,1} refers to whether to stop the search behavior in advance. 0 means to continue searching, and 1 means to stop searching in advance. Since the second behavior P of the reinforcement model is constructed, the search will be stopped in advance under corresponding circumstances. There are two situations in which the stopping operation occurs: The first is that all sentences have been processed; The second is that the agent has enough information to prove that it has obtained the sentence that best expresses the selected relationship; The agent determines the optimal behavior through calculation. The calculated value is represented by Q(St, A), which represents the value of action A in state St. It is composed of the value of behavior U and behavior P in state St, namely Q(St, U) and Q(St, P). The final behavior A * Equal to argmax A Q(St, A); The Q function acts as a guide to guide the Agent to obtain higher cumulative rewards.

5. The method for extracting relations based on reinforcement learning according to claim 4, characterized in that: In step S2, the step of reinforcement learning specifically includes: Step S21: For training packet B = {(e1, e2), (r1, ..., rl), {S1 ..., S n }}, e1, e2 represent the target entity pair, (r1, ..., r l ) represents the relationship label expressed by the entity pair in the knowledge base, {S1,...,S n } represents a sentence set, and defines that the Agent executes a series of actions according to a certain strategy until the end is an episode. Start n episodes, and each episode represents a relationship label, expressed as Ep1, Ep2, ..., Ep n , the initial state is in The two 0s represent no selected semantic features and no confidence, respectively, and p(r|S1, θ) represents the confidence of each relation in sentence S1; Step S22: The reinforcement learning agent will take the best action according to Q(St, A), and the ultimate goal is to select the sentence that best expresses the purpose relationship label; Step S23: When multiple episodes select the current sentence as the selected sentence, the Agent selects the sentence with the highest Q(St, U=1) value and updates this sentence as the selected sentence. The remaining update processes will be prohibited. Step S24: When the initial state of each package is the same, a heuristic initialization method is used. First, t sentences are found from the package. For the first sentence, the confidence of p(r|S1, θ) for each relation in the package is calculated, and the relation label with the highest confidence is set as the relation label of this sentence. Then, the sentence is assigned to the relation r with the highest confidence among the candidate relations. * ←argmax r p(r|S1, θ), and the initial state of the episode corresponding to this relationship label is modified to [e1; e2; S1; p(r|S1, θ); S l+1 ; p(r|S l+1 ,θ)]; Step S25: Process the second sentence according to the remaining relations, and repeat the operation until all the sentences in the bag are processed in this way.

6. The method for extracting relations based on reinforcement learning according to claim 5, characterized in that: In step S4, the training is the joint training of the instance selector based on reinforcement learning and the relationship classifier based on the PCNN model. The input of the relationship classifier is the sentence that best expresses the package relationship label selected in step S2, thereby optimizing the instance selection process and the relationship classification process.

Images