Text entity joint relation extraction model training method and device, and computer readable storage medium

Abstract

The embodiment of the application provides a kind of text entity joint relationship extraction model training method, device and computer readable storage medium, the method comprises: the input text is encoded to obtain the original feature representation of each word element corresponding task contained in input text respectively;Based on the dynamic fusion of the original feature representation of each word element based on multi-head attention mechanism to obtain the fusion feature of corresponding task respectively;Based on the entity recognition of double-direction marking, the entity recognition is carried out in combination with fusion feature, and all entity pairs with potential relationship are extracted;The relationship of each entity pair is assigned based on the double affine network structure of baseline model, and the relationship probability corresponding to entity pair is obtained;First loss function of entity recognition and second loss function of relationship assignment are respectively constructed, and total loss function is obtained by combining first loss function and second loss function, and total loss function is minimized to output model.The model trained in this embodiment can effectively improve the representation and learning ability.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of computer text recognition, in particular to a text entity joint relation extraction model training method and device and a computer readable storage medium. BACKGROUND

[0002] The text entity joint relation extraction system is a natural language processing (NLP) system, which aims to identify entities from text (entity recognition task) and determine the relationship between entities (relation extraction task), which is usually applied to text information extraction, knowledge graph construction, etc. At present, the text entity joint relation extraction system usually adopts the text entity joint relation extraction model obtained by training.

[0003] A text entity joint relation extraction model is a joint relation extraction model based on a bidirectional extraction framework, and its main training process includes: using a pre-trained language model to encode the text to extract context-related word vector representations; using a bidirectional labeled entity pair recognition method to combine the fusion feature representations to perform entity recognition in the head entity-tail entity and tail entity-head entity directions, respectively, to extract all entity pairs with potential relationships from the input text; using a double affine module to assign relationships to all entity pairs to assign corresponding relationship types; and constructing loss functions for entity recognition and relation extraction, respectively, using a joint training method to minimize the loss functions to obtain a trained model.

[0004] However, the inventors found in specific implementation that although the above method uses two parallel networks to perform entity recognition tasks and relation extraction tasks, the two tasks interact through shared input features and each encodes a corresponding specific representation. However, this method does not fully exploit the potential correlation between the two tasks, resulting in limited representation and learning ability of the model generated by training. SUMMARY

[0005] The technical problem to be solved by the embodiment of the present application is to provide a text entity joint relation extraction model training method, which can effectively improve the representation and learning ability of the model obtained by training.

[0006] The technical problem to be solved by the embodiment of the present application is to provide a text entity joint relation extraction model training device, which can effectively improve the representation and learning ability of the model obtained by training.

[0007] The technical problem to be solved by the embodiment of the present application is to provide a computer readable storage medium to store a computer program that can effectively improve the representation and learning ability of the model obtained by training.

[0008] To solve the above technical problems, the embodiment of the present application provides the following technical solutions: a training method of a text entity joint relation extraction model, comprising the following steps:

[0009] encoding the input text to obtain a first original feature representation of each word unit contained in the input text in an entity recognition task and a second original feature representation in a relation extraction task, respectively;

[0010] based on a multi-head attention mechanism, dynamically fusing the first original feature representation and the second original feature representation of each word unit in the entity recognition task processing process to obtain a first fused feature corresponding to each word unit, and fusing the first original feature representation and the second original feature representation of each word unit in the relation extraction task processing process to obtain a second fused feature corresponding to each word unit;

[0011] based on a bidirectional labeled entity pair recognition method, combining the first fused feature and the second fused feature to perform entity recognition from two complementary directions, respectively, to extract all entity pairs with potential relations from the input text;

[0012] using a double-affine network structure based on a baseline model to assign a relation to each of the entity pairs to calculate a relation probability corresponding to the entity pair; and

[0013] combining the relation probability to construct a first loss function of the entity recognition and a second loss function of the relation assignment, respectively, combining the first loss function and the second loss function to obtain a total loss function, minimizing the total loss function to output a text entity joint relation extraction model.

[0014] Further, the encoding of the input text to obtain the first original feature representation of each word unit contained in the input text in the entity recognition task and the second original feature representation in the relation extraction task specifically comprises:

[0015] using a pre-trained language model as an encoder to extract a word vector of the input text; and

[0016] generating the first original feature representation based on the word vector of the input text and a first trainable projection matrix of the entity recognition task, and generating the second original feature representation based on the word vector of the input text and a second trainable projection matrix of the relation extraction task.

[0017] Further, the dynamic fusion of the first original feature representation and the second original feature representation of each word unit based on the multi-head attention mechanism to obtain the fused feature representation corresponding to each word unit specifically comprises:

[0018] extracting, based on the multi-layer perception network model, first attention scores and second attention scores corresponding to the entity recognition features and the relation extraction features of each word piece in the entity recognition task respectively in combination with the first original feature representation and the second original feature representation, and extracting, based on the multi-layer perception network model, third attention scores and fourth attention scores corresponding to the entity recognition features and the relation extraction features of each word piece in the relation extraction task in combination with the first original feature representation and the second original feature representation;

[0019] normalizing the first attention scores and the second attention scores to obtain a first normalized weight matrix of the entity recognition task, and normalizing the third attention scores and the fourth attention scores to obtain a second normalized weight matrix of the relation extraction task;

[0020] weighting and fusing the first original feature representation and the second original feature representation based on the first normalized weight matrix to obtain a single entity recognition fusion feature corresponding to the entity recognition task in a single subspace, and weighting and fusing the first original feature representation and the second original feature representation based on the second normalized weight matrix to obtain a single relation extraction fusion feature in a single subspace; and

[0021] performing weighted averaging on outputs of all attention heads in combination with the single entity recognition fusion feature based on a first trainable weight matrix and applying a Dropout operation to obtain a first fusion feature in the entity recognition task, and performing weighted averaging on outputs of all attention heads in combination with the single relation extraction fusion feature based on a second trainable weight matrix and applying a Dropout operation to obtain a second fusion feature in the relation extraction task.

[0022] Further, the entity pair recognition method based on bidirectional labeling combines the first fusion feature and the second fusion feature to perform entity recognition from two complementary directions, specifically comprising:

[0023] multiplying the first fusion feature with a first trainable parameter matrix and a second trainable parameter matrix respectively to obtain a first context feature representation for head entity extraction and a second context feature representation for tail entity extraction;

[0024] The first head entity marker calculates a first head entity start probability and a first head entity end probability of each word token being a start position and an end position of a head entity in the input text based on a pointer network method combined with a first context feature representation for the head entity, a third trainable parameter matrix and a fourth trainable parameter matrix, and the first tail entity marker calculates a first tail entity start probability and a first tail entity end probability of each word token being a start position and an end position of a tail entity in the input text based on a pointer network method combined with a second context feature representation for the tail entity, a fifth trainable parameter matrix and a sixth trainable parameter matrix;

[0025] The first head entity start probability and the first head entity end probability of each word token are determined based on the first head entity start probability and the first head entity end probability of each word token in the input text, and the feature representation of each tail entity in the input text is determined based on the first tail entity start probability and the first tail entity end probability of each word token;

[0026] The second context feature for the tail entity is input as an input feature, and the feature representation of the head entity is input as a conditional function into a first CLN network model to calculate a first text encoding representation that integrates the head entity feature, and the first context feature for the head entity is input as an input feature, and the feature representation of the tail entity is input as a conditional function into a second CLN network model to calculate a second text encoding representation that integrates the tail entity feature;

[0027] The second tail entity marker calculates a second tail entity start probability and a second tail entity end probability of each word token being a start position and an end position of a tail entity based on a pointer network method combined with the first text encoding representation, the fifth trainable parameter matrix and the sixth trainable parameter matrix, and the second head entity marker calculates a second head entity start probability and a second head entity end probability of each word token being a start position and an end position of a head entity based on a pointer network method combined with the second text encoding representation, the third trainable parameter matrix and the fourth trainable parameter matrix; and

[0028] The first head entity start probability and the first head entity end probability of each word token in the input text are based on the first head entity start probability and the first head entity end probability of each word token in the input text, and the second tail entity start probability and the second tail entity end probability are extracted as a pair of entity pairs based on the first tail entity start probability and the first tail entity end probability of each word token in the input text, and the second head entity start probability and the second head entity end probability are extracted as a pair of entity pairs based on the first tail entity start probability and the first tail entity end probability of each word token in the input text.

[0029] Further, the relationship assignment of each pair of entity pairs based on the baseline model double affine network structure is used to calculate the relationship probability corresponding to the entity pair, which specifically includes:

[0030] calculating feature representations of head entities and tail entities in the entity pair respectively by using a double-affine network structure based on a baseline model; and

[0031] calculating a relationship probability corresponding to the entity pair based on the feature representations of the head entities and the tail entities in the entity pair and a trainable relationship parameter matrix.

[0032] Further, the first loss function is represented as:

[0033] ;

[0034] The second loss function is represented as:

[0035] ;

[0036] wherein, ;

[0037] p represents the relationship probability, t represents a true label, represents a start or end position marker of the head entity or the tail entity in the entity pair, respectively represent the first tail entity marker, the first head entity marker, the second tail entity marker, and the second head entity marker, g represents a number of word pieces contained in the input text, and G is a predefined relationship set.

[0038] Further, the total loss function is represented as: wherein, a and β are function weights respectively used for flexible adjustment according to importance of a current data set task.

[0039] Further, the pre-trained language model, the first trainable projection matrix, the second trainable projection matrix, the first trainable weight matrix, the second trainable weight matrix, the first trainable parameter matrix, the second trainable parameter matrix, the third trainable parameter matrix, the fourth trainable parameter matrix, the fifth trainable parameter matrix, the sixth trainable parameter matrix, the first CLN network model, the second CLN network model, and the trainable relationship parameter matrix are corrected to minimize the total loss function.

[0040] In another aspect, to solve the above further technical problems, the embodiments of the present application further provide the following technical solutions: a training device of a text entity joint relationship extraction model, the device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the training method of the text entity joint relationship extraction model as described in any one of the above embodiments when executing the computer program.

[0041] In still another aspect, to solve the above further technical problem, the embodiment of the present application further provides the following technical solution: a computer readable storage medium, the computer readable storage medium comprises a stored computer program, wherein the computer program controls the device where the computer readable storage medium is located to execute the training method of the text entity joint relation extraction model according to any one of the above when the computer program runs.

[0042] After adopting the above technical solution, the embodiment of the present application has at least the following beneficial effects: after the embodiment of the present application encodes the input text to obtain the first original feature representation of each word unit in the entity recognition task and the second original feature representation in the relation extraction task, the original feature representations of each word unit in the entity recognition task and the relation extraction task are dynamically fused based on the multi-head attention mechanism to obtain the corresponding first fusion feature and the second fusion feature, respectively, the fine-grained feature interaction between entities and relations is captured in multiple subspaces, the contribution of each subspace feature is dynamically adjusted, thereby enhancing the modeling capability of the model for the interaction between subtasks, and the representation and learning capability of the model can be improved; after extracting all entity pairs with potential relations from the input text based on the entity pair recognition method of bidirectional labeling, a double affine network structure based on a baseline model is used to assign relations to each of the entity pairs, and finally a first loss function of entity recognition and a second loss function of the relation assignment are constructed in combination with the relation probability, the total loss function obtained by minimizing the joint first loss function and the second loss function, and the finally generated text entity joint relation extraction model has good performance indicators. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The step flow chart of an optional embodiment of the training method of the text entity joint relation extraction model of the present application.

[0044] Figure 2 The schematic diagram of feature fusion in step S2 of an optional embodiment of the training method of the text entity joint relation extraction model of the present application.

[0045] Figure 3 The schematic diagram of the CLN network model of an optional embodiment of the training method of the text entity joint relation extraction model of the present application.

[0046] Figure 4 The principle block diagram of an optional embodiment of the training device of the text entity joint relation extraction model of the present application.

[0047] Figure 5 The functional module diagram of an optional embodiment of the training device of the text entity joint relation extraction model of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present application and are not intended to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0049] As shown in the figure, an optional embodiment of the present application provides a training method of a text entity joint relation extraction model, comprising the following steps: Figure 1

[0050] S1: encoding the input text to obtain a first original feature representation of each word unit contained in the input text in an entity recognition task and a second original feature representation in a relation extraction task, respectively;

[0051] S2: dynamically fusing the first original feature representation and the second original feature representation of each word unit in the entity recognition task processing process based on the multi-head attention mechanism to obtain the first fusion feature corresponding to each word unit, and fusing the first original feature representation and the second original feature representation of each word unit in the relation extraction task processing process to obtain the second fusion feature corresponding to each word unit;

[0052] S3: combining the first fusion feature and the second fusion feature to perform entity recognition from two complementary directions based on a bidirectional labeled entity pair recognition method, to extract all entity pairs with potential relations from the input text;

[0053] S4: assigning a relation to each of the entity pairs using a double affine network structure based on a baseline model to calculate the relation probability corresponding to the entity pairs; and

[0054] S5: constructing a first loss function of the entity recognition and a second loss function of the relation assignment based on the relation probability, respectively, combining the first loss function and the second loss function to obtain a total loss function, and minimizing the total loss function to output a text entity joint relation extraction model.

[0055] ​After the embodiment of the present application encodes the input text to obtain the first original feature representation of each word unit in the entity recognition task and the second original feature representation in the relation extraction task, the original feature representations of each word unit in the entity recognition task and the relation extraction task are dynamically fused based on the multi-head attention mechanism to obtain the corresponding first fusion feature and the second fusion feature, respectively, to capture the fine-grained feature interaction between entities and relations in multiple subspaces, dynamically adjust the contribution of each subspace feature, thereby enhancing the modeling capability of the model for the interaction between subtasks, and improving the representation and learning ability of the model; further based on the entity pair recognition method of bidirectional labeling, all entity pairs with potential relationships are extracted from the input text, then a double affine network structure based on the baseline model is used to assign relationships to each of the entity pairs, and finally a first loss function of entity recognition and a second loss function of the relationship assignment are constructed by combining the relationship probability, and the total loss function obtained by minimizing the joint first loss function and the second loss function, the finally generated text entity joint relation extraction model has good performance indicators.

[0056] In a specific implementation, in step S3, the bidirectional entity pair recognition method based on conditional layer normalization combines the first fusion feature and the second fusion feature to perform entity recognition from two complementary directions, respectively, to extract all entity pairs with potential relationships from the input text.

[0057] In an optional embodiment of the present application, step S1 specifically includes:

[0058] A pre-trained language model is used as an encoder to extract the word vector of the input text, wherein the input text is represented as wherein x CLS and x SEP represent the start token and the split token of the input text, respectively, and the word vector is represented as: wherein, b represents the batch size, l represents the sequence length of the input text, and d represents the hidden layer dimension; and

[0059] The first original feature representation is generated based on the word vector of the input text and the first trainable projection matrix of the entity recognition task, and the second original feature representation is generated based on the word vector of the input text and the second trainable projection matrix of the relation extraction task, wherein,

[0060] The first original feature representation is: ,

[0061] The second original feature representation is: ,

[0062] and respectively represent the first trainable projection matrix and the second trainable projection matrix, and are corresponding bias terms, e and r respectively represent the reference labels of the entity recognition task and the relation extraction task.

[0063] In the embodiment, a pre-trained language model is used as an encoder to extract word vectors of the input text, and then generate original feature representations corresponding to the entity recognition task and the relation extraction task, which has high encoding efficiency. In a specific implementation, the pre-trained language model can be determined according to the data set of the corresponding task, so as to improve the compatibility of the model. For example, when the data set is Chinese, the pre-trained language model can be selected as RoBERTa-wwm-ext model.

[0064] In an optional embodiment of the present application, step S2 specifically comprises:

[0065] S21: extracting, based on a multi-layer perception network model, first attention scores and second attention scores corresponding to entity recognition features and relation extraction features of each word piece in the entity recognition task based on the first original feature representation and the second original feature representation, and extracting, based on the multi-layer perception network model, third attention scores and fourth attention scores corresponding to the entity recognition features and the relation extraction features of each word piece in the relation extraction task based on the first original feature representation and the second original feature representation;

[0066] wherein the first attention score and the second attention score of the i-th word piece are respectively represented as: , the third attention score and the fourth attention score of the i-th word piece are respectively represented as: , ;

[0067] S22: normalizing the first attention score and the second attention score to obtain a first normalized weight matrix of the entity recognition task, and normalizing the third attention score and the fourth attention score to obtain a second normalized weight matrix of the relation extraction task;

[0068] wherein the first normalized weight matrix and the second normalized weight matrix are respectively represented as:

[0069] , ;

[0070] S23: weighting and fusing the first original feature representation and the second original feature representation based on the first normalized weight matrix to obtain a single entity recognition fusion feature corresponding to the entity recognition task in a single subspace, and weighting and fusing the first original feature representation and the second original feature representation based on the second normalized weight matrix to obtain a single relation extraction fusion feature in a single subspace;

[0071] wherein the single entity recognition fusion feature and the single relation extraction fusion feature are respectively represented as:

[0072]

[0073]

[0074] wherein [..., 0:1] and [..., 1:2] represent a slicing operation for extracting the first column or the second column from the last dimension of the corresponding normalized weight matrix, and k represents the kth attention head;

[0075] S24: weighting and averaging the outputs of all attention heads based on the first trainable weight matrix in combination with the single entity recognition fusion feature and applying a Dropout operation to obtain a first fusion feature in the entity recognition task, and weighting and averaging the outputs of all attention heads based on the second trainable weight matrix in combination with the single relation extraction fusion feature and applying a Dropout operation to obtain a second fusion feature in the relation extraction task;

[0076] wherein the first trainable weight matrix and the second trainable weight matrix are respectively represented as:

[0077]

[0078] The initial values of the first trainable weight matrix and the second trainable weight matrix are all-one matrices;

[0079] The first fusion feature is:

[0080] The second fusion feature is:

[0081] wherein

[0082] and respectively represent the weight proportion corresponding to the kth attention head, and H represents the total number of attention heads.

[0083] ​​​​​​​​In this embodiment, based on the multi-head attention mechanism, the fine-grained feature interaction between entities and relations is captured in multiple subspaces. Through the way of weighted fusion, the contribution of each subspace feature is dynamically adjusted, thereby enhancing the modeling ability of the model to the interaction between subtasks and improving the representation and learning ability of the model.

[0084] In steps S21-S24, as shown in the figure, Figure 2 two symmetric and independent feature fusion units (Feature Fusion) can be used for calculation to obtain the fusion features corresponding to the entity recognition task and the relation extraction task respectively. In step S24, in order to improve the generalization ability of the model and reduce overfitting, after weighting and averaging the outputs of all attention heads for the single entity recognition fusion feature and the single relation extraction fusion feature respectively, the Dropout operation needs to be applied.

[0085] In an optional embodiment of the present application, step S3 specifically comprises:

[0086] S31: multiplying the first fusion feature with a first trainable parameter matrix and a second trainable parameter matrix respectively to obtain a first context feature representation for head entity extraction and a second context feature representation for tail entity extraction;

[0087] wherein the first context feature representation and the second context feature representation are represented as: , , W s and W o respectively represent the first trainable parameter matrix and the second trainable parameter matrix, b is the corresponding bias term, and s and o are the reference labels for head entity and tail entity respectively;

[0088] S32: using a first head entity labeler to calculate the first head entity start probability and the first head entity end probability of each word token as the start position of the head entity in the input text based on the pointer network method combined with the first context feature representation for the head entity, a third trainable parameter matrix and a fourth trainable parameter matrix, and using a first tail entity labeler to calculate the first tail entity start probability and the first tail entity end probability of each word token as the start position of the tail entity in the input text based on the pointer network method combined with the second context feature representation for the tail entity, a fifth trainable parameter matrix and a sixth trainable parameter matrix;

[0089] wherein the first head entity start probability and the first head entity end probability of the i-th word token are represented as: , , and respectively represent the third trainable parameter matrix and the fourth trainable parameter matrix, the first tail entity start probability and the first tail entity end probability of the ith wordpiece are represented as , , and respectively represent the fifth trainable parameter matrix and the sixth trainable parameter matrix, σ is a sigmoid activation function, sta represents a pointer annotation of a start position of a head entity or a tail entity, and end represents a pointer annotation of an end position of the head entity or the tail entity;

[0090] S33: determining a feature representation of a head entity in the input text based on the first head entity start probability and the first head entity end probability of each wordpiece and determining a feature representation of each tail entity in the input text based on the first tail entity start probability and the first tail entity end probability of each wordpiece;

[0091] wherein the feature representation of the head entity in the input text is , and the feature representation of each tail entity in the input text is wherein, represents a vector span representation of a head entity from a start position to an end position, represents a vector span representation of a tail entity from a start position to an end position, and maxpool represents a max-pooling operation;

[0092] S34: inputting the second context feature for the tail entity as an input feature and the feature representation of the head entity as a conditional function into a first CLN network model to calculate a first text encoding representation of fused head entity features, and inputting the first context feature for the head entity as an input feature and the feature representation of the tail entity as a conditional function into a second CLN network model to calculate a second text encoding representation of fused tail entity features;

[0093] wherein the first text encoding representation of the fused head entity features is , and the second text encoding representation of the fused tail entity features is It can be understood that the first CLN network model and the second CLN network model are the same network structure;

[0094] S35: obtaining, by the second tail entity labeler, the second tail entity start probability and the second tail entity end probability of each wordpiece as the start position of the tail entity based on the pointer network method combined with the first text encoding representation, the fifth trainable parameter matrix and the sixth trainable parameter matrix, and obtaining, by the second head entity labeler, the second head entity start probability and the second head entity end probability of each wordpiece as the start position of the head entity based on the pointer network method combined with the second text encoding representation, the third trainable parameter matrix and the fourth trainable parameter matrix; and

[0095] wherein the second tail entity start probability and the second tail entity end probability of the ith wordpiece are represented as: , the second head entity start probability and the second head entity end probability of the ith wordpiece are represented as , ;

[0096] S36: extracting the head entity and the tail entity with potential relationship as a pair of entity pairs based on the first head entity start probability and the first head entity end probability and the second tail entity start probability and the second tail entity end probability of each wordpiece in the input text, and extracting the head entity and the tail entity with potential relationship as a pair of entity pairs based on the first tail entity start probability and the first tail entity end probability and the second head entity start probability and the second head entity end probability of each wordpiece in the input text.

[0097] In the embodiment, in step S31, since the entity recognition task needs to extract entity pairs in two opposite directions, there is a slight difference in entity extraction between the two directions, therefore, the fusion features corresponding to the entity recognition task are multiplied by the first parameter matrix and the second parameter matrix which are trainable and different, so as to obtain the context feature representation for head entity and tail entity extraction;

[0098] In steps S32-S35, since the calculation processes of the two extraction directions are similar, the head entity-tail entity extraction direction is described. Since it is based on the pointer network method, the start and end positions of the head entity in the labeled text sequence are identified, the pointer network adopts a simple 1 / 0 labeling method, first, the probability of each wordpiece as the start and end position of the entity is calculated by the sigmoid activation function, if the probability is greater than a preset threshold, the wordpiece position is marked as 1, otherwise it is marked as 0, for example: the head entity start position of the 7th wordpiece in the input text is marked as 1, and the tail entity end position of the 9th wordpiece is marked as 1, then the head entity is composed of the 7th to 9th wordpieces;

[0099] After the head entity is determined, the tail entity is labeled, which is equivalent to two interrelated processes of entity pair extraction, that is, the extraction of the tail entity is guided by the previously extracted head entity. After the feature representation of the head entity is obtained, a CLN (Conditional Layer Normalization) network model is introduced. The CLN network model is a technology based on layer normalization (LN). By introducing conditional information, the model can dynamically adjust its normalization process according to different input conditions, thereby improving the performance in the conditional generation task, as shown in the following formula (1) : Figure 3 In the CLN network model, the input feature x is first standardized, then a condition function c is introduced in the normalization structure, and the condition function c is converted into the same dimension as the scaling factor γ and the offset β through linear transformation, so that the transformation process of each feature can be dynamically adjusted according to the input condition information. The specific calculation process of the CLN network model is as follows:

[0100] (Formula 1);

[0101] (Formula 2);

[0102] (Formula 3);

[0103] (Formula 4);

[0104] Wherein, μ and σ represent the mean and standard deviation of the second context feature representation h o of the tail entity extraction, W γ and W γ are the scaling matrix and the offset matrix, which need to be corrected when minimizing the total loss function, b γ and b γ are the corresponding bias terms, in addition, x in the above formula is the second context feature representation h o of the tail entity extraction, and c is the feature representation of the head entity By introducing the feature representation of the head entity into the context feature representation of the tail entity extraction, the fusion mode of different entity and text features can be flexibly adjusted according to the context information, so that the feature fusion process is more precise, thereby improving the accuracy and stability of the relationship extraction task.

[0105] In an optional embodiment of the present application, step S4 specifically comprises:

[0106] S41: a biaffine model based on a baseline model is used to calculate the feature representation of the head entity and the tail entity in the entity pair respectively;

[0107] Wherein, the entity pair is represented as (s_k, o_j) Wherein, s_k represents the head entity in the entity pair, and o_j represents the tail entity in the entity pair.

[0108] The feature representation of the head entity in the entity pair is:

[0109]

[0110] The feature representation of the tail entity in the entity pair is:

[0111]

[0112] S42: the relationship probability corresponding to the entity pair is calculated based on the feature representation of the head entity and the tail entity in the entity pair and the trainable relationship parameter matrix.

[0113] Wherein, the relationship probability is represented as:

[0114]

[0115] Wherein, T represents a transposed matrix, Indicates that the trainable relationship parameter matrix for maintaining the yth relationship.

[0116] In the embodiment, the biaffine network structure based on the baseline model is used, compared with the traditional linear layer, a relationship parameter matrix is allocated for each relationship, so that the characteristics of the relationship can be accurately modeled; Secondly, it can output the corresponding relationship probability for each relationship, so as to facilitate the mining of the interaction between two entities and improve the expression ability of the trained model.

[0117] In an optional embodiment of the present application, the first loss function is represented as:

[0118]

[0119] The second loss function is represented as:

[0120]

[0121] Wherein,

[0122] P represents the relationship probability, t represents the true label, Indicates the start or end position mark of the head entity or tail entity in the entity pair, ​​​​​​respectively represent the first tail entity tagger, the first head entity tagger, the second tail entity tagger, the second head entity tagger, g represents the number of wordpieces contained in the input text, and G is a predefined relation set.

[0123] In the embodiment, when performing entity recognition, the pointer network tagging method is used for entity tagging, and four taggers (i.e., the first tail entity tagger, the first head entity tagger, the second tail entity tagger, and the second head entity tagger) are included. Each tagger tags the start and end positions of an entity as 1 and tags the remaining positions as 0. Therefore, the entity recognition task can be regarded as a binary classification problem, and the binary cross-entropy loss function can be used to train the entity recognition task. For the relation assignment task, the relation assignment of an entity pair can be regarded as a multi-classification problem, and the cross-entropy loss function can be used to train the module. Based on the above principle, the first loss function and the second loss function can be constructed accordingly.

[0124] In an optional embodiment of the present application, the total loss function is represented as: wherein α and β are function weights used to flexibly adjust the importance of the current data set task. In the embodiment, by setting two different function weights, the first loss function and the second loss function are combined to form the total loss function, so that the function weights can be flexibly adjusted according to the importance of the current data set task. For example, when the difficulty or importance of the entity recognition task and the relation extraction task is generally equivalent, α = β = 1 can be set; and when the difficulty or importance of the entity recognition task is higher than that of the relation extraction task, α = 2 and β = 1 can be set.

[0125] In an optional embodiment of the present application, the following parameters are modified to minimize the total loss function: the pre-trained language model, the first trainable projection matrix, the second trainable projection matrix, the first trainable weight matrix, the second trainable weight matrix, the first trainable parameter matrix, the second trainable parameter matrix, the third trainable parameter matrix, the fourth trainable parameter matrix, the fifth trainable parameter matrix, the sixth trainable parameter matrix, the first CLN network model, the second CLN network model, and the trainable relation parameter matrix. In the embodiment, the parameters of the pre-trained language model need to be modified to adapt to the corresponding downstream task, and the parameters of each matrix or network model of the built model also need to be updated accordingly to minimize the total loss function and improve the learning and expression ability of the model.

[0126] On the other hand, as Figure 4As shown, one optional embodiment of the present application further provides a text entity joint relation extraction model training device 1, which comprises a processor 10, a memory 12, and a computer program stored in the memory 12 and configured to be executed by the processor 10, and the processor 10 implements the text entity joint relation extraction model training method according to any one of the above embodiments when executing the computer program.

[0127] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the text entity joint relation extraction model training device 1. For example, the computer program can be divided into Figure 5 The text entity joint relation extraction model training device 1 comprises a function module, wherein the text encoding module 31, the task feature interaction module 32, the entity recognition module 33, the relation distribution module 34, and the loss function construction and optimization module 35 correspond to steps S1-S5 respectively.

[0128] The text entity joint relation extraction model training device 1 can be a desktop computer, a notebook, a palm computer, a cloud server, and the like. The text entity joint relation extraction model training device 1 can include, but is not limited to, the processor 10 and the memory 12. Those skilled in the art can understand that the schematic diagram is only an example of the text entity joint relation extraction model training device 1, and does not constitute a limitation on the text entity joint relation extraction model training device 1, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the text entity joint relation extraction model training device 1 can further include an input / output device, a network access device, a bus, and the like.

[0129] The processor 10 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor 10 is a control center of the text entity joint relation extraction model training apparatus 1, and is connected to various parts of the text entity joint relation extraction model training apparatus 1 through various interfaces and lines.

[0130] The memory 12 can be used to store the computer programs and / or modules, and the processor 10 realizes various functions of the text entity joint relation extraction model training apparatus 1 by running or executing the computer programs and / or modules stored in the memory 12, and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a graph recognition function, a graph layering function, etc.), etc.; and the data storage area can store data (such as graph data, etc.) created according to the use of the control device, etc. In addition, the memory 12 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0131] The functions of the embodiments of the present application, if implemented in the form of software function modules or units, and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such an understanding, the embodiments of the present application implement all or part of the processes in the above-described embodiment methods, and can also be completed by computer programs instructing related hardware. The computer program can be stored in a computer readable storage medium, and when the processor 10 executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms of the above. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable medium contains content that can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0132] In still another aspect, an optional embodiment of the present application further provides a computer readable storage medium, including a stored computer program, wherein the computer program controls the device where the computer readable storage medium is located to execute the training method of the text entity joint relation extraction model as described in any of the above embodiments when the computer program is running.

[0133] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other.

[0134] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection scope of the present application.

Claims

1. A method for training a text entity joint relation extraction model, characterized in that, The method comprises the following steps: encoding the input text to obtain a first original feature representation of each token contained in the input text in an entity recognition task and a second original feature representation in a relation extraction task, respectively; dynamically fusing the first original feature representation and the second original feature representation of each token in the process of entity recognition task processing based on a multi-head attention mechanism to obtain a first fusion feature corresponding to each token, and fusing the first original feature representation and the second original feature representation of each token in the process of relation extraction task processing to obtain a second fusion feature corresponding to each token; combining the first fusion feature and the second fusion feature to perform entity recognition from two complementary directions respectively based on a bidirectional labeled entity pair recognition method, so as to extract all entity pairs with potential relations from the input text; adopting a double affine network structure based on a baseline model to assign relations to each of the entity pairs to calculate and obtain a relation probability corresponding to the entity pair; and combining the relation probability to construct a first loss function of the entity recognition and a second loss function of the relation assignment respectively, combining the first loss function and the second loss function to obtain a total loss function, and minimizing the total loss function to output a text entity joint relation extraction model; wherein the dynamic fusion of the first original feature representation and the second original feature representation of each token in the process of entity recognition task processing based on the multi-head attention mechanism to obtain the first fusion feature corresponding to each token and the dynamic fusion of the first original feature representation and the second original feature representation of each token in the process of relation extraction task processing to obtain the second fusion feature corresponding to each token specifically comprises: extracting a first attention score and a second attention score corresponding to an entity recognition feature and a relation extraction feature of each token in the entity recognition task based on a multi-layer perception network model combining the first original feature representation and the second original feature representation, and extracting a third attention score and a fourth attention score corresponding to the entity recognition feature and the relation extraction feature of each token in the relation extraction task based on the multi-layer perception network model combining the first original feature representation and the second original feature representation; normalizing the first attention score and the second attention score to obtain a first normalized weight matrix of the entity recognition task, and normalizing the third attention score and the fourth attention score to obtain a second normalized weight matrix of the relation extraction task; weighting the fusion of the first original feature representation and the second original feature representation based on the first normalized weight matrix to obtain a single entity recognition fusion feature corresponding to the entity recognition task in a single subspace, and weighting the fusion of the first original feature representation and the second original feature representation based on the second normalized weight matrix to obtain a single relation extraction fusion feature in a single subspace; and The first fusion feature in the entity recognition task is obtained by weighting and averaging the outputs of all attention heads based on a first trainable weight matrix and applying a Dropout operation in combination with the single entity recognition fusion feature, and the second fusion feature in the relation extraction task is obtained by weighting and averaging the outputs of all attention heads based on a second trainable weight matrix and applying a Dropout operation in combination with the single relation extraction fusion feature. 2.The method of claim 1, wherein, The encoding of the input text to obtain the first original feature representation of each token included in the input text in the entity recognition task and the second original feature representation in the relation extraction task specifically comprises: adopting a pre-trained language model as an encoder to extract word vectors of the input text; and generating the first original feature representation based on the word vectors of the input text and a first trainable projection matrix of the entity recognition task, and generating the second original feature representation based on the word vectors of the input text and a second trainable projection matrix of the relation extraction task. 3.The method of claim 2, wherein, The entity pair recognition method based on bidirectional labeling combines the first fusion feature and the second fusion feature to perform entity recognition from two complementary directions, respectively, to extract all entity pairs with potential relationships from the input text, specifically comprising: multiplying the first fusion feature with a first trainable parameter matrix and a second trainable parameter matrix, respectively, to obtain a first context feature representation for head entity extraction and a second context feature representation for tail entity extraction; adopting a first head entity labeler to calculate a first head entity start probability and a first head entity end probability of each token in the input text as a start position and an end position of a head entity based on a pointer network method in combination with the first context feature representation for the head entity, a third trainable parameter matrix and a fourth trainable parameter matrix, and adopting a first tail entity labeler to calculate a first tail entity start probability and a first tail entity end probability of each token in the input text as a start position and an end position of a tail entity based on a pointer network method in combination with the second context feature representation for the tail entity, a fifth trainable parameter matrix and a sixth trainable parameter matrix; determining a feature representation of the head entity in the input text based on the first head entity start probability and the first head entity end probability of each token, and determining a feature representation of each tail entity in the input text based on the first tail entity start probability and the first tail entity end probability of each token; inputting the second context feature for the tail entity as an input feature and the feature representation of the head entity as a conditional function into a first CLN network model to calculate and generate a first text encoding representation of a fusion head entity feature, and inputting the first context feature for the head entity as an input feature and the feature representation of the tail entity as a conditional function into a second CLN network model to calculate and generate a second text encoding representation of a fusion tail entity feature; obtaining, by a second tail entity tagger, a second tail entity start probability and a second tail entity end probability for each token being a tail entity start position and end position based on a pointer network method in combination with the first text encoding representation, the fifth trainable parameter matrix and the sixth trainable parameter matrix, and obtaining, by a second head entity tagger, a second head entity start probability and a second head entity end probability for each token being a head entity start position and end position based on a pointer network method in combination with the second text encoding representation, the third trainable parameter matrix and the fourth trainable parameter matrix; and extracting, based on the first head entity start probability and the first head entity end probability and the second tail entity start probability and the second tail entity end probability for each token in the input text, a head entity and a tail entity that have a potential relationship as a pair of entity pairs, and extracting, based on the first tail entity start probability and the first tail entity end probability and the second head entity start probability and the second head entity end probability for each token in the input text, a head entity and a tail entity that have a potential relationship as a pair of entity pairs. 4.The method of claim 3, wherein, the assigning, by the dual affine network structure based on the baseline model, a relationship to each of the pair of entity pairs to calculate a relationship probability corresponding to the pair of entity pairs specifically comprises: calculating, by the dual affine network structure based on the baseline model, a feature representation of the head entity and the tail entity in the pair of entity pairs, respectively; and calculating, based on the feature representation of the head entity and the tail entity in the pair of entity pairs and a trainable relationship parameter matrix, a relationship probability corresponding to the pair of entity pairs. 5.The method of claim 4, wherein, the first loss function is represented as: ； the second loss function is represented as: ； wherein ; p represents the relation probability, t represents the true label, position markers indicating the start or end of the head entity or the tail entity in the entity pair, respectively represent the first tail entity marker, the first head entity marker, the second tail entity marker, the second head entity marker, g represents the number of word pieces contained in the input text, and G is a predefined relation set. 6.The method of claim 5, wherein, The total loss function is represented as: where a and β are function weights for flexible adjustment according to the importance of the current data set task, respectively.

7. The method of claim 4, wherein the method further comprises: minimizing the total loss function by modifying the pre-trained language model, the first trainable projection matrix, the second trainable projection matrix, the first trainable weight matrix, the second trainable weight matrix, the first trainable parameter matrix, the second trainable parameter matrix, the third trainable parameter matrix, the fourth trainable parameter matrix, the fifth trainable parameter matrix, the sixth trainable parameter matrix, the first CLN network model, the second CLN network model and the trainable relationship parameter matrix. 8.A device for training a text entity joint relation extraction model, characterized in that, The apparatus comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the training method of the text entity joint relationship extraction model according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the training method of the text entity joint relationship extraction model according to any one of claims 1 to 7 when the computer program is running.

Images