Text processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a text processing method and device, electronic equipment and a storage medium, and belongs to the technical field of natural language process.The method comprises the steps that sequence features and word features of a to-be-processed text segment after the to-be-processed text segment is input into a BERT model are obtained, position codes of the to-be-processed text segment are obtained, the word features and the position codes are fused to obtain fused features, and the fused features are stored in a storage medium; the method comprises the following steps: carrying out operation on a fusion feature through a multi-layer multi-head attention mechanism to obtain an attention distribution feature, processing the attention distribution feature through an activation function to obtain a microscopic attention weight, optimizing the fusion feature based on the attention distribution feature and the microscopic attention weight, and further obtaining a sequence feature based on the sequence feature and the optimized fusion feature. According to the method, the relation between the entities in the to-be-processed text segment is determined, and due to the fact that the optimized fusion feature can better express the semantic feature of the to-be-processed text, the effect of extracting the relation between the entities from the complex to-be-processed text segment can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text processing method, device, electronic device and storage medium. Background Art

[0002] With the development of natural language processing technology, there is a need to extract entities and relationships between entities from a text segment. The methods in the related art work well when processing texts with simple structures and standardized formats, but the effect is greatly reduced when facing some complex texts such as medical texts. Summary of the invention

[0003] Embodiments of the present application provide a text processing method, device, electronic device, and storage medium to solve the problem in the related art that the effect of extracting entities and relationships between entities from complex texts is relatively poor.

[0004] In a first aspect, an embodiment of the present application provides a text processing method, comprising:

[0005] Obtaining sequence features and word features of the text segment to be processed after it is input into the BERT model, and obtaining the position code of the text segment to be processed;

[0006] Fusing the word feature and the position code to obtain a fused feature;

[0007] The fusion features are operated through a multi-layer multi-head attention mechanism to obtain attention allocation features;

[0008] The attention allocation feature is processed by an activation function to obtain a micro attention weight;

[0009] Optimizing the fusion feature based on the attention allocation feature and the micro attention weight;

[0010] Based on the sequence features and the optimized fusion features, entities in the text segment to be processed and relationships between entities are determined.

[0011] In some embodiments, based on the attention allocation feature and the micro attention weight, the fusion feature is optimized, including:

[0012] Calculating the Hadamard product of the attention allocation feature and the micro attention weight;

[0013] Inputting the Hadamard product and the fused feature into a first residual network;

[0014] The output result of the first residual network is standardized to obtain optimized fusion features.

[0015] In some embodiments, it also includes:

[0016] Performing expansion processing on the sequence features to obtain sequence expansion features;

[0017] Performing mask processing on the sequence expansion features;

[0018] Inputting the sequence expansion features after mask processing into a feedforward neural network to update the sequence expansion features;

[0019] Optimizing the sequence features based on the updated sequence expansion features and the fusion features;

[0020] Determining entities and relationships between entities in the text segment to be processed based on the sequence features and the optimized fusion features includes:

[0021] Based on the optimized sequence features and the optimized fusion features, the relationships between the entities are determined.

[0022] In some embodiments, determining the relationship between the entities based on the optimized sequence features and the optimized fusion features includes:

[0023] Stacking the optimized sequence features and the optimized fusion features;

[0024] Performing mapping processing on the stacking result to reduce the dimension of the stacking result;

[0025] Processing the stacking result after the mapping process by an adaptive weighted average algorithm to obtain an optimized first weight of the sequence feature and an optimized second weight of the fusion feature;

[0026] Based on the first weight and the second weight, the optimized sequence feature and the optimized fusion feature are fused to obtain a target feature;

[0027] The target features are analyzed by the GPLinker algorithm to obtain the relationships between the entities.

[0028] In some embodiments, the sequence feature is optimized based on the updated sequence expansion feature and the fusion feature, including:

[0029] Inputting the updated sequence expansion features and the fusion features into a second residual network;

[0030] The output result of the second residual network is standardized to obtain optimized sequence features.

[0031] In some embodiments, the multi-layer multi-head attention mechanism includes four layers of multi-head attention mechanism, and each layer of multi-head attention mechanism includes four single-head attentions.

[0032] In a second aspect, an embodiment of the present application provides a text processing device, including:

[0033] An acquisition module, used to acquire sequence features and word features of the text segment to be processed after it is input into the BERT model, and to acquire the position code of the text segment to be processed;

[0034] A local information processing module is used to fuse the word feature and the position code to obtain a fused feature; operate the fused feature through a multi-layer multi-head attention mechanism to obtain an attention allocation feature; process the attention allocation feature through an activation function to obtain a micro-attention weight; and optimize the fused feature based on the attention allocation feature and the micro-attention weight;

[0035] A determination module is used to determine the entities and the relationships between the entities in the text segment to be processed based on the sequence features and the optimized fusion features.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0037] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned text processing method.

[0038] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned text processing method.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned text processing method when executed by a processor.

[0040] In the embodiment of the present application, the sequence features and word features of the text segment to be processed are obtained after the BERT model is input, and the position code of the text segment to be processed is obtained, the word features and the position code are fused to obtain fused features, and then the fused features are operated by a multi-layer multi-head attention mechanism to obtain attention allocation features, the attention allocation features are processed by an activation function to obtain micro-attention weights, and the fused features are optimized based on the attention allocation features and the micro-attention weights, and then the entities and the relationships between entities in the text segment to be processed are determined based on the sequence features and the optimized fused features. Compared with the word features output by the BERT model, the local features of the words in the attention allocation features are enhanced, which are more in line with the local correlation characteristics of complex texts. The micro-attention weights obtained by the attention allocation features can also better highlight the importance of words. With the help of the attention allocation features and the micro-attention weights, more accurate and richer fused features with semantic expression can be obtained, so the effect of extracting entities and the relationships between entities from complex text segments to be processed can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A flowchart of a text processing method provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of another text processing method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a text processing process provided in an embodiment of the present application;

[0045] Figure 4 A structural diagram of a text processing device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing a text processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to solve the problem in the related art that the effect of extracting entities and relationships between entities from complex texts is relatively poor, the embodiments of the present application provide a text processing method, device, electronic device and storage medium.

[0048] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0049] See also Figure 1 , Figure 1 A flowchart of a text processing method provided in an embodiment of the present application, the method can be applied to electronic devices such as servers and desktop computers, and the method includes the following steps.

[0050] In step 101, the sequence features and word features of the text segment to be processed are obtained after being input into the BERT model, and the position code of the text segment to be processed is obtained.

[0051] Generally, the text segment to be processed includes a paragraph, which includes several punctuation marks such as commas, periods, exclamation marks, question marks, etc. After the text segment to be processed is input into the BERT model, sequence features and word features can be obtained, among which the sequence features are generally of fixed length, that is, the BERT model will map the text segment to be processed of any length into sequence features of fixed length, and the sequence features are generally vectors, while the word features are generally in matrix form, and the word features contain the feature vectors of each word in the text segment to be processed.

[0052] The position coding of the text segment to be processed can be obtained through static position coding, rotation position coding and other coding methods, and the position coding of the text segment to be processed is also in matrix form, and the position coding contains the position vector of each word in the text segment to be processed.

[0053] In addition, it should be noted that the words in the embodiments of the present application are equivalent to characters, and word features are equivalent to character features.

[0054] In step 102, the word feature and the position code are fused to obtain a fused feature.

[0055] Among them, the matrix dimensions of word features and position codes are the same. Therefore, the word features and position codes can be fused by matrix addition to obtain fused features.

[0056] In step 103, the fusion features are operated through a multi-layer multi-head attention mechanism to obtain attention allocation features.

[0057] For example, a 4-layer multi-head attention mechanism is used, and each layer of the attention mechanism includes 4 parallel single-head attentions. Generally, the attention allocation feature is in the form of a matrix, and the attention allocation feature contains the attention allocation information of each word to other words in the text segment to be processed.

[0058] In step 104, the attention allocation features are processed by an activation function to obtain micro attention weights.

[0059] The activation function is such as the SoftMax function. The micro-attention weight includes the weight of each word in the text segment to be processed, which is used to represent the importance of this word to the text segment to be processed. The value of the weight is positively correlated with the importance.

[0060] In step 105, the fused features are optimized based on the attention allocation features and the micro-attention weights.

[0061] For example, the Hadamard product of the attention allocation feature and the micro-attention weight is calculated to amplify the element value of the important words in the text segment to be processed in the attention allocation feature. Considering that in practical applications, important words may not be the words that are focused on during entity extraction and relationship extraction, in order to improve the robustness of the algorithm, the Hadamard product and the fusion feature can be input into the first residual network, and then the output result of the first residual network is normalized (i.e., Normalization) to obtain the optimized fusion feature.

[0062] In step 106, entities and relationships between entities in the text segment to be processed are determined based on the sequence features and the optimized fusion features.

[0063] For example, the sequence features and the optimized fusion features are stacked, and the stacking results are mapped to reduce the dimension of the stacking results. Then, the stacking results after the mapping are processed by an adaptive weighted average algorithm to obtain the weight of the sequence features and the weight of the optimized fusion features. Based on the weight of the sequence features and the weight of the optimized fusion features, the sequence features and the optimized fusion features are fused to obtain the target features. For example, the weight of the sequence features and the Hadamard product of the sequence features are calculated, the weight of the optimized fusion features and the Hadamard product of the optimized fusion features are calculated, and the two Hadamard product results are tensor spliced ​​to obtain the target features. Then, the target features can be analyzed by the GPLinker algorithm to obtain the relationship between entities.

[0064] Among them, the GPLinker algorithm mainly improves the decoding part of the entity-relationship joint extraction task. When decoding, it does not distinguish between the subject and the object, but decodes the entity uniformly; the relationship recognition is obtained by predicting the pairing of the head and tail positions of the entity pair, and then the entity-relationship joint extraction task can be defined as predicting all possible combinations in the sequence. The specific prediction process can be broken down into two parts: the first part predicts the head and tail parts of the subject and the head and tail parts of the object; the second part further predicts the relationship between the subject and the object based on the prediction of the subject and object fragments respectively.

[0065] See also Figure 2 , Figure 2 A flowchart of another text processing method provided in an embodiment of the present application, the method includes the following steps.

[0066] In step 201, the sequence features and word features of the text segment to be processed are obtained after being input into the BERT model, and the position code of the text segment to be processed is obtained.

[0067] In step 202, the word feature and the position code are fused to obtain a fused feature.

[0068] In step 203, the fusion features are operated through a multi-layer multi-head attention mechanism to obtain attention allocation features.

[0069] In step 204, the attention allocation features are processed by an activation function to obtain micro attention weights.

[0070] In step 205, the fused features are optimized based on the attention allocation features and the micro-attention weights.

[0071] For example, the Hadamard product of the attention allocation feature and the micro-attention weight is calculated, the Hadamard product and the fusion feature are input into the first residual network, and then the output result of the first residual network is normalized (i.e., Normalization) to obtain the optimized fusion feature.

[0072] In step 206, the sequence features are expanded to obtain sequence expansion features.

[0073] That is, the sequence features in vector form are expanded into sequence expanded features in matrix form.

[0074] In step 207, mask processing is performed on the sequence expansion features.

[0075] The masking process refers to covering part of the text content in the text to be processed.

[0076] In step 208, the sequence expansion features after mask processing are input into a feed-forward neural network to update the sequence expansion features.

[0077] In practical applications, the feedforward neural network will generate new sequence expansion features based on the sequence expansion features after masking. The new sequence expansion features will not only contain the information of the masked text content, but also the information of the unmasked text content will be different from the previous ones.

[0078] Here, the masked text in the text segment to be processed will force the feedforward neural network to re-understand the entities and the relationships between entities in the text segment to be processed, improve the probability of the feedforward neural network splitting the complex entities in the text segment to be processed into different entities, enhance the understanding of global information, and facilitate the extraction of more entity relationship information from the text segment to be processed.

[0079] In step 209, the sequence features are optimized based on the updated sequence expansion features and fusion features.

[0080] For example, the updated sequence expansion features and fusion features are input into the second residual network, and then the output result of the second residual network is normalized (i.e., Normalization) to obtain the optimized sequence features. Compared with the sequence features before optimization, the optimized sequence features have better robustness.

[0081] In step 210, entities and relationships between entities in the text segment to be processed are determined based on the optimized sequence features and the optimized fusion features.

[0082] For example, the optimized sequence features and the optimized fusion features are stacked, and the stacking results are mapped to reduce the dimension of the stacking results. Then, the stacking results after the mapping are processed by an adaptive weighted average algorithm to obtain the first weight of the optimized sequence features and the second weight of the optimized fusion features. Based on the first weight and the second weight, the optimized sequence features and the optimized fusion features are fused to obtain the target features. For example, the Hadamard product of the first weight and the optimized sequence features is calculated, and the Hadamard product of the second weight and the optimized fusion features is calculated. The two Hadamard product results are tensor spliced ​​to obtain the target features. Then, the target features are analyzed by the GPLinker algorithm to obtain the relationship between entities.

[0083] The text processing scheme of the present application is based on the encoder-decoder as the basic architecture. The encoder is designed based on the Bidirectional Encoder Representations from Transformers (BERT) model. For the sequence features (global information) of the text segment to be processed output by the BERT model, an entity mask matrix is ​​added to the sequence features and the global information is enhanced with the help of a feedforward neural network. For the word features (local information) of the text segment to be processed output by the BERT model, a multi-layer multi-head attention mechanism is used to reuse local information, and micro-attention is used to enhance the understanding of complex entities such as medical professional terms. Then, an adaptive weighted balancing mechanism is designed to fuse the enhanced global information (i.e., the optimized sequence features) and the reused local information (i.e., the optimized fusion features). Finally, the decoder is designed based on the GPLinker algorithm. For the fusion result (i.e., the target feature) of the adaptive weighted balancing mechanism, an entity and relationship pointer decoder is constructed to interactively utilize the entity and relationship information and constrain the extraction results to achieve knowledge triple pairing.

[0084] Specifically, to address the problem of semantic modeling of complex texts, a Local Information Reuse (LIR) module is designed to extract local information through a multi-layer attention mechanism, and use micro-attention enhancement algorithm to perceive fine-grained features, so as to alleviate the problem of inaccurate modeling caused by the complex semantics of complex text data; to address the problem of nested relationship triples, a Global Information Enhancement (GIE) module is designed to fully learn the overall representation by adding a mask matrix to the sequence features, so as to avoid the problem of insufficient feature information acquisition caused by too long text; to address the problem of long-tail distribution of categories, an Adaptive Weighted Balancing Fusion (AWBF) module is designed to utilize the dynamic complementarity of global and local information to improve the algorithm's recognition ability of sparse relations, so as to enhance the ability to distinguish multiple nested relations in complex text data.

[0085] See also Figure 3 , Figure 3 A schematic diagram of a text processing process provided in an embodiment of the present application, assuming that the text segment to be processed is: Horner syndrome, characterized by ptosis, enophthalmos and anhidrosis on the same side. Figure 3 An introduction to the text processing process.

[0086] In practical applications, in order to avoid the algorithm losing the relative position information of words in the multi-layer network, for the text segment to be processed, the word feature matrix M of this text segment can be obtained by mapping the BERT model. T∈R s×d Add the position encoding matrix to get the matrix (i.e., fusion features), where s is the length of the sequence feature (i.e., the number of elements contained in the sequence feature), d is the feature dimension (the vector representation dimension of a single word in the word feature matrix), and the calculation formula for the elements in the position encoding matrix is ​​shown in formula (1).

[0087]

[0088] Among them, pos represents the position of the word in the text segment to be processed, the dimension of the position encoding matrix is ​​the same as the dimension of the word feature matrix, PE (pos,2i) and PE (pos,2i+1) They represent the elements in the posth row, 2ith column and the posth row, 2i+1th column in the position encoding matrix respectively.

[0089] 1. LIR module.

[0090] The design of the local information reuse module aims to achieve micro-modeling of local information, with the aim of alleviating the problem of inaccurate modeling caused by the complex semantics of complex texts.

[0091] This module is based on the encoder architecture design of the Transformer model. It increases the weight of entity information in the algorithm learning process by optimizing the attention mechanism, thereby capturing local key information. This module can avoid the problem of insufficient feature information acquisition caused by too long text, and at the same time ensure that the algorithm pays more attention to the feature information of complex text fragments, reduce the interference of useless fragment information, and enhance the algorithm's key knowledge learning ability.

[0092] In order to fully capture the deep features, this module adopts a multi-layer multi-head attention architecture. Specifically, a single attention head first performs a mapping transformation by transforming the matrix With the weight matrix W T ∈R d×d After three transformations, we get the query matrix Q of the same dimension. l , key matrix K l , value matrix V l , (l has no special meaning and is only used as a subscript to distinguish the matrices in the LIR module). Next, the query matrix Q l and key matrix K l Do a dot product operation to get the relative attention score matrix of the word, and then, with the help of the activation function (i.e. Figure 3 The SoftMax function in ) obtains the micro attention weights.

[0093] Different from the traditional scaled dot product attention, this module does not scale the micro attention weights to preserve local information to the greatest extent, but directly multiplies them by the value matrix V l The final result is obtained, and the calculation process is shown in formula (2):

[0094]

[0095] The multi-head attention mechanism consists of multiple single-head attention modules, including multiple groups of Q l , K l 、V l Matrix, map the matrix to a low-dimensional space, go through the same calculation process as the single-head attention module multiple times, and finally concatenate the outputs of multiple parallel attention heads to get the final result. The calculation process is shown in Equations (3) and (4).

[0096]

[0097]

[0098] in, It represents the calculation result of the i-th attention head, i is used to distinguish the attention heads, and n represents the total number of attention heads.

[0099] The traditional self-attention mechanism has a computational bottleneck problem. In order to improve the computational efficiency of the algorithm, this section adopts the low-rank matrix approximation method to decompose the original scaled dot product attention into multiple smaller attention sub-matrices through linear projection. By introducing low-rank matrix approximation, the computational matrix size of the self-attention mechanism can be reduced, that is, formula (3) can be transformed into:

[0100]

[0101] Among them, E i ,F i ∈R s×k is the key matrix K l , value matrix V l The projection matrix maps the original s×d dimensional matrix to a k×d dimensional matrix, and constrains k<s to achieve matrix dimensionality reduction.

[0102] This module can use a 4-layer multi-head attention module. Each layer of attention module consists of 4 parallel single-head attention modules. The 4-layer multi-head attention module finally calculates the local information enhancement feature matrix M L ′ (i.e., attention allocation feature). On this basis, this paper also proposes a micro-attention enhancement mechanism, which retains the attention score matrix X of the last layer of multi-layer multi-head attention Att , for the attention score matrix X AttAfter the mapping dimension is reduced, an activation function such as the SoftMax function is used for calculation to obtain the micro attention weight W Att , through this weight, the network's modeling of local information is further strengthened. The calculation of the micro attention module is expressed as:

[0103]

[0104] The micro-attention enhancement mechanism enables the algorithm to focus on specific word fragments, and improves the algorithm's recognition accuracy and learning efficiency for complex word fragments by extracting and integrating local information.

[0105] Finally, this module uses residual connection and normalization to obtain the local feature enhancement matrix M L (i.e. optimized fusion features), strengthen the network's overall and local interaction modeling, and improve the generalization ability of the algorithm. The formula is as follows:

[0106]

[0107] 2. GIE module.

[0108] Conventional fine-tuning pre-training model methods only use word features and ignore sequence features, which is not conducive to distinguishing multiple nested relationships in complex text data. For example, the Beijing Municipal Government is located in Tongzhou District. This text data contains two entity relationship pairs, namely (Beijing Municipal Government, located in, Tongzhou District) and (Beijing City, contains, Tongzhou District). The entity relationship pair extraction scheme in the related art usually ignores the latter entity relationship pair. It should be noted that the text segment to be processed in the embodiment of the present application usually contains multiple sentences. Here, only one sentence is cited to illustrate the multiple nested relationships. In actual applications, the text segment to be processed will contain more words, but it will generally be controlled within a certain length such as 300 words.

[0109] In the embodiment of the present application, in order to make full use of the global information of long sequence text, the global feature sequence output by the BERT model is first preprocessed. SG∈R d (i.e., sequence features), and by paying attention to the sequence SA∈R s The overall mask is implemented to hide part of the information in the text segment to be processed. At the same time, the dimension expansion and alignment operations are performed on the attention sequence and the global feature sequence to obtain the entity mask matrix X A ∈R 1×d and the global feature sequence matrix X G ∈R 1×s Then, a mask matrix is ​​used to cover part of the sequence feature information to enhance the algorithm's ability to extract global information:

[0110]

[0111] Among them, the Einsum function is used to quickly implement the dot product of two matrices. is the global feature matrix after mask processing, s is the length of the input sequence, and d is the embedding dimension of the last layer output of the BERT model.

[0112] The global feature matrix after masking cannot grasp the overall structure and context information, so a position-based feedforward neural network (FNN) is used to add nonlinear transformation to further learn the mask sequence feature matrix, so that the network module has the ability to perceive global information and pay attention to word order details. The FNN calculation process is shown in formula (9):

[0113]

[0114] Among them, W1 and W2 are pre-learned weight matrices, b1 and b2 are pre-learned biases, and f(·) represents the ReLU activation function. The feature matrix that extracts global information is obtained by projecting the first matrix to a low-dimensional space and then re-dimensioning it by the second matrix.

[0115] Taking the example of "Beijing Municipal Government is located in Tongzhou District", through masking processing, the text in this sentence can be arbitrarily covered. The covered text can increase the probability of the feedforward neural network splitting the Beijing Municipal Government into different entities, enhance the understanding of global information, and increase the probability of subsequent extraction of the relationship between entities (Beijing Municipal Government, including Tongzhou District).

[0116] In order to avoid information loss after multi-layer network transmission and improve the robustness of the algorithm, this module also uses residual connections to introduce word-level feature encoding with added position information, and normalizes it to obtain the global information enhancement feature matrix M G (i.e., optimized sequence features). This processing method can realize interactive learning of sequence and word information elements and enhance the robustness of the algorithm. G The function is defined as:

[0117]

[0118] 3. AWBF module.

[0119] In order to deal with the long-tail distribution problem of categories in complex text data, an adaptive weighted balanced fusion feature strategy is designed to fully realize the complementary fusion of global information and local information. This module integrates the two parts of information by stacking the global feature matrix (i.e., the optimized sequence features) and the local feature matrix (i.e., the optimized fusion features) to reduce the negative disturbance caused by directly superimposing the two matrices. Then, the integrated matrix is ​​mapped to the low-dimensional space to improve the knowledge enrichment of the network, and finally the activation function is used to calculate the dynamic weight matrix W α , the dynamic fusion process is as follows:

[0120]

[0121]

[0122] M fuse =W α ·M L +(1-W α )·M G , (13)

[0123] Among them, M fuse is the target feature.

[0124] This fusion method can adaptively balance the weights of global information and local information, ensure the effective fusion of information, and enhance the algorithm's ability to recognize sparse relationships.

[0125] 4. Entity and relationship decoding.

[0126] The decoder aims to identify all possible triples (subject, relation, object) in the text through the semantic information output by the encoder. In order to simplify the algorithm structure and realize single-step extraction, a global pointer decoding method can be used.

[0127] The decoding process includes two core steps: entity decoding and relation decoding. Relation decoding is further divided into head decoding and tail decoding. In the decoding matrix, the row represents the starting position and the column represents the ending position. Figure 3 As shown in the figure, the entity matrix shows the principle of entity pointer decoding. The "1" in the matrix means that the fragment corresponding to the row and column index is an entity. Here, the subject and object are not distinguished, but the upper triangular matrix is ​​used to ensure that the starting position is less than or equal to the ending position. The head relationship matrix and the tail relationship matrix represent the entity head and tail positions of the matching relationship, respectively. The rows and columns represent the starting and ending positions of the entity head and tail, respectively. Entity decoding and relationship decoding perform inference prediction under the constraints of the Schema, and the decoding results of these two parts will filter each other. This decoding method can implicitly use entity category information to assist relationship extraction, so as to directly predict the Schema, and jointly realize named entity recognition and relationship extraction with the idea of ​​global normalization.

[0128] In practical applications, the GPLinker algorithm can be used to identify entity recognition and the relationship between entities from target features.

[0129] In the embodiment of the present application, two modules, global information enhancement and local information reuse, are designed in the encoder part to extract macro and micro features respectively, and realize efficient complementarity between the two through adaptive weighted balance fusion. Among them, the local information reuse module enables the algorithm to focus on local information and effectively deal with complex text semantic problems; the global information enhancement module can improve the recognition ability of multiple nested relationships in complex text data; the adaptive weighted balance strategy is conducive to the fusion of global and local information, and effectively deal with the long-tail distribution problem.

[0130] Based on the same technical concept, the embodiment of the present application also provides a text processing device. The principle of solving the problem by the text processing device is similar to that of the above-mentioned text processing method. Therefore, the implementation of the text processing device can refer to the implementation of the text processing method, and the repeated parts will not be repeated.

[0131] Figure 4 A structural diagram of a text processing device provided in an embodiment of the present application includes:

[0132] An acquisition module 401 is used to acquire sequence features and word features of a text segment to be processed after it is input into a BERT model, and to acquire a position code of the text segment to be processed;

[0133] The local information processing module 402 is used to fuse the word feature and the position code to obtain a fused feature; operate the fused feature through a multi-layer multi-head attention mechanism to obtain an attention allocation feature; process the attention allocation feature through an activation function to obtain a micro attention weight;

[0134] Optimizing the fusion feature based on the attention allocation feature and the micro attention weight;

[0135] The determination module 403 is used to determine the entities and the relationships between the entities in the text segment to be processed based on the sequence features and the optimized fusion features.

[0136] In some embodiments, the local information processing module 402 is specifically configured to:

[0137] Calculating the Hadamard product of the attention allocation feature and the micro attention weight;

[0138] Inputting the Hadamard product and the fused feature into a first residual network;

[0139] The output result of the first residual network is standardized to obtain optimized fusion features.

[0140] In some embodiments, it also includes:

[0141] The global information processing module 404 is used to perform expansion processing on the sequence features to obtain sequence expansion features; perform mask processing on the sequence expansion features; input the masked sequence expansion features into a feedforward neural network to update the sequence expansion features; and optimize the sequence features based on the updated sequence expansion features and the fusion features;

[0142] The determination module 403 is further configured to determine the entities and the relationships between the entities based on the optimized sequence features and the optimized fusion features.

[0143] In some embodiments, the determination module 403 is specifically configured to:

[0144] Stacking the optimized sequence features and the optimized fusion features;

[0145] Performing mapping processing on the stacking result to reduce the dimension of the stacking result;

[0146] Processing the stacking result after the mapping process by an adaptive weighted average algorithm to obtain an optimized first weight of the sequence feature and an optimized second weight of the fusion feature;

[0147] Based on the first weight and the second weight, the optimized sequence feature and the optimized fusion feature are fused to obtain a target feature;

[0148] The target features are analyzed by the GPLinker algorithm to obtain the relationships between the entities.

[0149] In some embodiments, the global information processing module 404 is specifically configured to:

[0150] Inputting the updated sequence expansion features and the fusion features into a second residual network;

[0151] The output result of the second residual network is standardized to obtain optimized sequence features.

[0152] In some embodiments, the multi-layer multi-head attention mechanism includes four layers of multi-head attention mechanism, and each layer of multi-head attention mechanism includes four single-head attentions.

[0153] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0154] After introducing the text processing method and apparatus according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0155] Refer to the following Figure 5 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 5 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0156] like Figure 5 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0157] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0158] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0159] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0160] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0161] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned text processing method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any text processing method provided in the embodiments of the present application.

[0163] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.

[0164] Furthermore, the computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0165] The program product for text processing in the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0166] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0167] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0168] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0169] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0170] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0176] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.

Claims

1. A text processing method, characterized in that: include: Obtaining sequence features and word features of the text segment to be processed after it is input into the BERT model, and obtaining the position code of the text segment to be processed; Fusing the word feature and the position code to obtain a fused feature; The fusion features are operated through a multi-layer multi-head attention mechanism to obtain attention allocation features; The attention allocation feature is processed by an activation function to obtain a micro attention weight; Optimizing the fusion feature based on the attention allocation feature and the micro attention weight; Based on the sequence features and the optimized fusion features, entities in the text segment to be processed and relationships between entities are determined.

2. The method according to claim 1, characterized in that Based on the attention allocation feature and the micro attention weight, the fusion feature is optimized, including: Calculating the Hadamard product of the attention allocation feature and the micro attention weight; Inputting the Hadamard product and the fused feature into a first residual network; The output result of the first residual network is standardized to obtain optimized fusion features.

3. The method according to claim 1 or 2, characterized in that Also includes: Performing expansion processing on the sequence features to obtain sequence expansion features; Performing mask processing on the sequence expansion features; Inputting the sequence expansion features after mask processing into a feedforward neural network to update the sequence expansion features; Optimizing the sequence features based on the updated sequence expansion features and the fusion features; Determining entities and relationships between entities in the text segment to be processed based on the sequence features and the optimized fusion features includes: Based on the optimized sequence features and the optimized fusion features, the relationships between the entities are determined.

4. The method according to claim 3, characterized in that Determining the relationship between the entities based on the optimized sequence features and the optimized fusion features includes: Stacking the optimized sequence features and the optimized fusion features; Performing mapping processing on the stacking result to reduce the dimension of the stacking result; Processing the stacking result after the mapping process by an adaptive weighted average algorithm to obtain an optimized first weight of the sequence feature and an optimized second weight of the fusion feature; Based on the first weight and the second weight, the optimized sequence feature and the optimized fusion feature are fused to obtain a target feature; The target features are analyzed by the GPLinker algorithm to obtain the relationships between the entities.

5. The method according to claim 3, characterized in that Based on the updated sequence expansion feature and the fusion feature, the sequence feature is optimized, including: Inputting the updated sequence expansion features and the fusion features into the second residual network; The output result of the second residual network is standardized to obtain optimized sequence features.

6. The method according to claim 1, characterized in that The multi-layer multi-head attention mechanism includes four layers of multi-head attention mechanisms, and each layer of multi-head attention mechanism includes four single-head attentions.

7. A text processing device, characterized in that: include: An acquisition module, used to acquire sequence features and word features of the text segment to be processed after it is input into the BERT model, and to acquire the position code of the text segment to be processed; A local information processing module is used to fuse the word feature and the position code to obtain a fused feature; and to calculate the fused feature through a multi-layer multi-head attention mechanism to obtain an attention allocation feature; The attention allocation feature is processed by an activation function to obtain a micro attention weight; Optimizing the fusion feature based on the attention allocation feature and the micro attention weight; A determination module is used to determine the entities and the relationships between the entities in the text segment to be processed based on the sequence features and the optimized fusion features.

8. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images