A text processing method, apparatus, electronic device, and storage medium

Abstract

This application discloses a text processing method, apparatus, electronic device, and storage medium, belonging to the field of natural language processing technology. In this method, the sequence features and word features of the text segment to be processed after inputting into the BERT model are obtained, and the positional encoding of the text segment to be processed is obtained. The word features and positional encoding are fused to obtain fused features. The fused features are processed through a multi-layer multi-head attention mechanism to obtain attention allocation features. The attention allocation features are processed through an activation function to obtain micro-attention weights. Based on the attention allocation features and micro-attention weights, the fused features are optimized. Then, based on the sequence features and the optimized fused features, the entities and relationships between entities in the text segment to be processed are determined. Since the optimized fused features can better express the semantic features of the text to be processed, the effect of extracting entities and relationships between entities from complex text segments to be processed can be improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of natural language processing technology, there has been a need to extract entities and relationships between entities from a text segment. While existing methods perform well with simple, well-formatted text, their effectiveness drops significantly when dealing with complex texts, such as those in the medical field. Summary of the Invention

[0003] This application provides a text processing method, apparatus, electronic device, and storage medium to solve the problem of poor performance in extracting entities and relationships between entities from complex text in related technologies.

[0004] In a first aspect, embodiments of this application provide a text processing method, including:

[0005] Obtain the sequence features and word features of the text segment to be processed after inputting it into the BERT model, and obtain the positional encoding of the text segment to be processed;

[0006] The word features and the positional encoding are fused to obtain fused features;

[0007] The fused features are processed by a multi-layer, multi-head attention mechanism to obtain attention allocation features;

[0008] The attention allocation features are processed by an activation function to obtain micro-attention weights;

[0009] The fusion feature is optimized based on the attention allocation features and the micro-attention weights;

[0010] Based on the sequence features and the optimized fusion features, the entities in the text segment to be processed and the relationships between entities are determined.

[0011] In some embodiments, the fusion features are optimized based on the attention allocation features and the micro-attention weights, including:

[0012] Calculate the Hadamard product of the attention allocation features and the micro attention weights;

[0013] The Hadamard product and the fused feature are input into the first residual network;

[0014] The output of the first residual network is standardized to obtain the optimized fusion features.

[0015] In some embodiments, it also includes:

[0016] The sequence features are augmented to obtain augmented sequence features;

[0017] The sequence augmentation features are masked.

[0018] The sequence augmentation features after masking are input into a feedforward neural network to update the sequence augmentation features;

[0019] The sequence features are optimized based on the updated sequence augmentation features and the fusion features.

[0020] Based on the sequence features and the optimized fusion features, the entities and relationships between entities in the text segment to be processed are determined, including:

[0021] Based on the optimized sequence features and the optimized fusion features, the entities and the relationships between them are determined.

[0022] In some embodiments, determining the entities and relationships between entities based on the optimized sequence features and the optimized fusion features includes:

[0023] The optimized sequence features and the optimized fusion features are stacked;

[0024] The stacking results are mapped to reduce their dimensionality.

[0025] The stacked result after mapping is processed by an adaptive weighted average algorithm to obtain the first weight of the optimized sequence feature and the second weight of the optimized fusion feature;

[0026] Based on the first weight and the second weight, the optimized sequence features and the optimized fusion features are fused to obtain the target features;

[0027] The target features are analyzed using the GPLinker algorithm to obtain the entities and the relationships between them.

[0028] In some embodiments, the sequence features are optimized based on the updated sequence augmentation features and the fusion features, including:

[0029] The updated sequence augmentation features and the fusion features are input into the second residual network;

[0030] The output of the second residual network is standardized to obtain the optimized sequence features.

[0031] In some embodiments, the multi-layer multi-head attention mechanism includes a four-layer multi-head attention mechanism, with each layer including four single-head attention mechanisms.

[0032] Secondly, embodiments of this application provide a text processing apparatus, including:

[0033] The acquisition module is used to acquire the sequence features and word features of the text segment to be processed after it is input into the BERT model, and to acquire the positional encoding of the text segment to be processed;

[0034] A local information processing module is used to fuse the word features and the positional encoding to obtain fused features; to process the fused features through a multi-layer multi-head attention mechanism to obtain attention allocation features; to process the attention allocation features through an activation function to obtain micro-attention weights; and to optimize the fused features based on the attention allocation features and the micro-attention weights.

[0035] The determination module is used to determine the entities and relationships between entities in the text segment to be processed based on the sequence features and the optimized fusion features.

[0036] Thirdly, embodiments of this application provide an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0037] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the text processing method described above.

[0038] Fourthly, embodiments of this application provide a storage medium in which the electronic device can execute the above-described text processing method when a computer program in the storage medium is executed by a processor of an electronic device.

[0039] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described text processing method.

[0040] In this embodiment, sequence features and word features of the text segment to be processed are obtained after inputting into the BERT model, and positional encoding of the text segment is also obtained. The word features and positional encoding are fused to obtain fused features. Then, the fused features are processed through a multi-layer multi-head attention mechanism to obtain attention allocation features. The attention allocation features are processed through an activation function to obtain micro-attention weights. Based on the attention allocation features and micro-attention weights, the fused features are optimized. Finally, based on the sequence features and the optimized fused features, the entities and relationships between entities in the text segment to be processed are determined. Compared with the word features output by the BERT model, the local features of words in the attention allocation features are enhanced, which is more in line with the local correlation characteristics of complex text. The micro-attention weights obtained through the attention allocation features can also better highlight the importance of words. With the help of attention allocation features and micro-attention weights, fused features with more accurate and richer semantic expression can be obtained. Therefore, the effect of extracting entities and relationships between entities from complex text segments to be processed can be improved. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 A flowchart illustrating a text processing method provided in an embodiment of this application;

[0043] Figure 2 A flowchart illustrating yet another text processing method provided in this application embodiment;

[0044] Figure 3 A schematic diagram illustrating a text processing procedure provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the hardware structure of an electronic device for implementing a text processing method, provided as an embodiment of this application. Detailed Implementation

[0047] To address the issue of poor performance in extracting entities and relationships between entities from complex text in related technologies, embodiments of this application provide a text processing method, apparatus, electronic device, and storage medium.

[0048] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0049] See Figure 1 , Figure 1 This is a flowchart of a text processing method provided in an embodiment of this application. The method can be applied to electronic devices such as servers and desktop computers, and the method includes the following steps.

[0050] In step 101, the sequence features and word features of the text segment to be processed after being input into the BERT model are obtained, and the positional encoding of the text segment to be processed is obtained.

[0051] Generally, the text segment to be processed consists of a paragraph containing several punctuation marks such as commas, periods, exclamation marks, and question marks. After the text segment is input into the BERT model, sequence features and word features are obtained. The sequence features are generally of fixed length, meaning that the BERT model will map any length of text segment to be processed into a fixed-length sequence feature. Furthermore, the sequence features are generally vectors, while the word features are generally in matrix form, containing the feature vectors of each word in the text segment to be processed.

[0052] The positional encoding of the text segment to be processed can be obtained through encoding methods such as static positional encoding and rotational positional encoding. Furthermore, the positional encoding of the text segment to be processed is also in matrix form, containing the position vectors of each word in the text segment to be processed.

[0053] In addition, it should be noted that the words in the embodiments of this application are equivalent to characters, and the word features are equivalent to character features.

[0054] In step 102, word features and positional encoding are fused to obtain fused features.

[0055] Since word features and positional encoding have the same matrix dimension, word features and positional encoding can be fused by matrix addition to obtain fused features.

[0056] In step 103, the fused features are processed through a multi-layer multi-head attention mechanism to obtain attention allocation features.

[0057] For example, a four-layer multi-head attention mechanism can be used, with each layer consisting of four parallel single-head attention mechanisms. Generally, the attention allocation features are in matrix form, containing information about the attention allocation of each word in the text segment to other words.

[0058] In step 104, the attention allocation features are processed by an activation function to obtain micro-attention weights.

[0059] The activation function is such as the SoftMax function. The micro-attention weights contain the weight of each word in the text segment to be processed, representing the importance of that word in the text segment; the weight value is positively correlated with importance.

[0060] In step 105, the fusion features are optimized based on attention allocation features and micro attention weights.

[0061] For example, the Hadamard product of attention allocation features and micro-attention weights is calculated to amplify the element values ​​of important words in the attention allocation features of the text segment to be processed. Considering that in practical applications, important words may not be the words of interest in entity extraction and relation extraction, in order to improve the robustness of the algorithm, the Hadamard product and the fused features can be input into the first residual network. Then, the output of the first residual network is normalized to obtain the optimized fused features.

[0062] In step 106, based on sequence features and optimized fusion features, the entities in the text segment to be processed and the relationships between entities are determined.

[0063] For example, sequence features and optimized fused features are stacked, and the stacked results are mapped to reduce their dimensionality. Then, an adaptive weighted average algorithm is used to process the mapped stacked results to obtain the weights of the sequence features and the optimized fused features. Based on these weights, the sequence features and optimized fused features are fused to obtain the target features. For example, the weights of the sequence features and their Hadamard product are calculated, as are the weights of the optimized fused features and their Hadamard product. The two Hadamard product results are then concatenated using a tensor to obtain the target features. Finally, the GPLinker algorithm can be used to analyze the target features to obtain the entities and the relationships between them.

[0064] The GPLinker algorithm primarily improves the decoding part of the entity relation joint extraction task. During decoding, it does not distinguish between subject and object, but decodes them uniformly to obtain entities. Relationship recognition is obtained by predicting the head and tail positions of entity pairs. Therefore, the entity relation joint extraction task can be defined as predicting all possible combinations in the sequence. The specific prediction process can be broken down into two parts: the first part predicts the head and tail of the subject and the head and tail of the object; the second part, based on the separately predicted subject and object segments, further predicts the relationships between the subject and object.

[0065] See Figure 2 , Figure 2 A flowchart of another text processing method provided in an embodiment of this application is shown, the method including the following steps.

[0066] In step 201, the sequence features and word features of the text segment to be processed after being input into the BERT model are obtained, and the positional encoding of the text segment to be processed is obtained.

[0067] In step 202, word features and positional encoding are fused to obtain fused features.

[0068] In step 203, the fused features are processed through a multi-layer multi-head attention mechanism to obtain attention allocation features.

[0069] In step 204, the attention allocation features are processed by an activation function to obtain micro-attention weights.

[0070] In step 205, the fusion features are optimized based on attention allocation features and micro attention weights.

[0071] For example, the Hadamard product of attention allocation features and micro attention weights is calculated, and the Hadamard product and fused features are input into the first residual network. Then, the output of the first residual network is normalized to obtain the optimized fused features.

[0072] In step 206, the sequence features are augmented to obtain augmented sequence features.

[0073] That is, the sequence features in vector form are expanded into sequence extended features in matrix form.

[0074] In step 207, the sequence augmentation features are masked.

[0075] Masking refers to covering up part of the text content in the text to be processed.

[0076] In step 208, the masked sequence augmentation features are input into the feedforward neural network to update the sequence augmentation features.

[0077] In practical applications, the feedforward neural network generates new sequence augmentation features based on the sequence augmentation features after masking. The new sequence augmentation features not only contain information about the masked text content, but also the information about the unmasked text content will be different from the previous ones.

[0078] Here, the obscured text in the text segment to be processed forces the feedforward neural network to reinterpret the entities and relationships between them, increasing the probability that the feedforward neural network can split the complex entities in the text segment into different entities, enhancing the understanding of global information, and facilitating the extraction of more information on the relationships between entities from the text segment to be processed.

[0079] In step 209, the sequence features are optimized based on the updated sequence augmentation and fusion features.

[0080] For example, the updated sequence augmentation and fusion features are input into a second residual network. Then, the output of the second residual network is normalized to obtain optimized sequence features. Compared to the unoptimized sequence features, the optimized sequence features are more robust.

[0081] In step 210, based on the optimized sequence features and the optimized fusion features, the entities in the text segment to be processed and the relationships between entities are determined.

[0082] For example, the optimized sequence features and optimized fused features are stacked, and the stacked results are mapped to reduce the dimensionality of the stacked results. Then, the stacked results after mapping are processed by an adaptive weighted average algorithm to obtain the first weight of the optimized sequence features and the second weight of the optimized fused features. Based on the first and second weights, the optimized sequence features and optimized fused features are fused to obtain the target features. For example, the Hadamard product of the first weight and the optimized sequence features is calculated, and the Hadamard product of the second weight and the optimized fused features is calculated. The two Hadamard product results are then tensor-concatenated to obtain the target features. Finally, the GPLinker algorithm is used to analyze the target features to obtain the entities and the relationships between entities.

[0083] The text processing scheme in this application uses an encoder-decoder architecture. The encoder is designed based on the Bidirectional Encoder Representations from Transformers (BERT) model. For the sequence features (global information) of the text segment to be processed output by the BERT model, an entity mask matrix is ​​added to the sequence features and a feedforward neural network is used to enhance the global information. For the word features (local information) of the text segment to be processed output by the BERT model, a multi-layer multi-head attention mechanism is used to reuse the local information. At the same time, micro-attention is used to enhance the understanding of complex entities such as medical terminology. Then, an adaptive weighted balancing mechanism is designed to fuse the enhanced global information (i.e., the optimized sequence features) and the reused local information (i.e., the optimized fused features). Finally, the decoder is designed based on the GPLinker algorithm. For the fusion result (i.e., the target features) of the adaptive weighted balancing mechanism, an entity and relation pointer decoder is constructed. The entity and relation information are interactively utilized and the extraction results are constrained to achieve knowledge triple pairing.

[0084] Specifically, for the semantic modeling problem of complex text, a Local Information Reuse (LIR) module is designed. This module extracts local information through a multi-layer attention mechanism and uses a micro-attention enhancement algorithm to improve the perception of fine-grained features, thereby alleviating the modeling inaccuracies caused by the semantic complexity of complex text data. For the nested relation triple problem, a Global Information Enhancement (GIE) module is designed. This module learns the overall representation by adding a mask matrix to the sequence features, thereby avoiding the problem of insufficient feature information acquisition due to excessive text length. For the long-tailed distribution of categories, an Adapted Weighted Balancing Fusion (AWBF) module is designed. This module utilizes the dynamic complementarity of global and local information to improve the algorithm's ability to identify sparse relations, thereby enhancing its ability to distinguish nested multiple relations in complex text data.

[0085] See Figure 3 , Figure 3 This is a schematic diagram of a text processing procedure provided in an embodiment of this application. Assume the text segment to be processed is: Horner's syndrome, characterized by ptosis, enophthalmos, and ipsilateral anhidrosis. The following is in conjunction with... Figure 3 The text processing process is introduced.

[0086] In practical applications, to avoid the algorithm losing the relative position information of words in multi-layer networks, for the text segment to be processed, the word feature matrix M of this text segment obtained by the BERT model can be used. T∈R s×d Adding a position encoding matrix to the matrix yields the matrix. (i.e., fusion features), where s is the length of the sequence feature (i.e. the number of elements contained in the sequence feature), d is the feature dimension (the vector representation dimension of a single word in the word feature matrix), and the calculation formula for the elements in the position coding matrix is ​​shown in Equation (1).

[0087]

[0088] Where pos represents the position of a word in the text segment to be processed, and the dimension of the position encoding matrix is ​​the same as the dimension of the word feature matrix, PE (pos,2i) and PE (pos,2i+1) These represent the elements in the pos-th row and 2i-th column of the position coding matrix, and the elements in the pos-th row and 2i+1-th column, respectively.

[0089] 1. LIR module.

[0090] The design of the local information reuse module aims to achieve micro-modeling of local information, with the goal of alleviating the problem of inaccurate modeling caused by the semantic complexity of complex texts.

[0091] This module is based on the encoder architecture of the Transformer model. By optimizing the attention mechanism, it increases the weight of entity information in the algorithm's learning process, thereby capturing key local information. This module avoids the problem of insufficient feature information acquisition caused by excessively long texts, while also ensuring that the algorithm focuses more on the feature information of complex text segments, reducing interference from useless fragment information, and strengthening the algorithm's ability to learn key knowledge.

[0092] To fully capture deep features, this module adopts a multi-layer, multi-head attention architecture. Specifically, each attention head first performs a mapping transformation, by transforming the matrix... with weight matrix W T ∈R d×d Multiplying and transforming three times yields a query matrix Q of the same dimension. l Key matrix K l Value matrix V l (l has no special meaning; it is only used as a subscript to distinguish matrices in the LIR module). Next, for the query matrix Q... l Bond matrix K l Perform a dot product operation to obtain the relative attention score matrix for each word, and then, with the help of an activation function (i.e., ... Figure 3 The micro attention weights are obtained using the SoftMax function in the function.

[0093] Unlike traditional scaled dot product attention, this module, in order to preserve local information to the greatest extent, does not scale the micro-attention weights but directly multiplies them by the value matrix V. l The final result is obtained, and the calculation process is shown in equation (2):

[0094]

[0095] The multi-head attention mechanism consists of multiple single-head attention modules, containing multiple sets of Q-values. l K l V l The matrix is ​​mapped to a low-dimensional space. After multiple calculations similar to those of the single-head attention module, the outputs of multiple parallel attention heads are concatenated to obtain the final result. The calculation process is shown in equations (3) and (4).

[0096]

[0097]

[0098] in, This represents the calculation result of the i-th attention head, where i is used to distinguish attention heads, and n represents the total number of attention heads.

[0099] Traditional self-attention mechanisms suffer from computational bottlenecks. To improve the computational efficiency of the algorithm, this section employs a low-rank matrix approximation method, decomposing the original scaled dot product attention into multiple smaller attention sub-matrices through linear projection. By introducing low-rank matrix approximation, the computational matrix size of the self-attention mechanism is reduced, i.e., equation (3) can be transformed into:

[0100]

[0101] Among them, E i ,F i ∈R s×k The key matrix K l Value matrix V l The projection matrix maps the original s×d dimensional matrix to a k×d dimensional matrix, and the constraint k < s achieves the dimensionality reduction of the matrix.

[0102] This module can use a 4-layer multi-head attention module, with each layer consisting of 4 parallel single-head attention modules. The 4-layer multi-head attention module ultimately calculates the local information enhancement feature matrix M. L (i.e., attention allocation features). Based on this, this paper also proposes a micro-attention enhancement mechanism that preserves the attention score matrix X of the last layer of a multi-layered multi-head attention system. Att For the attention score matrix X AttAfter dimensionality reduction, the micro-attention weights W are calculated using activation functions such as the SoftMax function. Att This weighting further strengthens the network's modeling of local information. The computation of the micro-attention module is represented as follows:

[0103]

[0104] The micro-attention enhancement mechanism enables the algorithm to focus on specific word segments. By extracting and integrating local information, it improves the algorithm's accuracy in recognizing complex word segments and its learning efficiency.

[0105] Finally, this module employs residual connection and normalization to obtain the local feature enhancement matrix M. L (That is, the optimized fusion feature), which strengthens the network's modeling of the interaction between the whole and the local, and improves the algorithm's generalization ability. The formula is as follows:

[0106]

[0107] 2. GIE module.

[0108] Conventional fine-tuning methods for pre-trained models only use word features and ignore sequence features, making it difficult to distinguish nested multiple relationships in complex text data. For example, the text "The Beijing Municipal Government is located in Tongzhou District" contains two entity relationship pairs: (Beijing Municipal Government, located in, Tongzhou District) and (Beijing, contains, Tongzhou District). Entity relationship pair extraction schemes in related technologies typically ignore the latter. It should be noted that the text segment to be processed in this embodiment usually contains multiple sentences; this example only illustrates multiple nested relationships using a single sentence. In actual applications, the text segment to be processed will contain more words, but it will generally be kept within a certain length, such as 300 words.

[0109] In this embodiment of the application, in order to make full use of the global information of long text sequences, the global feature sequence output by the BERT model is first preprocessed. SG∈R d (i.e., sequence features), and through attention sequences SA∈R s A global mask is implemented to hide partial information in the text segment to be processed. Simultaneously, dimensionality expansion and alignment operations are performed on the attention sequence and global feature sequence to obtain the entity mask matrix X. A ∈R 1×d and global feature sequence matrix X G ∈R 1×s Then, a mask matrix is ​​used to mask some sequence feature information to enhance the algorithm's ability to extract global information:

[0110]

[0111] The Einsum function is used to quickly implement the dot product of two matrices. is the global feature matrix after masking, s is the length of the input sequence, and d is the embedding dimension of the last layer output of the BERT model.

[0112] The global feature matrix after masking cannot grasp the overall structure and context information. Therefore, a position-based feedforward neural network (FNN) is used to add nonlinear transformations to further learn the feature matrix of the mask sequence, enabling the network module to perceive global information and pay attention to word order details. The FNN calculation process is shown in equation (9):

[0113]

[0114] Where W1 and W2 are pre-learned weight matrices, b1 and b2 are pre-learned biases, and f(·) represents the ReLU activation function. The feature matrix, which extracts global information, is obtained by projecting the first matrix into a low-dimensional space and then raising the dimension again using the second matrix.

[0115] Taking the Beijing Municipal Government located in Tongzhou District as an example, by masking, the text in this sentence can be arbitrarily covered. The covered text can increase the probability that the feedforward neural network can split the Beijing Municipal Government into different entities, enhance the understanding of global information, and increase the probability of extracting the relationship between the entities (Beijing, including, Tongzhou District) in the future.

[0116] To avoid information loss during multi-layer network transmission and improve the robustness of the algorithm, this module also employs residual connections to introduce word-level feature encoding with added positional information, and then standardizes it to obtain a global information-enhanced feature matrix M. G (That is, the optimized sequence features). This processing method enables interactive learning of sequence and word information elements, enhancing the robustness of the algorithm. M G The function is defined as:

[0117]

[0118] 3. AWBF module.

[0119] To address the long-tailed distribution of categories in complex text data, an adaptive weighted balanced feature fusion strategy is designed to fully realize the complementary fusion of global and local information. This module integrates the two parts of information by stacking the global feature matrix (i.e., the optimized sequence features) and the local feature matrix (i.e., the optimized fused features) to mitigate the negative perturbations caused by directly superimposing the two matrices. Next, the integration matrix is ​​mapped to a low-dimensional space to enhance the network's knowledge enrichment. Finally, an activation function is used to calculate the dynamic weighting matrix W. α The dynamic fusion process is as follows:

[0120]

[0121]

[0122] M fuse =W α ·M L +(1-W α )·M G (13)

[0123] Among them, M fuse The target feature.

[0124] This fusion method can adaptively balance the weights of global and local information, ensuring effective information fusion and enhancing the algorithm's ability to identify sparse relationships.

[0125] 4. Decoding entities and relationships.

[0126] The decoder aims to identify all possible triples (subject, relation, object) in the text through the semantic information output by the encoder. To simplify the algorithm structure and achieve single-step extraction, a global pointer decoding method can be adopted.

[0127] The decoding process comprises two core stages: entity decoding and relation decoding. Relationship decoding is further subdivided into header decoding and tail decoding. In the decoding matrix, rows represent the starting position, and columns represent the ending position. For example... Figure 3 As shown, the entity matrix illustrates the principle of entity pointer decoding. A "1" in the matrix represents a segment corresponding to a row or column index that is an entity; there is no distinction between subject and object, but the upper triangular matrix ensures that the starting position is less than or equal to the ending position. The head relation matrix and tail relation matrix represent the head and tail positions of the matched entities, respectively, with rows and columns representing the start and end positions of the entity head and tail. Entity decoding and relation decoding are performed under schema constraints for inference and prediction, and the results of these two decoding parts are mutually filtered. This decoding method implicitly utilizes entity category information to assist relation extraction, thus directly predicting based on the schema, and jointly achieving named entity recognition and relation extraction using a global normalization approach.

[0128] In practical applications, the GPLinker algorithm can be used to identify entities and the relationships between entities from target features.

[0129] In this embodiment, the encoder section is designed with two modules: global information enhancement and local information reuse. These modules extract macroscopic and microscopic features respectively, and achieve efficient complementarity through adaptive weighted balancing fusion. Specifically, the local information reuse module enables the algorithm to focus on local information, effectively addressing complex text semantic problems; the global information enhancement module improves the ability to identify multiple nested relationships in complex text data; and the adaptive weighted balancing strategy facilitates the fusion of global and local information, effectively addressing long-tail distribution problems.

[0130] Based on the same technical concept, this application also provides a text processing device. The principle of the text processing device in solving the problem is similar to that of the above-mentioned text processing method. Therefore, the implementation of the text processing device can refer to the implementation of the text processing method, and the repeated parts will not be described again.

[0131] Figure 4 A schematic diagram of a text processing device provided in this application embodiment includes:

[0132] The acquisition module 401 is used to acquire the sequence features and word features of the text segment to be processed after it is input into the BERT model, and to acquire the position encoding of the text segment to be processed;

[0133] The local information processing module 402 is used to fuse the word features and the positional encoding to obtain fused features; to process the fused features through a multi-layer multi-head attention mechanism to obtain attention allocation features; and to process the attention allocation features through an activation function to obtain micro-attention weights.

[0134] The fusion feature is optimized based on the attention allocation features and the micro-attention weights;

[0135] The determination module 403 is used to determine the entities and relationships between entities in the text segment to be processed based on the sequence features and the optimized fusion features.

[0136] In some embodiments, the local information processing module 402 is specifically used for:

[0137] Calculate the Hadamard product of the attention allocation features and the micro attention weights;

[0138] The Hadamard product and the fused feature are input into the first residual network;

[0139] The output of the first residual network is standardized to obtain the optimized fusion features.

[0140] In some embodiments, it also includes:

[0141] The global information processing module 404 is used to expand the sequence features to obtain expanded sequence features; to mask the expanded sequence features; to input the masked expanded sequence features into a feedforward neural network to update the expanded sequence features; and to optimize the sequence features based on the updated expanded sequence features and the fused features.

[0142] The determining module 403 is further configured to determine the entity and the relationship between entities based on the optimized sequence features and the optimized fusion features.

[0143] In some embodiments, the determining module 403 is specifically used for:

[0144] The optimized sequence features and the optimized fusion features are stacked;

[0145] The stacking results are mapped to reduce their dimensionality.

[0146] The stacked result after mapping is processed by an adaptive weighted average algorithm to obtain the first weight of the optimized sequence feature and the second weight of the optimized fusion feature;

[0147] Based on the first weight and the second weight, the optimized sequence features and the optimized fusion features are fused to obtain the target features;

[0148] The target features are analyzed using the GPLinker algorithm to obtain the entities and the relationships between them.

[0149] In some embodiments, the global information processing module 404 is specifically used for:

[0150] The updated sequence augmentation features and the fusion features are input into the second residual network;

[0151] The output of the second residual network is standardized to obtain the optimized sequence features.

[0152] In some embodiments, the multi-layer multi-head attention mechanism includes a four-layer multi-head attention mechanism, with each layer including four single-head attention mechanisms.

[0153] The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. Coupling between modules can be achieved through interfaces, typically electrical communication interfaces, but mechanical interfaces or other types of interfaces are also possible. Therefore, modules described as separate components may or may not be physically separate; they can be located in one place or distributed across different locations on the same or different devices. The integrated modules described above can be implemented in hardware or as software functional modules.

[0154] Having introduced the text processing methods and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0155] The following reference Figure 5 To describe an electronic device 130 implemented according to this embodiment of the present application. Figure 5 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 5 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0157] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0158] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0159] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0160] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0161] In an exemplary embodiment, a storage medium is also provided, which enables the electronic device to perform the text processing method described above when a computer program in the storage medium is executed by a processor of the electronic device. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0162] In an exemplary embodiment, the electronic device of this application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it can cause the at least one processor to perform the steps of any text processing method provided in the embodiments of this application.

[0163] In an exemplary embodiment, a computer program product is also provided, which, when executed by an electronic device, enables the electronic device to implement any of the exemplary methods provided in this application.

[0164] Furthermore, computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] The program product for text processing in the embodiments of this application may be a CD-ROM and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0166] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0167] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0168] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0169] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0170] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0171] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0176] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, then this application also includes such modifications and variations.

Claims

1. A text processing method, characterized in that, include: Obtain the sequence features and word features of the text segment to be processed after inputting it into the BERT model, and obtain the positional encoding of the text segment to be processed; The word features and the positional encoding are fused to obtain fused features; The fused features are processed by a multi-layer, multi-head attention mechanism to obtain attention allocation features; The attention allocation features are processed by an activation function to obtain micro-attention weights; The fusion feature is optimized based on the attention allocation features and the micro-attention weights; Based on the sequence features and the optimized fusion features, the entities in the text segment to be processed and the relationships between entities are determined; Also includes: The sequence features are augmented to obtain augmented sequence features; The sequence augmentation features are masked. The sequence augmentation features after masking are input into a feedforward neural network to update the sequence augmentation features; The sequence features are optimized based on the updated sequence augmentation features and the fusion features. Based on the sequence features and the optimized fusion features, the entities and relationships between entities in the text segment to be processed are determined, including: Based on the optimized sequence features and the optimized fusion features, the entities and the relationships between them are determined.

2. The method as described in claim 1, characterized in that, Based on the attention allocation features and the micro-attention weights, the fusion features are optimized, including: Calculate the Hadamard product of the attention allocation features and the micro attention weights; The Hadamard product and the fused feature are input into the first residual network; The output of the first residual network is standardized to obtain the optimized fusion features.

3. The method as described in claim 1, characterized in that, Based on the optimized sequence features and the optimized fusion features, the entities and the relationships between them are determined, including: The optimized sequence features and the optimized fusion features are stacked; The stacking results are mapped to reduce their dimensionality. The stacked result after mapping is processed by an adaptive weighted average algorithm to obtain the first weight of the optimized sequence feature and the second weight of the optimized fusion feature; Based on the first weight and the second weight, the optimized sequence features and the optimized fusion features are fused to obtain the target features; The target features are analyzed using the GPLinker algorithm to obtain the entities and the relationships between them.

4. The method as described in claim 1, characterized in that, Based on the updated sequence augmentation features and the fusion features, the sequence features are optimized, including: The updated sequence augmentation features and the fusion features are input into the second residual network; The output of the second residual network is standardized to obtain the optimized sequence features.

5. The method as described in claim 1, characterized in that, The multi-layer multi-head attention mechanism includes four layers of multi-head attention mechanisms, and each layer of multi-head attention mechanisms includes four single-head attention mechanisms.

6. A text processing device, characterized in that, include: The acquisition module is used to acquire the sequence features and word features of the text segment to be processed after it is input into the BERT model, and to acquire the positional encoding of the text segment to be processed; The local information processing module is used to fuse the word features and the positional encoding to obtain fused features; and to perform calculations on the fused features through a multi-layer multi-head attention mechanism to obtain attention allocation features. The attention allocation features are processed by an activation function to obtain micro-attention weights; The fusion feature is optimized based on the attention allocation features and the micro-attention weights; The determination module is used to determine the entities and relationships between entities in the text segment to be processed based on the sequence features and the optimized fusion features; Also includes: A global information processing module is used to augment the sequence features to obtain augmented sequence features; to mask the augmented sequence features; to input the masked augmented sequence features into a feedforward neural network to update the augmented sequence features; and to optimize the sequence features based on the updated augmented sequence features and the fused features. The determining module is further configured to determine the entity and the relationship between entities based on the optimized sequence features and the optimized fusion features.

7. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor, wherein: The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-5.

8. A storage medium, characterized in that, When the computer program in the storage medium is executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images