Training method and device of natural language processing model, storage medium

Abstract

The present disclosure relates to a training method of a natural language processing model, comprising: obtaining a sample text of natural language; determining a triple in the sample text, the triple comprising two entities in the sample text and a relationship between the two entities; processing the sample text based on the triple to obtain a knowledge fusion vector; and inputting the knowledge fusion vector into a natural language processing model to obtain a target model. According to the present disclosure, the sample text can be processed based on the triple in the sample text to obtain a knowledge fusion vector. Compared with the sample text, the knowledge fusion vector contains other knowledge information in addition to the information of the sample text itself, such as entity information in the text and relationship information between entities, so that the computer can more accurately obtain the true meaning of the sample text. Therefore, the target model obtained by training the natural language processing model based on the knowledge fusion vector has a better effect on natural language processing.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to training methods for natural language processing models, training devices for natural language processing models, and computer-readable storage media. Background Technology

[0002] Current Natural Language Processing (NLP) techniques primarily use natural language text itself as samples when training NLP models. However, the features contained in natural language text itself are relatively limited, resulting in NLP models trained based on this having less than ideal processing performance. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a training method for a natural language processing model, a training apparatus for a natural language processing model, and a computer-readable storage medium to address the problems in related technologies.

[0004] According to a first aspect of the present disclosure, a method for training a natural language processing model is proposed, comprising:

[0005] Obtain sample text in natural language;

[0006] Determine the triples in the sample text, wherein the triples include two entities in the sample text and the relationship between the two entities;

[0007] The sample text is processed based on the triples to obtain a knowledge fusion vector;

[0008] The knowledge fusion vector is input into a natural language processing model for training to obtain the target model.

[0009] Optionally, the step of inputting the knowledge fusion vector into a natural language processing model for training to obtain the target model includes:

[0010] Construct at least one pre-training task, and train the natural language processing model based on the at least one pre-training task to obtain the target model.

[0011] Optionally, the at least one pre-training task includes:

[0012] The first pre-training task includes a masking task, which includes masking the sample text based on the triples to obtain the knowledge fusion vector.

[0013] Optionally, the step of masking the sample text based on the triples to obtain the knowledge fusion vector includes:

[0014] The tokens corresponding to the triples in the sample text are masked to obtain the knowledge fusion vector.

[0015] Optionally, masking the tokens corresponding to the triples in the sample text to obtain the knowledge fusion vector includes:

[0016] The tokens corresponding to entities and relations in the triples in the sample text are masked to obtain the knowledge fusion vector.

[0017] Optionally, masking the sample text based on the triples to obtain the knowledge fusion vector includes:

[0018] The sample text is divided into multiple sub-texts based on the lexical units corresponding to the triples, and the multiple sub-texts are masked to obtain the knowledge fusion vector.

[0019] Optionally, the step of dividing the sample text into multiple sub-texts based on the lexical units corresponding to the triples includes:

[0020] The sample text is divided into multiple subtexts by dividing the word at the beginning or end of the corresponding triplet.

[0021] Optionally, masking the plurality of sub-texts to obtain the knowledge fusion vector includes:

[0022] In the subtext, the tokens corresponding to the triples are masked to obtain the knowledge fusion vector.

[0023] Optionally, when the sample text contains n triples, the step of dividing the sample text into multiple subtexts based on the lexical units corresponding to the triples includes:

[0024] The sample text is divided into multiple subtexts according to the word corresponding to the m-th triplet among the n triplets, where n is an integer greater than 1, m is a positive integer less than or equal to n, and m is a specified value or a random value.

[0025] Optionally, when the sample text contains n triples, the step of dividing the sample text into multiple sub-texts based on the tokens corresponding to the triples, and masking the tokens corresponding to the triples in the sample text to obtain the knowledge fusion vector, includes:

[0026] Starting from i=1 and continuing until i=n, ​​repeat the following steps;

[0027] The sample text is divided according to the word element corresponding to the i-th triplet among the n triplets to obtain the i-th sub-text group composed of multiple sub-texts corresponding to the i-th triplet;

[0028] The tokens corresponding to the triples in the i-th subtext group are masked to obtain the knowledge fusion vector; where n is an integer greater than 1 and i is a positive integer less than or equal to n.

[0029] Optionally, training is performed after masking the tokens corresponding to the triples in the i-th sub-text group as one round of training, and k rounds of training are performed, where k is an integer greater than or equal to n.

[0030] Optionally, determining the triples in the sample text includes:

[0031] Determine the triples corresponding to the sample text in the knowledge graph, or obtain the triples in the sample text based on the triple extraction model.

[0032] Optionally, determining the triples corresponding to the sample text in the knowledge graph includes:

[0033] Determine the domain to which the sample text belongs;

[0034] Determine the knowledge graph for the domain;

[0035] Determine the triples corresponding to the sample text in the knowledge graph of the domain.

[0036] Optionally, the knowledge graph includes at least one of the following:

[0037] Pre-built knowledge graph;

[0038] Pre-stored knowledge graph.

[0039] Optionally, when the knowledge graph includes a pre-constructed knowledge graph, the method further includes, before determining the triples corresponding to the sample text in the knowledge graph:

[0040] A triplet extraction model is determined, wherein the triplet extraction model is used to extract triples from the sample text;

[0041] The knowledge graph is constructed based on the extracted triples.

[0042] Optionally, the triple extraction model includes:

[0043] A label determination model is used to determine the label of the token in the sample text; wherein the label is used to determine the entity in the sample text;

[0044] A relationship determination model is used to determine the relationship between at least two entities determined by the label determination model and / or the position of the relationship in the sample text.

[0045] Optionally, the label determination model includes:

[0046] A first coding layer, the input of which includes the terminology;

[0047] A feature extraction layer, wherein the input of the feature extraction layer includes the output of the first encoding layer;

[0048] A first activation layer, the input of which includes the output of the feature extraction layer;

[0049] The first normalization layer, the input of which includes the output of the first activation layer;

[0050] The first output layer, whose input includes the output of the first normalization layer, is used to output the label;

[0051] The input corresponding to the (i+1)th word in the sample text in the first activation layer also includes the output corresponding to the ith word in the sample text in the first output layer, where i is an integer greater than or equal to 1.

[0052] Optionally, the relationship determination model includes:

[0053] A second coding layer, the input of which includes the entity pairs;

[0054] A fully connected layer, the input of which includes the output of the second coding layer and the output of the feature extraction layer;

[0055] A second activation layer, the input of which includes the output of the fully connected layer;

[0056] The second normalization layer, whose input includes the output of the second activation layer, is used to output the relation and / or the position of the relation in the sample text.

[0057] Optionally, determining the triples corresponding to the sample text in the knowledge graph includes:

[0058] Identify multiple sub-knowledge graphs;

[0059] Entities with the same semantics in different sub-knowledge graphs are merged to obtain a merged knowledge graph;

[0060] Determine the triples corresponding to the sample text in the merged knowledge graph.

[0061] Optionally, merging entities with the same semantics from different sub-knowledge graphs includes:

[0062] The distance between the embedding vectors of any two entities in different sub-knowledge graphs is calculated based on the structural feature vector hs and attribute feature vector ha corresponding to the two entities. If the distance is less than the distance threshold, the two entities are merged.

[0063] Optionally, the structural feature vector hs and attribute feature vector ha of the entity are calculated by the structural feature vector hs-1 and attribute feature vector ha-1 of the previous entity and the connectivity matrix A through a graph neural network (GCN) model.

[0064] Optionally, the at least one pre-training task further includes:

[0065] The second pre-training task is used to split the sample document into fragments at at least one level of granularity, randomly arrange and combine the fragments after each level of splitting, and train based on the rearranged samples to output the relationship type between sentences in the rearranged samples.

[0066] Optionally, the at least one pre-training task further includes:

[0067] The third pre-training task is used to perform semantic recognition training by masking the lexical units corresponding to the triples in the sample text.

[0068] Optionally, training the natural language processing model according to the at least one pre-training task to obtain the target model includes:

[0069] Perform at least one round of the following steps:

[0070] Pre-training tasks are selected and superimposed according to a preset order. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks.

[0071] Optionally, training the natural language processing model according to the at least one pre-training task to obtain the target model includes:

[0072] Pre-training tasks are randomly selected and superimposed. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks. The learning rate of the natural language processing model is positively correlated with the number of samples input to the natural language processing model for each training iteration.

[0073] Optionally, the relationship between the learning rate of the natural language processing model during training and the number of samples input to the natural language processing model for each training iteration is as follows:

[0074] Y = 5.57e -6 ·log2X-4.75e-6 ;

[0075] Where X is the batchsize of the samples input to the natural language processing model for each training iteration, and Y is the learning rate.

[0076] Optionally, the method further includes:

[0077] Determine the target sample set based on the target training task;

[0078] The overall model is trained based on the target sample set, wherein the overall model includes the target model and the fine-tuned model.

[0079] Optionally, the fine-tuning model includes at least one of the following:

[0080] Convolutional Neural Network (CNN) model;

[0081] Recurrent Neural Network (RNN) model;

[0082] Gate-controlled loop unit (GRU) model;

[0083] Fully connected layer (FC);

[0084] Long Short-Term Memory (LSTM) model.

[0085] Optionally, the process of processing the sample text based on the triples to obtain the knowledge fusion vector includes:

[0086] The sample text and the triples in the sample text are fused to obtain a knowledge fusion vector.

[0087] Optionally, fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes:

[0088] The triples in the sample text are inserted into the sample text to obtain the knowledge fusion vector.

[0089] Optionally, fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes:

[0090] Determine the text embedding vector corresponding to the sample text;

[0091] Determine the knowledge embedding vector corresponding to the triple;

[0092] The text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

[0093] Optionally, the triplet includes at least one of the following:

[0094] Head entity, relation, tail entity.

[0095] Optionally, determining the knowledge embedding vector corresponding to the triple includes:

[0096] The embedding vectors of the head entity, the relation, and the tail entity are fused to obtain the knowledge embedding vector:

[0097] e k =concat(h e +r e +t e );

[0098] Among them, e k Represents knowledge embedding vector, h e The embedding vector r of the head entity represents the embedding vector of the head entity. e The embedding vector representing the relationship, t e The embedding vector of the tail entity is represented by and concat() is represented by the concatenation function.

[0099] Optionally, the text embedding vector may include at least one of the following:

[0100] Sentence embedding vector, position embedding vector corresponding to the word element of the triple, and task type embedding vector.

[0101] Optionally, the knowledge embedding vector includes at least one of the following:

[0102] The embedding vector of the entity belonging to the triple;

[0103] The embedding vector of the relation belonging to the triple.

[0104] Optionally, fusing the text embedding vector and the knowledge embedding vector includes:

[0105] The text feature vector is obtained by extracting features from the text embedding vector through the feature extraction layer;

[0106] The knowledge feature vector is obtained by extracting features from the knowledge embedding vector through the feature extraction layer;

[0107] The text feature vector and the knowledge feature vector are fused to obtain the knowledge fusion vector.

[0108] Optionally, fusing the text feature vector and the knowledge feature vector includes:

[0109] The text feature vector and the knowledge feature vector are fused through multiple layers of neural network.

[0110] Optionally, fusing the text embedding vector and the knowledge embedding vector includes:

[0111] The text embedding vector and the knowledge embedding vector are fused according to the knowledge fusion model:

[0112]

[0113] Among them, f j Represents the knowledge fusion vector, e t Represents text embedding vector, e k Represents knowledge embedding vector, W t The weights W of the text embedding vectors are represented by these weights. e denoted by , b represents the weight of the knowledge embedding vector, b represents the bias, and i represents the i-th word in the corresponding sample text.

[0114] Optionally, the natural language processing model includes at least one feature extraction layer, which is used for:

[0115] A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix.

[0116] A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the sample text and the sample text.

[0117] A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the sample text.

[0118] Optionally, the at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

[0119] Optionally, the at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

[0120] Optionally, the third feature extraction layer is located after the first feature extraction layer.

[0121] Optionally, when the training task is a generation task, the second feature extraction layer is frozen; when the training task is a semantic understanding task, the third feature extraction layer is frozen.

[0122] Optionally, a splitting layer is further included before the second feature extraction layer and the third feature extraction layer, for splitting the output of the first feature extraction layer into knowledge information and text information;

[0123] The first dimension transformation layer is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the second feature extraction layer, or to transform the dimension of the knowledge information to obtain a dimension-reduced matrix as the input of the second feature extraction layer.

[0124] The first feature extraction layer and the third feature extraction layer further include a second dimension transformation layer, which is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the third feature extraction layer, or to transform the dimension of the text information to obtain a dimension-reduced matrix as the input of the third feature extraction layer.

[0125] According to a second aspect of the embodiments of this disclosure, a text processing method is provided, comprising:

[0126] Get the target text;

[0127] Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities;

[0128] The target text is processed based on the triples to obtain a knowledge fusion vector;

[0129] The knowledge fusion vector is input into the target model to obtain the processing result of the target text.

[0130] Optionally, the process of processing the target text based on the triples to obtain the knowledge fusion vector includes:

[0131] The target text and the triples in the target text are fused to obtain a knowledge fusion vector.

[0132] Optionally, inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes:

[0133] The triples in the target text are inserted into the target text to obtain the knowledge fusion vector.

[0134] Optionally, inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes:

[0135] Determine the text embedding vector corresponding to the target text;

[0136] Determine the knowledge embedding vector corresponding to the triple;

[0137] The text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

[0138] Optionally, the target model includes at least one feature extraction layer, the feature extraction layer being used for:

[0139] A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix.

[0140] A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the target text and the target text.

[0141] A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the target text.

[0142] Optionally, the at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

[0143] Optionally, the at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

[0144] According to a third aspect of the present disclosure, a text processing apparatus is provided, including a processor configured to:

[0145] Get the target text;

[0146] Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities;

[0147] The target text is processed based on the triples to obtain a knowledge fusion vector;

[0148] The knowledge fusion vector is input into the target model to obtain the processing result of the target text.

[0149] Optionally, the device further includes:

[0150] The interaction module is used to determine whether to obtain the triples in the target text based on the knowledge graph according to the user's operation;

[0151] Specifically, when determining the triples in the target text based on the knowledge graph, the triples corresponding to the target text in the knowledge graph are determined.

[0152] When it is determined that the triples in the target text are not obtained based on a knowledge graph, the triples in the target text are determined.

[0153] Optionally, the interaction module is further configured to display the triples corresponding to the target text in the knowledge graph; and

[0154] The displayed triples are adjusted based on the user's actions, and the adjusted triples are used as the triples corresponding to the target text in the knowledge graph.

[0155] Optionally, the interaction module is further configured to receive a triple input by the user as the triple corresponding to the target text in the knowledge graph when determining that the target model is obtained by training based on the knowledge graph.

[0156] Optionally, the interaction module is further configured to select the target model based on user operation, wherein the target model includes at least one of the following:

[0157] CNN, RNN, GRU, LSTM, Transformer, Transformer-XL.

[0158] Optionally, when the target model includes Transformer and / or Transformer-XL, the interaction module is also used to determine the fine-tuning model based on user operations.

[0159] Optionally, the fine-tuning model includes at least one of the following:

[0160] CNN, RNN, GRU, LSTM.

[0161] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the training method of the natural language processing model described above and / or the steps in the text processing method described above.

[0162] According to embodiments of this disclosure, a knowledge fusion vector can be obtained by processing the sample text based on the triples in the sample text. Compared with the sample text, the knowledge fusion vector contains other knowledge information besides the information in the sample text itself, such as entity information in the text and the relationship information between entities, so as to enable the computer to obtain the true meaning of the sample text more accurately. Therefore, the target model obtained by training the natural language processing model based on the knowledge fusion vector has a better effect on natural language processing. Attached Figure Description

[0163] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0164] Figure 1 This is a schematic flowchart illustrating a training method for a natural language processing model according to an embodiment of the present disclosure.

[0165] Figure 2 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0166] Figure 3 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0167] Figure 4 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0168] Figure 5 This is a schematic diagram illustrating a triplet extraction model according to an embodiment of the present disclosure.

[0169] Figure 6 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0170] Figure 7 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0171] Figure 8 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0172] Figure 9 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0173] Figure 10 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0174] Figure 11 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0175] Figure 12This is a schematic diagram illustrating the relationship between a fusion process and a natural language processing model according to an embodiment of the present disclosure.

[0176] Figure 13 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0177] Figure 14 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0178] Figure 15 This is a schematic diagram illustrating the relationship between another fusion process and a natural language processing model according to embodiments of this disclosure.

[0179] Figure 16 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure.

[0180] Figure 17A This is a schematic diagram illustrating the relationship between another fusion process and a natural language processing model according to embodiments of the present disclosure.

[0181] Figure 17B This is a schematic diagram illustrating a feature extraction calculation process according to an embodiment of the present disclosure.

[0182] Figure 18 This is a schematic diagram illustrating the relationship between feature extraction layers according to embodiments of the present disclosure.

[0183] Figure 19 This is a schematic diagram illustrating another relationship between feature extraction layers according to an embodiment of the present disclosure.

[0184] Figure 20 This is a schematic diagram illustrating a dimension transformation according to an embodiment of the present disclosure.

[0185] Figure 21 This is a schematic diagram illustrating another dimension transformation according to an embodiment of the present disclosure.

[0186] Figure 22A This is a schematic diagram of the structure of a first feature extraction layer according to an embodiment of the present disclosure.

[0187] Figure 22B This is a schematic diagram illustrating the calculation process of a first feature extraction layer according to an embodiment of the present disclosure.

[0188] Figure 23 This is a schematic diagram of the structure of a first sub-model according to an embodiment of the present disclosure.

[0189] Figure 24This is a schematic diagram of the structure of a second sub-model according to an embodiment of the present disclosure.

[0190] Figure 25 This is an interactive schematic diagram illustrating an embodiment of the present disclosure.

[0191] Figure 26 This is a schematic block diagram illustrating an apparatus for training a natural language processing model according to embodiments of the present disclosure. Detailed Implementation

[0192] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0193] Figure 1 This is a schematic flowchart illustrating a training method for a natural language processing model according to an embodiment of the present disclosure. The training method for the natural language processing model shown in this embodiment can be applied to terminals or servers. The terminals include, but are not limited to, electronic devices such as mobile phones, tablets, wearable devices, and personal computers. The servers include, but are not limited to, local servers and cloud servers.

[0194] like Figure 1 As shown, the training method for the natural language processing model may include the following steps:

[0195] In step S101, sample text of natural language is obtained;

[0196] In step S102, triples in the sample text are determined, wherein the triples include two entities in the text and the relationship between the two entities;

[0197] In step S103, the sample text is processed based on the triples to obtain a knowledge fusion vector;

[0198] In step S104, the knowledge fusion vector is input into the natural language processing model for training to obtain the target model.

[0199] It should be noted that in this embodiment, "entity" refers to a distinguishable and independently existing thing. Examples include a person, a city, a plant, or a commodity. Everything in the world is composed of concrete objects, which are entities, such as "United States" or "United Kingdom." Entities can exist in a knowledge graph, for example, as the most basic element of a knowledge graph, or they can exist independently of a knowledge graph. Different entities can have different relationships. In this embodiment, "relationship" refers to the association between different entities; for example, the association between London and the United Kingdom is that London is the capital city. "Relationship" can also refer to attributes, such as a person's age, height, or weight.

[0200] The "knowledge graph" mentioned in subsequent embodiments aims to describe various entities or concepts existing in the real world and their relationships. It constitutes a huge semantic network graph, where nodes represent entities or concepts, and edges are composed of attributes or relationships. Currently, "knowledge graph" is used to refer generally to various large-scale knowledge bases.

[0201] In one embodiment, the triple may include two entities and a relationship between them. The two entities may include a head entity and a tail entity. For example, the sample text is "Levi is a civil servant of country A", where the head entity includes "Levi", the tail entity includes "civil servant of country A", and the relationship includes "position". It should be noted that the relationship in the triple described in this disclosure may include the relationship itself, and may further include attributes of the entities. The embodiments of this disclosure mainly illustrate the case where the relationship only includes the relationship itself.

[0202] As can be seen, the entities in the triple exist in the sample text, while the relations in the triple may not exist in the sample text. Of course, this is just an example; in some cases, the relations in the triple may also exist in the sample text.

[0203] In one embodiment, a sample text may contain not only one triplet, but also multiple triplets. Regarding the cases where a sample text contains one triplet or multiple triplets, the sample text includes at least one of the following: a sentence, multiple sentences, a paragraph, or an article. The following example primarily focuses on the case where the sample text contains only one sentence.

[0204] For example, the sample text states: "After the Tumu Crisis, Yu Qian, an outstanding national hero of the Ming Dynasty, was created."

[0205] It can be determined that in the sample text, the head entity SUBJ includes "Yu Qian", the tail entity OBJ includes "Ming Dynasty", and the relation includes "Dynasty".

[0206] In this sample text, there is only one triple [[“Yu Qian”, “Dynasty”, “Ming Dynasty”]].

[0207] For example, the sample text reads: "Song Yingxing was born in Fengxin, Jiangxi Province in 1587 AD. He lived through the final period of the Ming Dynasty's corruption and demise."

[0208] It can be determined that in the sample text, the head entity SUBJ includes "Song Yingxing", the tail entity OBJ includes "Fengxin, Jiangxi" and "1587", and the relations include "birth location" and "birth time".

[0209] In this sample text, there are two triples: one is [["Song Yingxing", "birth location", "Fengxin, Jiangxi"]], and the other is [["Song Yingxing", "birth time", "1587"]].

[0210] In addition, the sample text may include, but is not limited to, Chinese, and may also be a foreign language, such as English. Therefore, the triples in the sample text may also be in English.

[0211] According to embodiments of this disclosure, a knowledge fusion vector can be obtained by processing sample text based on triples in the sample text. Since the triples in the sample text contain both the original features of the entities within the triples and the features of the relations within the triples, the resulting knowledge fusion vector can also contain the features of the relations within the triples. Entity extraction can uncover the core meaning of the sample text, while relations can comprehensively reflect the relationships between entities in the sample text. Therefore, through feature extraction of entities and / or relations, the model can better obtain parameters that reflect the meaning of the sample text during subsequent model training. Thus, compared to the sample text itself, the knowledge fusion vector contains other knowledge information beyond the sample text itself, such as entity information and relationship information between entities, thereby enabling the computer to more accurately obtain the true meaning of the sample text. Therefore, the target model trained on the natural language processing model based on the knowledge fusion vector performs better in natural language processing.

[0212] In one embodiment, the natural language processing model includes, but is not limited to, any one of the following models: self-attention mechanism model, multilayer perceptron model, recurrent neural network model, convolutional neural network, deep convolutional neural network, and self-attention transformer.

[0213] Figure 2 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of this disclosure. Figure 2As shown, the step of inputting the knowledge fusion vector into the natural language processing model for training to obtain the target model includes:

[0214] In step S201, at least one pre-training task is constructed, and the natural language processing model is trained according to the at least one pre-training task to obtain the target model.

[0215] In one embodiment, the model used in practice can be called the overall model, which may include the target model and the fine-tuned model. The target model may be obtained in the pre-training task stage, and the fine-tuned model may be obtained in the subsequent fine-tuning stage.

[0216] In the pre-training phase, the embedding vectors obtained from the sample text can be input into the natural language processing model, and training can be performed according to the constructed pre-training task. Based on the difference between the output of the natural language processing model and the input and / or the correct result corresponding to the pre-training task in this training phase, the parameters of the natural language processing model can be adjusted so that the difference between the processing result of the natural language processing model and the corresponding correct result becomes smaller and smaller, and eventually falls within a certain threshold. Then, the training of the natural language processing model based on the pre-training task can be considered complete.

[0217] When multiple pre-training tasks exist, the natural language processing (NLP) model can be trained individually for each pre-training task, or it can be trained together using multiple pre-training tasks, depending on the needs. During the pre-training phase, a target model can be obtained after training the NLP model. This target model can be extracted and used to provide features for different downstream tasks, or it can be directly fine-tuned for other downstream tasks.

[0218] In another embodiment, the target model obtained from the pre-training task can also be used as the model for actual application.

[0219] The following embodiments first describe the pre-training task phase, and the subsequent embodiments describe the fine-tuning phase.

[0220] Training a natural language processing model can be done based on pre-training tasks. The specific pre-training tasks can be set as needed, and the number of pre-training tasks can also be set as needed, for example, there can be one or multiple pre-training tasks.

[0221] In one embodiment, the at least one pre-training task includes:

[0222] The first pre-training task includes a masking task, which involves masking the sample text based on the triples to obtain a knowledge fusion vector.

[0223] The masking task can be understood as masking one or more tokens in the sample text (replacing the token with [mask]) to obtain an initial text vector. In the pre-training task, prediction training is performed, that is, the model predicts the masked tokens based on the unmasked tokens (context) to finally obtain the trained text vector. The vector representation of the corresponding mask position in the trained text vector integrates the meaning of the token and the context, and can more accurately represent the meaning of the text.

[0224] In one embodiment, the first pre-training task can be a task of masking sample text based on triples. This can be understood as introducing knowledge information (triples) during the masking process of the sample text, thereby enabling more targeted masking and resulting in a more accurate trained knowledge fusion vector. Masking sample text based on triples includes at least two cases: 1) Masking all or part of the words in the sample text corresponding to triples. Triples in the sample text are usually key information, and predicting based on them yields better results. The proportion of masked words in the total words of the sample text can be set as needed, for example, 10% to 20%, where 15% can be set. If the number of triples is greater than this proportion, only some triples can be masked; if the number of triples is less than this proportion, other words in the sample text can be masked simultaneously. 2) Dividing the sample text into multiple sub-texts based on triples and masking each sub-text separately. The masked words have a certain degree of randomness. By segmenting the text using triples, each part of the text has the potential to be masked, resulting in more uniform masking and improved prediction performance. These two methods can be implemented individually or simultaneously; no specific restrictions are placed here.

[0225] In one embodiment, processing the sample text based on the triples to obtain a knowledge fusion vector may include masking words in the sample text to obtain the knowledge fusion vector, inputting the knowledge fusion vector into a natural language model for training to obtain a target model, or training the natural language model to predict the masked words. The trained target model can output the trained knowledge fusion vector. In this case, the obtained knowledge fusion vector may include the embedding vector of words, or further, the embedding vector of the entire sample text.

[0226] For example, the knowledge fusion vector can be obtained by masking the tokens corresponding to triples in the sample text. That is, after masking the tokens corresponding to entities and / or relations in triples in the sample text, the masked tokens are predicted. In this case, the knowledge fusion vector can include the embedding vector of the tokens corresponding to triples, and can also include the embedding vector of the tokens outside triples. Furthermore, the embedding vector of the entire sample text can also be obtained.

[0227] The target model trained based on the first pre-training task can at least output the embedding vectors of masked words, and can also output the embedding vectors of unmasked words. Furthermore, it can also obtain the embedding vector of the entire sample text.

[0228] In one embodiment, masking the sample text based on the triples to obtain the knowledge fusion vector includes:

[0229] The tokens corresponding to the triples in the sample text are masked to obtain the knowledge fusion vector. The tokens in the triples can be tokens corresponding to entities, tokens corresponding to relations, or tokens corresponding to both entities and relations.

[0230] In one embodiment, masking the lexical units corresponding to the triples in the sample text to obtain the knowledge fusion vector includes:

[0231] The tokens corresponding to entities and relations in the triples in the sample text are masked to obtain the knowledge fusion vector.

[0232] The knowledge fusion vector is obtained by masking the lexical units corresponding to the triples in the sample text. This can be done by masking only the lexical units corresponding to entities in the triples, thus obtaining the embedding vectors for the entities; or by masking the lexical units corresponding to both entities and relations in the triples, thus obtaining the embedding vectors for the entities and relations; or by obtaining the embedding vector for the triples as a whole.

[0233] Whether to mask the entities in the triples to obtain the knowledge fusion vector, or to mask both the entities and relations, can be chosen as needed. In one embodiment, masking the sample text based on the triples to obtain the knowledge fusion vector includes:

[0234] The sample text is divided into multiple sub-texts based on the lexical units corresponding to the triples, and the multiple sub-texts are masked to obtain the knowledge fusion vector.

[0235] In one embodiment, dividing the sample text into multiple sub-texts based on the lexical units corresponding to the triples includes:

[0236] The sample text is divided into multiple subtexts by dividing the word at the beginning or end of the corresponding triplet.

[0237] In one embodiment, masking the plurality of sub-texts to obtain the knowledge fusion vector includes:

[0238] In the subtext, the tokens corresponding to the triples are masked to obtain the knowledge fusion vector.

[0239] In one embodiment, due to the diversity of sample texts, the triples in the sample texts may be evenly distributed throughout the sample texts. In this case, the knowledge fusion vector is obtained by directly masking the words in the sample texts. The trained target model can determine the features of each part of the sample texts relatively accurately and achieve relatively accurate predictions.

[0240] However, in some cases, the triples in the sample text are not evenly distributed throughout the text, but rather concentrated in a certain part. For example, if the sample text contains 30 characters, and the triples only exist in the first 20 characters, then when masking the words corresponding to the triples in the sample text to obtain the knowledge fusion vector, the prediction training is mainly based on the features of words near the masked words. Therefore, the target model can relatively accurately determine the features of the first 20 characters in the sample text, but its determination of the features of the last 10 characters is relatively inaccurate. In this case, the prediction accuracy based on the target model will also be relatively low.

[0241] Therefore, before performing prediction training, this embodiment can first divide the sample text according to the corresponding triplets. For example, for a sample text containing one triplet, the text can be divided once along the words in the sample text from front to back, and the corresponding triplet is encountered, resulting in two sub-texts for each division. For example, for a sample text containing multiple triplets, for each triplet, the text can be divided once along the words in the sample text from front to back, and the corresponding triplet is encountered, resulting in multiple sub-texts, each division corresponding to two sub-texts. This results in multiple sub-texts, and the corresponding triplets are more evenly distributed in the sub-texts compared to the sample text before division.

[0242] It should be noted that since a triple includes a head entity, a tail entity, and a relation, the sample text can be divided in three ways: once when a word corresponding to the head entity is encountered, once when a word corresponding to the relation is encountered, or once when a word corresponding to the tail entity is encountered. In this case, one side can be divided to obtain two sub-texts.

[0243] Furthermore, multiple divisions can be performed. For example, a division can be performed whenever a word is encountered in the head entity, tail entity, or relation. In this case, if the sample text does not include the relation, it can be divided into three sub-texts (i.e., the head entity and tail entity are used as the dividing boundaries). If the sample text includes the relation, it can be divided into four sub-texts (i.e., the head entity, tail entity, and relation are used as the dividing boundaries).

[0244] In addition, when making the division, it can be done at the beginning of the word in the corresponding triplet or at the end of the word in the corresponding triplet.

[0245] The following examples are mainly for the case of performing a single partition and partitioning the lexical endings of the corresponding triples.

[0246] For example, in the sample text of 30 characters mentioned above, the head entities in the triple are the 9th and 10th characters, and the tail entities are the 18th and 19th characters. Then, a division can be made at the end of the 10th character, with the first to 10th characters forming the first sub-text, and the 11th to 30th characters forming the second sub-text. Alternatively, it can be divided into multiple segments, for example, further dividing the 11th to 30th characters into segments 11 to 20, 21 to 30, etc.

[0247] Because the first subtext has fewer characters, although the head entity is located at the end of the first subtext, the corresponding triplet tokens are still relatively evenly distributed in the first sample. In the second subtext, the tail entity is located in the 8th and 9th characters out of 20 characters, which is approximately the center of the sample text, so the distribution is also relatively even.

[0248] For example, consider the sample text: "After the Tumu Crisis, Yu Qian, an outstanding national hero of the Ming Dynasty, was created." The triad [["Yu Qian", "Dynasty", "Ming Dynasty"]] can be used to divide the sample text based on the entities "Yu Qian" and "Ming Dynasty." The first subtext is: "After the Tumu Crisis, the Ming Dynasty was created"; the second subtext is: "Outstanding national hero Yu Qian."

[0249] As can be seen, the corresponding entity tokens "Yu Qian" and "Ming Dynasty" are relatively evenly distributed in the two sub-texts. Based on this, the tokens corresponding to the triples in the sub-texts are masked to obtain the knowledge fusion vector. The trained target model can relatively accurately determine the features of each part of the sample text before segmentation and achieve relatively accurate prediction.

[0250] In one embodiment, when the sample text contains n (n is a positive integer) triples, the step of dividing the sample text into multiple sub-texts based on the lexical units corresponding to the triples, and masking the lexical units corresponding to the triples in the sample text to obtain the knowledge fusion vector, includes:

[0251] Starting from i=1 and continuing until i=n, ​​repeat the following steps;

[0252] The sample text is divided according to the word element corresponding to the i-th triplet among the n triplets to obtain the i-th sub-text group composed of multiple sub-texts corresponding to the i-th triplet;

[0253] In the i-th subtext group, the tokens corresponding to the triples are masked to obtain the knowledge fusion vector; where n is an integer greater than 1 and i is a positive integer less than or equal to n.

[0254] When a sample text contains a single triplet, it can be partitioned. Similarly, when a sample text contains multiple triplets, such as n triplets, it can also be partitioned. For each triplet, we can proceed along the tokens in the sample text from beginning to end, performing a partition at the end of the first corresponding token in the sample text. This results in multiple partitions corresponding to two sub-texts.

[0255] For example, the sample text reads: "Song Yingxing was born in Fengxin, Jiangxi Province in 1587 AD. He lived through the final period of the Ming Dynasty's corruption and demise."

[0256] It can be determined that in the sample text, the head entity SUBJ includes "Song Yingxing", the tail entity OBJ includes "Fengxin, Jiangxi" and "1587", and the relations include "birth location" and "birth time".

[0257] In this sample text, there are two triples: one is [["Song Yingxing", "birth location", "Fengxin, Jiangxi"]], and the other is [["Song Yingxing", "birth time", "1587"]].

[0258] We can first divide the sample text according to the first triplet. For example, the first subtext corresponding to the first triplet is: Song Yingxing was from Fengxin, Jiangxi; the second subtext is: a person, born in 1587 AD, who lived through the final period of the Ming Dynasty's corruption and demise.

[0259] Then, the sample text is divided according to the second triplet. For example, the first subtext corresponding to the second triplet is: Song Yingxing was from Fengxin, Jiangxi, born in 1587 AD; the second subtext is: born, and lived through the final period of the Ming Dynasty from its corruption to its demise.

[0260] It should be noted that when performing multiple partitions, the lexical units on which the partitions are based can also be considered. For example, it is possible to ensure that the results of partitioning based on each triplet are different. Since the head entities of the two triplets mentioned above are the same, the partitions are based on the tail entities.

[0261] After the partitioning, we can first mask the words corresponding to the triples in the two sub-texts corresponding to the first triple to obtain the knowledge fusion vector, and complete one epoch of training; then we can mask the words corresponding to the triples in the two sub-texts corresponding to the second triple to obtain the knowledge fusion vector, and complete the next epoch of training.

[0262] In one embodiment, training is performed after masking the tokens corresponding to the triples in the i-th subtext group as one epoch of training, and k epochs of training are performed, where k is an integer greater than or equal to n.

[0263] The number of training rounds k can be greater than or equal to n. For example, if k equals n, then from position i=1 to position i=n, ​​the word corresponding to the triple in the i-th sub-text group is masked to obtain the knowledge fusion vector, that is, n rounds of training are performed.

[0264] When k is greater than n, for example, k can be an integer multiple of n, then k steps can be executed:

[0265] Starting from position i=1 to position i=n, ​​the tokens corresponding to the triples in the i-th sub-text group are masked to obtain the knowledge fusion vector. This completes k rounds of training, and the value of k can be set as needed.

[0266] Understandably, each epoch of training performs masking training on one of the sub-text groups. After n epochs, all sub-text groups have undergone masking training once. Performing k epochs of training means repeatedly looping through the n epochs. This training method allows the model to learn all the knowledge derived from dividing the text into n sub-text groups, improving its accuracy.

[0267] In one embodiment, when the sample text contains n triples, the step of dividing the sample text into multiple subtexts based on the lexical units corresponding to the triples includes:

[0268] The sample text is divided into multiple subtexts according to the word corresponding to the m-th triplet among the n triplets, where n is an integer greater than 1, m is a positive integer less than or equal to n, and m is a specified value or a random value.

[0269] For example, in the sample text of the above embodiment, there are two triples: one is [["Song Yingxing", "birth location", "Fengxin, Jiangxi"]], and the other is [["Song Yingxing", "birth time", "1587"]]. The sample text can be divided according to one of the triples, or it can be divided according to the two triples separately.

[0270] In one embodiment, when the sample text contains multiple triples, such as n triples, it can be divided multiple times according to the above embodiment, or it can be divided only once as needed. For example, the n triples can be labeled first (the labeling logic can be determined as needed, such as labeling according to the first occurrence position of each triple), and then the m-th triple can be determined from the n triples. Then, the sample text can be divided into multiple sub-texts according to the word corresponding to the m-th triple.

[0271] In one embodiment, masking the lexical units corresponding to the triples in the sample text to obtain the knowledge fusion vector includes:

[0272] The knowledge fusion vector is obtained by masking the lexical units corresponding to the triples in the sample text according to a first probability (which may be less than 100%).

[0273] In one embodiment, masking the lexical units corresponding to the triples in the sample text to obtain the knowledge fusion vector further includes:

[0274] The knowledge fusion vector is obtained by masking the uncorresponding lexical units of the triple in the sample text according to the second probability.

[0275] In one embodiment, the first probability is greater than the second probability.

[0276] In one embodiment, the first probability is 60% to 80%, for example, set to 80%, and / or the second probability is 20% to 40%, for example, set to 20%, and the sum of the first probability and the second probability is 100%.

[0277] This disclosure allows for masking the corresponding words in the sample text based on a first probability to obtain the knowledge fusion vector, and also allows for masking the words in the sample text that do not correspond to the triples based on a second probability to obtain the knowledge fusion vector. For example, if the first probability is 80%, then there is an 80% probability of selecting the word to be masked as the word corresponding to the triple, and a 20% probability of selecting the word that does not correspond to the triple. Therefore, it can be ensured that the trained target model can accurately predict both the words corresponding to the triples and the words that do not correspond to the triples.

[0278] It's important to note that when masking the tokens corresponding to triples in the sample text, if masking is performed with 100% probability—that is, removing the masked tokens and replacing the original words with a [mask] tag—and the natural language processing model then tries to correctly predict the removed words, a problem arises: the [mask] tag is frequently seen during training, but it doesn't exist in the actual text used. This leads the natural language processing model to believe that the output is for the [mask] tag, but since the tag isn't actually present in the text, it causes some issues.

[0279] To avoid this problem, this embodiment masks selected words that need to be masked according to a preset ratio. For other words that need to be masked, some can be randomly replaced with another word, while others can remain unchanged. For example, masking is performed with an 80% probability; that is, the selected words that need to be masked have an 80% probability of actually being replaced with the [mask] tag. Other words that need to be masked have a 10% probability of being randomly replaced with another word, and a 10% probability of remaining unchanged.

[0280] Figure 3 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 3 As shown, determining the triples in the sample text includes:

[0281] In step S301, the triples corresponding to the sample text in the knowledge graph (KG) are determined, or a triple extraction model (e.g.) is used. Figure 4 The triplet extraction model in the illustrated embodiment obtains triples from the sample text.

[0282] In one embodiment, there are multiple ways to determine triples in a sample text. For example, a triple extraction model can be determined first, and then the triples in the sample text can be directly obtained based on the triple extraction model. Alternatively, a knowledge graph can be used to determine triples. For example, the head entity and tail entity of the sample text can be determined, and then the relationship between the head entity and tail entity can be determined in the knowledge graph.

[0283] Since knowledge graphs contain explicit entities and relationships between them, the corresponding triples of a sample text can be determined within the knowledge graph. For example, after identifying the head and tail entities in a triple in the sample text, the relationship between the head and tail entities in the knowledge graph can be determined based on the identified head and tail entities, which is beneficial for accurately determining the relationships within the triples.

[0284] It should be noted that even if the sample text contains the relationship in the triplet, since the sample text is an independent text and has no connection with other texts, the relationship between entities determined based on the sample text may still be incomplete. Knowledge graphs, on the other hand, contain a large number of entities and have constructed a large number of relationships between them, so determining relationships based on knowledge graphs is more comprehensive and accurate.

[0285] In one embodiment, determining the triple corresponding to the sample text in the knowledge graph includes:

[0286] Determine the domain to which the sample text belongs;

[0287] Determine the knowledge graph for the domain;

[0288] Determine the triples corresponding to the sample text in the knowledge graph of the domain.

[0289] In one embodiment, because the same entity can express different meanings in different domains, the relationships between the same entity and other entities can vary significantly across different domain-specific knowledge graphs. For example, the entity "apple" generally refers to a type of fruit in the agricultural domain, so in an agricultural knowledge graph, "apple" is generally related to other fruits; however, in the communications domain, it generally refers to a type of mobile phone, so in a communications knowledge graph, "apple" is generally related to other mobile phones. Therefore, if triples are determined based on the knowledge graph without distinguishing between domains, the determined triples are likely to be inaccurate.

[0290] In this embodiment, the domain to which the sample text belongs can be determined first, followed by the determination of the knowledge graph of that domain, and finally the determination of the triples corresponding to the sample text in the knowledge graph of that domain. For example, if the sample text belongs to the agricultural domain, then the triples corresponding to the sample text in the agricultural domain's knowledge graph can be determined; if the sample text belongs to the communication domain, then the triples corresponding to the sample text in the agricultural domain's communication knowledge graph can be determined. This helps ensure the accuracy of the determined triples.

[0291] In one embodiment, the knowledge graph includes at least one of the following:

[0292] Pre-built knowledge graph;

[0293] Pre-stored knowledge graph.

[0294] The knowledge graph used in this embodiment can be a pre-stored knowledge graph, such as an existing knowledge graph, or a pre-built knowledge graph. The following describes how to construct a knowledge graph through several exemplary embodiments.

[0295] In one embodiment, when the knowledge graph includes a pre-constructed knowledge graph, the method further includes constructing the knowledge graph before determining the triple corresponding to the sample text in the knowledge graph. This embodiment allows for the independent construction of the knowledge graph as needed, which helps ensure that the knowledge graph containing the triple meets the requirements.

[0296] Figure 4 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 4 As shown, constructing the knowledge graph includes:

[0297] In step S401, a triplet extraction model is determined, wherein the triplet extraction model is used to extract triples from the sample text;

[0298] In step S402, the knowledge graph is constructed based on the extracted triples.

[0299] In one embodiment, since the main content of a knowledge graph is triples, in order to construct a knowledge graph, this embodiment can first determine a triple extraction model, and then extract triples from a large number of sample texts using the triple extraction model, which is beneficial to improve the efficiency of triple extraction, and then construct a knowledge graph based on the extracted triples.

[0300] Figure 5 This is a schematic diagram illustrating a triplet extraction model according to an embodiment of the present disclosure. Figure 5 As shown, the triplet extraction model includes:

[0301] The label determination model 501 is used to determine the label of the token (the granularity of the segmentation can be a character or a word, which can be selected as needed) in the sample text; wherein, the label is used to determine the entity in the sample text;

[0302] Relationship determination model 502 is used to determine the relationship between at least two entities (e.g., entity pairs, Span Entities) determined by the label determination model and / or the position of the relationship in the sample text.

[0303] In one embodiment, the most basic element in the sample text can be a token. Therefore, the entities in the triples of the sample text can be determined at the token level. To determine which tokens can be entities in the triples, annotation methods can be used, such as the BILOU(beginning, inside, outside, last, unit) annotation method. The tokens in the sample text are annotated with labels B, I, L, O, and U (the granularity can be characters or words, depending on the needs). Entities in the sample text can then be determined based on the annotations. Here, label B represents the beginning of an entity, label I represents the middle of an entity, label L represents the end of an entity, label O represents a non-entity token, and label U represents a single word entity. Multiple tokens in the sample text that sequentially (or even consecutively) correspond to the label BIL can then be identified as entities.

[0304] Therefore, in order to accurately determine the label of each word in the sample text, this embodiment first constructs a label determination model to determine the label of the word in the sample text, and then the entity in the sample text can be determined based on the label of the word.

[0305] However, as can be seen from the embodiments described above, the entities in the triple are generally located in the sample text, while the relations in the triple may not be located in the sample text. Moreover, even if they are located in the sample text, it is necessary to accurately determine the position of the relations in the sample text in order to accurately determine the relations in the triple in the sample text.

[0306] Therefore, in order to determine the relation in the triple and / or the position of the relation in the sample text, this embodiment also constructs a relation determination model, which can be used to determine the relationship between two entities in an entity pair and / or the position of the relation in the sample text.

[0307] Therefore, triples can be accurately extracted from sample text using the triple extraction model. Specifically, the entity extracted from the triples can be determined by the label in the triple extraction model, and the relation extracted from the triples can be determined by the relation in the triple extraction model.

[0308] In one embodiment, such as Figure 5 As shown, the label determination model 501 includes:

[0309] First coding layer 5011, the input of the first coding layer includes the word character;

[0310] Feature extraction layer 5012, the input of which includes the output of the first encoding layer;

[0311] The first activation layer 5013, the input of the first activation layer includes the output of the feature extraction layer;

[0312] The first normalization layer 5014, the input of the first normalization layer includes the output of the first activation layer;

[0313] The first output layer 5015, whose input includes the output of the first normalization layer, is used to output the label.

[0314] In one embodiment, such as Figure 5 As shown, the sample text can be divided into multiple tokens and then input into the first encoding layer. The first encoding layer can be used to determine the embedding vector of the tokens. Optionally, the first encoding layer can include a BERT model and / or a word2vec model. The BERT model and / or word2vec model can extract features from the entire sentence of the sample text to obtain the embedding vector of the tokens.

[0315] The output of the first encoding layer can be input to the feature extraction layer, which may include, for example, a bidirectional long short-term memory (Bi-LSTM) model to extract features from the embedding vector. The feature extraction layer is based on the Bi-LSTM model.

[0316] The output of the feature extraction layer can be input into the first activation layer, which may include, for example, a Tanh layer;

[0317] The output of the first activation layer can be input to the first normalization layer, which may include, for example, a softmax layer.

[0318] The output of the first normalization layer can be input into the first output layer, and the first output layer finally outputs the tag corresponding to the word.

[0319] By using the label determination model, the label corresponding to each token in the sample text can be determined. For example, for a sample text containing n tokens, the label of token(1) can be determined as label(1), the label of token(2) can be determined as label(2), ..., the label of token(n) can be determined as label(n).

[0320] In one embodiment, such as Figure 5 As shown, the input corresponding to the (i+1)th word in the sample text in the first activation layer also includes the output corresponding to the ith word in the sample text in the first output layer, where i is an integer greater than or equal to 1.

[0321] In one embodiment, if the first activation layer, the first normalization layer, and the first output layer process each word independently, although a label for each word can be obtained, the accuracy of the labels may not be very high due to a lack of consideration for the relationships between words. Therefore, this embodiment uses the output of the i-th word in the first output layer as the input of the (i+1)-th word in the first activation layer. This ensures that the label of the i-th word is taken into account when determining the label of the (i+1)-th word, thus guaranteeing that the determined words satisfy the relationships between words and improving the accuracy of the label determination.

[0322] In one embodiment, such as Figure 5 As shown, the relationship determination model 502 includes:

[0323] Second coding layer 5021, the input of the second coding layer includes the entity pair;

[0324] Fully connected layer 5022, the input of which includes the output of the second coding layer;

[0325] The second activation layer 5023, the input of the second activation layer includes the output of the fully connected layer;

[0326] The second normalization layer 5024, whose input includes the output of the second activation layer, is used to output the relation and / or the position of the relation in the sample text.

[0327] In one embodiment, after determining the tags corresponding to the words in the sample text, the entities belonging to the triples in the sample text can be further determined based on the tags. After the entities are determined, each pair of entities can be formed into an entity pair, and the entity pairs can be used as input to the relation determination model.

[0328] For example, entity pairs can first be input into a second coding layer, which can determine the embedding vector of the entity pair;

[0329] The output of the second coding layer can be used as the input of the fully connected layer.

[0330] The output of the fully connected layer can be input to a second activation layer, which may include, for example, a Tanh layer;

[0331] The output of the second activation layer can be input to the second normalization layer, which may include, for example, a softmax layer.

[0332] By determining the relationship through the model, the relationships between entity pairs in the input model can be identified. If the relationship exists in the sample text, its position within the sample text can be further determined, and the lexical units at that position can be extracted as the relationship. Based on this, triples can be extracted.

[0333] In one embodiment, such as Figure 5 As shown, the input of the fully connected layer also includes the output of the feature extraction layer. Since determining relationships solely based on entity pairs lacks consideration of the contextual relationships between entity pairs within the sample text, the accuracy of the determined entity and / or relationship positions may be relatively low.

[0334] Therefore, in this embodiment, the output of the feature extraction layer in the label determination model is used as the input of the fully connected layer in the relation determination model. Since the output of the feature extraction layer can well characterize the relationship between entities in an entity pair within the context of the sample text, the relation determination model can also consider the relationship between entities in an entity pair within the context of the sample text when determining the location of entities and / or relations, which helps to ensure the accuracy of determining the location of entities and / or relations.

[0335] Figure 6 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 6 As shown, determining the triplet corresponding to the sample text in the knowledge graph includes:

[0336] In step S601, multiple sub-knowledge graphs are determined;

[0337] In step S602, entities with the same semantics in different sub-knowledge graphs are merged to obtain a merged knowledge graph;

[0338] In step S603, the triples corresponding to the sample text in the merged knowledge graph are determined.

[0339] In one embodiment, since the same entity may be represented by different lexical units in different knowledge graphs (for example, the entity "pineapple" can also be represented by "pineapple"), the triples containing the entity "pineapple" and the triples containing the entity "pineapple" may differ slightly in different knowledge graphs. Therefore, merging semantically identical entities in different knowledge graphs can avoid redundancy caused by using different lexical units to represent entities with the same meaning, and it also helps to expand the triples containing entities, ensuring the comprehensiveness of the knowledge graph. For ease of description, the knowledge graphs that need to be merged are called sub-knowledge graphs.

[0340] For example, in sub-knowledge graph KG1, there is an entity called "pineapple", and the corresponding triple includes [["pineapple", "type", "fruit"]]; while in sub-knowledge graph KG2, there is an entity called "pineapple", and the corresponding triple includes [["pineapple", "type", "fruit"]].

[0341] In essence, these two triples express the same meaning, but because the head entities are represented by different semantically identical lemmas, there is a certain degree of redundancy. This may lead to different triples being determined when identifying the lemmas corresponding to the entities "pineapple" and "pineapple," or in some cases, only one of these two entities can be identified as a triple.

[0342] According to this embodiment, entities with the same semantics in different sub-knowledge graphs can be merged to obtain a merged knowledge graph. For example, the entities "pineapple" and "pineapple" in KG1 and KG2 can be merged. Then, the triples corresponding to the sample text in the merged knowledge graph can be determined. So, no matter whether the word in the sample text is "pineapple" or "pineapple", the same triples can be successfully determined in the merged indicator graph.

[0343] The following examples illustrate the methods for merging semantically identical entities in the above embodiments.

[0344] In one embodiment, merging semantically identical entities from different sub-knowledge graphs includes:

[0345] Calculate the distance between the embedding vectors of any two entities in different sub-knowledge graphs. If the distance is less than a distance threshold, then merge the two entities.

[0346] In one embodiment, for each entity in a sub-knowledge graph, a corresponding embedding vector can be determined. For example, the embedding vector for each entity in the sub-knowledge graph can be determined in the same vector space to calculate the distance between the embedding vectors. When the distance between two embedding vectors is sufficiently small, such as less than a distance threshold, it can be determined that the entities corresponding to these two embedding vectors have the same semantics. Therefore, the entities corresponding to these two embedding vectors can be merged. This avoids the problem of semantically identical entities appearing with different text in the merged knowledge graph, which leads to unclear triple relationships in the knowledge graph, thus enabling accurate extraction of triple information.

[0347] For example, an entity might be represented as (A) in knowledge graph KG1, (B) in knowledge graph KG2, (C) in knowledge graph KG3, and (D) in knowledge graph KG4. After merging, (A), (B), (C), and (D) can be combined into a list (A, B, C, D). When any item in this list appears in the sample text, it can be mapped to other items in the list. For example, if (A) appears in the sample text, the triple corresponding to (A) can also be used as the triple corresponding to (B), (C), and (D) in the list.

[0348] In one embodiment, calculating the distance between the embedding vectors of any two entities in different sub-knowledge graphs includes:

[0349] The distance is calculated based on the structural feature vector hs and attribute feature vector ha corresponding to the two entities.

[0350] In one embodiment, for each entity in different sub-knowledge graphs, two feature vectors can be assigned (e.g., the embedding vector corresponding to the entity in the vector space contains these two feature vectors), namely a structural feature vector hs and an attribute feature vector ha. The distance can then be calculated based on the structural feature vector hs and the attribute feature vector ha, for example, based on the following formula:

[0351]

[0352] Among them, e i and v j D(e) represents two entities in different sub-knowledge graphs. i ,v j ) represents the distance between the embedding vectors corresponding to these two entities, h s (e i ) represents e i The corresponding structural feature vector, h s (v j ) represents v j The corresponding structural feature vector, ha (e i ) represents e i The corresponding attribute feature vector, h a (v j ) represents v j The corresponding attribute feature vector, d s It is the dimension of the structural feature vector, d a is the dimension of the attribute feature vector, β is a hyperparameter used to balance the attribute feature vector and the structural feature vector, and f(x,y)=||xy||1.

[0353] In one embodiment, the structural feature vector h of the entity s and attribute feature vector h a The structural feature vector h of the previous entity s-1 and attribute feature vector h a-1 The connectivity matrix A is calculated using a graph neural network (GCN) model. For example, it can be determined using the following formula:

[0354]

[0355] Among them, H s H represents the structural feature matrix of an entity. a The attribute feature matrix representing the entity. This represents the structural feature weight matrix of the l-th layer. This represents the attribute feature weight matrix of the l-th layer. A is an n×n connectivity matrix, and I is the identity matrix. yes The degree matrix of the diagonal nodes, where σ is the activation function.

[0356] In one embodiment, a can be set ij ∈A, representing the degree of alignment information propagation from the i-th entity to the j-th entity. For each relation in the triple, two measures can be computed, called the function and the inverse function, respectively:

[0357]

[0358]

[0359] To measure the influence of the i-th entity on the j-th entity, a can be calculated using the following formula: ij ∈A:

[0360]

[0361] Where G represents a knowledge graph.

[0362] In one embodiment, the GCN model can be trained by minimizing the following distance-based ranking loss function:

[0363]

[0364]

[0365] The method for creating negative examples involves randomly selecting an entity from the two merged knowledge graphs to replace e and v in (e,v), γ s and γ a All values ​​greater than 0 indicate a distance hyperparameter used to separate positive and negative alignment. s It is the loss function for structural embedding, L a It is the loss function for feature (also called attribute) embedding, L s and L a These are independent of each other and can be optimized separately. For example, the above loss function can be minimized using the stochastic steepest descent method (SGD).

[0366] In one embodiment, the at least one pre-training task further includes:

[0367] The second pre-training task is used to split the sample document containing the sample text into segments and then rearrange them for training.

[0368] The target model trained based on the second pre-training task can be used to determine the distance between sample text and other texts, and then determine the relationship between sample text and other texts based on the predicted distance. For example, it can predict whether sample text and other texts belong to the same document, and whether they belong to the neighbor relationship in the same document.

[0369] In one embodiment, the step of splitting the sample document containing the sample text into multiple segments and then reordering the training includes:

[0370] The sample document is split into fragments at least one level of granularity. After each level of splitting, the resulting fragments are randomly arranged and combined to rearrange the sample.

[0371] Training is performed based on the rearranged samples to output the relationship types between sentences in the rearranged samples.

[0372] In one embodiment, the relationship type includes at least one of the following:

[0373] They belong to the same document and are adjacent to each other;

[0374] They belong to the same document but are not adjacent;

[0375] They belong to different documents.

[0376] For example, taking a two-level granularity approach, for the sample document containing the sample text, the sample document can be split into multiple segments at the first granularity (e.g., paragraphs), for example, into three segments [C1, C2, C3]. The order of the segments is randomly shuffled, for example, after shuffling, they become [C2, C1, C3]. Then, training is performed to restore the correct order. Furthermore, [C1, C2, C3] can be further split into smaller segments at the second granularity (e.g., sentences), with the second granularity being smaller than the first granularity, for example, into [S0, S1, S2, ..., Sn]. The order of the segments is randomly shuffled, for example, after shuffling, they become [S2, S9, Sn, ..., S3]. Then, training is performed to restore the correct order.

[0377] Specifically, the distance between segments can be learned through document-level information. This task can be a 3-class classification problem. For example, output 0 means that the two segments belong to the same document and are adjacent; output 1 means that the two segments belong to the same document and are not adjacent; output 2 means that the two segments belong to different documents.

[0378] It should be noted that the granularity of document splitting and rearranging is not limited to the two levels of granularity mentioned above. It can also include larger or smaller granularities. Taking a smaller granularity as an example, a sample text can be divided into multiple segments using triples, and then the order of these segments can be shuffled and rearranged for training to restore the correct order. The methods for splitting sample text according to triples include, but are not limited to, the methods described in the previous embodiments, and will not be repeated here.

[0379] In one embodiment, the at least one pre-training task further includes:

[0380] The third pre-training task is used to train semantic recognition on the sample text.

[0381] The target model trained based on the third pre-training task can be used to recognize the semantics of sample text. In an embodiment of this disclosure, the semantic recognition training of the third pre-training task can be training the relationship between two sentences in the sample text. This can be understood as training to learn the possible causal, progressive, and adversative logical relationships between the two sentences by recognizing words such as "because," "further," and "but." In another embodiment, the semantic recognition training of the third pre-training task can be the degree of relevance between a query and the title of a passage in the sample text. This can be understood as a multi-classification task; for example, an output of 0 indicates a strong correlation, meaning the user will continue to click on the title of the passage in the sample text after entering the query; an output of 1 indicates a weak correlation, meaning the user will not click on the title of the passage in the sample text after entering the query; and an output of 2 indicates no relevance, i.e., randomly generated samples. These training samples can be data obtained through a search engine.

[0382] It should be noted that, in addition to the first, second, and third pre-training tasks mentioned above, at least one training task can also include other pre-training tasks as needed. These pre-training tasks can be supervised or unsupervised, and can be set according to the actual task situation. Furthermore, the natural language processing model can be trained based on one or more pre-training tasks. When training with multiple pre-training tasks, the resulting target model can have multiple functions, thereby improving the performance of the target model.

[0383] Figure 7 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 7 As shown, training the natural language processing model according to the at least one pre-training task to obtain the target model includes:

[0384] In step S701, at least one round of the following steps is performed:

[0385] Pre-training tasks are selected and superimposed according to a preset order. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks.

[0386] In one embodiment, the order of each pre-training task can be preset. For example, taking the first pre-training task Task1, the second pre-training task Task2, and the third pre-training task Task3 mentioned above, the order of the three pre-training tasks from front to back is Task1, Task2, and Task3.

[0387] We can first select Task1 to train the natural language processing model; then select Task2 and stack it together with Task1, and use the two pre-training tasks, Task1 and Task2, to continue training the natural language processing model after the previous training; then select Task3 and stack it together with Task1 and Task2, and use the three pre-training tasks, Task1, Task2 and Task3, to continue training the natural language processing model after the previous training.

[0388] Here, "overlay" refers to superimposing the loss functions corresponding to the pre-training tasks used in the training together to form a comprehensive loss function for training. The comprehensive loss function can be obtained by directly superimposing the loss functions corresponding to each pre-training task, or by superimposing the loss functions corresponding to each pre-training task and then averaging them, or by weighted summing the loss functions corresponding to each pre-training task. This disclosure does not limit the scope of the embodiments.

[0389] This completes one round of training. The number of training rounds can be set as needed, or determined based on set cutoff conditions (such as the training results converging to a specified degree).

[0390] The number of training iterations based on each stacked pre-training task during each training round can be flexibly adjusted. For example, one training iteration can be performed based on each stacked pre-training task, such as one training iteration based on Task1, then one training iteration based on Task1 and Task2 combined, and then one training iteration based on Task1, Task2, and Task3 combined. Alternatively, multiple training iterations can be performed based on each stacked pre-training task, such as two training iterations based on Task1, then three training iterations based on Task1 and Task2 combined, and then four training iterations based on Task1, Task2, and Task3 combined.

[0391] Figure 8 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 8 As shown, training the natural language processing model according to the at least one pre-training task to obtain the target model includes:

[0392] In step S801, pre-training tasks are randomly selected and superimposed. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks.

[0393] In one embodiment, the order of each pre-training task can be preset. For example, taking the first pre-training task Task1, the second pre-training task Task2, and the third pre-training task Task3 mentioned above, the order of the three pre-training tasks from front to back is Task1, Task2, and Task3.

[0394] Then, a pre-training task can be randomly selected from these three pre-training tasks. The number of pre-training tasks randomly selected each time can be preset or random.

[0395] Taking the example of pre-selecting an incrementally increasing number of randomly chosen pre-training tasks each time, for instance, if Task2 is randomly selected from the three pre-training tasks, the natural language processing model will be trained first using Task2. Then, if Task2 and Task3 are randomly selected, the previously trained natural language processing model will be trained again using these two pre-training tasks. Finally, if Task1, Task2, and Task3 are randomly selected, the previously trained natural language processing model will be trained again using these three pre-training tasks. Each randomly selected pre-training task may or may not include previously selected pre-training tasks; the specific random selection rules can be set as needed. Training can stop when the training results reach convergence.

[0396] about Figure 7 and Figure 8 The illustrated embodiment allows for training tasks to be selected sequentially or randomly, depending on the needs. Furthermore, priority can be given based on the importance of each pre-training task, with a certain probability of selection.

[0397] In one embodiment, the learning rate for training the natural language processing model is positively correlated with the number of samples input to the natural language processing model each time it is trained.

[0398] In one embodiment, the relationship between the learning rate and the number of samples input to the natural language processing model each time is as follows:

[0399] Y = 5.57e -6 ·log2X-4.75e -6 ;

[0400] Where X is the batch size, which is the number of samples input to the natural language processing model for each training iteration, ranging from 24 ≤ X ≤ 256, and Y is the learning rate. It should be noted that during actual training, the calculated value of Y can be fine-tuned, within the range of [Y - 1.0e-5, Y + 1.0e-5]. Training using the relationship between batch size and learning rate described above can lead to faster model convergence. Typically, X is a power of 2, but this application does not impose a limitation. For values ​​of X outside the range of [24, 256], the learning rate can also be selected using the above formula.

[0401] Figure 9 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 9 As shown, the method further includes:

[0402] In step S901, the target sample set is determined according to the target training task;

[0403] In step S902, an overall model is trained based on the target sample set, wherein the overall model includes the target model and the fine-tuning model.

[0404] In one embodiment, the target model obtained by training a natural language model through at least one pre-training task can be a pre-trained model. This pre-trained model, through a large corpus and multi-task learning, enables the target model to better learn general grammatical and semantic knowledge, forming a general model with good generalization ability. After obtaining the target model, this embodiment can also construct a target training task (fine-tuning task) according to a specific application scenario. For the training samples of the target training task, a target sample set smaller than the pre-training sample set can be obtained according to the application scenario. The training model for the fine-tuning task can be a fine-tuned model added to the trained target model as the overall model for training, thereby more accurately adapting to the needs of the specific application scenario. The pre-training + fine-tuning training method is an application of transfer learning and has good scalability. Because the pre-trained model has good generalization ability, when training the model for a new application scenario, the parameters of the pre-trained model can be used for initialization, and fine-tuning the overall model can achieve good training results.

[0405] In one embodiment, the fine-tuning model includes at least one of the following:

[0406] Convolutional Neural Network (CNN) model;

[0407] Recurrent Neural Network (RNN) model;

[0408] Gate-controlled loop unit (GRU) model;

[0409] Fully connected layer (FC);

[0410] Long Short-Term Memory (LSTM) model.

[0411] about Figure 1 The embodiments shown in the figure process the sample text based on the triples to obtain a knowledge fusion vector. There are two main embodiments. One embodiment is the aforementioned "masking the lexical units in the sample text corresponding to the entities in the triples to obtain the knowledge fusion vector" to obtain the embedding vector of the lexical units corresponding to the triples. The other embodiment is the fusion of sample text and triples.

[0412] These two embodiments can be implemented independently or in combination. For example, based on obtaining the embedding vector of the word corresponding to the triplet in "masking the word corresponding to the triplet in the sample text to obtain the knowledge fusion vector", the sample text and the triplet can be fused to obtain the knowledge fusion vector, or the word corresponding to the triplet can be masked during the fusion process of the sample text and the triplet. No limitation is made here. The following examples illustrate the embodiment of processing the sample text based on the triplet to obtain the knowledge fusion vector, including obtaining the knowledge fusion vector from the sample text and the triplet.

[0413] Figure 10 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 10 As shown, the process of processing the sample text based on the triples to obtain the knowledge fusion vector includes:

[0414] In step S1001, the sample text and the triples in the sample text are fused to obtain a knowledge fusion vector.

[0415] In one embodiment, sample text and triples in the sample text can be fused to obtain a knowledge fusion vector. The obtained knowledge fusion vector is then used as a sample to form a sample set, and a natural language processing model is trained based on the training sample set. The training task used for training may include one or more of the pre-training tasks described in the previous embodiments.

[0416] Since the knowledge fusion vector used as a sample contains not only the features of the sample text but also the features of the triples in the sample text, it contains more features than the sample text. It can more comprehensively reflect the relationships between entities in the sample text and thus more accurately represent the sample text. Therefore, the target model obtained by training the natural language processing model based on the knowledge fusion vector has better performance in natural language processing.

[0417] Regarding how to fuse the sample text and its triples to obtain a knowledge fusion vector, the following embodiments mainly illustrate two fusion methods: one is to insert the triples from the sample text into the sample text to obtain the knowledge fusion vector; the other is to determine the text embedding vector corresponding to the sample text, determine the knowledge embedding vector corresponding to the triples, and then fuse the text embedding vector and the knowledge embedding vector to obtain the knowledge fusion vector. However, in specific applications, the fusion method is not limited to these two and can be selected as needed.

[0418] Figure 11This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 11 As shown, the process of fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes:

[0419] In step S1101, the triples in the sample text are inserted into the sample text to obtain the knowledge fusion vector.

[0420] In one embodiment, triples can be inserted into the sample text, for example, between characters in the sample text, after the last character, or before the first character. The insertion position of the triples in the sample text can be set as needed, or it can be determined randomly.

[0421] In one embodiment, the sample text with inserted triples can be processed by word embedding to obtain a knowledge fusion vector. The word embedding method includes at least one of one-hot encoding, word2vec, word list conversion, etc.

[0422] In one embodiment, the method for word embedding processing of the sample text with inserted triples can also be one-hot encoding followed by multiplication with trainable weights to obtain the word embedding vector, or one-hot encoding followed by multiplication with trainable weights followed by word2vec to obtain the word embedding vector. This application does not limit this.

[0423] In one embodiment, the knowledge fusion vector may further include at least one of the following:

[0424] Sentence embedding vector, position embedding vector corresponding to the word element of the triple, and task type embedding vector.

[0425] It should be noted that when inserting triples into sample text, the triples can be inserted as a whole, or the head entity, relation, and tail entity of the triples can be inserted as three separate parts into the sample text. For example, these three parts can be inserted into different positions in the sample text.

[0426] Figure 12 This is a schematic diagram illustrating the relationship between a fusion process and a natural language processing model according to an embodiment of the present disclosure.

[0427] like Figure 12As shown, triples in the sample text can be identified. Each triple includes two entities in the sample text and the relationship between them. These triples can then be inserted into the sample text to obtain a knowledge fusion vector. Finally, this knowledge fusion vector can be used as input to a natural language processing model for training. Triples can be considered key information in the sample text. Inserting triples into the sample text for model training allows the model to repeatedly learn the key information within the sample text, thereby improving the model's training performance.

[0428] In one embodiment, in Figure 12 Based on the implementation examples, when the first training task includes a masking task that masks sample text based on triples, the inserted triples and the original sample text can be masked separately. This ensures that both triples and non-triple information in the sample text are masked.

[0429] Figure 13 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 13 As shown, the process of fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes:

[0430] In step S1301, the text embedding vector corresponding to the sample text is determined;

[0431] In step S1302, the knowledge embedding vector corresponding to the triple is determined;

[0432] In step S1303, the text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

[0433] In one embodiment, on the one hand, the text embedding vector corresponding to the sample text can be determined, and on the other hand, the knowledge embedding vector corresponding to the triple can be determined. For example, the knowledge embedding vector can be obtained by training a preset algorithm (such as the graph embedding transformation algorithm Translate). The preset algorithm includes, but is not limited to, TransR, TransE, TransH, TransD, etc.

[0434] In one embodiment, the method for obtaining the corresponding text embedding vector by word embedding processing of sample text includes at least one of one-hot encoding, word2vec, word list conversion, etc.

[0435] In one embodiment, the method for word embedding processing of sample text can also be one-hot encoding followed by multiplication with trainable weights to obtain the word embedding vector, or one-hot encoding followed by multiplication with trainable weights followed by word2vec to obtain the word embedding vector. This application does not limit this.

[0436] In one embodiment, the triple includes at least one of the following:

[0437] Head entity, relation, tail entity.

[0438] The graph embedding transformation algorithm can be used to obtain the embedding vector h of the head entity h. e Embedding vector t of tail entity t e The embedding vector r of relation r e .

[0439] In one embodiment, the specific steps for obtaining the knowledge embedding vector using the TransR algorithm are as follows:

[0440] 1) Project entities h and t in the entity space through the projection matrix M r Projecting onto the relation space r, we obtain h. e and t e , where h e =hM r , t e =tM r ;

[0441] 2) Objective function for model training Training based on the loss function minimizes the objective function, thus yielding the embedding vectors h corresponding to h, r, and t, respectively. e r e and t e .

[0442] The target loss function could be, for example:

[0443]

[0444] Where h' and t' represent the head and tail entities after random replacement, respectively. The negative sample is the part where the head or tail entity is randomly replaced based on the correct triple (h,r,t) and the replaced triple is different from the original triple.

[0445] In one embodiment, determining the knowledge embedding vector corresponding to the triple includes:

[0446] The embedding vectors of the head entity, the relation, and the tail entity are fused to obtain the knowledge embedding vector:

[0447] e k =concat(h e +r e +t e );

[0448] Among them, e k Represents knowledge embedding vector, h e The embedding vector r of the head entity represents the embedding vector of the head entity. e The embedding vector representing the relationship, t e The embedding vector of the tail entity is represented by and concat() is represented by the concatenation function.

[0449] In one embodiment, the text embedding vector may include the text embedding vector itself, and may also include at least one of the following:

[0450] Sentence embedding vector, position embedding vector corresponding to the word element of the triple, and task type embedding vector.

[0451] Based on the fusion of the text embedding vector and the knowledge embedding vector to obtain the knowledge fusion vector, other embedding vectors can be further fused to obtain the knowledge fusion vector. These other embedding vectors can be determined according to the training task and are not limited to the types mentioned above. For example, if a second pre-training task is required, other embedding vectors may include sentence embedding vectors; if a fourth pre-training task is required, other embedding vectors may include task type embedding vectors, and the fourth pre-training task may be a task used to classify training task types.

[0452] Regarding how to fuse the text embedding vector and the knowledge embedding vector, the following embodiments mainly illustrate two fusion methods. However, in specific applications, the fusion method is not limited to the following two and can be selected as needed.

[0453] Figure 14 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 14 As shown, the fusion of the text embedding vector and the knowledge embedding vector includes:

[0454] In step S1401, the text embedding vector is subjected to feature extraction by the feature extraction layer to obtain the text feature vector;

[0455] In step S1402, the knowledge embedding vector is subjected to feature extraction by the feature extraction layer to obtain the knowledge feature vector (the execution order of step S1402 and step S1401 is not important and can be set as needed, for example, they can be executed in parallel).

[0456] In step S1403, the text feature vector and the knowledge feature vector are fused to obtain the knowledge fusion vector.

[0457] In one embodiment, a text feature vector can be obtained by extracting features from the text embedding vector using at least one feature extraction layer. For example, for a text embedding vector, a text feature vector can be obtained by extracting features using feature extraction layer A.

[0458] For knowledge embedding vectors, feature extraction can be performed through feature extraction layer B to obtain knowledge feature vectors.

[0459] Finally, the text feature vector and the knowledge feature vector can be fused to obtain the knowledge fusion vector.

[0460] The fusion of the text feature vector and the knowledge feature vector includes:

[0461] The text feature vector and the knowledge feature vector are fused using a multi-layer perceptron (MLP) neural network.

[0462] The knowledge embedding vector includes at least one of the following:

[0463] The embedding vector of the entity belonging to the triple;

[0464] The embedding vector of the relation belonging to the triple.

[0465] In other words, for a knowledge embedding vector, feature extraction layer B can extract features from only the entity embedding vectors in the triplet to obtain the knowledge feature vector, or it can extract features from only the relation embedding vectors in the triplet to obtain the knowledge feature vector, or it can extract features from both the entity and relation embedding vectors in the triplet together to obtain the knowledge feature vector. It can be understood that there can be one or more feature extraction layers A and B.

[0466] Figure 15 This is a schematic diagram illustrating the relationship between another fusion process and a natural language processing model according to embodiments of this disclosure.

[0467] like Figure 15 As shown, on the one hand, the text embedding vector corresponding to the sample text can be determined; on the other hand, triples can be determined in the sample text, and then for each triple, the knowledge embedding vector corresponding to the triple in the knowledge graph can be determined.

[0468] Next, on the one hand, the text embedding vector can be input into the feature extraction layer A to obtain the extraction result as the text feature vector; on the other hand, the knowledge embedding vector can be input into the feature extraction layer B to obtain the knowledge feature vector.

[0469] Finally, the text feature vector and knowledge feature vector can be fused through a multi-layer neural network MLP layer to obtain a knowledge fusion vector, which can then be used as a sample input to train the natural language processing model.

[0470] It is understandable that the choice of feature extraction layers A and B is related to the natural language processing model. That is, the feature extraction layers in the fusion process are the same as the feature extraction layers in the natural language processing model, so as to ensure that the natural language processing model can directly calculate the knowledge fusion vector.

[0471] Figure 16 This is a schematic flowchart illustrating another method for training a natural language processing model according to embodiments of the present disclosure. Figure 16 As shown, fusing the text embedding vector and the knowledge embedding vector includes:

[0472] In step S1601, the text embedding vector and the knowledge embedding vector are fused according to the knowledge fusion model:

[0473]

[0474] Among them, f j Represents the knowledge fusion vector, e t Represents text embedding vector, e k Represents knowledge embedding vector, W t The weights W of the text embedding vectors are represented by these weights. e The weights of the knowledge embedding vector, b i Let σ represent the bias, i represent the i-th word in the corresponding sample text, and σ be the activation function.

[0475] In the embodiments of this application, W t and W e These are the weights of the text embedding vector and the knowledge embedding vector, respectively, which can be used as parameters for training. (W...) t and W e The settings can act as a filter, reducing the impact of interference or useless terms and knowledge on the fusion results.

[0476] In one embodiment, the fusion of the text embedding vector and the knowledge embedding vector can be performed using an activation function σ during the fusion process.

[0477] Figure 17A This is a schematic diagram illustrating the relationship between another fusion process and a natural language processing model according to embodiments of the present disclosure.

[0478] like Figure 17AAs shown, the triples in the sample text can be determined. The triples include two entities in the sample text and the relationship between the two entities. Then, the knowledge embedding vector corresponding to the triples can be determined. The knowledge embedding vector and the text embedding vector are then fused to obtain the knowledge fusion vector. Finally, the knowledge fusion vector can be used as a sample input to the natural language processing model.

[0479] like Figure 17A As shown, on the one hand, the text embedding vector corresponding to the sample text can be determined; on the other hand, triples can be determined in the sample text, and then for each triple, the knowledge embedding vector corresponding to the triple in the knowledge graph can be determined.

[0480] Next, we can use the knowledge fusion model... Embed the text vector e t and knowledge embedding vector e k By fusing the data, we obtain the knowledge embedding vector f. i Finally, the knowledge fusion vector is used as a sample input to train the natural language processing model.

[0481] In one embodiment, the natural language processing model includes at least one feature extraction layer, the feature extraction layer being used for:

[0482] A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix.

[0483] A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the sample text and the sample text.

[0484] A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the sample text.

[0485] Figure 17B This is a schematic diagram illustrating a feature extraction calculation process according to an embodiment of the present disclosure.

[0486] like Figure 17B As shown, from a computational perspective, the function of the feature extraction layer can be described as follows:

[0487] Multiplying the knowledge embedding vectors by the training matrix yields the first word vector matrix, the second word vector matrix, and the third word vector matrix.

[0488] Multiply the first word vector matrix and the second word vector matrix (transposed) to obtain the fourth word vector matrix;

[0489] The fifth word vector matrix is ​​obtained by normalizing the fourth word vector matrix;

[0490] The sixth word vector matrix is ​​obtained by weighted summation of the third word vector matrix based on the fifth word vector matrix.

[0491] Before normalizing the fourth word vector matrix, a scaling transformation can be performed, that is, the fourth word vector matrix is ​​divided by... d k The dimension of the word embedding vector.

[0492] Understandably, based on Figure 17B In this embodiment, the first word vector matrix is ​​the first knowledge fusion matrix, the third word vector matrix is ​​the second knowledge fusion matrix, the fourth or fifth word vector matrix is ​​the association matrix, and the sixth word vector matrix is ​​the weight matrix. The second word vector matrix is ​​the third knowledge fusion matrix. The third word vector matrix is ​​a weighted sum of the fifth word vector matrix, i.e., the fifth word vector matrix is ​​multiplied by the third word vector matrix. The training matrix is ​​a matrix whose parameters are continuously adjusted during training. The knowledge embedding vector is multiplied by the training matrix with different parameters obtained during training, thus obtaining the first, second, and third word vector matrices. All three word vector matrices include at least textual information and knowledge information (entities) from the knowledge embedding vectors, and therefore can all be called knowledge fusion matrices. To obtain the association matrix representing the association information between at least one entity in the sample text and each word in the sample text, it can be calculated by the inner product of two fusion matrices (i.e., multiplying the transposes of the first and second word vector matrices) or by methods such as covariance and Euclidean distance; no limitation is made here. By multiplying the obtained association relationships by their own fusion matrix (i.e., by weighted summation of the third word vector matrix based on the fifth word vector matrix), the weight of each word element (including entities) relative to the entire sample text can be obtained, i.e., the weight matrix. Matrix multiplication can also be multiplied by the transpose of the matrix, which can be adjusted according to the actual situation and is not limited here.

[0493] In one embodiment, the feature extraction layer can perform multiple parallel computations and then concatenate the results as a whole before inputting them into the next layer model. This allows the model to be mapped to different subspaces for computation, thereby improving the model's accuracy.

[0494] In one embodiment, the feature extraction layer includes, but is not limited to, an attention layer, a self-attention layer, or a multi-head self-attention layer. When the feature extraction layer includes a self-attention layer, the natural language processing model can be a Transformer model, a TransformerXL model, a BERT model, a GPT model, etc., and this application does not impose any limitations on it.

[0495] In one embodiment, the dimension of the natural language model is variable, and the dimension refers to the dimension of the knowledge fusion vector of the input model.

[0496] In one embodiment, the dimension of the natural language model is variable, including the ability to adjust the dimension of the natural language model according to preset conditions, such as user settings, hardware resources, etc.

[0497] In another embodiment, the dimensionality of the natural language model is variable, including: the natural language model includes at least two feature extraction layers, for example, at least a first feature extraction layer and a second feature extraction layer, wherein the first feature extraction layer and the second feature extraction layer have different dimensions. It is understood that the different dimensions of the first feature extraction layer and the second feature extraction layer mean that the dimensions of the association matrix and / or weight matrix of the first feature extraction layer and the second feature extraction layer are different.

[0498] In one embodiment, the at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer. In one embodiment, the dimension of the second feature extraction layer is smaller than the dimension of the first feature extraction layer. During deep learning of multiple feature extraction layers in a natural language model, gradually reducing the dimension of the feature extraction layers helps to reduce the number of parameters, improve computational efficiency, and enhance real-time performance. The number of times the dimension is reduced can be once or two to three times; this application does not limit this.

[0499] In one embodiment, the at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix used to partially mask the correlation matrix. For example, when the first feature extraction layer and the second feature extraction layer are both multi-head self-attention layers, the third feature extraction layer can be a masked multi-head self-attention layer. The following embodiments mainly illustrate this situation.

[0500] In one embodiment, the third feature extraction layer is located after the first feature extraction layer.

[0501] In one embodiment, the dimension of the third feature extraction layer can be smaller than the dimension of the first feature extraction layer, but equal to the dimension of the second feature extraction layer.

[0502] Figure 18 This is a schematic diagram illustrating the relationship between feature extraction layers according to embodiments of the present disclosure. Figure 19 This is a schematic diagram illustrating another relationship between feature extraction layers according to an embodiment of the present disclosure.

[0503] In one embodiment, such as Figure 18 As shown, the third feature extraction layer is located after the first feature extraction layer, or it can be located after the first extraction layer and after the second feature extraction layer.

[0504] It should be noted that in all embodiments of this disclosure, a layer following another means that the output of the preceding layer serves as the input of the following layer; a layer preceding another means that the input of the following layer includes the output of the preceding layer. The output of the preceding layer can be directly used as the input of the following layer, or it can be used as the input of the following layer after other calculations.

[0505] In one embodiment, such as Figure 19 As shown, the third feature extraction layer is located after the first feature extraction layer. This can be achieved by placing the third feature extraction layer alongside the second feature extraction layer after the first feature extraction layer, based on the second feature extraction layer being located after the first feature extraction layer.

[0506] The output of the target model can include the output after passing through all feature extraction layers, or it can be the output after passing through some feature extraction layers, such as the output after passing through only at least one first feature extraction layer, or the output after passing through at least one first feature extraction layer and at least one second feature extraction layer, or the output after passing through at least one first feature extraction layer and at least one third feature extraction layer. It is understood that if the output passes through at least one first feature extraction layer and at least one second feature extraction layer, then the third feature extraction layer is frozen and does not participate in the training process; similarly, if the output passes through at least one first feature extraction layer and at least one third feature extraction layer, then the second feature extraction layer is frozen and does not participate in the training process.

[0507] In one embodiment, when the training task is a generation task, the second feature extraction layer is frozen; when the training task is a semantic understanding task, the third feature extraction layer is frozen. The training task may include the aforementioned pre-training tasks (which may include the three pre-training tasks mentioned above, as well as generation tasks), and may also include subsequent fine-tuning tasks. For example, the first, second, and third pre-training tasks can be considered as semantic understanding tasks; therefore, a fifth pre-training task can also be constructed, which can be used to generate a prediction task of the content to be input based on the input text. This fifth pre-training task can be considered as a generation task.

[0508] The third feature extraction layer can include a masking matrix to perform prediction tasks. In this case, the second feature extraction layer can be used for semantic understanding, while the third feature extraction layer can be used to predict the following content, such as predicting the text to be input based on the already input text and generating the text to be input.

[0509] Therefore, when the training task is a generation task, it can be achieved based on the third feature extraction layer, so the second feature extraction layer can be frozen; when the training task is a semantic understanding task, it can be achieved based on the second feature extraction layer, so the third feature extraction layer can be frozen. This simplifies the training process.

[0510] In one embodiment, a first dimension transformation layer is further included between the first feature extraction layer and the second feature extraction layer. This layer is used to transform the dimension of the weight matrix obtained by the first feature extraction layer and use the dimension-transformed output matrix as the input of the second feature extraction layer, thereby realizing the dimension transformation between the feature extraction layers.

[0511] Since the dimensions of the first feature extraction layer and the second feature extraction layer can be different, for example, the dimension of the first feature extraction layer is greater than the dimension of the second feature extraction layer, such as the dimension of the first feature extraction layer being 4096 and the dimension of the second feature extraction layer being 768.

[0512] In order to input the output of the first feature extraction layer into the second feature extraction layer, it is first necessary to transform the dimension of the output of the first feature extraction layer into the dimension of the second feature extraction layer. Therefore, a first dimension transformation layer can be set between the first and second feature extraction layers to transform the dimension of the weight matrix obtained by the first feature extraction layer and obtain the dimension-reduced output matrix as the input of the second feature extraction layer.

[0513] For example, when the dimension of the first feature extraction layer is 4096 and the dimension of the second feature extraction layer is 768, the dimension transformation can specifically be dimensionality reduction.

[0514] In one embodiment, the first dimension transformation layer may consist of at least one dimension transformation unit, which includes a linear layer.

[0515] For example, for a dimension transformation unit, the output of the first feature extraction layer includes an output embedding vector. The output embedding vector is input to a linear layer. The linear layer performs dimensionality reduction on the output of the first feature extraction layer, which can adjust the dimension of the output of the first feature extraction layer. Then, the dimensionality-reduced output of the first feature extraction layer is input to the next feature extraction layer, such as the second feature extraction layer and / or the third feature extraction layer.

[0516] In one embodiment, after processing by linear layers in multiple dimensionality transformation units, the output of the first feature extraction layer can be gradually reduced to the same dimension as the second feature extraction layer.

[0517] Figure 20 This is a schematic diagram illustrating a dimension transformation according to an embodiment of the present disclosure.

[0518] In one embodiment, such as Figure 20 As shown, a second dimension transformation layer is further included between the first feature extraction layer and the third feature extraction layer. This second layer transforms the dimension of the weight matrix obtained by the first feature extraction layer, resulting in a dimension-reduced output matrix that serves as the input to the third feature extraction layer. It should be noted that the first and second dimension transformation layers can be configured as follows: Figure 20 The images show different dimensional transformation layers, but they can also be set to the same dimensional transformation layer as needed.

[0519] In one embodiment, the dimension of the third feature extraction layer can be the same as the dimension of the second feature extraction layer, for example, 768. Since the dimension of the output of the first feature extraction layer needs to be transformed to the dimension of the third feature extraction layer in order to input the output of the first feature extraction layer into the third feature extraction layer, a second dimension transformation layer can be set between the first and third feature extraction layers. This transforms the dimension of the weight matrix obtained by the first feature extraction layer, resulting in a dimension-reduced output matrix that serves as the input to the third feature extraction layer.

[0520] The principle of the second-dimensional transformation layer is similar to that of the first-dimensional transformation layer, and will not be elaborated here.

[0521] Figure 21 This is a schematic diagram illustrating another dimension transformation according to an embodiment of the present disclosure.

[0522] In one embodiment, such as Figure 21 As shown, a splitting layer is also included before the second feature extraction layer and the third feature extraction layer, which is used to split the output of the first feature extraction layer into knowledge information and text information;

[0523] The first dimension transformation layer is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the second feature extraction layer, or to transform the dimension of the knowledge information to obtain a dimension-reduced matrix as the input of the second feature extraction layer.

[0524] The first feature extraction layer and the third feature extraction layer further include a second dimension transformation layer, which is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the third feature extraction layer, or to transform the dimension of the text information to obtain a dimension-reduced matrix as the input of the third feature extraction layer.

[0525] In one embodiment, since the input of the first feature extraction layer includes a knowledge fusion vector, which is obtained by fusing the text embedding vector corresponding to the sample text and the knowledge embedding vector corresponding to the triple, the output of the first feature extraction layer will also include knowledge information related to the knowledge embedding vector and text information related to the text embedding vector.

[0526] Furthermore, as can be seen from the foregoing embodiments, the second feature extraction layer is mainly used for semantic understanding tasks, while the third feature extraction layer is mainly used for generation tasks. Since semantic understanding tasks mainly focus on the words and relationships between words in the text, they can be expressed relatively accurately through knowledge information. In contrast, generation tasks mainly focus on the words themselves in the text, with less attention paid to the relationships between words, and can therefore be expressed relatively accurately through textual information.

[0527] Therefore, for the second feature extraction layer, only knowledge information can be used as input, so the first dimension transformation layer can only perform dimension transformation on the knowledge information before inputting it into the second feature extraction layer; for the third feature extraction layer, only text information can be used as input, so the second dimension transformation layer can only perform dimension transformation on the text information before inputting it into the third feature extraction layer. By setting up split layers, redundant data can be reduced, further improving computational efficiency.

[0528] In one embodiment, after inserting triples from the sample text into the sample text to obtain the knowledge fusion vector, the position of the triple corresponding to the inserted text can be marked. After obtaining the output vector matrix through the first feature extraction layer, the splitting layer can split the matrix information corresponding to the knowledge information (e.g., triples) and text information in the output vector matrix according to the position marked for the triples, thereby generating a knowledge information matrix and a sample text information matrix.

[0529] Of course, the text information and knowledge information can also be transformed together through the first dimension transformation layer and then input into the second feature extraction layer, or the text information and knowledge information can be transformed together through the second dimension transformation layer and then input into the third feature extraction layer.

[0530] In one embodiment, where the natural language processing model includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer, the natural language processing model may further include a residual network, a feedforward neural network layer, etc.

[0531] Figure 22A This is a schematic diagram of the structure of a first feature extraction layer according to an embodiment of the present disclosure.

[0532] In one embodiment, such as Figure 22A As shown, the natural language processing model includes:

[0533] The first feature extraction layer and the first residual network (the specific calculation process includes superposition and normalization (Add & Norm)) layer, wherein the input of the first feature extraction layer includes the knowledge fusion vector, and the input of the first residual network includes the knowledge fusion vector and the output of the feature extraction layer;

[0534] The natural language processing model also includes:

[0535] The system consists of a first feedforward neural network layer and a second residual network. The input of the first feedforward neural network layer includes the output of the first module, and the input of the second residual network includes the output of the first feedforward neural network layer and the output of the first module.

[0536] In one embodiment, the positional encoding of the lexical units in the sample text can also be determined, and then the positional encoding features and the knowledge fusion vector are superimposed and input into the first feature extraction layer.

[0537] In another embodiment, the knowledge fusion vector input to the first feature extraction layer may not include positional encoding; instead, a sequential encoding matrix is ​​used in the first feature extraction layer to supplement positional information. This sequential encoding matrix consists of an upper triangular matrix and a lower triangular matrix, the upper and lower triangular matrices being different, and no element in the sequential encoding matrix is ​​zero. In one embodiment, the elements of both the upper and lower triangular matrices are identical. Further, the elements of the upper triangular matrix can be 1 or -1, and the elements of the lower triangular matrix can be -1 or 1. The sequential encoding matrix can be located between the correlation matrix and the weight matrix (serving as positional encoding), such as by multiplying the correlation matrix and the sequential encoding matrix to obtain positional information. Through the upper and lower triangular matrices, the sequential encoding matrix allows the model to learn the relative positional information of the sample text and the sequential relationship between at least one entity and the sample text, thereby improving the model training effect.

[0538] Figure 22B This is a schematic diagram illustrating the calculation process of a first feature extraction layer according to an embodiment of the present disclosure.

[0539] like Figure 22B As shown, from the perspective of the computational process, the function of the first feature extraction layer can be described as follows:

[0540] Multiplying the knowledge embedding vectors by the training matrix yields the first word vector matrix, the second word vector matrix, and the third word vector matrix.

[0541] Multiply the first word vector matrix and the second word vector matrix to obtain the fourth word vector matrix;

[0542] The fourth word vector matrix is ​​encoded using a sequential encoding matrix to obtain the fifth word vector matrix;

[0543] The sixth word vector matrix is ​​obtained by padding the fifth word vector matrix.

[0544] The sixth word vector matrix is ​​normalized to obtain the seventh word vector matrix;

[0545] The eighth word vector matrix is ​​obtained by weighted summation of the third word vector matrix based on the seventh word vector matrix.

[0546] In one embodiment, after receiving the knowledge fusion vector, the first feature extraction layer can repeat the functions of the first feature extraction layer, the first residual network, the first feedforward network, and the second residual network multiple times before outputting the result.

[0547] Figure 23This is a schematic diagram of the structure of a first sub-model according to an embodiment of the present disclosure. Figure 24 This is a schematic diagram of the structure of a second sub-model according to an embodiment of the present disclosure.

[0548] In one embodiment, the second feature extraction layer includes a first sub-model and / or a second sub-model;

[0549] like Figure 23 As shown, the first sub-model includes:

[0550] The second feature extraction layer and the third residual network, wherein the input of the second feature extraction layer includes the output of the second residual network, and the input of the third residual network includes the output of the second residual network and the output of the third feature extraction layer;

[0551] The second feedforward neural network layer and the fourth residual network are provided. The input of the second feedforward neural network layer includes the output of the third residual network, and the input of the fourth residual network includes the output of the second feedforward neural network layer and the output of the third residual network.

[0552] In one embodiment, the positional encoding of the lexical units in the sample text can also be determined, and then the superimposed embedding vector obtained by superimposing the positional encoding features and the output of the first feature extraction layer is input into the first sub-model.

[0553] In one embodiment, after receiving the knowledge fusion vector, the first sub-model can repeat the functions of the second feature extraction layer, the third residual network, the second feedforward neural network layer, and the fourth residual network multiple times before outputting the result.

[0554] like Figure 24 As shown, the second sub-model includes:

[0555] The third feature extraction layer and the fifth residual network are provided. The input of the third feature extraction layer includes the output of the second residual network, and the input of the fifth residual network includes the output of the second residual network and the output of the third feature extraction layer.

[0556] A third feedforward neural network layer and a sixth residual network, wherein the input of the third feedforward neural network layer includes the output of the fifth residual network, and the input of the sixth residual network includes the output of the third feedforward neural network layer and the output of the fifth residual network.

[0557] In one embodiment, the positional encoding of the lexical units in the sample text can also be determined, and then the superimposed embedding vector obtained by superimposing the positional encoding features and the output of the first feature extraction layer is input into the second sub-model.

[0558] In one embodiment, after receiving the knowledge fusion vector, the second sub-model can repeat the functions of the third feature extraction layer, the fifth residual network, the third feedforward neural network layer, and the sixth residual network multiple times before outputting the result.

[0559] In one embodiment, when the training task is a generation task, the second feature extraction layer is frozen; when the training task is a semantic understanding task, the third feature extraction layer is frozen, specifically meaning:

[0560] The first sub-model is frozen when the training task is a generation task, and the second sub-model is frozen when the training task is a semantic understanding task.

[0561] The target processing model training method provided in this application involves natural language text processing. Specifically, it can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (such as sample text and initial knowledge fusion vector in this application) to finally obtain a trained target processing model.

[0562] Furthermore, the text processing method provided in this application embodiment can utilize the above-mentioned trained target processing model to input input data (such as the text to be processed in this application) into the trained target processing model and obtain output data (such as the processing result corresponding to the target task in this application).

[0563] It should be noted that the target processing model training method and the text processing method provided in the embodiments of this application are inventions based on the same concept, and can also be understood as two parts of a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0564] Embodiments of this disclosure also propose a text processing method, including:

[0565] Get the target text;

[0566] Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities;

[0567] The target text is processed based on the triples to obtain a knowledge fusion vector;

[0568] The knowledge fusion vector is input into the target model (e.g., the target model in the method described in any of the above embodiments) to obtain the processing result of the target text.

[0569] The processing result includes at least one of the following: translation information of the text data, response information of the text data, classification information of the text data, or correlation information between the text data and other reference text data.

[0570] In one embodiment, processing the target text based on the triples to obtain a knowledge fusion vector includes fusing the target text and the triples in the target text to obtain a knowledge fusion vector.

[0571] In one embodiment, inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes:

[0572] The triples in the target text are inserted into the target text to obtain the knowledge fusion vector.

[0573] In one embodiment, inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes:

[0574] Determine the text embedding vector corresponding to the target text;

[0575] Determine the knowledge embedding vector corresponding to the triple;

[0576] The text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

[0577] In one embodiment, the target model includes at least one feature extraction layer, the feature extraction layer being used for:

[0578] A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix.

[0579] A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the target text and the target text.

[0580] A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the target text.

[0581] In one embodiment, the at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

[0582] In one embodiment, the at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

[0583] A text processing apparatus includes a processor, the processor being configured to:

[0584] Get the target text;

[0585] Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities;

[0586] The target text is processed based on the triples to obtain a knowledge fusion vector;

[0587] The knowledge fusion vector is input into the target model to obtain the processing result of the target text.

[0588] In one embodiment, the device includes, but is not limited to, a terminal and a server. The terminal includes, but is not limited to, electronic devices such as mobile phones, tablets, wearable devices, personal computers, and Internet of Things devices. The server includes, but is not limited to, a local server and a cloud server.

[0589] In one embodiment, the processor is configured to fuse the target text and the triples in the target text to obtain a knowledge fusion vector.

[0590] In one embodiment, the processor is configured to insert triples from the target text into the target text to obtain the knowledge fusion vector.

[0591] In one embodiment, the processor is configured to determine a text embedding vector corresponding to the target text; determine a knowledge embedding vector corresponding to the triple; and fuse the text embedding vector and the knowledge embedding vector to obtain the knowledge fusion vector.

[0592] In one embodiment, the target model includes at least one feature extraction layer, the feature extraction layer being used for:

[0593] A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix.

[0594] A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the target text and the target text.

[0595] A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the target text.

[0596] In one embodiment, the at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

[0597] In one embodiment, the at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

[0598] After the device trains the target model based on the method described in the above embodiments, it can store the target model, and then subsequently process the input target text. For example, the processing of the target text includes, but is not limited to, at least one of the following:

[0599] Semantic understanding, for example, the ability to generate the semantics of target text;

[0600] Translation, for example, can generate the corresponding content of the target text in other languages;

[0601] Predictive generation, for example, can predict what will appear next in target text and display the predicted content;

[0602] Word order adjustment, for example, can adjust the word order of the target text to the correct word order.

[0603] In one embodiment, based on the obtained target model, a fine-tuning model can be further trained to obtain an overall model. The output of the overall model includes at least one of the following: translation information of the text data, response information of the text data, classification information of the text data (e.g., sentiment classification), or correlation information between the text data and other reference text data. Different fine-tuning models can be trained to make the overall model output different information.

[0604] In one embodiment, the apparatus further includes:

[0605] The interaction module is used to determine whether to obtain the triples in the target text based on the knowledge graph according to the user's operation;

[0606] Specifically, when determining the triples in the target text based on the knowledge graph, the triples corresponding to the target text in the knowledge graph are determined.

[0607] When it is determined that the triples in the target text are not obtained based on a knowledge graph, the triples in the target text are determined.

[0608] Figure 25 This is an interactive schematic diagram illustrating an embodiment of the present disclosure.

[0609] like Figure 25 As shown, for example, the interaction module includes a screen with integrated touch functionality, displaying an input area and an output area. Users can input target text in the input area, and after the target text is processed by the target model, the processing result can be displayed in the output area.

[0610] In one embodiment, the interaction module is further configured to display the triples corresponding to the target text in the knowledge graph;

[0611] Specifically, when determining the target model obtained by training based on the knowledge graph, the triplet corresponding to the target text in the knowledge graph is determined;

[0612] When it is determined that the target model is not obtained by training based on a knowledge graph, the triples in the target text are determined.

[0613] In one embodiment, the user can also operate on the interactive module to control the process of training the natural language processing model. For example, options can be provided on the screen for the user to determine whether to train the target model based on a knowledge graph.

[0614] If the user chooses to train based on a knowledge graph, then the triples corresponding to the target text in the knowledge graph can be determined; if the user chooses not to train based on a knowledge graph, then the triples in the target text can be determined directly, without considering the knowledge graph.

[0615] In one embodiment, the interaction module is further configured to, when determining that the target model is obtained by training based on the knowledge graph, display the triples corresponding to the target text in the knowledge graph; and

[0616] The displayed triples are adjusted based on the user's actions, and the adjusted triples are used as the triples corresponding to the target text in the knowledge graph.

[0617] In one embodiment, the interaction module can also display the determined triples. The user can determine whether the triples are reasonable. If they are not reasonable, the user can adjust the displayed triples. The interaction module will then use the adjusted triples as the triples corresponding to the target text in the knowledge graph.

[0618] In one embodiment, the interaction module is further configured to receive user-input triples as the triples corresponding to the target text in the knowledge graph when determining that the target model is obtained by training based on the knowledge graph. That is, the user can directly input triples as needed as the triples corresponding to the target text in the knowledge graph.

[0619] In one embodiment, the interaction module is further configured to select the target model based on user operation, wherein the target model includes at least one of the following:

[0620] CNN, RNN, GRU, LSTM, Transformer, Transformer-XL.

[0621] Users can interact with the module to select the type of target model. For example, the drop-down menu can display the types of target models mentioned above, and users can choose one of them for training.

[0622] In one embodiment, when the target model includes Transformer and / or Transformer-XL, the interaction module is further configured to determine the fine-tuning model based on user actions.

[0623] When the natural language processing model includes Transformer and / or Transformer-XL, the training process mainly involves pre-training, followed by fine-tuning the model based on the specific task. In this case, users can interact with the module to select the type of fine-tuning model; for example, a dropdown menu may display several types of fine-tuning models, allowing users to choose one for training.

[0624] In one embodiment, the fine-tuning model includes at least one of the following:

[0625] CNN, RNN, GRU, LSTM.

[0626] Embodiments of this disclosure also propose a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the training method of the natural language processing model described in any of the above embodiments, and / or the steps in the text processing method described in any of the above embodiments.

[0627] Figure 26This is a schematic block diagram illustrating an apparatus 2600 for training a natural language processing model according to embodiments of the present disclosure. For example, apparatus 2600 may be a terminal, such as a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.; it may also be a server.

[0628] Reference Figure 26 The device 2600 may include one or more of the following components: a processing component 2602, a memory 2604, a power supply component 2606, a multimedia component 2608, an audio component 2610, an input / output (I / O) interface 2612, a sensor component 2614, and a communication component 2616.

[0629] Processing component 2602 typically controls the overall operation of device 2600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 2602 may include one or more processors 2620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 2602 may include one or more modules to facilitate interaction between processing component 2602 and other components. For example, processing component 2602 may include a multimedia module to facilitate interaction between multimedia component 2608 and processing component 2602.

[0630] Memory 2604 is configured to store various types of data to support the operation of device 2600. Examples of this data include instructions for any application or method operating on device 2600, contact data, phonebook data, messages, pictures, videos, etc. Memory 2604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0631] Power supply component 2606 provides power to various components of device 2600. Power supply component 2606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 2600.

[0632] Multimedia component 2608 includes a screen that provides an output interface between the device 2600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 2608 includes a front-facing camera and / or a rear-facing camera. When the device 2600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0633] Audio component 2610 is configured to output and / or input audio signals. For example, audio component 2610 includes a microphone (MIC) configured to receive external audio signals when device 2600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 2604 or transmitted via communication component 2616. In some embodiments, audio component 2610 also includes a speaker for outputting audio signals.

[0634] I / O interface 2612 provides an interface between processing component 2602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0635] Sensor assembly 2614 includes one or more sensors for providing status assessments of various aspects of device 2600. For example, sensor assembly 2614 may detect the on / off state of device 2600, the relative positioning of components such as the display and keypad of device 2600, changes in the position of device 2600 or a component of device 2600, the presence or absence of user contact with device 2600, the orientation or acceleration / deceleration of device 2600, and temperature changes of device 2600. Sensor assembly 2614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 2614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 2614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0636] Communication component 2616 is configured to facilitate wired or wireless communication between device 2600 and other devices. Device 2600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 2616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 2616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0637] In an exemplary embodiment, the apparatus 2600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0638] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 2604 including instructions, which can be executed by a processor 2620 of the device 2600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0639] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0640] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0641] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0642] The methods and apparatus provided in the embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A training method for a natural language processing model, characterized in that, include: Obtain sample text in natural language; Determine the triples in the sample text, wherein the triples include two entities in the sample text and the relationship between the two entities; The sample text is processed based on the triples to obtain a knowledge fusion vector; The knowledge fusion vector is input into a natural language processing model for training to obtain the target model; The natural language processing model includes at least one feature extraction layer, which is used for: A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix. A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the sample text and the sample text. A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the sample text.

2. The method according to claim 1, characterized in that, The step of inputting the knowledge fusion vector into a natural language processing model for training to obtain the target model includes: Construct at least one pre-training task, and train the natural language processing model based on the at least one pre-training task to obtain the target model.

3. The method according to claim 2, characterized in that, The at least one pre-training task includes: The first pre-training task includes a masking task, which includes masking the sample text based on the triples to obtain the knowledge fusion vector.

4. The method according to claim 3, characterized in that, The process of masking the sample text based on the triples to obtain the knowledge fusion vector includes: The tokens corresponding to the triples in the sample text are masked to obtain the knowledge fusion vector.

5. The method according to claim 4, characterized in that, The step of masking the tokens corresponding to the triples in the sample text to obtain the knowledge fusion vector includes: The tokens corresponding to entities and relations in the triples in the sample text are masked to obtain the knowledge fusion vector.

6. The method according to claim 3, characterized in that, The knowledge fusion vector is obtained by masking the sample text based on the triplet, including: The sample text is divided into multiple sub-texts based on the lexical units corresponding to the triples, and the multiple sub-texts are masked to obtain the knowledge fusion vector.

7. The method according to claim 6, characterized in that, The step of dividing the sample text into multiple sub-texts based on the lexical units corresponding to the triples includes: The sample text is divided into multiple subtexts by dividing the word at the beginning or end of the corresponding triplet.

8. The method according to claim 6, characterized in that, The process of masking the multiple sub-texts to obtain the knowledge fusion vector includes: In the subtext, the tokens corresponding to the triples are masked to obtain the knowledge fusion vector.

9. The method according to claim 6, characterized in that, When the sample text contains n triples, the step of dividing the sample text into multiple subtexts based on the lexical units corresponding to the triples includes: The sample text is divided into multiple subtexts according to the word corresponding to the m-th triplet among the n triplets, where n is an integer greater than 1, m is a positive integer less than or equal to n, and m is a specified value or a random value.

10. The method according to claim 6, characterized in that, When the sample text contains n triples, the steps of dividing the sample text into multiple sub-texts based on the lexical units corresponding to the triples, and masking the lexical units corresponding to the triples in the sample text to obtain the knowledge fusion vector include: Starting from i=1 and continuing until i=n, ​​repeat the following steps; The sample text is divided according to the word element corresponding to the i-th triplet among the n triplets to obtain the i-th sub-text group composed of multiple sub-texts corresponding to the i-th triplet; The tokens corresponding to the triples in the i-th subtext group are masked to obtain the knowledge fusion vector; where n is an integer greater than 1 and i is a positive integer less than or equal to n.

11. The method according to claim 10, characterized in that, In the i-th subtext group, the tokens corresponding to the triples are masked and then trained as one round of training. This training is performed for k rounds, where k is an integer greater than or equal to n.

12. The method according to claim 1, characterized in that, Determining the triples in the sample text includes: Determine the triples corresponding to the sample text in the knowledge graph, or obtain the triples in the sample text based on the triple extraction model.

13. The method according to claim 12, characterized in that, Determining the triplet corresponding to the sample text in the knowledge graph includes: Determine the domain to which the sample text belongs; Determine the knowledge graph for the domain; Determine the triples corresponding to the sample text in the knowledge graph of the domain.

14. The method according to claim 12, characterized in that, The knowledge graph includes at least one of the following: Pre-built knowledge graph; Pre-stored knowledge graph.

15. The method according to claim 14, characterized in that, In the case where the knowledge graph includes a pre-constructed knowledge graph, the method further includes, before determining the triples corresponding to the sample text in the knowledge graph: A triplet extraction model is determined, wherein the triplet extraction model is used to extract triples from the sample text; The knowledge graph is constructed based on the extracted triples.

16. The method according to claim 15, characterized in that, The triplet extraction model includes: A label determination model is used to determine the label of the token in the sample text; wherein the label is used to determine the entity in the sample text; A relationship determination model is used to determine the relationship between at least two entities determined by the label determination model and / or the position of the relationship in the sample text.

17. The method according to claim 16, characterized in that, The label determination model includes: A first coding layer, the input of which includes the terminology; A feature extraction layer, wherein the input of the feature extraction layer includes the output of the first encoding layer; A first activation layer, the input of which includes the output of the feature extraction layer; The first normalization layer, the input of which includes the output of the first activation layer; The first output layer, whose input includes the output of the first normalization layer, is used to output the label; The input corresponding to the (i+1)th word in the sample text in the first activation layer also includes the output corresponding to the ith word in the sample text in the first output layer, where i is an integer greater than or equal to 1.

18. The method according to claim 17, characterized in that, The relationship determination model includes: A second coding layer, the input of which includes the entity pairs; A fully connected layer, the input of which includes the output of the second coding layer and the output of the feature extraction layer; A second activation layer, the input of which includes the output of the fully connected layer; The second normalization layer, whose input includes the output of the second activation layer, is used to output the relation and / or the position of the relation in the sample text.

19. The method according to claim 12, characterized in that, Determining the triplet corresponding to the sample text in the knowledge graph includes: Identify multiple sub-knowledge graphs; Entities with the same semantics in different sub-knowledge graphs are merged to obtain a merged knowledge graph; Determine the triples corresponding to the sample text in the merged knowledge graph.

20. The method according to claim 19, characterized in that, Merging semantically identical entities from different sub-knowledge graphs includes: Based on the structural feature vector h corresponding to the two entities s and attribute feature vector h a Calculate the distance between the embedding vectors of any two entities in different sub-knowledge graphs. If the distance is less than a distance threshold, then merge the two entities.

21. The method according to claim 20, characterized in that, The structural feature vector h of the entity s and attribute feature vector h a The structural feature vector h of the previous entity s-1 and attribute feature vector h a-1 The connectivity matrix A is calculated using a graph neural network (GCN) model.

22. The method according to claim 2, characterized in that, The at least one pre-training task also includes: The second pre-training task is used to split the sample document into fragments at at least one level of granularity, randomly arrange and combine the fragments after each level of splitting, and train based on the rearranged samples to output the relationship type between sentences in the rearranged samples.

23. The method according to claim 2, characterized in that, The at least one pre-training task also includes: The third pre-training task is used to perform semantic recognition training by masking the lexical units corresponding to the triples in the sample text.

24. The method according to claim 2, characterized in that, The step of training the natural language processing model according to the at least one pre-training task to obtain the target model includes: Perform at least one round of the following steps: Pre-training tasks are selected and superimposed according to a preset order. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks.

25. The method according to claim 2, characterized in that, The step of training the natural language processing model according to the at least one pre-training task to obtain the target model includes: Pre-training tasks are randomly selected and superimposed. After each selection of a pre-training task, the natural language processing model is trained based on the superimposed pre-training tasks. The learning rate of the natural language processing model is positively correlated with the number of samples input to the natural language processing model for each training iteration.

26. The method according to claim 1, characterized in that, The relationship between the learning rate of the natural language processing model during training and the number of samples input into the natural language processing model for each training iteration is as follows: ； Where X is the batchsize of the samples input to the natural language processing model for each training iteration, and Y is the learning rate.

27. The method according to claim 1, characterized in that, The method further includes: Determine the target sample set based on the target training task; The overall model is trained based on the target sample set, wherein the overall model includes the target model and the fine-tuned model.

28. The method according to claim 27, characterized in that, The fine-tuning model includes at least one of the following: Convolutional Neural Network (CNN) model; Recurrent Neural Network (RNN) model; Gate-controlled loop unit (GRU) model; Fully connected layer (FC); Long Short-Term Memory (LSTM) model.

29. The method according to claim 1, characterized in that, The process of processing the sample text based on the triples to obtain the knowledge fusion vector includes: The sample text and the triples in the sample text are fused to obtain a knowledge fusion vector.

30. The method according to claim 29, characterized in that, The step of fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes: The triples in the sample text are inserted into the sample text to obtain the knowledge fusion vector.

31. The method according to claim 29, characterized in that, The step of fusing the sample text and the triples in the sample text to obtain a knowledge fusion vector includes: Determine the text embedding vector corresponding to the sample text; Determine the knowledge embedding vector corresponding to the triple; The text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

32. The method according to claim 31, characterized in that, The triplet includes at least one of the following: Head entity, relation, tail entity.

33. The method according to claim 32, characterized in that, Determining the knowledge embedding vector corresponding to the triplet includes: The embedding vectors of the head entity, the relation, and the tail entity are fused to obtain the knowledge embedding vector: ； in, Represents knowledge embedding vectors, The embedding vector of the head entity, The embedding vector representing the relationship, The embedding vector representing the tail entity, This represents the concatenation function.

34. The method according to claim 31, characterized in that, The text embedding vector includes the text embedding vector itself, and also includes at least one of the following: Sentence embedding vector, position embedding vector corresponding to the word element of the triple, and task type embedding vector.

35. The method according to claim 31, characterized in that, The knowledge embedding vector includes at least one of the following: The embedding vector of the entity belonging to the triple; The embedding vector of the relation belonging to the triple.

36. The method according to claim 31, characterized in that, The fusion of the text embedding vector and the knowledge embedding vector includes: The text feature vector is obtained by extracting features from the text embedding vector through the feature extraction layer; The knowledge feature vector is obtained by extracting features from the knowledge embedding vector through the feature extraction layer; The text feature vector and the knowledge feature vector are fused to obtain the knowledge fusion vector.

37. The method according to claim 36, characterized in that, The fusion of the text feature vector and the knowledge feature vector includes: The text feature vector and the knowledge feature vector are fused through multiple layers of neural network.

38. The method according to claim 31, characterized in that, The fusion of the text embedding vector and the knowledge embedding vector includes: The text embedding vector and the knowledge embedding vector are fused according to the knowledge fusion model: ； in, Represents knowledge fusion vectors, Represents text embedding vectors, Represents knowledge embedding vectors, The weights of the text embedding vectors are represented. The weights of the knowledge embedding vector are represented. The symbol represents the bias, and i represents the i-th word in the corresponding sample text.

39. The method according to claim 1, characterized in that, The at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

40. The method according to claim 39, characterized in that, The at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

41. The method according to claim 40, characterized in that, The third feature extraction layer is located after the first feature extraction layer.

42. The method according to claim 41, characterized in that, When the training task is a generation task, the second feature extraction layer is frozen; when the training task is a semantic understanding task, the third feature extraction layer is frozen.

43. The method according to claim 41, characterized in that, A splitting layer is included before the second feature extraction layer and the third feature extraction layer, which is used to split the output of the first feature extraction layer into knowledge information and text information. The first feature extraction layer and the second feature extraction layer further include a first dimension transformation layer, which is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the second feature extraction layer, or to transform the dimension of the knowledge information to obtain a dimension-reduced matrix as the input of the second feature extraction layer. The first feature extraction layer and the third feature extraction layer further include a second dimension transformation layer, which is used to transform the dimension of the weight matrix obtained by the first feature extraction layer to obtain a dimension-reduced output matrix as the input of the third feature extraction layer, or to transform the dimension of the text information to obtain a dimension-reduced matrix as the input of the third feature extraction layer.

44. A text processing method, characterized in that, include: Get the target text; Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities; The target text is processed based on the triples to obtain a knowledge fusion vector; The knowledge fusion vector is input into the target model to obtain the processing result of the target text, wherein the target model includes at least one feature extraction layer, the feature extraction layer being used for: A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix. A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the target text and the target text. A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the target text.

45. The method according to claim 44, characterized in that, The process of processing the target text based on the triples to obtain the knowledge fusion vector includes: The target text and the triples in the target text are fused to obtain a knowledge fusion vector.

46. ​​The method according to claim 45, characterized in that, The step of inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes: The triples in the target text are inserted into the target text to obtain the knowledge fusion vector.

47. The method according to claim 45, characterized in that, The step of inputting the knowledge fusion vector into the target model to obtain the processing result of the target text processing includes: Determine the text embedding vector corresponding to the target text; Determine the knowledge embedding vector corresponding to the triple; The text embedding vector and the knowledge embedding vector are fused to obtain the knowledge fusion vector.

48. The method according to claim 47, characterized in that, The at least one feature extraction layer includes at least a first feature extraction layer and a second feature extraction layer, wherein the dimension of the correlation matrix in the second feature extraction layer is variable relative to the dimension of the weight matrix in the first feature extraction layer.

49. The method according to claim 48, characterized in that, The at least one feature extraction layer further includes at least one third feature extraction layer, the third feature extraction layer including a masking matrix, the masking matrix being used to partially mask the correlation matrix.

50. A text processing device, characterized in that, Includes a processor, the processor being configured to: Get the target text; Determine the triples in the target text, wherein the triples include two entities in the target text and the relationship between the two entities; The target text is processed based on the triples to obtain a knowledge fusion vector; The knowledge fusion vector is input into the target model to obtain the processing result of the target text, wherein the target model includes at least one feature extraction layer, the feature extraction layer being used for: A linear transformation is performed on the knowledge fusion vector to obtain at least two knowledge fusion matrices, including a first knowledge fusion matrix and a second knowledge fusion matrix. A correlation matrix is ​​determined based on the first knowledge fusion matrix, and the correlation matrix is ​​used to characterize the correlation relationship information between at least one entity in the target text and the target text. A weight matrix is ​​determined based on the second knowledge fusion matrix and the association matrix, and the weight matrix is ​​used to characterize the weight information between the at least one entity and the target text.

51. The apparatus according to claim 50, characterized in that, Also includes: The interaction module is used to determine whether to obtain the triples in the target text based on the knowledge graph according to the user's operation; Specifically, when determining the triples in the target text based on the knowledge graph, the triples corresponding to the target text in the knowledge graph are determined. When it is determined that the triples in the target text are not obtained based on a knowledge graph, the triples in the target text are determined.

52. The apparatus according to claim 51, characterized in that, The interaction module is also used to display the triplet corresponding to the target text in the knowledge graph; and The displayed triples are adjusted based on the user's actions, and the adjusted triples are used as the triples corresponding to the target text in the knowledge graph.

53. The apparatus according to claim 51, characterized in that, The interaction module is also used to receive user-input triples as the triples corresponding to the target text in the knowledge graph when determining the target model obtained by training based on the knowledge graph.

54. The apparatus according to claim 51, characterized in that, The interaction module is further configured to select the target model based on user operation, wherein the target model includes at least one of the following: CNN, RNN, GRU, LSTM, Transformer, Transformer-XL.

55. The apparatus according to claim 54, characterized in that, When the target model includes Transformer and / or Transformer-XL, the interaction module is also used to determine the fine-tuning model based on user operations.

56. The apparatus according to claim 55, characterized in that, The fine-tuning model includes at least one of the following: CNN, RNN, GRU, LSTM.

57. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the training method for the natural language processing model as described in claims 1 to 43 and / or the text processing method as described in any one of claims 44 to 49.

Images