Language model pre-training method and device, computer storage medium and electronic device

Abstract

The present application discloses a cross-language language model pre-training method and device, a cross-language named entity recognition method based on transfer learning, a cross-language sentiment analysis method based on transfer learning, a cross-language sentence relationship inference method based on transfer learning, a computer storage medium and an electronic device, wherein the pre-training method comprises: inputting text information into a pre-trained language model, obtaining a first context word vector of a word in the text information; selecting a word sense vector matching the first context word vector in a constructed word sense clustering set; determining a second context word vector corresponding to the word sense vector in a context word vector set output by the pre-trained language model; determining the word sense vector probability of the word based on the second context word vector and the first context word vector. Thus, the context word vectors are aligned at the word sense level granularity.

Description

Technical Field

[0001] The present application relates to natural language processing in the field of computer technology, and more specifically to a cross-language language model pre-training method and device. The present application also relates to a cross-language named entity recognition method based on transfer learning, a cross-language sentiment analysis method based on transfer learning, a cross-language sentence relationship inference method based on transfer learning, and a computer storage medium and electronic device. Background Art

[0002] Natural language processing (NLP) is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods for effective communication between humans and computers through natural language. Furthermore, natural language processing is a science that covers linguistics, computer science, and mathematics.

[0003] With the continuous development of artificial intelligence technology and big data, the application scope of natural language processing is becoming more and more extensive. In the application scenarios of natural language processing with different language corpora, such as multilingual sentiment analysis, translation, etc., it is necessary to train corresponding language models for each language to achieve cross-language learning. This process requires the annotation of a large amount of data, which takes a lot of time and effort, and will lead to low learning efficiency and waste of resources.

[0004] With the continuous development of natural language processing technology, an unsupervised multi-corpus language model training method is proposed to solve the problems of low learning efficiency and resource waste caused by establishing language models for each language. This method trains multiple different languages ​​in the same language model to generate context word vectors, and uses word vector alignment technology to achieve analysis between different languages. Existing word vector alignment technologies involve sentence-level alignment technology and word-level alignment technology. In cross-language translation application scenarios, sentence-level word vector alignment methods rely on a large number of mutually translated sentence pairs to achieve better performance, but translation data also requires a lot of manpower and time to prepare. When there is no translation data or when translation data is scarce, word-level word vector alignment technology can be used, that is, natural language learning can be achieved by controlling the distribution distance between words. However, different words have different meanings in different language environments. The word-level alignment method only aligns by the average value of the distance between words. Obviously, it will cause the alignment granularity to be relatively coarse, resulting in many problems such as inaccurate downstream translation data. Summary of the invention

[0005] The present application provides a cross-language language model pre-training method to solve the inaccuracy problem in word vector alignment in the prior art.

[0006] The present application provides a cross-language language model pre-training method, comprising:

[0007] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0008] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0009] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0010] The word meaning vector probability of the vocabulary is determined according to the second context word vector and the first context word vector.

[0011] In some embodiments, it also includes:

[0012] Obtaining the meaning of the vocabulary;

[0013] The word sense clustering set is constructed according to the word sense vectors of the word senses.

[0014] In some embodiments, constructing the word sense clustering set according to the word sense vector of the word sense includes:

[0015] Determining the number of cluster sets of the word meaning cluster sets;

[0016] Determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets;

[0017] According to the central word sense vector, the word sense clustering set matching the number of the clustering sets is constructed.

[0018] In some embodiments, selecting a word sense vector that matches the first context word vector from the constructed word sense clustering set includes:

[0019] According to the distance between the first context word vector and the central word sense vector, the word sense vector that matches the first context word vector is selected.

[0020] In some embodiments, determining a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model includes:

[0021] When the pre-trained language model is running in the initial training phase, randomly initializing the context word vector set output by the pre-trained language model;

[0022] Determine the second context word vector corresponding to the word meaning vector according to the randomly initialized context word vector set;

[0023] or,

[0024] Updating the context word vector set output by the pre-trained language model according to the determined word meaning vector probability;

[0025] According to the updated context word vector set, the second context word vector corresponding to the word meaning vector is determined.

[0026] In some embodiments, determining the word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector includes:

[0027] Determine the word meaning vector probability of the vocabulary word according to the similarity between the second context word vector and the first context word vector.

[0028] In some embodiments, inputting the text information into a pre-trained language model to obtain a first context word vector of a word in the text information includes:

[0029] When the pre-trained language model is a bidirectional language model, inputting the text information into the bidirectional language model to obtain the first context word vector of the next word adjacent to the current word in the text information;

[0030] When the pre-trained language model is a masked language model, the text information is input into the masked language model to obtain the first context word vector of the current vocabulary in the text information.

[0031] In some embodiments, it also includes:

[0032] The training parameters of the pre-trained language model are adjusted according to the word sense vector probability, or the word sense vector probability is used as an input parameter and input into a new language training model.

[0033] Accordingly, the present application also provides a cross-language language model pre-training device, comprising:

[0034] An acquisition unit, used to input text information into a pre-trained language model to acquire a first context word vector of a word in the text information;

[0035] A selection unit, configured to select a word sense vector that matches the first context word vector from the constructed word sense clustering set;

[0036] A first determining unit, configured to determine a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model;

[0037] The second determining unit is used to determine the word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector.

[0038] The present application also provides a cross-language language model pre-training method, including:

[0039] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0040] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0041] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0042] The word meaning vector probability of the vocabulary is determined according to the second context word vector, the second context word vector set and the current context word vector output by the pre-trained language model.

[0043] In some embodiments, it also includes:

[0044] Obtaining the meaning of the vocabulary;

[0045] The word sense clustering set is constructed according to the word sense vectors of the word senses.

[0046] In some embodiments, constructing the word sense clustering set according to the word sense vector of the word sense includes:

[0047] Determining the number of cluster sets of the word meaning cluster sets;

[0048] Determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets;

[0049] According to the central word sense vector, the word sense clustering set matching the number of the clustering sets is constructed.

[0050] In some embodiments, selecting a word sense vector that matches the first context word vector from the constructed word sense clustering set includes:

[0051] According to the distance between the first context word vector and the central word sense vector, the word sense vector that matches the first context word vector is selected.

[0052] In some embodiments, determining a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model includes:

[0053] When the pre-trained language model is running in the initial training phase, randomly initializing the context word vector set output by the pre-trained language model;

[0054] Determine the second context word vector corresponding to the word meaning vector according to the randomly initialized context word vector set;

[0055] or,

[0056] Updating the context word vector set output by the pre-trained language model according to the determined word meaning vector probability;

[0057] According to the updated context word vector set, the second context word vector corresponding to the word meaning vector is determined.

[0058] In some embodiments, determining the word meaning vector probability of the vocabulary according to the second context word vector, the second context word vector set, and the current context word vector output by the pre-trained language model includes:

[0059] The word meaning vector probability corresponding to the vocabulary is determined according to the dot product value of the second context word vector and the current context word vector, and the dot product value of the current context word vector and the context word vector set.

[0060] In some embodiments, inputting the text information into a pre-trained language model to obtain a first context word vector of a word in the text information includes:

[0061] When the pre-trained language model is a bidirectional language model, inputting the text information into the bidirectional language model to obtain the first context word vector of the next word adjacent to the current word in the text information;

[0062] When the pre-trained language model is a masked language model, the text information is input into the masked language model to obtain the first context word vector of the current vocabulary in the text information.

[0063] In some embodiments, it also includes:

[0064] The training parameters of the pre-trained language model are adjusted according to the word sense vector probability, or the word sense vector probability is used as an input parameter and input into a new language training model.

[0065] The present application also provides a cross-language language model pre-training device, comprising:

[0066] An acquisition unit, used to input text information into a pre-trained language model to acquire a first context word vector of a word in the text information;

[0067] A selection unit, configured to select a word sense vector that matches the first context word vector from the constructed word sense clustering set;

[0068] A first determining unit, configured to determine a second context word vector corresponding to the word meaning vector according to the context word vector set output by the pre-trained language model;

[0069] The second determining unit is used to determine the word meaning vector probability corresponding to the vocabulary according to the second context word vector, the context word vector set and the current context word vector output by the pre-trained language model.

[0070] The present application also provides a cross-language named entity recognition method based on transfer learning, including:

[0071] Obtaining text information to be recognized in a first language;

[0072] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a named entity recognition task; wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0073] Based on the language model, a target recognition result is obtained.

[0074] This application also provides a cross-language sentiment analysis method based on transfer learning, including:

[0075] Obtaining text information to be recognized in a first language;

[0076] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0077] Based on the language model, a target analysis result is obtained.

[0078] The present application also provides a cross-language sentence relationship inference method based on transfer learning, including:

[0079] Obtaining text information to be recognized in a first language;

[0080] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0081] Based on the language model, a target relationship inference result is obtained.

[0082] The present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0083] When the program is read and executed, the following steps are performed:

[0084] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0085] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0086] Determining a second context word sense vector corresponding to the word sense vector from the context word vector set output by the pre-trained language model;

[0087] Determine a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector;

[0088] Alternatively, when the program is read and executed, the following steps are performed:

[0089] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0090] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0091] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0092] Determine the word meaning vector probability of the vocabulary according to the second context word vector, the second context word vector set, and the current context word vector output by the pre-trained language model;

[0093] Alternatively, when the program is read and executed, the following steps are performed:

[0094] Obtaining text information to be recognized in a first language;

[0095] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a named entity recognition task; wherein the cross-language pre-training method uses text information in a second language as a pre-training sample;

[0096] Based on the language model, obtaining a target recognition result;

[0097] Alternatively, when the program is read and executed, the following steps are performed:

[0098] Obtaining text information to be recognized in a first language;

[0099] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a language sentiment analysis task, wherein the cross-language pre-training method uses text information in a second language as a pre-training sample;

[0100] Based on the language model, obtaining a target analysis result;

[0101] Alternatively, when the program is read and executed, the following steps are performed:

[0102] Obtaining text information to be recognized in a first language;

[0103] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0104] Based on the language model, a target relationship inference result is obtained.

[0105] The present application also provides an electronic device, a processor;

[0106] The memory is used to store a program for processing data generated by the network platform, and when the program is read and executed by the processor, the following steps are performed:

[0107] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0108] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0109] Determining a second context word sense vector corresponding to the word sense vector from the context word vector set output by the pre-trained language model;

[0110] Determine a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector;

[0111] Alternatively, when the program is read and executed, the following steps are performed:

[0112] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0113] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0114] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0115] Determine the word meaning vector probability of the vocabulary according to the second context word vector, the second context word vector set, and the current context word vector output by the pre-trained language model;

[0116] Alternatively, when the program is read and executed, the following steps are performed:

[0117] Obtaining text information to be recognized in a first language;

[0118] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a named entity recognition task; wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0119] Based on the language model, obtaining a target recognition result;

[0120] Alternatively, when the program is read and executed, the following steps are performed:

[0121] Obtaining text information to be recognized in a first language;

[0122] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0123] Based on the language model, obtaining a target analysis result;

[0124] Alternatively, when the program is read and executed, the following steps are performed:

[0125] Obtaining text information to be recognized in a first language;

[0126] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0127] Based on the language model, a target relationship inference result is obtained.

[0128] Compared with the prior art, this application has the following advantages:

[0129] The present application provides a cross-language language model pre-training method, which can select a word sense vector that matches the first context word vector in the constructed word sense vector clustering set according to the first context word vector obtained, determine a second context word vector that matches the word sense vector in the context word vector set output by the language model according to the word sense vector, determine the word sense vector probability of the vocabulary through the second context word vector and the first context word vector, or determine the word sense vector probability of the vocabulary through the second context word vector, the context word vector set and the current context word vector output by the pre-trained language model. In this way, the distribution of the second context word vector and the current context word vector at the word sense level can be known, and the context word vectors can be aligned at the fine-grained level of word sense; and the word sense vector probability can provide a reference for downstream tasks, so that the language model can adjust the training parameters according to the probability to achieve a more accurate context word vector output. BRIEF DESCRIPTION OF THE DRAWINGS

[0130] Figure 1 is a flowchart of an embodiment of a cross-language language model pre-training method provided by the present application;

[0131] Figure 2 It is a structural schematic diagram of word meaning clustering set construction in an embodiment of a cross-language language model pre-training method provided by the present application;

[0132] Figure 3 It is a training logic diagram when a bidirectional language model is used as a pre-training language model in an embodiment of a cross-language language model pre-training method provided by the present application;

[0133] Figure 4 It is a training logic diagram when a masked language model is used as a pre-training language model in an embodiment of a cross-language language model pre-training method provided by the present application;

[0134] Figure 5 It is a structural schematic diagram of an embodiment of a cross-language language model pre-training device provided by the present application;

[0135] Figure 6 It is a flowchart of an embodiment of a cross-language named entity recognition method based on transfer learning provided by the present application;

[0136] Figure 7 It is a flowchart of an embodiment of a cross-language sentiment analysis method based on transfer learning provided by the present application;

[0137] Figure 8 It is a flowchart of an embodiment of a cross-language sentence relationship inference method based on transfer learning provided by the present application;

[0138] Fig. 9It is a structural schematic diagram of an electronic device embodiment provided by the present application. DETAILED DESCRIPTION

[0139] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0140] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0141] As mentioned in the background technology, the prior art mainly uses word vector alignment technology to implement cross-language application scenarios, such as translation, human-computer dialogue, etc. A word vector can be a feature vector that represents a word. The word vector of the corresponding word is obtained by inputting the corpus into a language model, such as the word2vec language model. The core idea of ​​the word2vec language model is to use the language model to train the context of the word to obtain a vectorized representation of the word. However, only a static word vector representation is obtained through the word2vec language model. The so-called static means that after the word vector is obtained through word training, it is fixed in any semantic environment and will not change with the change of the context scenario. Therefore, the problem of polysemy cannot be solved. Then, in cross-language application scenarios, it is bound to affect the execution and implementation of downstream tasks, such as the accuracy of translation and the accuracy of semantic recognition.

[0142] Based on the problems of obtaining word vectors based on the word2vec language model, dynamic word vector acquisition methods have emerged, such as ELMO, GPT, BERT and other language models. Taking the ELMO language model as an example: its core idea is to use a language model to learn the word vector (word embedding) of a word in advance. When it is used, the word already has a specific contextual representation meaning. Then the word vector is adjusted according to the contextual representation meaning. On the surface, it solves the polysemy problem, but in fact, the dynamic adjustment process is only carried out between words, that is, it depends on the average value of the word vector distribution distance for dynamic adjustment. This dynamic adjustment method can only shorten the distribution distance between words, and cannot accurately express the meaning of the word in the contextual semantic environment.

[0143] In view of the above problems, this application provides a dynamic adjustment method based on word meaning, so as to shorten the distribution distance of vectors with the same or similar meanings in terms of word meaning, so as to achieve finer-grained alignment. Figure 1 As shown, Figure 1 : is a flowchart of an embodiment of a cross-language language model pre-training method provided by the present application, and the embodiment includes:

[0144] Step S101: input text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0145] First, the terms in step S101 are explained:

[0146] The text information can be understood as original corpus information, which can be corpus information in any language, such as sentences.

[0147] The vocabulary is the unit that constitutes the corpus. For different languages, the number of words that make up the vocabulary is not limited. For example, for Chinese, it can be a vocabulary composed of multiple individual characters, or a vocabulary composed of a single character; for English, it can be a word composed of multiple letters, or a phrase vocabulary composed of multiple words, etc. Therefore, there are no requirements for the organizational structure or composition form of the vocabulary, and it can have an independent meaning or be composed of other words to express the meaning.

[0148] The first contextual word vector, where "first" is only a way of expressing the difference between contextual word vectors, is not a restriction on the order or number of contextual word vectors. Contextual word vectors can also be understood as contextual word vectors, which generate word vectors of the word according to the meaning of the word in the text information combined with the context.

[0149] The language model in the pre-trained language model is a neural network model applied in natural language processing (NLP). Natural language processing belongs to an application sub-field in artificial intelligence technology. The pre-trained language model is a language model trained on a large amount of corpus to learn the distribution of words or vocabulary in the text information.

[0150] In this embodiment, the pre-trained language model can be an ELMO (Embeddings from LanguageModels) model and a BERT (BidirectionalEncoder Representations from Transformer) model, or other models. The ELMO model mainly adopts a bidirectional language model, which can be used to predict the word vector of the next word; the BERT model is a multi-layer bidirectional transformer encoder based on fine-tuning, which can be used to predict the word vector of the masked word.

[0151] The specific implementation process of step S101 may be:

[0152] The text information is input into a pre-trained language model, and the first context word vector of the vocabulary in the text information can be obtained through the pre-trained language model.

[0153] When the pre-trained language model is an ELMO model, the ELMO model can be a bidirectional language model (idirectional language models) and a bidirectional LSTM (long-short term memory) neural network structure. Given a sequence containing N tokens (vocabulary) represented as: t1, t2, ..., t N , then the forward representation of the ELMO model can be:

[0154]

[0155] Then, the LSTM neural network model can generate the first context word vector for each vocabulary or word (token) using the following formula:

[0156]

[0157] Among them, k' represents word t k The index in the vocabulary, V represents the size of the vocabulary; w1, ...w2 represent the output word vectors of the words in the vocabulary, and the words in the vocabulary are the meanings of the vocabulary. A vocabulary can include multiple meanings, and a meaning can be regarded as a word, which corresponds to the meaning. It can be seen from the above formula that when the LSTM neural network model generates the first context word vector, it represents the word vector through the linear combination of the output values ​​of all layers. Because the above formula is a description of the forward structure of LSTM, the word vector representation that is independent of the context uses Correspondingly, the backward representation and generation method is similar to the forward one. The forward one uses the previous context to predict the next context, and the backward one uses the next context to predict the previous one, that is, predicting the representation of the next word in the context, that is, the context word vector. Therefore, the backward structure is not described here. The LSTM neural network model is also an existing model, and is only described here in summary.

[0158] Of course, the pre-trained language model can also be a BERT model, that is, a masked language model. The text information input into the BERT model is randomly masked, and then the masked words are predicted. Specifically, a sequence (t1, t2, ..., t N ,) masks the words in the text information according to a certain ratio, and generates a sequence with word masking represented as: m1, m2…, m N ; Among them, m k is the masked word. Then, the following formula can be used to learn to generate context representation (context word vector) through the BERT model:

[0159]

[0160] Then, the context word vector generated by the above formula can be used as the first context word vector obtained. Because the BERT model is also one of the commonly used pre-training models in the prior art, its principle will not be described in detail here.

[0161] Combined with the above description of step S101, it can be seen that in this embodiment, the purpose of step S101 is to obtain the first context word vector in the text information. The method of obtaining it can be based on the neural network in the ELMO model or the BERT model. Of course, other models that can obtain the first context word vector are not excluded.

[0162] It should be noted that, in this embodiment, the context word vector can also be called a context word vector, that is, what is obtained is the representation of the word in the context, which can be expressed in the form of a word vector. The word vector can be understood as a form of representation in two-dimensional space or high-dimensional space so that the computer can recognize and process it.

[0163] Based on the above content, the specific implementation process of step S101 may include at least two methods.

[0164] The first way is:

[0165] When the pre-trained language model is a bidirectional language model (i.e., LSTM in the ELMO model), the text information is input into the bidirectional language model to obtain the first context word vector of the next word adjacent to the current word in the text information, such as Figure 3 As shown, the first context word vector obtained is h k,L , the context word vector of the current word is h k-1,L ; That is, the LSTM neural network model is based on the input vocabulary t k-1,L The context word vector h k-1,L , predict the next word t k The context word vector h k,L , the context word vector h k,L As the first context word vector.

[0166] The second way is:

[0167] When the pre-trained language model is a masked language model, the text information is input into the masked language model to obtain the first context word vector of the current word in the text information. Figure 4 As shown, the masking word is t k-1 , predict the masked word t k-1 The context word vector is the first context word vector h k,L .

[0168] Step S102: selecting a word sense vector that matches the first context word vector according to the word sense cluster set constructed based on the word sense of the vocabulary;

[0169] The purpose of step S102 is to select a word meaning vector that matches the first context word vector.

[0170] Based on the above content, it can be known that in general, there are multiple meanings of words in text information, which express different meanings in different contexts. Therefore, step S102 needs to select a matching word vector for the first context word vector. The so-called word vector is the meaning representation of the words in the text information.

[0171] Please refer to Figure 2 As shown, the word sense vector that matches the first context word vector can be selected by the constructed word sense clustering set. Specifically, it can be determined whether the first context word vector matches the central word sense vector in the word sense clustering set according to the distance between the two. If the first context word vector is closer to the central word sense vector of a certain word sense clustering set than to the central word sense vector of other word sense clustering sets, the word sense vector with the closest distance is determined as the word sense vector that matches the first context word vector.

[0172] Therefore, please refer to Figure 2 As shown, this embodiment also includes:

[0173] Obtaining the meaning of the vocabulary;

[0174] The word sense clustering set is constructed according to the word sense vectors of the word senses.

[0175] The meaning of the vocabulary can be obtained through an electronic dictionary, and of course can also be obtained through the Internet. Therefore, obtaining the meaning of the word is the purpose, and there are many ways to obtain the meaning of the word.

[0176] In this embodiment, the word sense clustering set can be constructed in real time according to the vocabulary in the text information obtained in step S101. The construction method adopted can be an on-line K-means algorithm. Therefore, the word sense clustering set constructed according to the word sense vector of the word sense can include:

[0177] Determining the number of cluster sets of the word meaning cluster sets;

[0178] Determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets;

[0179] According to the central word sense vector, the word sense clustering set matching the number of the clustering sets is constructed.

[0180] It is understandable that the determination of the central word sense vector can achieve the desired clustering effect through multiple iterations, that is, after determining the central word sense vector of the center position of each word sense clustering set, the word sense vector of the center position can be recalculated. When the distance between the recalculated central word sense vector of the center position and the central word sense vector of the original center position (the last center position) is less than a certain set threshold, there is no need to recalculate, which means that the position of the recalculated central word sense vector does not change much, and the clustering of the central word sense vector converges. Of course, it is also possible to first perform dimensionality reduction processing on the word sense vector and then perform clustering. The dimensionality reduction process can be performed through PCA (principal component analysis) to reduce the dimensionality, thereby avoiding noise interference and reducing the computational complexity of clustering processing.

[0181] It should be noted that when constructing the word sense clustering set for the first time, the central word sense vector at the center position can be initialized and then recalculated and iterated. The K-means clustering algorithm belongs to the prior art, so it is only briefly described here. In this embodiment, in the process of word sense clustering using the clustering algorithm, a word sense clustering set for the word sense can be constructed based on the vocabulary in the input text information.

[0182] Based on the word meaning clustering set constructed above, the specific implementation process of step S102 may include:

[0183] According to the distance between the first context word vector and the central word sense vector, the word sense vector that matches the first context word vector is selected.

[0184] The following is an example to illustrate the selection of word sense vectors.

[0185] When the vocabulary of the text information is the English word bank, its corresponding Japanese translation can be bank, river bank, etc., as well as the sub-word meanings of bank and river bank, such as: storehouse, deposit, embankment, accumulation, etc. to construct a word meaning cluster set, such as Figure 2 shown. Figure 2 Medium k ,L can be expressed as the first context word vector of the bank in the financial context, c k’2 is the central word sense vector of the cluster set of the vocabulary bank; c a,1 , c a,2 , c b,1 , c b,2 ...etc. represent the different sub-word meanings of the Japanese translation bank and riverbank in the cluster set. Of course, it can also be a cluster set of bank translated into other meanings in Japanese, and c b,2 After PCA (Principal Component Analysis), h k,L The center word vector of the closest cluster set. Figure 2 It is only a representation of a clustering set. In fact, after each word meaning is clustered by a clustering algorithm, it is in a high-dimensional space. After PCA dimensionality reduction, the word meaning vector that matches the first context word vector can be obtained.

[0186] Step S103: determining a second context word vector corresponding to the word meaning vector according to the context word vector set output by the pre-trained language model;

[0187] The second context word vector in step S103 is actually determined based on the word meaning vector. In other words, the second context word vector can be understood as a word vector at the word meaning level.

[0188] The specific implementation process of step S103 may include two methods, because the selection of the word sense vector in step S102 is a dynamic process, that is, dynamically constructing the word sense clustering set according to the input of the pre-trained language model and selecting the word sense vector matching the first context word vector. Therefore, when the pre-trained language model does not have an output context word vector (output embeddings) set, the context word vector set can be randomly initialized first. Of course, if there is a context word vector set, it can be used directly. Therefore, the specific implementation process of step S103 may include:

[0189] When the pre-trained language model is running in the initial training phase, randomly initializing the context word vector set output by the pre-trained language model;

[0190] Determine the second context word vector corresponding to the word meaning vector according to the randomly initialized context word vector set;

[0191] or,

[0192] Updating the context word vector set output by the pre-trained language model according to the determined word meaning vector probability;

[0193] According to the updated context word vector set, the second context word vector corresponding to the word meaning vector is determined.

[0194] Whether it is a randomly initialized context word vector set or an updated context word vector set, it is necessary to determine the second context word vector in combination with the selected word meaning vector. The determination process can be performed by comparing the two vectors, or by searching for a context word vector with the same identifier as the word meaning vector in the context word meaning vector combination and using it as the second context word vector. Normally, the word vector is represented by identification information, so the identifier of the selected word meaning vector can be compared with the identifier in the context word vector set.

[0195] Step S104: determining the word meaning vector probability of the vocabulary according to the second context word vector, the context word vector set and the current context word vector output by the pre-trained language model.

[0196] It is understandable that the word sense vector probability of the vocabulary can also be determined based on the second context word vector and the first context word vector. For example, the word sense vector probability of the vocabulary is determined based on the similarity between the second context word vector and the first context word vector.

[0197] The specific implementation process of step S104 may include:

[0198] Step S104-1: Determine the word meaning vector probability of the vocabulary according to the dot product value of the second context word vector and the current context word vector, and the dot product value of the current context word vector and the context word vector set.

[0199] The specific implementation process of step S104-1 can be calculated by using the word meaning cross entropy loss function, and specifically can be calculated using the following formula:

[0200]

[0201] in, is the current context word vector, wk’,s’ is the second context word vector, w i,s is a set of context word vectors.

[0202] The significance of the formula is that the word meaning vector probability is determined by calculating the ratio of the dot product value of the second context word vector and the current context word vector to the dot product value of the current context word vector and the set of context word vectors, so that the gap between the current context word vector output by LSTM and the second context word vector selected according to the word meaning vector can be judged according to the word meaning vector probability, that is, the larger the word meaning vector probability, the smaller the distance between the current context word vector and the second context word vector, and the closer to the meaning to be expressed by the input text; the smaller the word meaning vector probability, the larger the distance between the current context word vector and the second context word vector, and the more it deviates from the meaning to be expressed by the input text, so it is necessary to shorten the distribution distance between the two to make the word meaning vector probability larger, so as to achieve the alignment of the context word vectors at the word meaning level.

[0203] Therefore, this embodiment also includes:

[0204] The training parameters of the pre-trained language model are adjusted according to the word sense vector probability, or the word sense vector probability is used as an input parameter and input into a new language training model.

[0205] By adjusting the training parameters of the pre-trained language model through the word meaning vector probability, the current context word vector output by the pre-trained language model can be closer to the meaning to be expressed by the text information, thereby making the current context word vector output by the pre-trained language model more consistent with the meaning to be expressed by the text information.

[0206] In the process of adjusting the training parameters of the pre-trained language model, this embodiment aligns the context word vectors on the specific word meanings, that is, clusters the word meanings corresponding to the context word vectors, and determines the word meaning vector that matches the context word vector based on the word meaning clustering set, and then determines the second context word vector that matches the word meaning vector in the context vector set output by the pre-trained language model, and then uses the cross entropy loss function to calculate the second context word vector, the current context word vector output by the pre-trained language model, and the set of all context word vectors output by the pre-trained language model to determine whether the distribution distance between the second context word vector and the current context word vector is close. The second context word vector is determined based on the word meaning vector, so it can be regarded as a word meaning vector. For example, the contextual word vector of the English word "bank" and the word meaning vector of the word "bank" are closer in distribution distance. However, in the prior art, only the translation data "bank" of bank can be input into the training model as a vocabulary, and the word vector is generated for bank. When the word vector is aligned, it can only be aligned based on the word vectors. Bank also includes other word meanings or other sub-word meanings. Therefore, the prior art can only achieve word vector alignment at the word level, and cannot be extended to a more fine-grained level, such as word meaning alignment on word meaning or sub-word meaning; and it also needs to rely on translation data. Therefore, this word-level alignment of the prior art will inevitably make the training results of the language model inaccurate, and will also cause the problem of poor recognition accuracy in the output of transfer learning. Because, in the prior art, the training data of the cross-language pre-training model is not used as supervision, so the alignment granularity is low, and only word-level alignment can be performed.

[0207] It should be noted that the text information in this embodiment can be text information connected by corpora in two or more languages. Therefore, when constructing a word sense clustering set, a joint vocabulary can also be constructed based on words in different languages. The word sense clustering set constructed based on the joint vocabulary has the same construction method as described above, and the method for determining the probability of a word sense vector through a pre-trained model is also the same.

[0208] It is understandable that the selection of translation word meaning vectors can be selected by cosine similarity, for example: when the dictionary is for t k There are T translations, each corresponding to S sub-word meanings, then t k There may be T×S word meaning translations, and the cluster set to which the first context word vector belongs can be determined according to its maximum cosine similarity in the cluster set, thereby determining the translation word meaning vector corresponding to the first context word vector.

[0209] When cross-language pre-training is performed on multiple languages ​​in this embodiment, that is, joint training, the loss calculation of the translation meaning can be performed as follows:

[0210]

[0211] Among them, L JOINT represents the joint loss, L SENSE represents the word meaning cross entropy loss, L TRAN represents the translation cross entropy loss.

[0212] The translation cross entropy loss can be obtained by existing methods as follows:

[0213]

[0214] Among them, V j,s Indicates the meaning of the translation word.

[0215] Of course, when the word has no translation meaning in the dictionary, the loss can be L JOINT =L SENSE Among them, L SENSE It can be understood as:

[0216]

[0217] The above is a specific description of an embodiment of a cross-language language model pre-training method provided by the present application. Corresponding to the embodiment of a cross-language language model pre-training method provided above, the present application also discloses an embodiment of a cross-language language model pre-training device, please refer to Figure 5 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0218] like Figure 5 As shown, the device embodiment includes:

[0219] An acquisition unit 501 is used to input text information into a pre-trained language model to acquire a first context word vector of a word in the text information;

[0220] The acquisition unit 501 includes:

[0221] a next word acquisition unit, configured to input the text information into the bidirectional language model when the pre-trained language model is a bidirectional language model, and acquire the first context word vector of the next word adjacent to the current word in the text information;

[0222] The masked vocabulary acquisition unit is used to input the text information into the masked language model when the pre-trained language model is a masked language model, and acquire the first context word vector of the current vocabulary in the text information.

[0223] A selection unit 502, configured to select a word sense vector that matches the first context word vector according to the word sense cluster set constructed from the word senses of the vocabulary;

[0224] The selection unit is specifically configured to select the word sense vector that matches the first context word vector according to the distance between the first context word vector and the central word sense vector.

[0225] A first determining unit 503 is used to determine a second context word vector corresponding to the word meaning vector according to the context word vector set output by the pre-trained language model;

[0226] The first determining unit 503 includes:

[0227] An initialization subunit, used for randomly initializing the context word vector set output by the pre-trained language model when the pre-trained language model runs in the initial training phase;

[0228] A determination subunit is used to determine the second context word vector corresponding to the word meaning vector based on the randomly initialized context word vector set.

[0229] Or, include:

[0230] An updating subunit, configured to update the context word vector set output by the pre-trained language model according to the determined word meaning vector probability;

[0231] A determination subunit is used to determine the second context word vector corresponding to the word meaning vector based on the updated context word vector set.

[0232] The second determining unit 504 is used to determine the word meaning vector probability of the vocabulary according to the second context word vector, the context word vector set and the current context word vector output by the pre-trained language model.

[0233] The determining unit 504 is specifically configured to determine the word meaning vector probability of the vocabulary word according to the dot product value of the second context word vector and the current context word vector, and the dot product value of the current context word vector and the context word vector set.

[0234] Also includes:

[0235] A word meaning acquisition unit, used to acquire the word meaning of the word;

[0236] A construction unit is used to construct the word sense clustering set according to the word sense vector of the word sense.

[0237] The construction unit comprises: a first determining subunit, a second determining subunit and a construction subunit;

[0238] Among them, the first determination subunit is used to determine the number of cluster sets of the word sense cluster set; the second determination subunit is used to determine the central word sense vector at the center position of each word sense cluster set based on the number of cluster sets; the construction subunit is used to construct the word sense cluster set that matches the number of cluster sets based on the central word sense vector.

[0239] It also includes: an adjustment unit or an input unit;

[0240] The adjustment unit is used to adjust the training parameters of the pre-trained language model according to the word sense vector probability;

[0241] The input unit is used to input the word sense vector probability as an input parameter into a new language training model.

[0242] The above is a brief description of an embodiment of a cross-language language model pre-training device provided in the present application. For specific content, please refer to the description of the above cross-language language model pre-training method embodiment.

[0243] Based on the above content, the present application also provides an embodiment of a cross-language named entity recognition method based on transfer learning. Figure 6 As shown, this embodiment includes:

[0244] Step S601: obtaining text information to be recognized in a first language;

[0245] Step S602: inputting the text information to be recognized into the language model adjusted according to the above-mentioned embodiment of the cross-language language model pre-training method to perform a named entity recognition task; wherein, in the embodiment of the cross-language language model pre-training method, text information in the second language is used as a pre-training sample;

[0246] Step S603: Obtaining a target recognition result based on the language model.

[0247] The following table provides experimental data for cross-language named entity recognition based on German, Spanish, and Chinese. It is obvious that based on the above solution with less data and model parameters, the language model adjusted by the above cross-language language model pre-training method makes German have better results in transfer learning of named entity recognition.

[0248]

[0249] Based on the above content, the present application also embodies an embodiment of a cross-language sentiment analysis method based on transfer learning. Figure 7 As shown, this embodiment includes:

[0250] Step S701: obtaining text information to be recognized in a first language;

[0251] Step S702: inputting the text information to be recognized into the language model adjusted according to the above-mentioned embodiment of the cross-language language model pre-training method, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0252] Step S703: Obtain a target analysis result based on the language model.

[0253] The following table provides experimental data based on German and Japanese in language transfer learning for product review sentiment analysis. It is obvious that the recognition accuracy has been significantly improved.

[0254]

[0255] Based on the above content, the present application also provides an embodiment of a cross-language sentence relationship inference method based on transfer learning. Figure 8 As shown, this embodiment includes:

[0256] Step S801: obtaining text information to be recognized in a first language;

[0257] Step S802: inputting the text information to be recognized into the language model adjusted according to the embodiment of the cross-language language model pre-training method provided above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in the second language as a pre-training sample;

[0258] Step S803: Based on the language model, a target relationship inference result is obtained.

[0259] The following table provides experimental data on cross-language relationship inference tasks based on German, Spanish, and Chinese. It is clear that the accuracy of language model inference has been significantly improved.

[0260]

[0261] According to the above experimental data and the description of the method embodiment, it can be seen that in the cross-language language model pre-training method embodiment provided by the present application, no additional data is required as supervision. Instead, a word meaning clustering set is constructed for each word vocabulary input into the pre-trained language model. Therefore, when aligning the context word vectors, there is no need to rely on external translation data or other annotation data. The distribution of the context word vectors can be analyzed at the word meaning level, and then the language model parameters can be adjusted to shorten the distribution distance of different languages ​​at the word meaning level, so that the language model parameters can achieve a higher recognition effect, thereby realizing fine-grained context word vector alignment.

[0262] Based on the above, it can be understood that a cross-language language model pre-training method embodiment provided by the present application can be applied to machine translation, including any translation scenario such as online machine translation and offline machine translation. It can be based on multilingual translation or single language translation (one language to another language). It should be noted that the use scenario involved in machine translation is not a simple translation scenario, but can also be other scenarios with translation needs, such as: cross-border e-commerce platform needs, cross-border resource service platform needs, etc.

[0263] In summary, the present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0264] When the program is read and executed, the following steps are performed:

[0265] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0266] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0267] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0268] Determine a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector;

[0269] Alternatively, when the program is read and executed, the following steps are performed:

[0270] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0271] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0272] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0273] Determining a word meaning vector probability of the vocabulary according to the second context word vector, the context word vectors in the context word vector set, and the current context word vector output by the pre-trained language model;

[0274] Alternatively, when the program is read and executed, the following steps are performed:

[0275] Obtaining text information to be recognized in a first language;

[0276] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a named entity recognition task; wherein the cross-language pre-training method uses text information in a second language as a pre-training sample;

[0277] Based on the language model, obtaining a target recognition result;

[0278] Alternatively, when the program is read and executed, the following steps are performed:

[0279] Obtaining text information to be recognized in a first language;

[0280] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a language sentiment analysis task, wherein the cross-language pre-training method uses text information in a second language as a pre-training sample;

[0281] Based on the language model, obtaining a target analysis result;

[0282] Alternatively, when the program is read and executed, the following steps are performed:

[0283] Obtaining text information to be recognized in a first language;

[0284] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0285] Based on the language model, a target relationship inference result is obtained.

[0286] The present application also provides an electronic device, a processor 901;

[0287] The memory 902 is used to store a program for processing data generated by the network platform. When the program is read and executed by the processor, the following steps are performed:

[0288] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0289] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0290] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0291] Determining a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector;

[0292] Alternatively, when the program is read and executed, the following steps are performed:

[0293] Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information;

[0294] In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector;

[0295] Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector;

[0296] Determining a word meaning vector probability of the vocabulary according to the second context word vector, the context word vectors in the context word vector set, and the current context word vector output by the pre-trained language model;

[0297] Alternatively, when the program is read and executed, the following steps are performed:

[0298] Obtaining text information to be recognized in a first language;

[0299] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a named entity recognition task; wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0300] Based on the language model, obtaining a target recognition result;

[0301] Alternatively, when the program is read and executed, the following steps are performed:

[0302] Obtaining text information to be recognized in a first language;

[0303] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0304] Based on the language model, obtaining a target analysis result;

[0305] Alternatively, when the program is read and executed, the following steps are performed:

[0306] Obtaining text information to be recognized in a first language;

[0307] Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method as described above, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample;

[0308] Based on the language model, a target relationship inference result is obtained.

[0309] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0310] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0311] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0312] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0313] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A cross-language language model pre-training method, characterized in that: include: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: determining the number of cluster sets of the word sense cluster set; According to the number of cluster sets, determining the central word sense vector of the center position of each word sense cluster set; according to the central word sense vector, constructing the word sense cluster sets matching the number of cluster sets; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector; The word meaning vector probability of the vocabulary is determined according to the second context word vector and the first context word vector.

2. The cross-language language model pre-training method according to claim 1, characterized in that: In the constructed word sense clustering set, a word sense vector matching the first context word vector is selected, including: According to the distance between the first context word vector and the central word sense vector, the word sense vector that matches the first context word vector is selected.

3. The cross-language language model pre-training method according to claim 1, characterized in that: Determining a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model includes: When the pre-trained language model is running in the initial training phase, randomly initializing the context word vector set output by the pre-trained language model; Determine the second context word vector corresponding to the word meaning vector according to the randomly initialized context word vector set; or, Updating the context word vector set output by the pre-trained language model according to the determined word meaning vector probability; According to the updated context word vector set, the second context word vector corresponding to the word meaning vector is determined.

4. The cross-language language model pre-training method according to claim 1, characterized in that: The determining the word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector comprises: Determine the word meaning vector probability of the vocabulary word according to the similarity between the second context word vector and the first context word vector.

5. The cross-language language model pre-training method according to claim 1, characterized in that: The step of inputting the text information into the pre-trained language model and obtaining the first context word vector of the vocabulary in the text information includes: When the pre-trained language model is a bidirectional language model, inputting the text information into the bidirectional language model to obtain the first context word vector of the next word adjacent to the current word in the text information; When the pre-trained language model is a masked language model, the text information is input into the masked language model to obtain the first context word vector of the current vocabulary in the text information.

6. The cross-language language model pre-training method according to claim 1, characterized in that: Also includes: The training parameters of the pre-trained language model are adjusted according to the word sense vector probability, or the word sense vector probability is used as an input parameter and input into a new language training model.

7. A cross-language language model pre-training device, characterized in that: include: An acquisition unit, used to input text information into a pre-trained language model to acquire a first context word vector of a word in the text information; A construction unit, used for constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: a first determining subunit, a second determining subunit and a construction subunit; the first determining subunit is used for determining the number of cluster sets of the word sense cluster set; the second determining subunit is used for determining the central word sense vector of the center position of each word sense cluster set according to the number of cluster sets; the construction subunit is used for constructing the word sense cluster set matching the number of cluster sets according to the central word sense vector; A selection unit, configured to select a word sense vector that matches the first context word vector from the constructed word sense clustering set; A first determining unit, configured to determine a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model; The second determining unit is used to determine the word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector.

8. A cross-language language model pre-training method, characterized in that: include: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: determining the number of cluster sets of the word sense cluster set; According to the number of cluster sets, determining the central word sense vector of the center position of each word sense cluster set; according to the central word sense vector, constructing the word sense cluster sets matching the number of cluster sets; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector; The word meaning vector probability of the vocabulary is determined according to the second context word vector, the context word vectors in the context word vector set, and the current context word vector output by the pre-trained language model.

9. The cross-language language model pre-training method according to claim 8, characterized in that: The step of selecting a word sense vector that matches the first context word vector from the constructed word sense clustering set includes: According to the distance between the first context word vector and the central word sense vector, the word sense vector that matches the first context word vector is selected.

10. The cross-language language model pre-training method according to claim 8, characterized in that: The step of determining a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model includes: When the pre-trained language model is running in the initial training phase, randomly initializing the context word vector set output by the pre-trained language model; Determine the second context word vector corresponding to the word meaning vector according to the randomly initialized context word vector set; or, Updating the context word vector set output by the pre-trained language model according to the determined word meaning vector probability; According to the updated context word vector set, the second context word vector corresponding to the word meaning vector is determined.

11. The cross-language language model pre-training method according to claim 8, characterized in that: The determining, according to the second context word vector, the context word vector set, and the current context word vector output by the pre-trained language model, the word sense vector probability corresponding to the vocabulary includes: The word meaning vector probability corresponding to the vocabulary is determined according to the dot product value of the second context word vector and the current context word vector, and the dot product value of the current context word vector and the context word vector set.

12. The cross-language language model pre-training method according to claim 8, characterized in that: The step of inputting the text information into the pre-trained language model and obtaining the first context word vector of the vocabulary in the text information includes: When the pre-trained language model is a bidirectional language model, inputting the text information into the bidirectional language model to obtain the first context word vector of the next word adjacent to the current word in the text information; When the pre-trained language model is a masked language model, the text information is input into the masked language model to obtain the first context word vector of the current vocabulary in the text information.

13. The cross-language language model pre-training method according to claim 8, characterized in that: Also includes: The training parameters of the pre-trained language model are adjusted according to the word sense vector probability, or the word sense vector probability is used as an input parameter and input into a new language training model.

14. A cross-language language model pre-training device, characterized in that: include: An acquisition unit, used to input text information into a pre-trained language model to acquire a first context word vector of a word in the text information; A construction unit, used for constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: a first determining subunit, a second determining subunit and a construction subunit; the first determining subunit is used for determining the number of cluster sets of the word sense cluster set; the second determining subunit is used for determining the central word sense vector of the center position of each word sense cluster set according to the number of cluster sets; the construction subunit is used for constructing the word sense cluster set matching the number of cluster sets according to the central word sense vector; A selection unit, configured to select a word sense vector that matches the first context word vector from the constructed word sense clustering set; A first determining unit, configured to determine a second context word vector corresponding to the word meaning vector from the context word vector set output by the pre-trained language model; The second determining unit is used to determine the word meaning vector probability corresponding to the vocabulary according to the second context word vector, the context word vectors in the context word vector set, and the current context word vector output by the pre-trained language model.

15. A cross-language named entity recognition method based on transfer learning, characterized in that: include: Obtaining text information to be recognized in a first language; Input the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and perform a named entity recognition task; wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, a target recognition result is obtained.

16. A cross-language sentiment analysis method based on transfer learning, characterized in that: include: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, a target analysis result is obtained.

17. A cross-language sentence relationship inference method based on transfer learning, characterized in that: include: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, a target relationship inference result is obtained.

18. A computer storage medium for storing data generated by a network platform and a program for processing the data generated by the network platform; When the program is read and executed, the following steps are performed: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense clustering set according to the word sense vectors of the vocabulary; comprising: Determining the number of cluster sets of the word meaning cluster sets; According to the number of cluster sets, determining the central word sense vector of the center position of each word sense cluster set; according to the central word sense vector, constructing the word sense cluster sets matching the number of cluster sets; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determining a second context word sense vector corresponding to the word sense vector from the context word vector set output by the pre-trained language model; Determine a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector; Alternatively, when the program is read and executed, the following steps are performed: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: determining the number of cluster sets of the word sense cluster set; determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets; constructing the word sense cluster set matching the number of cluster sets according to the central word sense vector; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector; Determining a word meaning vector probability of the vocabulary according to the second context word vector, the context word vector of the context word vector set, and the current context word vector output by the pre-trained language model; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a named entity recognition task; wherein the cross-language pre-training method uses text information in a second language as a pre-training sample; Based on the language model, obtaining a target recognition result; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a language sentiment analysis task, wherein the cross-language pre-training method uses text information in a second language as a pre-training sample; Based on the language model, obtaining a target analysis result; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, a target relationship inference result is obtained.

19. An electronic device comprising: processor; The memory is used to store a program for processing data generated by the network platform, and when the program is read and executed by the processor, the following steps are performed: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: determining the number of cluster sets of the word sense cluster set; determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets; constructing the word sense cluster set matching the number of cluster sets according to the central word sense vector; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determining a second context word sense vector corresponding to the word sense vector from the context word vector set output by the pre-trained language model; Determine a word meaning vector probability of the vocabulary according to the second context word vector and the first context word vector; Alternatively, when the program is read and executed, the following steps are performed: Inputting text information into a pre-trained language model to obtain a first context word vector of a word in the text information; Constructing a word sense cluster set according to the word sense vector of the vocabulary; comprising: determining the number of cluster sets of the word sense cluster set; determining the central word sense vector at the center position of each word sense cluster set according to the number of cluster sets; constructing the word sense cluster set matching the number of cluster sets according to the central word sense vector; In the constructed word sense clustering set, selecting a word sense vector that matches the first context word vector; Determine, from the context word vector set output by the pre-trained language model, a second context word vector corresponding to the word meaning vector; Determining a word meaning vector probability of the vocabulary according to the second context word vector, the context word vectors in the context word vector set, and the current context word vector output by the pre-trained language model; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Input the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and perform a named entity recognition task; wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, obtaining a target recognition result; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a language sentiment analysis task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, obtaining a target analysis result; Alternatively, when the program is read and executed, the following steps are performed: Obtaining text information to be recognized in a first language; Inputting the text information to be recognized into the language model adjusted according to the cross-language language model pre-training method according to any one of claims 1 to 6 or 8 to 13, and performing a sentence relationship inference task, wherein the cross-language language model pre-training method uses text information in a second language as a pre-training sample; Based on the language model, a target relationship inference result is obtained.

Images