Language model pre-training method, text processing method and related equipment

Abstract

The present application discloses a model training method, a text processing method, and related equipment. The method includes: obtaining a sample text sequence and its corresponding relationship label, wherein the sample text sequence is composed of a first language text and a second language text, and the relationship label indicates whether the second language text is translated from the first language text; masking a target word in the sample text sequence to obtain a masked text sequence, wherein the target word includes a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech; encoding the masked text sequence through a language model to obtain a masked representation vector; predicting the word corresponding to the target word and the relationship between the first language text and the second language text based on the masked representation vector through a relationship prediction model; and adjusting the model parameters of the language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a language model pre-training method, a text processing method, and related equipment. Background Art

[0002] The recent development of pre-trained language models (PTMs) has elevated research in natural language processing (NLP) to a new level. These models can be used to learn universal language representations from massive amounts of data without manual labeling, significantly improving downstream task capabilities. The use of pre-trained language models as the backbone of downstream tasks, rather than learning models from scratch, has become a consensus within the artificial intelligence (AI) community. Therefore, pre-training language models is particularly important.

[0003] Currently, language model pre-training typically involves randomly extracting a certain number of words from the original training text and masking them. The unmasked words in the original training text are then used to predict the masked words. This allows the language model to learn the connections between words in the original training text and master the ability to understand text semantics, thereby learning universal language representations from massive corpora. However, because the masked words are randomly extracted from the original training text, the connections between words in the original training text are difficult to learn and interpret. Furthermore, these connections rely heavily on the grammatical correctness of the original training text, resulting in the pre-trained language model being unable to accurately represent the text semantics, ultimately affecting the effectiveness of downstream tasks. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a speech model pre-training method, a text processing method and related equipment, which are used to solve the problem that the current pre-trained language model cannot accurately represent the text semantics and ultimately affects the effectiveness of downstream tasks.

[0005] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for pre-training a language model, comprising:

[0007] Obtaining a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is composed of a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text;

[0008] performing masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech;

[0009] Encoding the masked text sequence through a language model to obtain a mask representation vector of the masked text sequence;

[0010] Predicting, by a relationship prediction model and based on the mask representation vector, a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text;

[0011] Based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, the model parameters of the language model are adjusted to obtain a pre-trained language model.

[0012] The language model pre-training method provided in the embodiment of the present application uses texts in two different languages ​​to splice into a sample text sequence, and uses whether the texts in the two languages ​​are in a translation relationship as the relationship label corresponding to the sample text sequence; further, the sample text sequence and its corresponding relationship label are used as corpus for pre-training the language model, and by adding a relationship prediction model downstream of the language model, introducing an MLM (Masked Language Model) training task and an inter-text relationship prediction task, the language model masters the ability to understand text semantics by learning the relationship between words in the sample text sequence and the relationship between texts in different languages, thereby being able to learn a common language representation from a large amount of corpus; on this basis, for the MLM training task, words with the same part of speech in texts in different languages ​​in the sample text sequence are masked, instead of randomly extracting words from the sample text sequence for masking, so that the masked text sequence obtained after masking can retain more multilingual comparison information, making the relationship between words in the sample text sequence easier to be learned and interpreted by the language model, This is conducive to improving the text representation accuracy of the language model; for the task of predicting the relationship between texts, from the perspective of language genealogy, the more distant the kinship, the more unique the language's expression of a certain spatiotemporal concept can be found in the comparison. By utilizing this objective natural law, the relationship label corresponding to the sample text sequence is used as the supervision signal for the task of predicting the relationship between texts. The relationship prediction model predicts whether there is a translation relationship between two different language texts based on the masked text sequence, so that the language model can utilize the relationship between different language texts and fully explore the expressions used to express specific spatiotemporal semantics in various languages, which is conducive to improving the text representation accuracy of the language model.

[0013] In a second aspect, an embodiment of the present application provides a method for training a text processing model, wherein the text processing model includes a pre-trained language model and a task processing model, wherein the pre-trained language model is trained based on the pre-training method of the language model described in the first aspect, and the task processing model is used to perform a target task, the method comprising:

[0014] Obtaining a sample text and its corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task;

[0015] Encoding the sample text using the pre-trained language model to obtain a text representation vector corresponding to the sample text;

[0016] Performing the target task on the sample text based on the text representation vector by the task processing model to obtain a text processing result corresponding to the sample text;

[0017] Based on the text processing results and task labels corresponding to the sample texts, the model parameters of the text processing model are adjusted.

[0018] The training method of the text processing model provided in the embodiment of the present application is as follows: after the language model is trained by the above-mentioned pre-training method, a task processing model for performing the target task is added downstream of the pre-trained language model according to the target task of text processing, so as to obtain the text processing model; further, since the pre-trained language model already has the ability to accurately represent the semantics of the text, after the sample text is encoded by the pre-trained language model to obtain the corresponding text representation vector, the task label corresponding to the sample text is used as the supervision signal of the target task, and the target task is performed on the sample text based on the supervision signal and the text representation vector through the text processing task, and the text processing model is fine-tuned (Fine Tune) based on the obtained text processing result and the task label corresponding to the sample text, so as to complete the training of the text processing model. This can not only improve the text processing accuracy of the text processing model, but also save a lot of computing resources and computing time compared to training a model from scratch (Training a model from scatch), thereby improving training efficiency.

[0019] In a third aspect, an embodiment of the present application provides a text processing method, including:

[0020] Get the target text to be processed;

[0021] The target text is input into a text processing model to obtain a text processing result corresponding to the target text; wherein the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in the first aspect; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0022] The text processing method provided in the embodiment of the present application is beneficial to improving the accuracy of text processing by using the text processing model trained by the above-mentioned text processing model training method to process the text to be processed.

[0023] In a fourth aspect, an embodiment of the present application provides a language model pre-training device, comprising:

[0024] an acquisition unit, configured to acquire a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is formed by concatenating a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text;

[0025] a masking unit configured to perform masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech;

[0026] an encoding unit, configured to encode the masked text sequence using a language model to obtain a mask representation vector of the masked text sequence;

[0027] a prediction unit, configured to predict a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text based on the mask representation vector using a relationship prediction model;

[0028] An adjustment unit is used to adjust the model parameters of the language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label to obtain a pre-trained language model.

[0029] In a fifth aspect, an embodiment of the present application provides a training device for a text processing model, wherein the text processing model includes a pre-trained language model and a task processing model, wherein the pre-trained language model is trained based on the pre-training method described in the first aspect, and the task processing model is used to perform a target task, and the device includes:

[0030] An acquisition unit, configured to acquire a sample text and a corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task;

[0031] An encoding unit, configured to encode the sample text using the pre-trained language model to obtain a text representation vector corresponding to the sample text;

[0032] a processing unit, configured to execute the target task on the sample text based on the text representation vector by using the task processing model, and obtain a text processing result corresponding to the sample text;

[0033] An adjustment unit is used to adjust the model parameters of the text processing model based on the text processing results and task labels corresponding to the sample text.

[0034] In a sixth aspect, an embodiment of the present application provides a text processing device, including:

[0035] An acquisition unit, used for acquiring target text to be processed;

[0036] A processing unit, configured to input the target text into a text processing model to obtain a text processing result corresponding to the target text;

[0037] In which, the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method described in the first aspect; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0038] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method described in the first aspect; or, the processor is configured to execute the instructions to implement the method described in the second aspect; or, the processor is configured to execute the instructions to implement the method described in the third aspect.

[0039] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method as described in the first aspect; or, when the instructions in the storage medium are executed by the processor of the electronic device, enables the electronic device to execute the method as described in the second aspect; or, when the instructions in the storage medium are executed by the processor of the electronic device, enables the electronic device to execute the method as described in the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 A flowchart of a language model pre-training method provided in accordance with an embodiment of the present application;

[0042] Figure 2 A flowchart of a language model pre-training method provided in accordance with another embodiment of the application;

[0043] Figure 3 A flowchart of a method for training a text processing model provided in accordance with an embodiment of the present application;

[0044] Figure 4 A flowchart of a method for training a text processing model provided in another embodiment of the present application;

[0045] Figure 5 A flowchart of a text processing method provided in one embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a language model pre-training device provided in one embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of a text processing model training device provided in one embodiment of the present application;

[0048] Figure 8 A schematic structural diagram of a text processing device provided in one embodiment of the present application;

[0049] Figure 9 A schematic structural diagram of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] The terms "first," "second," and the like in this specification and claims are used to distinguish similar objects and are not intended to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate so that the embodiments of the present application can be implemented in sequences other than those illustrated or described herein. In addition, the term "and / or" in this specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects in the preceding and following relationship are in an "or" relationship.

[0052] Some concept descriptions:

[0053] Natural Language Processing (NLP) is an important field in computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language.

[0054] Pre-trained language models (PTMs): Essentially, they learn good feature representations for words by running self-supervised learning methods on a large corpus of data. Examples include BERT (encoder architecture), Mengzi (encoder architecture), CPM (decoder architecture), and mT5 (encoder-decoder architecture).

[0055] BERT, short for Bidirectional Encoder Representation from Transformers, is a pre-trained language model. It emphasizes that instead of using traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training, it uses a new Masked Language Model (MLM) to generate deep bidirectional language representations.

[0056] Attention: The attention mechanism is a mechanism that captures long-range dependencies by focusing on different parts of the input vector.

[0057] As described in the background technology section, current language model pre-training typically involves randomly extracting a certain number of words from the original training text and masking them. The unmasked words in the original training text are then used to predict the masked words. This allows the language model to learn the connections between words in the original training text and master the ability to understand text semantics, thereby enabling it to learn universal language representations from massive corpora. However, because the masked words are randomly extracted from the original training text, the connections between words in the original training text are difficult to learn and interpret. Furthermore, these connections rely heavily on the grammatical correctness of the original training text, resulting in the pre-trained language model being unable to accurately represent the text semantics, ultimately affecting the effectiveness of downstream tasks.

[0058] In view of this, an embodiment of the present application aims to propose a pre-training method for a language model, which uses texts in two different languages ​​to splice into a sample text sequence, and uses whether the texts in the two languages ​​are in a translation relationship as the relationship label corresponding to the sample text sequence; further, the sample text sequence and its corresponding relationship label are used as corpus for pre-training the language model, and by adding a relationship prediction model downstream of the language model, introducing MLM training tasks and inter-text relationship prediction tasks, so that the language model can master the text semantic understanding ability by learning the connection between words in the sample text sequence and the connection between texts in different languages, so that it can learn a common language representation from a large amount of corpus; on this basis, for the MLM training task, the words with the same part of speech in texts of different languages ​​in the sample text sequence are masked to replace the random extraction of words from the sample text sequence. The language is masked so that the masked text sequence obtained after masking can retain more multilingual comparison information, making the connection between words in the sample text sequence easier to be learned and interpreted by the language model, which is beneficial to improving the text representation accuracy of the language model; for the task of predicting the relationship between texts, from the perspective of language pedigree relationships, the more distant the kinship, the more unique the language expression of a certain spatiotemporal concept can be found in the comparison of languages. By utilizing this objective natural law, the relationship label corresponding to the sample text sequence is used as the supervision signal for the task of predicting the relationship between texts. The relationship prediction model predicts whether there is a translation relationship between two different language texts based on the masked text sequence, so that the language model can utilize the relationship between texts in different languages ​​and fully explore the expressions used to express specific spatiotemporal semantics in various languages, which is beneficial to improving the text representation accuracy of the language model.

[0059] Furthermore, an embodiment of the present application also proposes a training method for a text processing model. After the language model is trained by the above-mentioned pre-training method, a task processing model for performing the target task is added downstream of the pre-trained language model according to the target task of processing the text, thereby obtaining a text processing model. Furthermore, since the pre-trained language model already has the ability to accurately represent the semantics of the text, after the sample text is encoded by the pre-trained language model to obtain the corresponding text representation vector, the task label corresponding to the sample text is used as the supervision signal of the target task. The target task is performed on the sample text based on the supervision signal and the text representation vector through the text processing task, and the text processing model is fine-tuned based on the obtained text processing results and the task label corresponding to the sample text. This can complete the training of the text processing model, which can not only improve the text processing accuracy of the text processing model, but also save a lot of computing resources and computing time compared to training the model from scratch, thereby improving training efficiency.

[0060] Furthermore, an embodiment of the present application also proposes a text processing method. Since the text processing model trained by the above-mentioned text processing model training method has high accuracy, using this text processing model to process the text to be processed is conducive to improving the accuracy of text processing.

[0061] It should be understood that the language model pre-training method, text processing model training method, and text processing method provided in the embodiments of the present application can be executed by an electronic device or software installed in an electronic device. The so-called electronic device here can include terminal devices, such as smart phones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices, smart home appliances, smart watches, vehicle terminals, aircraft, etc.; or, the electronic device can also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0062] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0063] See Figure 1 , is a flow chart of a method for pre-training a language model provided in one embodiment of the present application, the method may include the following steps:

[0064] S102: Obtain a sample text sequence and a relationship tag corresponding to the sample text sequence.

[0065] In the embodiment of the present application, a sample text sequence is formed by concatenating a first language text and a second language text. In practical applications, to facilitate the language model's accurate understanding of the overall semantics of the sample text sequence and the respective semantics of the first and second language texts from the sample text sequence, a start identifier [CLS] may be added before the first language text when concatenating the first and second language texts, and an end identifier [SEP] may be added before and after the second language text, respectively. In other words, the sample text sequence may be "[CLS] first language text [SEP] second language text [SEP]".

[0066] It's worth noting that first-language text refers to text written in the first language, while second-language text refers to text written in the second language. First and second languages ​​are two different languages, for example, Chinese is the first language, and Japanese is the second language. In practical applications, the first and second languages ​​can be determined based on the degree of kinship between the languages. Language kinship, also known as language lineage, refers to the relationship between independent languages ​​that have evolved from the same language and share common characteristics and corresponding relationships in phonetics, vocabulary, and grammar. Language families, language groups, and language branches are all established based on the "kinship" of languages ​​and are therefore called related languages.

[0067] From the perspective of language genealogy, the more distantly related languages ​​are, the more unique a language's ability to express certain spatiotemporal concepts can be found. To make it easier for language models to detect this uniqueness and fully explore the expressions used to express specific spatiotemporal semantics in various languages, the first and second languages ​​can come from different language families. For example, the first language could be Chinese and the second language could be English.

[0068] In the embodiment of the present application, the relationship tag corresponding to the sample text sequence is used to indicate whether the second language text in the sample text sequence is translated from the first language text. For example, if the second language text in the sample text sequence is translated from the first language text, then the relationship tag corresponding to the sample text sequence can be expressed as IsTranslate; if the second language text in the sample text sequence is not translated from the first language text, then the relationship tag corresponding to the sample text sequence can be expressed as NotTranslate. Two sample text sequences and their corresponding relationship tags are shown below:

[0069] Sample text sequence 1: [CLS] the man went to the store [SEP] The man just went to the store [SEP], the corresponding relation label Lable1 = IsTranslate;

[0070] Sample text sequence 2: [CLS]the man went to the store[SEP]He likes sports[SEP], and the corresponding relationship label Lable1 = NotTranslate.

[0071] S104, perform masking on the target words in the sample text sequence to obtain a masked text sequence.

[0072] Among them, the target words include the first word in the first language text and the second word in the second language text, and the first word and the second word have the same词性. By masking the words with the same词性 in different language texts in the sample text sequence, the masked text sequence obtained after masking can retain more multilingual comparison information, making the connections between words in the sample text sequence easier to be learned and interpreted by the language model, thus facilitating the improvement of the text representation accuracy of the language model.

[0073] In the embodiments of this application, the target words in the sample text sequence can be masked in any appropriate manner. Optionally, the target words in the sample text sequence can be replaced with a preset mask character to obtain a masked text sequence. Among them, the preset mask character can be set according to actual needs, such as setting it to [MASK] to clearly distinguish it from the unmasked words in the sample text sequence. The embodiments of this application do not limit the form of the preset mask character.

[0074] Exemplarily, taking the sample text sequence 1 and sample text sequence 2 mentioned above as examples, by replacing the first word "went" and the second word "去" in the sample text sequence 1 with the preset mask character respectively, the following masked text sequence 1 can be obtained; by replacing the first word "went" and the second word "喜欢" in the sample text sequence 2 with the preset mask character respectively, the following masked text sequence 2 can be obtained.

[0075] Masked text sequence 1: [CLS]the man[MASK]to the store[SEP]This man just [MASK] the store[SEP];

[0076] Masked text sequence 2: [CLS]the man[MASK]to the store[SEP]He [MASK] sports[SEP].

[0077] It can be understood that by replacing the target words in the sample text sequence with the preset mask character, the implementation is simple and conducive to improving the pre-training efficiency of the language model.

[0078] It should be noted that the Chinese character "词性" in the original text seems to be a misspelling. It might be intended to be "词性" (English: "word property" or "lexical category"). I translated it as "词性" for the sake of literal translation. If it has a specific meaning in your context, please adjust accordingly.Optionally, in the above S104, if there are too few masked words in the sample text sequence, it will lead to insufficient learning of the language model, thereby increasing the pre-training duration of the language model and reducing the pre-training efficiency; if there are too many masked words in the sample text sequence, it will lead to excessive loss of semantic information in the sample text sequence and make it difficult to be understood by the language model, affecting the accuracy of the text representation of the language model. In view of this, in order to improve the accuracy of the text representation of the language model on the basis of尽可能缩短语言模型的预训练时长 as much as possible, the above S104 may include the following steps: First, count the total number of word types in the sample text sequence; then, based on this total number and the word types of the words in the sample text sequence, select words that meet the preset masking conditions as target words from the sample text sequence, where the preset masking conditions include: the first word in the first language text and the second word in the second language text have the same word type, the proportion of the number of word types included in the target word in this total number is less than the preset ratio, and words of the same word type in the first language text can only be masked once; finally, perform masking processing on the target words in the sample text sequence to obtain a masked text sequence. In practical applications, the preset ratio can be set according to actual needs, and the embodiments of this application do not limit this.

[0079] Exemplarily, taking the above-mentioned sample text sequence 1 "[CLS]the man went to the store[SEP]这个男人刚刚去商店[SEP]" as an example, the first language text includes the article "the", the nouns "man" and "store", the verb "went", and the preposition "to", and the second language text includes the pronoun "这个", the nouns "男人" and "商店", the adverb "刚刚", and the verb "去", that is, the sample text sequence contains a total of 6 word types. Assuming that the preset ratio is 25%, then it can be determined that the number of word types included in the target word does not exceed 1.5, and further it can be determined that the number of word types included in the target word is 1; further, one word with the same word type can be selected from the first language text and the second language text respectively, for example, select the verb "went" from the first language text and the verb "去" from the second language text as the target word; finally, by performing masking processing on the target word in the sample text sequence 1, the masked text sequence 1 corresponding to the sample text sequence 1 "[CLS]theman[MASK]to the store[SEP]这个男人刚刚[MASK]商店[SEP]" can be obtained.

[0080] It should be noted that the part "尽可能缩短语言模型的预训练时长" in the translation of seems to be an incomplete or incorrect expression in Chinese. It might need to be further clarified in the original Chinese text for a more accurate translation.In the embodiment of the present application, the part of speech of each word in the sample text sequence can be determined by various appropriate methods, which can be selected according to actual needs, and the embodiment of the present application does not limit this. Optionally, in order to ensure the accuracy of the part of speech, before the total number of the parts of speech of the words in the above-mentioned statistical sample text sequence, the pre-training method of the language model provided in the embodiment of the present application can also include: step A1, performing word segmentation processing on the sample text sequence to obtain a plurality of words; step A2, based on the word vectors corresponding to each of the multiple words, the positions of each of the multiple words in the sample text sequence, and the language texts to which each of the multiple words belongs in the sample text sequence, encoding the sample text sequence to obtain a sample representation vector corresponding to the sample text sequence; step A3, performing part of speech prediction on each of the multiple words based on the sample representation vector to obtain the parts of speech of each of the multiple words.

[0081] Among them, in step A1, various word segmentation tools commonly used in the field can be used for implementation, which will not be repeated here. In step A2, for each word, a word embedding vector (word_embedding) representing the meaning of the word itself can be obtained by querying the word vector table, and the word is segmented and embedded (Segmentation Embedding) to obtain the text segmentation vector (segmentation_embedding) of the word, and the position of the word in the language text to which it belongs is embedded (Position Embedding) to obtain the position vector (position_embedding) of the word; then, the three vectors of the word are added and fused to obtain the representation vector embedding_α of the word; in the above step A3, the part of speech of the word can be predicted based on the matching status between the representation vector embedding_α of the word and the sample representation vectors of words with different parts of speech.

[0082] The embodiment of the present application shows some specific implementations of the above S104. Of course, it should be understood that the above S104 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0083] S106, encoding the masked text sequence through a language model to obtain a masked representation vector of the masked text sequence.

[0084] Among them, encoding the masked text sequence, that is, embedding the masked text sequence, refers to mapping the sample text sequence to a vector space of a certain dimension to obtain a vector (mask representation vector) that can be recognized and processed by a computer. This vector can be used to represent the masked text sequence.

[0085] In the embodiments of the present application, the language model can have any appropriate result and can be selected according to actual needs. The embodiments of the present application are not limited to this. In practical applications, the language model in the embodiments of the present application can be various language models with encoding capabilities commonly used in the field, such as, but not limited to, a combination of one or more of the following models: BERT model, RoBerta model, ERNIE model, SpanBERT model, etc.

[0086] In a more preferred solution, the language model can adopt an encoder-encoder-decoder (Decoder architecture). For example, the language model can be a BERT model, which includes a representation layer and a conversion layer, wherein the representation layer adopts the encoder result and the conversion layer adopts the encoder-decoder structure. Figure 2 As shown, it is assumed that the first language text in the sample text sequence includes N words (i.e., Tok1 to TokN) and the second language text includes M words (i.e., Tok1' to TokM'). After the sample text sequence is input into the language model, the representation layer of the language model performs three embedding representations on each word in the sample text sequence, namely, embedding the word itself (Word Embedding) to obtain the word embedding vector of the word, segmentation embedding of the word (Segmentation Embedding) to obtain the text segmentation vector of the word, and embedding the position of the word in the language text to which it belongs (Position Embedding) to obtain the position vector of the word. Then, the three vectors of each word are added and fused to obtain the representation vector of each word, i.e., E [CLS] 、E1~E N 、E [SEP] 、E1'~E M '、E [SEP] '; Then, the language model's transformer layer uses the attention mechanism to fuse the representation vector of each word with the representation vector of the word's context words to obtain the target representation vector of the word. The mask representation vector of the sample masked text sequence includes the target representation vector of each word in the masked text sequence, namely C, T1~T N 、T [SEP] 、T1'~T M '、T [SEP] ', where C is the target representation vector corresponding to the symbol [CLS], which is used to aggregate the representation information of the entire masked text sequence.

[0087] S108 , predicting the predicted word corresponding to the target word and the predicted relationship between the first language text and the second language text based on the mask representation vector using a relationship prediction model.

[0088] In the embodiment of this application, Figure 2 As shown in the figure, the relationship prediction model is equivalent to the output layer added downstream of the language model, which is used to perform MLM tasks and relationship prediction tasks. Among them, the MLM task refers to predicting the predicted words corresponding to the target words in the masked text sequence based on the masked representation vector output by the language model, which is equivalent to filling in the blanks at the positions of the masked words in the masked text sequence to obtain the predicted words corresponding to the target words (including the predicted words corresponding to the first word and the predicted words corresponding to the second word); the relationship prediction task refers to predicting the relationship between the first language text and the second language text in the sample text sequence based on the masked representation vector output by the language model. The relationship here refers to whether the second language text is translated from the first language text.

[0089] In practical applications, the relationship prediction model can be a pre-trained model with any appropriate results, such as a neural network, etc., and can be selected according to actual needs. The embodiments of the present application do not limit this.

[0090] S110 , adjusting model parameters of a language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label to obtain a pre-trained language model.

[0091] Among them, the model parameters of the language model may include the number of nodes (such as neurons) in each network layer in the language model, the connection relationship between nodes in different network layers and the connection edge weights, the bias corresponding to the nodes in each network layer, etc.

[0092] Optionally, the above S110 may include the following steps: S1101, determining the prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label.

[0093] Among them, the prediction loss corresponding to the sample text sequence is used to represent the difference between the output result of the mask representation vector of the mask text sequence corresponding to the sample text sequence by the relational prediction model and the expected result. It can quantitatively reflect the accuracy of the language model's text representation of the masked text sequence.

[0094] Since the relationship prediction model will perform MLM tasks and relationship prediction tasks based on the output of the language model, there will be a certain difference between the actual execution result of each task and the expected execution result corresponding to the task, that is, each task will generate a certain prediction loss. Based on this, in order to accurately calculate the prediction loss corresponding to the sample text sequence to improve the training effect of the language model, optionally, the above S1101 can be specifically implemented as follows: based on the target word and its corresponding predicted word, determine the word prediction loss corresponding to the sample text sequence; based on the predicted relationship between the first language text and the second language text in the sample text sequence and the relationship label corresponding to the sample text, determine the relationship prediction loss corresponding to the sample text sequence; further, perform a weighted sum of the word prediction loss and the relationship prediction loss to obtain the prediction loss corresponding to the sample text sequence.

[0095] Among them, the word prediction loss corresponding to the sample text sequence refers to the prediction loss caused by the MLM task performed by the relationship prediction model based on the output of the language model. The word prediction loss corresponding to the sample text sequence is used to represent the difference between the target word in the sample text sequence and the predicted word corresponding to the target word. For example, the difference between the target word in the sample text sequence and the predicted word corresponding to the target word can be calculated based on the first preset loss function to obtain the word prediction loss corresponding to the sample text sequence. The first preset loss function can be set according to actual needs, such as a distance function, etc., and the embodiment of the present application is not limited to this.

[0096] The relationship prediction loss corresponding to the sample text sequence refers to the prediction loss caused by the relationship prediction task performed by the relationship prediction model based on the output of the language model. The word prediction loss corresponding to the sample text sequence is used to represent the difference between the predicted relationship between the first language text and the second language text in the sample text sequence and the relationship label corresponding to the sample text sequence. For example, the difference between the predicted relationship and the relationship label can be calculated based on the second preset loss function to obtain the relationship prediction loss corresponding to the sample text sequence. The second preset loss function can be set according to actual needs, such as the cross entropy loss function, etc., and the embodiments of the present application are not limited to this.

[0097] The weights corresponding to the word prediction loss and the relationship prediction loss can be set according to actual needs, and the embodiments of the present application do not limit this.

[0098] S1102: Adjust the model parameters of the language model based on the prediction loss corresponding to the sample text sequence to obtain a pre-trained language model.

[0099] For example, a back propagation algorithm and a gradient descent algorithm may be used to adjust model parameters of a language model based on the prediction loss corresponding to a sample text sequence to obtain a pre-trained language model.

[0100] It can be understood that since the prediction loss corresponding to the sample text sequence can quantitatively reflect the accuracy of the language model's text representation of the masked text sequence, adjusting the model parameters of the language model based on the prediction loss corresponding to the sample text sequence is beneficial to accelerating the convergence of the language model and improving the pre-training efficiency of the language model on the basis of improving the representation accuracy of the language model.

[0101] The embodiment of the present application shows a specific implementation of the above S110. Of course, it should be understood that the above S110 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0102] It should be noted that, in actual applications, there are multiple sample text sequences, which include positive samples and negative samples. If the second language text in a sample text sequence is translated from the first language text, then the sample text sequence is called a positive sample; if the second language text in a sample text sequence is not translated from the first language text, then the sample text sequence is called a negative sample. For each sample text sequence, the predicted words corresponding to the target words in the sample text sequence and the predicted relationship between the first language text and the second language text in the sample text sequence can be obtained through S102 to S108 above; in S110 above, the prediction loss corresponding to each sample text sequence can be calculated according to the method described above, and then the prediction loss corresponding to all sample text sequences can be calculated.

[0103] The total prediction loss is obtained by adding the losses. Furthermore, the model parameters of the language model are adjusted based on the total prediction loss to obtain a pre-trained language model.

[0104] The above process is only a single model parameter adjustment process. In actual applications, the model parameters of the language model may need to be adjusted multiple times. Therefore, the above steps S102 to S110 can be repeatedly performed until the first preset training stop condition is met. The language model obtained at this time can be used as the pre-trained language model. The first preset training stop condition may include the total prediction loss of all sample text sequences being less than a preset loss threshold or the number of adjustments reaching a preset number threshold, etc. The specific setting can be based on actual needs and is not limited in this embodiment of the present application.

[0105] The language model pre-training method provided by one or more embodiments of the present application uses texts in two different languages ​​to splice into a sample text sequence, and uses whether the texts in the two languages ​​are in a translation relationship as the relationship label corresponding to the sample text sequence; further, the sample text sequence and its corresponding relationship label are used as the pre-training method for the language model.

[0106] By adding a relation prediction model to the downstream of the language model, introducing the MLM training task and the inter-text relation prediction task, the language model can learn the relationship between words in the sample text sequence and the relationship between texts in different languages.

[0107] The connection between the text and the semantic understanding of the text is mastered, so that the general language representation can be learned from the massive corpus. On this basis, for the MLM training task, the words with the same part of speech in the texts of different languages ​​in the sample text sequence are masked to replace the random extraction of words from the sample text sequence for masking, so that the masked text sequence obtained after masking is

[0108] It can retain more multilingual contrast information, making the connection between words in the sample text sequence easier for the language model to learn and explain, thereby improving the accuracy of the text representation of the language model; for the task of predicting the relationship between texts,

[0109] From the perspective of language genealogy, the more distant the kinship, the more unique a language can be found in expressing a certain concept of time and space. Based on this objective natural law, the relationship labels corresponding to the sample text sequences are used as the supervisory signals for the task of predicting the relationship between texts. The relationship prediction model predicts whether there is a translation relationship between two different language texts based on the masked text sequence.

[0110] The system makes predictions, enabling the language model to utilize the relationship between texts in different languages ​​and fully explore the expressions used to express specific spatiotemporal semantics in various languages, thereby improving the text representation accuracy of the language model.

[0111] Based on the language model pre-training method provided in the embodiment of the present application, the embodiment of the present application also provides a text processing model training method. The training process of the text processing model is described in detail below.

[0112] See Figure 3 , which is a flow chart of a method for training a text processing model provided in one embodiment of the present application

[0113] FIG, the method may include the following steps:

[0114] S302: Obtain sample text and the task label corresponding to the sample text.

[0115] The task label corresponding to the sample text represents the expected processing result of the sample text under the target task. Target tasks can include text generation tasks or text classification tasks. Text generation tasks can include, but are not limited to, determining response text, text translation, generating text summaries, and generating a paragraph based on input prompt information. Text classification tasks can include, but are not limited to, sentiment analysis, spam detection, and intent recognition.

[0116] The same sample text has different corresponding task labels for different target tasks. For example, if the target task is intent recognition, then the task label corresponding to the sample text under this target task is used to represent the intent of the sample text; for another example, if the target task is sentiment analysis, then the task label corresponding to the sample text under this target task is used to represent whether the sentiment of the sample text is positive or negative; for another example, if the target task is text summarization, then the task label corresponding to the sample text under this target task is the summary of the sample text, and so on.

[0117] S304: Encode the sample text using a pre-trained language model in the text processing model to obtain a text representation vector corresponding to the sample text.

[0118] Among them, the pre-trained language model is trained based on the pre-training method of the language model provided in the embodiment of the present application.

[0119] like Figure 4 As shown, the sample text is input into the pre-trained language model to obtain the text representation vector corresponding to the sample text. The specific implementation process of the above S304 is the same as the above Figure 1 The specific implementation process of S106 in the method shown is similar. For details, please refer to the detailed description of S106 above, which will not be repeated here.

[0120] S306 , executing the target task on the sample text based on the text representation vector through the task processing model in the text processing model to obtain a text processing result corresponding to the sample text.

[0121] like Figure 4 As shown, the task processing model is a model that matches the pre-trained language model and is used to perform the downstream target task. The text representation vector obtained by the pre-trained language model is input into the task processing model, which then performs the target task to obtain the text processing result corresponding to the sample text.

[0122] The task processing model varies depending on the target task. For example, if the target task is intent recognition, then the task processing model is an intent recognition model. By inputting the text representation vector of the sample text into the task processing model, the intent of the sample text can be obtained. For another example, if the target task is sentiment analysis, then the task processing model is a sentiment analysis model. By inputting the text representation vector of the sample text into the task processing model, the sentiment of the sample text can be obtained. For another example, if the target task is text summary generation, then the task processing model is a summary generation model. By inputting the text representation vector of the sample text into the task processing model, the summary of the sample text can be obtained, and so on.

[0123] In practical applications, the task processing model can have any appropriate structure, and can be specifically set according to actual needs, which is not limited in the embodiments of the present application. For example, the task processing model may include a linear layer (Linear) and a normalization layer (Softmax), wherein the linear layer is used to convert the text representation vector corresponding to the sample text into a target feature vector based on matrix multiplication; the normalization layer is used to use the Softmax function to perform the target task on the sample text based on the target feature vector corresponding to the sample text, and obtain the text processing result corresponding to the sample text. For example, taking the target task as an intent recognition task, the normalization layer can use the Softmax function to calculate the probability that the sample text belongs to multiple candidate intents based on the target feature vector corresponding to the sample text; further, the candidate intent with the largest probability is selected from the multiple candidate intents as the intent of the sample text.

[0124] S308: Adjust the model parameters of the text processing model based on the text processing results and task labels corresponding to the sample text.

[0125] Among them, the model parameters of the text processing model may include the model parameters of the task processing model and the model parameters of the pre-trained language model. The model parameters of each model in the text processing model may include the number of nodes (such as neurons) in each network layer in each model, the connection relationship between nodes in different network layers and the connection edge weights, the bias corresponding to the nodes in each network layer, etc. Of course, it should be understood that in actual applications, since the pre-trained language model already has the ability to accurately represent the semantics of the text, in the above S308, only the model parameters of the task processing model can be adjusted.

[0126] like Figure 4 As shown, in the above S308, the prediction loss of the text processing model can be calculated based on the text processing result and the task label corresponding to the sample text. The prediction loss of the text processing model is used to represent the difference between the text processing result corresponding to the sample text and the task label corresponding to the sample text, which can quantitatively reflect the accuracy of the text processing model in processing the sample text. Then, based on the prediction loss of the text processing model, the model parameters of the text processing model are adjusted. More specifically, the back propagation algorithm and the gradient descent algorithm can be used to adjust the model parameters of the text processing model based on the prediction loss of the text processing model.

[0127] It is understandable that since the prediction loss of the text processing model can quantitatively reflect the accuracy of the text processing model in processing sample text, adjusting the model parameters of the text processing model based on the prediction loss of the text processing model is beneficial to accelerating the convergence of the text processing model and improving the pre-training efficiency of the text processing model on the basis of improving the accuracy of the text processing model.

[0128] The embodiment of the present application shows a specific implementation of the above S308. Of course, it should be understood that the above S308 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0129] The above process is only a single model parameter adjustment process. In actual applications, the model parameters of the text processing model may need to be adjusted multiple times. Therefore, the above steps S302 to S308 can be repeatedly performed until the second preset training stop condition is met. At this point, the resulting text processing model can be used for text processing. The second preset training stop condition may include the prediction loss of the text processing model being less than a preset loss threshold or the number of adjustments reaching a preset number threshold, etc. The specific setting can be based on actual needs and is not limited in this embodiment of the application.

[0130] Through the training method of the text processing model provided by one or more embodiments of the present application, after the language model is trained by the above-mentioned pre-training method, a task processing model for performing the target task is added downstream of the pre-trained language model according to the target task of text processing, so that the text processing model can be obtained; further, since the pre-trained language model already has the ability to accurately represent the semantics of the text, after the sample text is encoded by the pre-trained language model to obtain the corresponding text representation vector, the task label corresponding to the sample text is used as the supervision signal of the target task, and the target task is performed on the sample text based on the supervision signal and the text representation vector through the text processing task, and the text processing model is fine-tuned based on the obtained text processing results and the task label corresponding to the sample text, so as to complete the training of the text processing model. This can not only improve the text processing accuracy of the text processing model, but also save a lot of computing resources and computing time compared to training the model from scratch, thereby improving training efficiency.

[0131] See Figure 5 , is a flowchart of a text processing method provided in one embodiment of the present application, the method may include the following steps:

[0132] S502: Obtain the target text to be processed.

[0133] The target text to be processed refers to the text that needs to be processed.

[0134] S504: Input the target text into the text processing model to obtain a text processing result corresponding to the target text.

[0135] The text processing model includes a pre-trained language model and a task processing model. The text processing model is trained using the language model pre-training method described in the embodiments of this application; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; and the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0136] The target task may include a text generation task or a text classification task, wherein the text generation task may include, but is not limited to: determining the response text, text translation, text summary generation, generating a paragraph based on the input prompt information, etc.; the text classification task may include, but is not limited to: sentiment analysis, spam identification, intent identification, etc.

[0137] Taking the target task as an example of intent recognition, the target text is "I want to handle business A". Through the above S502 to S504, the target text is subjected to intent recognition, and it can be identified that the intent of the target text is "the customer has the intention to handle business A".

[0138] The text processing method provided by one or more embodiments of the present application has a text processing model trained by the above-mentioned text processing model training method, which has high accuracy. Using this text processing model to process the text to be processed is conducive to improving the accuracy of text processing.

[0139] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] In addition, with the above Figure 1 Corresponding to the language model pre-training method shown in FIG, the present application embodiment also provides a language model pre-training device. Figure 6 , is a schematic structural diagram of a language model pre-training device 600 provided in one embodiment of the present application. The device 600 may include:

[0141] An acquiring unit 610 is configured to acquire a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is formed by concatenating a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text;

[0142] a masking unit 620 configured to perform masking on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, wherein the first word and the second word have the same part of speech;

[0143] an encoding unit 630, configured to encode the masked text sequence using a language model to obtain a mask representation vector of the masked text sequence;

[0144] A prediction unit 640 is configured to predict a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text based on the mask representation vector using a relationship prediction model;

[0145] The adjustment unit 650 is configured to adjust the model parameters of the language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label to obtain a pre-trained language model.

[0146] Optionally, the masking unit is specifically used to: count the total number of parts of speech of words in the sample text sequence; based on the total number and the parts of speech of the words in the sample text sequence, select words that meet preset masking conditions from the sample text sequence as target words, wherein the preset masking conditions include: a first word in the first language text and a second word in the second language text have the same part of speech, the number of parts of speech contained in the target word accounts for less than a preset proportion in the total number, and words with the same part of speech in the first language text can only be masked once; masking is performed on the target words in the sample text sequence to obtain a masked text sequence.

[0147] Optionally, the language model pre-training device further comprises: a word segmentation unit, configured to perform word segmentation processing on the sample text sequence to obtain a plurality of words before the masking unit counts the total number of parts of speech of the words in the sample text sequence;

[0148] The encoding unit is further configured to perform encoding processing on the sample text sequence based on the word vectors corresponding to the multiple words, the positions of the multiple words in the sample text sequence, and the language texts to which the multiple words belong in the sample text sequence, so as to obtain a sample representation vector corresponding to the sample text sequence;

[0149] The language model pre-training device further includes: a prediction unit, configured to perform part-of-speech prediction on each of the plurality of words based on the sample representation vectors to obtain the part-of-speech of each of the plurality of words.

[0150] Optionally, the adjustment unit is specifically used to: determine the prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label; and adjust the model parameters of the language model based on the prediction loss corresponding to the sample text sequence to obtain a pre-trained language model.

[0151] Optionally, the adjustment unit determines the prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, including: determining the word prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word; determining the relationship prediction loss corresponding to the sample text sequence based on the predicted relationship and the relationship label; and performing weighted summation of the word prediction loss and the relationship prediction loss to obtain the prediction loss corresponding to the sample text sequence.

[0152] Obviously, the language model pre-training device provided in the embodiment of the present application can be used as Figure 1 The execution body of the pre-training method of the language model shown, for example Figure 1 In the language model pre-training method shown in FIG, step S102 can be performed by Figure 6 The acquisition unit in the pre-training device of the language model shown in FIG. 1 is executed, and step S104 can be performed by Figure 6 The mask unit in the pre-training device of the language model shown in FIG. 1 is executed, and step S106 can be performed by Figure 6 The encoding unit in the pre-training device of the language model shown in FIG. 1 is executed, and step S108 can be performed by Figure 6 The prediction unit in the pre-training device of the language model shown in FIG. 1 is executed, and step S110 can be performed by Figure 6 The language model pre-training device shown is performed by the adjustment unit.

[0153] According to another embodiment of the present application, Figure 6 The various units in the pre-training device of the language model shown can be separately or all merged into one or several other units to constitute, or a certain (some) unit therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the pre-training device of the language model can also include other units. In practical applications, these functions can also be assisted by other units to achieve, and can be achieved by the collaboration of multiple units.

[0154] According to another embodiment of the present application, the program can be executed on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), and other processing elements and storage elements. Figure 1 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 6 The computer program may be recorded on, for example, a computer-readable storage medium, and transferred to an electronic device via the computer-readable storage medium and run therein.

[0155] In addition, with the above Figure 3 Corresponding to the training method of the text processing model shown in FIG, the present application embodiment also provides a training device for a text processing model. Figure 7 , is a structural diagram of a text processing model training device 700 provided in one embodiment of the present application, wherein the text processing model includes a pre-trained language model and a task processing model. The pre-trained language model is trained based on the pre-training method of the language model described in the embodiment of the present application, and the task processing model is used to perform the target task. Figure 7 As shown, the apparatus 700 may include:

[0156] An acquisition unit 710 is configured to acquire a sample text and a corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task;

[0157] An encoding unit 720 is configured to encode the sample text using the pre-trained language model to obtain a text representation vector corresponding to the sample text;

[0158] A processing unit 730 is configured to execute the target task on the sample text based on the text representation vector by using the task processing model to obtain a text processing result corresponding to the sample text;

[0159] The adjusting unit 740 is configured to adjust the model parameters of the text processing model based on the text processing result and the task label corresponding to the sample text.

[0160] Obviously, the training device of the text processing model provided in the embodiment of the present application can be used as Figure 3 The execution body of the training method of the text processing model shown, for example Figure 3 In the training method of the text processing model shown in FIG, step S302 can be performed by Figure 7 The acquisition unit in the training device of the text processing model shown in FIG. 1 is executed, and step S304 can be performed by Figure 7 The encoding unit in the training device of the text processing model shown in FIG. 1 is executed, and step S306 can be performed by Figure 7 The processing unit in the training device of the text processing model shown in FIG. 1 is executed, and step S308 can be performed by Figure 7 The text processing model is shown to be trained by an adjustment unit in the apparatus.

[0161] According to another embodiment of the present application, Figure 7 The various units in the training device of the text processing model shown can be individually or all combined into one or several other units to constitute, or one (or some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the training device of the text processing model may also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0162] According to another embodiment of the present application, the system can be executed on a general computing device such as a computer including processing elements such as a CPU, RAM, ROM and storage elements. Figure 3 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 7 The text processing model training device shown in and the text processing model training method for implementing the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and transferred to an electronic device through the computer-readable storage medium and run therein.

[0163] In addition, with the above Figure 5 Corresponding to the text processing method shown, the present application embodiment also provides a text processing device. Figure 8 , is a structural diagram of a text processing device 800 provided in one embodiment of the present application. The device 800 may include:

[0164] An acquisition unit 810 is used to acquire a target text to be processed;

[0165] The processing unit 820 is used to input the target text into a text processing model to obtain a text processing result corresponding to the target text; wherein the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in the embodiment of the present application; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0166] Obviously, the text processing device provided in the embodiment of the present application can be used as Figure 5 The execution body of the text processing method shown is, for example Figure 5 In the text processing method shown in FIG. 1 , step S502 can be performed by Figure 8 The acquisition unit in the text processing device shown in FIG. 1 is executed, and step S504 can be performed by Figure 8 The processing unit in the text processing device shown executes.

[0167] According to another embodiment of the present application, Figure 8 The various units in the text processing device shown can be individually or all combined into one or several other units to form a structure, or one (or some) of the units can be further divided into multiple functionally smaller units to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the text processing device may also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0168] According to another embodiment of the present application, the system can be executed on a general computing device such as a computer including processing elements such as a CPU, RAM, ROM and storage elements. Figure 5 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 8 The text processing device shown in and the text processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and transferred to an electronic device through the computer-readable storage medium and run therein.

[0169] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 9At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0170] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0171] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0172] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a pre-training device for the language model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0173] Obtaining a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is composed of a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text;

[0174] performing masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech;

[0175] Encoding the masked text sequence through a language model to obtain a mask representation vector of the masked text sequence;

[0176] Predicting, by a relationship prediction model and based on the mask representation vector, a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text;

[0177] Based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, the model parameters of the language model are adjusted to obtain a pre-trained language model.

[0178] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a training device for the text processing model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0179] Obtaining a sample text and its corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task;

[0180] Encoding the sample text using a pre-trained language model in a text processing model to obtain a text representation vector corresponding to the sample text;

[0181] Performing the target task on the sample text based on the text representation vector by a task processing model in the text processing model, obtaining a text processing result corresponding to the sample text;

[0182] Based on the text processing results and task labels corresponding to the sample texts, the model parameters of the text processing model are adjusted.

[0183] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a text processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0184] Get the target text to be processed;

[0185] The target text is input into a text processing model to obtain a text processing result corresponding to the target text; wherein the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in the embodiment of the present application; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0186] The above application Figure 1The method of executing the language model pre-training device disclosed in the embodiment shown in the present application Figure 3 The method for executing the training device of the text processing model disclosed in the embodiment and the above-mentioned application Figure 5 Any of the methods performed by the text processing device disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0187] The electronic device may also perform Figure 1 Method, and implement the language model pre-training device in Figure 1 Alternatively, the electronic device may also perform the functions of the embodiment shown. Figure 3 Method, and implement the training device of text processing model in Figure 3 Alternatively, the electronic device may also perform the functions of the embodiment shown. Figure 5 Method, and implement text processing device in Figure 5 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0188] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0189] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including multiple application programs, can enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0190] Obtaining a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is composed of a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text;

[0191] performing masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech;

[0192] Encoding the masked text sequence through a language model to obtain a mask representation vector of the masked text sequence;

[0193] Predicting, by a relationship prediction model and based on the mask representation vector, a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text;

[0194] Based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, the model parameters of the language model are adjusted to obtain a pre-trained language model.

[0195] Alternatively, when the instruction is executed by a portable electronic device including a plurality of applications, the portable electronic device can execute Figure 3 The method of the embodiment shown is specifically used to perform the following operations:

[0196] Obtaining a sample text and its corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task;

[0197] Encoding the sample text using a pre-trained language model in a text processing model to obtain a text representation vector corresponding to the sample text;

[0198] Performing the target task on the sample text based on the text representation vector by a task processing model in the text processing model, obtaining a text processing result corresponding to the sample text;

[0199] Based on the text processing results and task labels corresponding to the sample texts, the model parameters of the text processing model are adjusted.

[0200] Alternatively, when the instruction is executed by a portable electronic device including a plurality of applications, the portable electronic device can execute Figure 5 The method of the embodiment shown is specifically used to perform the following operations:

[0201] Get the target text to be processed;

[0202] The target text is input into a text processing model to obtain a text processing result corresponding to the target text; wherein the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in the embodiment of the present application; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

[0203] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0204] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0205] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0206] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0207] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A language model pre-training method, characterized in that: include: Obtaining a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is composed of a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text; performing masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech; Encoding the masked text sequence through a language model to obtain a mask representation vector of the masked text sequence; Predicting, by a relationship prediction model and based on the mask representation vector, a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text; Based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, the model parameters of the language model are adjusted to obtain a pre-trained language model.

2. The method according to claim 1, characterized in that The masking process is performed on the target words in the sample text sequence to obtain a masked text sequence, including: Counting the total number of parts of speech of words in the sample text sequence; Based on the total number and the parts of speech of the words in the sample text sequence, selecting words that meet a preset masking condition from the sample text sequence as target words, wherein the preset masking condition includes: a first word in the first language text and a second word in the second language text have the same part of speech, the number of parts of speech contained in the target word accounts for less than a preset proportion in the total number, and words with the same part of speech in the first language text can only be masked once; Masking is performed on the target words in the sample text sequence to obtain a masked text sequence.

3. The method according to claim 2, characterized in that Before counting the total number of parts of speech of words in the sample text sequence, the method further includes: Performing word segmentation processing on the sample text sequence to obtain multiple words; Based on the word vectors corresponding to the multiple words, the positions of the multiple words in the sample text sequence, and the language texts to which the multiple words belong in the sample text sequence, the sample text sequence is encoded to obtain a sample representation vector corresponding to the sample text sequence; Parts of speech are predicted for each of the multiple words based on the sample representation vectors to obtain the parts of speech of each of the multiple words.

4. The method according to claim 1, wherein The adjusting the model parameters of the language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label to obtain a pre-trained language model includes: Determining a prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label; Based on the prediction loss corresponding to the sample text sequence, the model parameters of the language model are adjusted to obtain a pre-trained language model.

5. The method according to claim 4, characterized in that The determining, based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label, of the prediction loss corresponding to the sample text sequence includes: Determining a word prediction loss corresponding to the sample text sequence based on the target word and its corresponding predicted word; Determining a relationship prediction loss corresponding to the sample text sequence based on the predicted relationship and the relationship label; A weighted sum is performed on the word prediction loss and the relationship prediction loss to obtain the prediction loss corresponding to the sample text sequence.

6. A training method for a text processing model, characterized in that: The text processing model includes a pre-trained language model and a task processing model, wherein the pre-trained language model is trained based on the pre-training method of the language model according to any one of claims 1 to 5, and the task processing model is used to perform the target task, and the method includes: Obtaining a sample text and its corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task; Encoding the sample text using the pre-trained language model to obtain a text representation vector corresponding to the sample text; Performing the target task on the sample text based on the text representation vector by the task processing model to obtain a text processing result corresponding to the sample text; Based on the text processing results and task labels corresponding to the sample texts, the model parameters of the text processing model are adjusted.

7. A text processing method, characterized in that: include: Get the target text to be processed; Inputting the target text into a text processing model to obtain a text processing result corresponding to the target text; In which, the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in any one of claims 1 to 5; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

8. A language model pre-training device, characterized in that: include: an acquisition unit, configured to acquire a sample text sequence and a relationship tag corresponding to the sample text sequence, wherein the sample text sequence is formed by concatenating a first language text and a second language text, and the relationship tag is used to indicate whether the second language text in the sample text sequence is translated from the first language text; a masking unit configured to perform masking processing on target words in the sample text sequence to obtain a masked text sequence, wherein the target words include a first word in the first language text and a second word in the second language text, and the first word and the second word have the same part of speech; an encoding unit, configured to encode the masked text sequence using a language model to obtain a mask representation vector of the masked text sequence; a prediction unit, configured to predict a predicted word corresponding to the target word and a predicted relationship between the first language text and the second language text based on the mask representation vector using a relationship prediction model; An adjustment unit is used to adjust the model parameters of the language model based on the target word and its corresponding predicted word, the predicted relationship between the first language text and the second language text, and the relationship label to obtain a pre-trained language model.

9. A training device for a text processing model, characterized in that: The text processing model includes a pre-trained language model and a task processing model, wherein the pre-trained language model is obtained by training based on the pre-training method of the language model according to any one of claims 1 to 5, and the task processing model is used to perform the target task. The device includes: An acquisition unit, configured to acquire a sample text and a corresponding task label, wherein the task label is used to indicate an expected processing result corresponding to the sample text under the target task; An encoding unit, configured to encode the sample text using the pre-trained language model to obtain a text representation vector corresponding to the sample text; a processing unit, configured to execute the target task on the sample text based on the text representation vector by using the task processing model, and obtain a text processing result corresponding to the sample text; An adjustment unit is used to adjust the model parameters of the text processing model based on the text processing results and task labels corresponding to the sample text.

10. A text processing device, characterized in that: include: An acquisition unit, used for acquiring target text to be processed; A processing unit, configured to input the target text into a text processing model to obtain a text processing result corresponding to the target text; In which, the text processing model includes a pre-trained language model and a task processing model, and the text processing model is trained based on the pre-training method of the language model described in any one of claims 1 to 5; the pre-trained language model is used to encode the target text to obtain a text representation vector corresponding to the target text; the task processing model is used to perform a target task on the target text based on the text representation vector corresponding to the target text to obtain a text processing result corresponding to the target text.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 5; or, the processor is configured to execute the instructions to implement the method according to claim 6; or, the processor is configured to execute the instructions to implement the method according to claim 7.

12. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 5; or, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method as claimed in claim 6; or, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method as claimed in claim 7.

Images