Cross-language task training

Abstract

In an embodiment of the present disclosure, a cross-language task training method is proposed. After obtaining a training sentence in one language, a certain word in the training sentence is masked, and then a corresponding word in another language corresponding to the masked word is obtained. Next, the masked sentence in one language and the corresponding word in another language are used to pre-train a cross-language model. The pre-trained cross-language model has the ability to understand and process multiple languages, and it can be further trained according to specific tasks. According to an embodiment of the present disclosure, when there is a large amount of training corpus in one language and less or even no training corpus in another language, the cross-language model can be pre-trained through a cross-language training method, so that the cross-language model can be applied to another language, thereby realizing model training for various languages.

Description

Technical Field

[0001] The present application generally relates to the field of computers, and in particular, to a method, device and computer program product for cross-language task training. Background Art

[0002] Natural language processing refers to the technology of using computers to process human natural language. Natural language processing enables computers to understand human language. Computers are trained through manually annotated corpora to generate semantic representations of natural language. Natural language processing is a hot topic in the field of artificial intelligence. It can be applied to semantic analysis, information retrieval, machine translation, automatic question answering, chatbots, etc.

[0003] A language model is a probability distribution of a sequence of words, which is the basis of natural language processing technology. Usually, a language model can be constructed by training a large corpus. For example, a neural network-based language model can use a three-layer feedforward neural network, which optimizes the parameters of the neural network-based language model based on back propagation technology during the training process. The trained language model can learn the relationship between sentences or words, and can therefore be used to predict the next word or sentence, etc. Summary of the invention

[0004] In an embodiment of the present disclosure, a cross-language task training method is proposed. After obtaining a training sentence in one language, a certain word in the training sentence is masked, and a corresponding word in another language corresponding to the masked word is obtained. Then, the masked sentence in one language and the corresponding word in another language are used to pre-train a cross-language model. The pre-trained cross-language model has the ability to understand and process multiple languages, and it can be further trained according to specific tasks. According to an embodiment of the present disclosure, when there is a large amount of training corpus in one language and less or even no training corpus in another language, the cross-language model can be pre-trained through a cross-language training method, so that the cross-language model can be applied to another language, thereby realizing model training of various languages.

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0007] Figure 1 A block diagram illustrating a computing device / server in which one or more embodiments of the present disclosure may be implemented;

[0008] Figure 2 A flowchart of a method for pre-training a cross-language model in which the present disclosure may be implemented is shown;

[0009] Figure 3 A schematic diagram showing a process for training a cross-lingual model for a specific task in which the present disclosure may be implemented is shown;

[0010] Figure 4 A schematic diagram showing an example of pre-training a cross-language model according to an embodiment of the present disclosure is shown;

[0011] Figure 5 A schematic diagram showing another example of pre-training a cross-language model according to an embodiment of the present disclosure is shown;

[0012] Figure 6 A schematic diagram showing another example of pre-training a cross-language model according to an embodiment of the present disclosure is shown;

[0013] Figure 7 A schematic diagram showing an example for further training a cross-language model according to an embodiment of the present disclosure is shown;

[0014] Figure 8 A schematic diagram showing an example for fine-tuning a cross-language model according to an embodiment of the present disclosure; and

[0015] Fig. 9 A diagram showing an example of providing a question and answer in a search engine according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0018] Traditionally, in order to train a language model for a certain language, a large amount of training corpus of the language is usually required to complete the training of the language model. However, for certain tasks, not every language has a large amount of training corpus. Since there are only a few training corpora or even no training corpora in many languages, language models for specific tasks in many languages ​​cannot be trained at all.

[0019] To this end, an embodiment of the present disclosure proposes a cross-language task training method. According to an embodiment of the present disclosure, when there is a training corpus for one language and there is less or even no training corpus for another language, a cross-language model can be pre-trained by cross-language training, so that the cross-language model can be applied to another language, thereby realizing model training for various languages. It can be seen that according to the cross-language training method proposed in an embodiment of the present disclosure, a multilingual model applicable to multiple languages ​​can be trained without the need for training corpora in multiple languages.

[0020] The following references Figures 1 to 9 To illustrate the basic principles and several example implementations of the present disclosure. Figure 1 1 shows a block diagram of a computing device / server 100 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 1 The computing device / server 100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0021] like Figure 1 As shown, computing device / server 100 is in the form of a general-purpose computing device. Components of computing device / server 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160. Processing unit 110 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device / server 100.

[0022] The computing device / server 100 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device / server 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 130 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device / server 100.

[0023] The computing device / server 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 1 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 120 may include a computer program product 125 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0024] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device / server 100 may be implemented in a single computing cluster or multiple computing machines that are capable of communicating via a communication connection. Thus, the computing device / server 100 may operate in a networked environment using a logical connection to one or more other servers, a network personal computer (PC), or another network node.

[0025] Input device 150 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 160 may be one or more output devices, such as a display, speaker, printer, etc. Computing device / server 100 may also communicate with one or more external devices (not shown) as needed, such as storage devices, display devices, etc., with one or more devices that enable a user to interact with computing device / server 100, or with any device that enables computing device / server 100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.) through communication unit 140. Such communication may be performed via an input / output (I / O) interface (not shown).

[0026] like Figure 1 As shown, a training corpus for training a cross-language model may exist in the storage device 130. The program product 125 can implement pre-training and fine-tuning of the cross-language model based on the training corpus stored in the storage device 130. The cross-language model trained according to the embodiments of the present disclosure can be used for a variety of tasks in a variety of languages, for example, a Chinese question-answering model can be trained with only a small number of Chinese question-answering corpora. It should be understood that although English is used as the first language and Chinese is used as the second language in the following embodiments, the second language may also be other languages, such as French, German, Japanese, etc. In addition, the first language is a language with a large amount of training corpus, which may also be a language other than English.

[0027] Figure 2 1 shows a flow chart of a method 200 for pre-training a cross-language model in which the present disclosure may be implemented. It should be understood that the method 200 may be referred to as Figure 1 The computing device / server 100 described here performs. For the sake of clarity Figure 2 Method 200, herein also referred to Figure 4 The example 400 shown for pre-training a cross-lingual model is described together.

[0028] At 202, based on the masking of the first word in the first sentence of the first language, a first masked sentence in the first language is generated. In some embodiments, the first language may be English, for example. Figure 4 , the first sentence can be an English sentence "This is an example", which can serve as a training corpus sentence, which can come from an English encyclopedia, an English book corpus, etc. Figure 4 , the English word "example" in the English sentence is masked to "[MASK]", thereby obtaining the English masked sentence 420 "This is an [MASK]", where the meaning of mask means blocking, shielding, etc. Figure 4 In the example of , assume that there is a large amount of training data in English (the first language) and little or no training data in Chinese (the second language).

[0029] At 204, a second word in a second language corresponding to the first word is determined, wherein the first language and the second language are different languages. Figure 4In the example, the second language can be, for example, Chinese, and the Chinese word corresponding to the masked English word "example" is the word 450 "example". In some embodiments, the Chinese word 450 can be a word that has been manually annotated. Alternatively, the word 450 can also be obtained by translating the masked English word "example" through a translation system. It should be understood that various existing or future-developed translation systems, platforms, programs, interfaces, etc. can be used in combination with the embodiments of the present disclosure.

[0030] At 206, a first masked sentence in a first language and a second word in a second language are used to pre-train a cross-lingual encoder, where the cross-lingual encoder is to be further trained based on a specific task. As Figure 4 shown, an English masked sentence 420 "This is an [MASK]" can be used as the input to the cross-lingual model 410, and the Chinese word 450 "example" can be used as the output of the cross-lingual model 410 to pre-train the cross-lingual model, where the cross-lingual model includes a cross-lingual encoder, or a cross-lingual encoder and a decoder. Therefore, the cross-lingual model trained according to the embodiments of the present disclosure can have multi-lingual understanding and processing capabilities.

[0031] It should be understood that in the embodiments of the present disclosure, the term "pre-training" means that there are at least two stages of the training process. In the first-stage pre-training process, the cross-lingual model is trained using general training data so that the cross-lingual model has multi-lingual understanding and processing capabilities; in the second-stage further training and / or fine-tuning process, for a specific task, the cross-lingual model is further trained so that the trained cross-lingual model can handle the specific task, such as handling a question-and-answer task, a natural language inference task, and so on.

[0032] Figure 3 shows a schematic diagram of a process 300 for training a cross-lingual model for a specific task according to the present disclosure. As Figure 3 shown, in order to train a cross-lingual model 340 for a specific task (such as a question-and-answer task), a training process of three stages 310-330 can be performed. Any known or future-developed machine learning and / or neural network technology can be used in combination with the embodiments of the present disclosure to implement the cross-lingual model 340. For example, the cross-lingual model 340 can be implemented by a Transformer model, a recurrent neural network (RNN), or a long short-term memory (LSTM) network.

[0033] In the cross-language pre-training stage 310, pre-training can be performed based on task-independent general training data 305, and the cross-language model can be trained together with at least two languages ​​so that the cross-language model has the ability to understand and process at least two languages. Figure 4-6 Several different example training modes of the cross-lingual pre-training stage 310 are shown.

[0034] In the cross-language task-related training stage 320, cross-language training can be further performed based on task-related training data 325, so that the trained cross-language model has cross-language task processing capabilities, wherein the task-related training data 325 can include, for example, a question-answering corpus in a certain language. Figure 7 An example training pattern of the cross-language task-related training phase 320 is shown.

[0035] In the single language task-related fine-tuning stage 330, single language fine-tuning can be performed based on the task-related training data 325, so that the trained cross-language model 340 for a specific task has the task processing capability of the target language. Through further fine-tuning of the single language, the task processing capability of the trained model for the target language can be further improved. For example, if the target language is Chinese, the cross-language model can be fine-tuned based on Chinese questions and Chinese answers. Figure 8 An example training pattern for a single language task-related fine-tuning phase 330 is shown.

[0036] Figure 4 A schematic diagram of an example 400 for pre-training a cross-language model according to an embodiment of the present disclosure is shown. Figure 4 The cross-language word alignment prediction process is shown, the English word "example" in the English sentence "This is an example" is masked to generate an English masked sentence 420 "This is an [MASK]". Then, the English masked sentence 420 is embedded as a token, the position information 430 of each word is embedded as a position, and the language information 440 is embedded as a language, which are input into the cross-language model 410 together, and the Chinese word 450 "example" is used as an output to implement pre-training of the cross-language model 410.

[0037] In some embodiments, the cross-language model 410 may have an encoder-decoder architecture based on an attention mechanism. Figure 4 As shown, a large number of paired English masked sentences and corresponding Chinese words can be used to train the cross-language model 410, so that the cross-language model has the ability to understand and process both English and Chinese. Figure 4The cross - language model 410 can be a bidirectional encoder representation. In some embodiments, if the subsequent model task is related to French, French can be used as the output for pre - training the cross - language model.

[0038] Figure 5 FIG. shows a schematic diagram of another example 500 for pre - training a cross - language model according to an embodiment of the present disclosure. Figure 5 A cross - language paraphrase classification process is shown. The model input is a combination of the English sentence “This is an example” and the Chinese sentence “这是一个例子”. Its output is a flag indicating whether the input English sentence and Chinese sentence have the same meaning, for example, “true” or “false”. As Figure 5 shown, the combination 520 of the English sentence “This is an example” and the Chinese sentence “这是一个例子” can be used as a token embedding, the position information 530 of each word in the corresponding sentence can be used as a position embedding, and the language information 540 can be used as a language embedding, and they are input into the cross - language model 410 together, and the flag 550 (such as “true”) is used as the output to implement the pre - training of the cross - language model 410. It should be understood that the pre - training method of example 500 is different from the training method of traditional translation systems because in example 500, sentences in two languages are both input into the cross - language model as model inputs, rather than generating an output in another language based on an input in one language.

[0039] Figure 6 FIG. shows a schematic diagram of yet another example 600 for pre - training a cross - language model according to an embodiment of the present disclosure. Figure 6 A cross - language next - sentence prediction process is shown. The model input is a combination of the English sentence “This is an example” and the Chinese sentence “明天出发去纽约”. Its output is a flag indicating whether the latter sentence is semantically the next sentence of the former sentence, for example, “true” or “false”. As Figure 6 shown, the combination 620 of the English sentence “This is an example” and the Chinese sentence “明天出发去纽约” is used as a token embedding, the position information 630 of each word in the corresponding sentence is used as a position embedding, and the language information 640 is used as a language embedding, and they are input into the cross - language model 410 together, and the flag 650 (such as “true”) is used as the output to implement the pre - training of the cross - language model 410. It should be understood that although Figure 4-6 only one example in each training mode is shown, in fact, for each training mode, a large amount of training data in this training mode is required for training.

[0040] Figure 7FIG. 7 is a schematic diagram of an example 700 for further training a cross-language model according to an embodiment of the present disclosure. Figure 7 As shown in FIG, after completing the pre-training of the cross-language model, further task-related training can be performed according to the specific task. For example, Figure 7 An example 700 for training a question-answering task is shown. Since there is only a large amount of English question-answering corpus but not enough Chinese question-answering corpus, a question-answering pair is first obtained from the English question-answering corpus, such as the English question "Why is the sky blue" and the English answer "The atmosphere of the Earth scatters blue light most". Then, a translation system can be used to automatically translate the English answer "The atmosphere of the Earth scatters blue light most" into the Chinese answer "The Earth's atmosphere scatters blue light most". Figure 7 As shown, the English question "Why is the sky blue" and the Chinese answer "The Earth's atmosphere scatters the most blue light" can be used as input 720, and then the cross-language model 410 generates input embeddings 730 for each token, and then generates context representations 740 for each token. In example 700, the output 750 is a true or false flag indicating whether the English question and the Chinese answer match.

[0041] In some embodiments, English questions may be translated into Chinese questions while keeping the English answers unchanged, and then the Chinese questions and English answers may be used to train the cross-language model 410 for question answering in the Chinese language.

[0042] Figure 8 FIG. 8 is a schematic diagram of an example 800 for fine-tuning a cross-language model according to an embodiment of the present disclosure. Figure 8 As shown, after completing Figure 7 After further task-related training of the cross-lingual model in , task-related fine-tuning can be performed according to specific tasks. For example, Figure 8An example 800 for training a question-answering task is shown. Since there is only a large amount of English question-answering corpus but not enough Chinese corpus, a question-answer pair is first obtained from the English question-answering corpus, such as the English question "Why is the sky blue" and the English answer "The atmosphere of the Earth scatters bluelight most". Then, the English question and the English answer can be automatically translated into the Chinese question "Why is the sky blue" and the Chinese answer "The Earth's atmosphere scatters blue light most" using a translation system, and the Chinese question and the Chinese answer are used as input 820, and then the cross-language model 410 generates input embeddings 830 for each tag, and then generates contextual representations 840 for each tag. In example 700, the output 850 is a true or false flag indicating whether the Chinese question and the Chinese answer match. That is, in the fine-tuning stage, the cross-language model for question-answering can be fine-tuned using the question in the target language and the answer in the target language.

[0043] Fig. 9 FIG. 9 is a diagram showing an example of providing a question and answer in a search engine according to an embodiment of the present disclosure. Figure 8 The trained cross-language model is used to process Chinese questions and answers. After receiving a Chinese query 910 in the search engine, the cross-language model determines a Chinese answer 920, and presents the Chinese answer 920 in the search result page of the search engine. In this way, even without a Chinese question and answer corpus, a question and answer model for Chinese can be trained and provided. It should be understood that Chinese is only an example of a target language, and other languages ​​can also be used in combination with the embodiments of the present disclosure as target languages.

[0044] In some embodiments, a sentence pair can be obtained from an English natural language reasoning library, and the sentence pair includes an English premise sentence and an English hypothesis sentence, wherein the English premise sentence and the English hypothesis sentence are marked with one of the following relationships: implication, contradiction, and neutrality. For example, the premise sentence "You don't have to stay there" and the hypothesis sentence "You can leave" are in an implication relationship. Optionally, the English hypothesis sentence can be translated from the English language into Chinese, and the English premise sentence, the Chinese hypothesis sentence, and the sentence relationship are used to train a cross-language model for natural language reasoning in the Chinese language. Alternatively, the English premise sentence can also be translated from the English language into Chinese, for example, the English premise sentence "You don't have to stay there" is translated into Chinese "You don't need to stay there", and the Chinese premise sentence "You don't need to stay there", the English hypothesis sentence "You can leave" and the sentence relationship (implication) are used to train a cross-language model for natural language reasoning in the Chinese language.

[0045] Therefore, according to an embodiment of the present disclosure, when there is a training corpus for one language and less or even no training corpus for another language, a cross-language model can be pre-trained through a cross-language training method so that the cross-language model can be applied to another language, thereby realizing model training for various languages.

[0046] The methods and functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0047] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0048] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0049] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0050] Some example implementations of the present disclosure are listed below.

[0051] In one aspect, a computer-implemented method is provided. The method includes: generating a first masked sentence in a first language based on masking of a first word in a first sentence in a first language; determining a second word in a second language corresponding to the first word, wherein the first language is different from the second language; and using the first masked sentence in the first language and the second word in the second language to pre-train a cross-language encoder, the cross-language encoder to be further trained based on a specific task.

[0052] In some embodiments, pre-training the cross-language encoder includes: determining a first mask sentence in a first language as a first input; determining a second word in a second language as a first output; and pre-training the cross-language encoder using the first input and the first output.

[0053] In some embodiments, the method also includes: obtaining a second sentence in a second language; combining the first sentence in the first language and the second sentence in the second language to obtain a second input; and pre-training a cross-language encoder using the second input and the second output, wherein the second output includes a flag indicating whether the first sentence and the second sentence have the same meaning.

[0054] In some embodiments, the method also includes: obtaining a third sentence in a second language; combining the first sentence in the first language and the third sentence in the second language to obtain a third input; and using the third input and the third output to pre-train a cross-language encoder, wherein the third output includes a flag indicating whether the third sentence is semantically the next sentence of the first sentence.

[0055] In some embodiments, the method also includes: after completing pre-training of the cross-language encoder, obtaining a first question-answer pair from a question-answer corpus in the first language, the first question-answer pair including a first question and a first answer in the first language; and performing at least one of the following: translating the first answer from the first language into a second language, and using the first question in the first language and the first answer in the second language to further train the cross-language encoder for question-answering in the second language; and translating the first question from the first language into the second language, and using the first question in the second language and the first answer in the first language to further train the cross-language encoder for question-answering in the second language.

[0056] In some embodiments, the method also includes: obtaining a second question-answer pair from a question-answer corpus, the second question-answer pair comprising a second question and a second answer in the first language; translating the second question and the second answer from the first language into a second language, respectively; and fine-tuning the cross-language encoder using the second question in the second language and the second answer in the second language.

[0057] In some embodiments, the method further includes: responding to receiving a user query in a second language in a search engine; determining an answer in the second language to the user query using a cross-language encoder; and presenting the answer in the second language in a search results page of the search engine.

[0058] In some embodiments, the method also includes: obtaining a first sentence pair from a natural language reasoning library in a first language, wherein the first sentence pair includes a first premise sentence and a first hypothesis sentence in the first language, and the first premise sentence and the first hypothesis sentence are marked with one of the following relations: implication, contradiction, and neutrality; and performing at least one of the following: translating the first hypothesis sentence from the first language into a second language, and using the first premise sentence of the first language, the first hypothesis sentence of the second language, and the relation to train a cross-language encoder for reasoning in the second language; and translating the first premise sentence from the first language into the second language, and using the first premise sentence of the second language, the first hypothesis sentence of the first language, and the relation to train a cross-language encoder for reasoning in the second language.

[0059] In some embodiments, the method further includes: determining a third word in a third language corresponding to the first word; and pre-training a cross-language encoder using the first mask sentence in the first language and the third word in the third language for a task associated with the third language.

[0060] In another aspect, an electronic device is provided. The electronic device includes: a processing unit; a memory coupled to the processing unit and storing instructions, the instructions performing actions when executed by the processing unit, the actions including: generating a first masked sentence in the first language based on a mask of a first word in a first sentence in the first language; determining a second word in a second language corresponding to the first word, the first language being different from the second language; and pre-training a cross-language encoder using the first masked sentence in the first language and the second word in the second language, wherein the cross-language encoder is to be further trained based on a specific task.

[0061] In some embodiments, pre-training the cross-language encoder includes: determining a first mask sentence in a first language as a first input; determining a second word in a second language as a first output; and pre-training the cross-language encoder using the first input and the first output.

[0062] In some embodiments, the actions also include: obtaining a second sentence in a second language; combining the first sentence in the first language and the second sentence in the second language to obtain a second input; and pre-training a cross-language encoder using the second input and the second output, wherein the second output includes a flag indicating whether the first sentence and the second sentence have the same meaning.

[0063] In some embodiments, the actions also include: obtaining a third sentence in a second language; combining the first sentence in the first language and the third sentence in the second language to obtain a third input; and pre-training a cross-language encoder using the third input and the third output, wherein the third output includes a flag indicating whether the third sentence is semantically the next sentence of the first sentence.

[0064] In some embodiments, the actions also include: after completing pre-training of the cross-language encoder, obtaining a first question-answer pair from a question-answer corpus in the first language, wherein the first question-answer pair includes a first question and a first answer in the first language; and performing at least one of the following: translating the first answer from the first language into a second language, and using the first question in the first language and the first answer in the second language to further train the cross-language encoder for question-answering in the second language; and translating the first question from the first language into the second language, and using the first question in the second language and the first answer in the first language to further train the cross-language encoder for question-answering in the second language.

[0065] In some embodiments, the action also includes: obtaining a second question-answer pair from a question-answer corpus, the second question-answer pair comprising a second question and a second answer in the first language; translating the second question and the second answer from the first language into a second language, respectively; and fine-tuning the cross-language encoder using the second question in the second language and the second answer in the second language.

[0066] In some embodiments, the actions further include: responding to receiving a user query in a second language in a search engine; determining an answer in the second language to the user query using a cross-language encoder; and presenting the answer in the second language in a search results page of the search engine.

[0067] In some embodiments, the action also includes: obtaining a first sentence pair from a natural language reasoning library in a first language, the first sentence pair including a first premise sentence and a first hypothesis sentence in the first language, wherein the first premise sentence and the first hypothesis sentence are marked with one of the following relations: implication, contradiction, and neutrality; and performing at least one of the following: translating the first hypothesis sentence from the first language into a second language, and using the first premise sentence in the first language, the first hypothesis sentence in the second language, and the relation to train a cross-language encoder for reasoning in the second language; and translating the first premise sentence from the first language into the second language, and using the first premise sentence in the second language, the first hypothesis sentence in the first language, and the relation to train a cross-language encoder for reasoning in the second language.

[0068] In some embodiments, the actions further include: determining a third word in a third language corresponding to the first word; and pre-training a cross-language encoder using the first mask sentence in the first language and the third word in the third language for a task associated with the third language.

[0069] In another aspect, a computer program product is provided. The computer program product is stored in a computer storage medium and includes machine executable instructions, which when executed in a device cause the device to: generate a first masked sentence in the first language based on a mask of a first word in a first sentence in the first language; determine a second word in a second language corresponding to the first word, wherein the first language is different from the second language; and use the first masked sentence in the first language and the second word in the second language to pre-train a cross-language encoder, the cross-language encoder to be further trained based on a specific task.

[0070] In some embodiments, pre-training the cross-language encoder includes: determining a first mask sentence in a first language as a first input; determining a second word in a second language as a first output; and pre-training the cross-language encoder using the first input and the first output.

[0071] In some embodiments, the machine executable instructions, when executed in a device, further cause the device to: obtain a second sentence in a second language; combine the first sentence in the first language and the second sentence in the second language to obtain a second input; and pre-train a cross-language encoder using the second input and the second output, the second output including a flag indicating whether the first sentence and the second sentence have the same meaning.

[0072] In some embodiments, the machine executable instructions, when executed in a device, further cause the device to: obtain a third sentence in a second language; combine the first sentence in the first language and the third sentence in the second language to obtain a third input; and pre-train a cross-language encoder using the third input and the third output, the third output including a flag indicating whether the third sentence is semantically the next sentence of the first sentence.

[0073] In some embodiments, the machine executable instructions, when executed in the device, further cause the device to: obtain a first question-answer pair from a question-answer corpus in a first language after completing pre-training of the cross-language encoder, wherein the first question-answer pair includes a first question and a first answer in the first language; and perform at least one of the following: translate the first answer from the first language into a second language, and use the first question in the first language and the first answer in the second language to further train the cross-language encoder for question-answering in the second language; and translate the first question from the first language into the second language, and use the first question in the second language and the first answer in the first language to further train the cross-language encoder for question-answering in the second language.

[0074] In some embodiments, the machine executable instructions, when executed in the device, further cause the device to: obtain a second question-answer pair from the question-answer corpus, wherein the second question-answer pair includes a second question and a second answer in the first language; translate the second question and the second answer from the first language into the second language, respectively; and fine-tune the cross-language encoder using the second question in the second language and the second answer in the second language.

[0075] In some embodiments, the machine executable instructions, when executed in a device, further cause the device to: respond to a user query received in a second language in a search engine; determine an answer in the second language to the user query using a cross-language encoder; and present the answer in the second language in a search results page of the search engine.

[0076] In some embodiments, the machine executable instructions, when executed in the device, further cause the device to: obtain a first sentence pair from a natural language reasoning library of a first language, wherein the first sentence pair includes a first premise sentence and a first hypothesis sentence in the first language, and the first premise sentence and the first hypothesis sentence are marked with one of the following relations: implication, contradiction, and neutrality; and perform at least one of the following: translate the first hypothesis sentence from the first language into a second language, and use the first premise sentence of the first language, the first hypothesis sentence of the second language, and the relation to train a cross-language encoder for reasoning in the second language; and translate the first premise sentence from the first language into the second language, and use the first premise sentence of the second language, the first hypothesis sentence of the first language, and the relation to train a cross-language encoder for reasoning in the second language.

[0077] In some embodiments, the machine executable instructions, when executed in the device, further cause the device to: determine a third word in a third language corresponding to the first word; and pre-train a cross-language encoder using the first mask sentence in the first language and the third word in the third language for a task associated with the third language.

[0078] Although the present disclosure has been described in language specific to structure and / or methodological logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: generating a first masked sentence in a first language based on masking of a first word in a first sentence in a first language; determining a second word in a second language corresponding to the first word, the first language being different from the second language; as well as pre-training a cross-language encoder using the first masked sentence in the first language and the second word in the second language, the cross-language encoder to be further trained based on a predetermined task; The pre-trained cross-language encoders include: determining the first masked sentence in the first language as a first input; determining the second word in the second language as a first output; and The cross-lingual encoder is pre-trained using the first input and the first output.

2. The method according to claim 1, further comprising: obtaining a second sentence in the second language; combining the first sentence in the first language and the second sentence in the second language to obtain a second input; as well as The cross-lingual encoder is pre-trained using the second input and the second output, the second output comprising a flag indicating whether the first sentence and the second sentence have the same meaning.

3. The method according to claim 1, further comprising: obtaining a third sentence in the second language; combining the first sentence in the first language and the third sentence in the second language to obtain a third input; as well as The cross-lingual encoder is pre-trained using the third input and a third output, the third output comprising a flag indicating whether the third sentence is semantically the next sentence of the first sentence.

4. The method according to claim 1, further comprising: After completing pre-training of the cross-language encoder, obtaining a first question-answer pair from a question-answer corpus in the first language, the first question-answer pair comprising a first question and a first answer in the first language; as well as Do at least one of the following: translating the first answer from the first language into the second language, and further training the cross-lingual encoder using the first question in the first language and the first answer in the second language for question answering in the second language, and The first question is translated from the first language into the second language, and the cross-language encoder is further trained using the first question in the second language and the first answer in the first language for question answering in the second language.

5. The method according to claim 4, further comprising: Obtain a second question-answer pair from the question-answer corpus, wherein the second question-answer pair includes a second question and a second answer in the first language; translating the second question and the second answer from the first language into the second language respectively; as well as The cross-lingual encoder is fine-tuned using the second question in the second language and the second answer in the second language.

6. The method according to claim 5, further comprising: in response to receiving a user query in the second language in a search engine; determining, using the cross-lingual encoder, an answer in a second language to the user query; as well as The answer in the second language is presented in a search result page of the search engine.

7. The method according to claim 1, further comprising: Obtaining a first sentence pair from a natural language inference library of the first language, the first sentence pair comprising a first premise sentence and a first hypothesis sentence of the first language, wherein the first premise sentence and the first hypothesis sentence are marked with one of the following relations: implication, contradiction, and neutrality; as well as Do at least one of the following: translating the first hypothesis sentence from the first language into the second language, and using the first premise sentence in the first language, the first hypothesis sentence in the second language, and the relationship to train the cross-language encoder for reasoning in the second language, and The first premise sentence is translated from the first language into the second language, and the cross-language encoder is trained using the first premise sentence in the second language, the first hypothesis sentence in the first language, and the relationship for reasoning in the second language.

8. The method according to claim 1, further comprising: determining a third word in a third language corresponding to the first word; as well as A cross-language encoder is pre-trained using the first masked sentence in the first language and a third word in the third language for a task associated with the third language.

9. An electronic device, comprising: Processing unit; a memory coupled to the processing unit and storing instructions, the instructions performing actions when executed by the processing unit, the actions comprising: generating a first masked sentence in a first language based on masking of a first word in a first sentence in a first language; determining a second word in a second language that corresponds to the first word, the first language being different from the second language; and The first masked sentence in the first language and the second words in the second language are used to pre-train a cross-language encoder, which is to be further trained based on a predetermined task.

10. The apparatus of claim 9, wherein pre-training a cross-language encoder comprises: determining the first masked sentence in the first language as a first input; determining the second word in the second language as a first output; as well as The cross-lingual encoder is pre-trained using the first input and the first output.

11. The apparatus of claim 9, the actions further comprising: obtaining a second sentence in the second language; combining the first sentence in the first language and the second sentence in the second language to obtain a second input; as well as The cross-lingual encoder is pre-trained using the second input and the second output, the second output comprising a flag indicating whether the first sentence and the second sentence have the same meaning.

12. The apparatus of claim 9, the actions further comprising: obtaining a third sentence in the second language; combining the first sentence in the first language and the third sentence in the second language to obtain a third input; as well as The cross-lingual encoder is pre-trained using the third input and a third output, the third output comprising a flag indicating whether the third sentence is semantically the next sentence of the first sentence.

13. The apparatus of claim 9, the actions further comprising: After completing pre-training of the cross-language encoder, obtaining a first question-answer pair from a question-answer corpus in the first language, the first question-answer pair comprising a first question and a first answer in the first language; as well as Do at least one of the following: translating the first answer from the first language into the second language, and further training the cross-lingual encoder using the first question in the first language and the first answer in the second language for question answering in the second language, and The first question is translated from the first language into the second language, and the cross-language encoder is further trained using the first question in the second language and the first answer in the first language for question answering in the second language.

14. The apparatus of claim 13, wherein the actions further comprise: Obtain a second question-answer pair from the question-answer corpus, wherein the second question-answer pair includes a second question and a second answer in the first language; translating the second question and the second answer from the first language into the second language respectively; as well as The cross-lingual encoder is fine-tuned using the second question in the second language and the second answer in the second language.

15. The apparatus of claim 14, the actions further comprising: in response to receiving a user query in the second language in a search engine; determining, using the cross-lingual encoder, an answer in a second language to the user query; as well as The answer in the second language is presented in a search result page of the search engine.

16. The apparatus of claim 9, the actions further comprising: Obtaining a first sentence pair from a natural language inference library of the first language, the first sentence pair comprising a first premise sentence and a first hypothesis sentence of the first language, wherein the first premise sentence and the first hypothesis sentence are marked with one of the following relations: implication, contradiction, and neutrality; as well as Do at least one of the following: translating the first hypothesis sentence from the first language into the second language, and using the first premise sentence in the first language, the first hypothesis sentence in the second language, and the relationship to train the cross-language encoder for reasoning in the second language, and The first premise sentence is translated from the first language into the second language, and the cross-language encoder is trained using the first premise sentence in the second language, the first hypothesis sentence in the first language, and the relationship for reasoning in the second language.

17. The apparatus of claim 9, the actions further comprising: determining a third word in a third language corresponding to the first word; as well as A cross-language encoder is pre-trained using the first masked sentence in the first language and a third word in the third language for a task associated with the third language.

18. A computer program product stored in a computer storage medium and comprising machine executable instructions which, when executed in a device, cause the device to: generating a first masked sentence in a first language based on masking of a first word in a first sentence in a first language; determining a second word in a second language corresponding to the first word, the first language being different from the second language; as well as The first masked sentence in the first language and the second words in the second language are used to pre-train a cross-language encoder, which is to be further trained based on a predetermined task.

19. The computer program product of claim 18, further comprising: obtaining a second sentence in the second language; combining the first sentence in the first language and the second sentence in the second language to obtain a second input; as well as The cross-lingual encoder is pre-trained using the second input and the second output, the second output comprising a flag indicating whether the first sentence and the second sentence have the same meaning.

Images