Cross-language data migration method and apparatus

Abstract

Embodiments of the present application provide a cross-language data migration method and device, and relate to the field of artificial intelligence. The method comprises: determining whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; if the word in the first language corresponds to the word in the second language, migrating the labeled data of the word in the first language to the word in the second language to obtain the labeled data of the word in the second language; and if the word in the first language does not correspond to the word in the second language, determining whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same labeled data; and if the extended word of the word in the first language corresponds to the word in the second language, migrating the labeled data of the extended word of the word in the first language to the word in the second language to obtain the labeled data of the word in the second language.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and in particular, to a cross-language data migration method and device. BACKGROUND

[0002] On high-resource languages like English and Chinese, a large amount of dictionary data has been artificially annotated over the past few decades, which enables supervised machine learning algorithms to be fully learned, and the learned models have achieved remarkable results in various natural language processing tasks. However, in many small languages such as Spanish and Russian, artificially annotated resources are very scarce, which makes traditional supervised machine learning algorithms unable to achieve good results, and the cost of annotating a large amount of data for each language in the world is very high, which means that a large amount of annotated data cannot be obtained for many languages. For example, in the business scenario of part-of-speech identification of various words in product reviews, there is a large amount of annotated data on Chinese and English. However, there is only a small amount of annotated data on some languages such as Spanish, which makes it very difficult to train models on these languages. SUMMARY

[0003] The purpose of the present application is to provide a cross-language data migration method and device to solve the problem of how to effectively improve the annotation efficiency of low-resource languages in the prior art.

[0004] According to a first aspect of embodiments of the present application, a cross-language data migration method is provided. The method comprises: determining whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; if the word in the first language corresponds to the word in the second language, migrating annotated data of the word in the first language to the word in the second language to obtain annotated data of the word in the second language; and if the word in the first language does not correspond to the word in the second language, determining whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same annotated data; and if the extended word of the word in the first language corresponds to the word in the second language, migrating annotated data of the extended word of the word in the first language to the word in the second language to obtain annotated data of the word in the second language.

[0005] According to a second aspect of the embodiments of the present application, a device for cross-language data migration is provided. The device comprises: a first determining module configured to determine whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; a first migration module configured to, if the word in the first language corresponds to the word in the second language, migrate labeled data of the word in the first language to the word in the second language to obtain labeled data of the word in the second language; a second determining module configured to, if the word in the first language does not correspond to the word in the second language, determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same labeled data; and a second migration module configured to, if the extended word of the word in the first language corresponds to the word in the second language, migrate labeled data of the extended word of the word in the first language to the word in the second language to obtain labeled data of the word in the second language.

[0006] According to a third aspect of the embodiments of the present application, an electronic device is provided, which comprises: one or more processors; and a computer readable medium configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-language data migration method according to the first aspect of the embodiments of the present application.

[0007] According to a fourth aspect of the embodiments of the present application, a computer readable medium is provided, which stores a computer program, when the computer program is executed by a processor, the cross-language data migration method according to the first aspect of the embodiments of the present application is implemented.

[0008] Through the technical scheme provided in the embodiments of the present application, whether a word in a first language corresponds to a word in a second language is determined based on the correspondence between the words in the first language and the words in the second language; if the word in the first language corresponds to the word in the second language, the annotation data of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language; if the word in the first language does not correspond to the word in the second language, whether an extended word of the word in the first language corresponds to the word in the second language is determined based on the correspondence between the words in the first language and the words in the second language, wherein the word in the first language and the extended word have the same annotation data; if the extended word of the word in the first language corresponds to the word in the second language, the annotation data of the extended word of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language. Compared with other existing methods, the annotation data of the word in the first language (high-resource language) can be migrated to the word in the second language (low-resource language) without manually defining cross-language features, which not only effectively improves the annotation efficiency of the word in the second language (low-resource language), but also effectively reduces the annotation cost of the word in the second language (low-resource language). BRIEF DESCRIPTION OF DRAWINGS

[0009] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0010] Figure 1 A step flowchart of the cross-language data migration method in the embodiment one of the present application;

[0011] Figure 2A A step flowchart of the cross-language data migration method in the embodiment two of the present application;

[0012] Figure 2B A schematic diagram of the cross-language data migration method according to the embodiment two of the present application;

[0013] Figure 3 A structural schematic diagram of the cross-language data migration device in the embodiment three of the present application;

[0014] Figure 4 A structural schematic diagram of the cross-language data migration device in the embodiment four of the present application;

[0015] Figure 5 A structural schematic diagram of the cross-language data migration device in the embodiment five of the present application;

[0016] Figure 6 A structural schematic diagram of the electronic device in the embodiment six of the present application;

[0017] Figure 7 A hardware structure of the electronic device in the embodiment seven of the present application. Detailed Implementation

[0018] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Reference Figure 1 The flowchart illustrates the steps of the cross-language data migration method according to Embodiment 1 of this application.

[0021] Specifically, the cross-language data migration method in this embodiment includes the following steps:

[0022] In step S101, based on the word correspondence between the first language and the second language, it is determined whether a word in the first language corresponds to a word in the second language.

[0023] In this embodiment, the first language can be a high-resource language, such as Chinese or English. A high-resource language can be understood as a language with abundant manually annotated dictionaries or training data. Due to the rich resources of high-resource languages, supervised learning based on high-resource languages ​​yields good results. The second language can be a low-resource language, such as French, Spanish, or Russian. A low-resource language can be understood as a language with very limited manually annotated dictionaries or training data. Due to the lack of resources in low-resource languages, supervised learning based on low-resource languages ​​yields poor results. The word correspondence between the first and second languages ​​specifically includes bilingual dictionaries of the first and second languages, mapping tables of the first and second languages, etc. It is understood that the above description is merely exemplary, and this application embodiment does not impose any limitations on it.

[0024] In step S102, if it is determined that a word in the first language corresponds to a word in the second language based on the word correspondence between the first language and the second language, then the annotation data of the word in the first language is transferred to the word in the second language to obtain the annotation data of the word in the second language.

[0025] In the embodiment, the annotation data can include part-of-speech annotation data, named entity recognition annotation data, and the like. The cross-language migration can be understood as migrating the knowledge or information of one language to another language, thereby obtaining the knowledge or information of another language, and generally migrating the knowledge or information of a high-resource language to a low-resource language to assist the learning of a model of the low-resource language. For example, the part-of-speech annotation data of the English word "Spanish" is a proper noun, and the part-of-speech annotation data of the English word "Spanish" is migrated to obtain the part-of-speech annotation data of the Spanish word "América" is a proper noun. It can be understood that the above description is only exemplary, and the embodiments of the present application do not limit this.

[0026] In a specific example, if it is determined based on the word correspondence relationship between the first language and the second language that the word of the first language corresponds to the word of the second language, the annotation data of the word of the first language is migrated to the word of the second language to obtain the annotation data of the word of the second language. After obtaining the annotation data of the word of the second language, the model based on the second language is directly trained using the annotation data of the word of the second language. In this way, without manually defining cross-language features, the annotation data of the word of the first language can be migrated to the word of the second language, which not only effectively improves the annotation efficiency of the word of the second language, but also effectively reduces the annotation cost of the word of the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not limit this.

[0027] In step S103, if it is determined based on the word correspondence relationship between the first language and the second language that the word of the first language does not have a corresponding word of the second language, it is determined based on the word correspondence relationship whether the extended word of the word of the first language corresponds to the word of the second language.

[0028] In the embodiment, the word of the first language and its extended word have the same annotation data. If it is determined based on the word correspondence relationship between the first language and the second language that the word of the first language does not have a corresponding word of the second language, it is determined based on the word correspondence relationship whether the extended word of the word of the first language corresponds to the word of the second language. If so, the annotation data of the word of the first language can still be migrated to the word of the second language to obtain the annotation data of the word of the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not limit this.

[0029] In some optional embodiments, before determining whether the extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, the method further includes: performing a semantic clustering operation on the word in the first language based on semantic representation data of the word in the first language, to obtain a semantic clustering result of the word in the first language; and determining the extended word of the word in the first language based on the semantic clustering result of the word in the first language. In this way, the extended word of the word in the first language can be accurately determined. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0030] In one specific example, before performing the semantic clustering operation on the word in the first language based on the semantic representation data of the word in the first language, the method further includes: training a word vector model to be trained using unlabeled text in the first language; and performing vectorization on the word in the first language by using the trained word vector model, to obtain the semantic representation data of the word in the first language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0031] In one specific example, the semantic representation data of the word in the first language can be a word vector representation of the word in the first language. The word vector representation can be understood as a high-dimensional vector representation of the word obtained by a word vector learning algorithm on unlabeled corpus in the first language, and such representation can learn the semantic information of the word. Specifically, the word vector representation of the word in the first language is obtained by training a large amount of unlabeled text in the first language, and the specific training method can be word2vec, glove, fasttext, etc. After obtaining the word vector representation of the word in the first language, the semantic clustering operation is performed on the word in the first language based on the word vector representation of the word in the first language, to obtain the semantic clustering result of the word in the first language. Specifically, the semantic clustering result of the word in the first language is obtained by calculating the neighbor words of the word in the first language using the word vector representation of the word in the first language. More specifically, the neighbor words of the word in the first language are calculated by the cosine distance between the word vector representations of the word in the first language, so as to obtain the semantic clustering result of the word in the first language. For example, the k neighbor words of the word x in the first language are calculated by using the following formula:

[0032]

[0033] wherein, kneighbour represents a set of words in the vocabulary V having the largest cosine distance with the word vector representation of the word x in the first language, cos(x, y) represents the cosine distance between the word vector representation y of a word in the vocabulary V and the word vector representation x of the word x in the first language, and argmax represents a function of parameter (set) of a function. In a specific calculation process, the threshold of the number k of the nearest neighbors of the word x in the first language can be set as 5, and the minimum threshold of the cosine distance can be set as 0.3. Of course, other manners can be used for replacement, for example, the CSLS distance can be used to replace the cosine distance. After obtaining the semantic clustering result of the word in the first language, the nearest neighbors in the semantic clustering result of the word in the first language are determined as the extended words of the word in the first language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0034] In step S104, if it is determined that the extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, the annotation data of the extended word of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language.

[0035] In the embodiment, if it is determined that the extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, the annotation data of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language. In this way, the word in the first language can be associated with the word in the second language through the extended word, thereby further improving the annotation efficiency of the word in the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0036] In step S105, if it is determined that the extended word of the word in the first language does not have a corresponding word in the second language based on the word correspondence relationship, it is determined whether to perform a discard operation on the word in the first language based on a preset probability event.

[0037] In the embodiment, if it is determined that the extended word of the word in the first language does not have a corresponding word in the second language based on the word correspondence relationship, it is determined whether to perform a discard operation on the word in the first language based on a preset probability event. The preset probability event can be a dice or a sample from a Bernoulli distribution. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0038] In step S106, if it is determined to perform the discard operation on the words in the first language based on the preset probability event, the discard operation is performed on the words in the first language.

[0039] In this embodiment, if it is determined to perform the discard operation on the words in the first language based on the preset probability event, the discard operation is performed on the words in the first language. For example, a machine rolls a die, if the number on the upper surface of the die is "1", the discard operation is performed on the words in the first language, or a sample is taken from a Bernoulli distribution with a probability P(1) = 0.1, if the sample is "1", the discard operation is performed on the words in the first language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0040] In step S107, if it is determined not to perform the discard operation on the words in the first language based on the preset probability event, the keep operation is performed on the words in the first language.

[0041] In this embodiment, if it is determined not to perform the discard operation on the words in the first language based on the preset probability event, the keep operation is performed on the words in the first language. For example, a machine rolls a die, if the number on the upper surface of the die is "2, 3, 4, 5, 6", the keep operation is performed on the words in the first language, or a sample is taken from a Bernoulli distribution with a probability P(1) = 0.1, if the sample is "0", the keep operation is performed on the words in the first language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0042] By applying the method provided in the embodiment, the labeled data in English can be migrated to Spanish, that is, a Spanish model can be trained. In this process, a large amount of unlabeled English and Spanish monolingual text is needed, the semantic relationship inside the monolingual is modeled through the word vector, and through the small-scale bilingual dictionary as a bridge, a large amount of labeled data in English can be migrated to Spanish, a Spanish part-of-speech tagging model is trained, and the Spanish text is directly part-of-speech tagged. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0043] The cross-language data migration method provided in the embodiment of the present application determines whether a word in a first language corresponds to a word in a second language based on the correspondence between the words in the first language and the words in the second language; if the word in the first language corresponds to the word in the second language, the annotation data of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language; if the word in the first language does not correspond to the word in the second language, the extended word of the word in the first language is determined based on the correspondence between the words in the first language and the words in the second language, wherein the word in the first language and the extended word have the same annotation data; if the extended word of the word in the first language corresponds to the word in the second language, the annotation data of the extended word of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language. Compared with other existing methods, the annotation data of the word in the first language (high-resource language) can be migrated to the word in the second language (low-resource language) without manually defining cross-language features, which not only effectively improves the annotation efficiency of the word in the second language (low-resource language), but also effectively reduces the annotation cost of the word in the second language (low-resource language).

[0044] The cross-language data migration method of the embodiment can be executed by any appropriate device with data processing capability, including but not limited to a camera, a terminal, a mobile terminal, a PC, a server, a vehicle-mounted device, an entertainment device, an advertising device, a personal digital assistant (PDA), a tablet computer, a notebook computer, a palm game console, smart glasses, a smart watch, a wearable device, a virtual display device or a display enhancement device (such as Google Glass, Oculus Rift, Hololens, Gear VR), and the like.

[0045] Referring to Figure 2A FIG. 2 shows a step flowchart of the cross-language data migration method of the second embodiment of the present application.

[0046] Specifically, the cross-language data migration method of the embodiment includes the following steps:

[0047] In step S201, it is determined whether a word in a first language corresponds to a word in a second language based on the correspondence between the words in the first language and the words in the second language.

[0048] In the embodiment of the present application, the word correspondence relationship includes a bilingual dictionary. When determining whether the word in the first language corresponds to a word in the second language based on the word correspondence relationship between the first language and the second language, the word in the first language to be processed is searched from the bilingual dictionary, and it is determined whether the word in the first language corresponds to a word in the second language according to the language correspondence relationship in the bilingual dictionary. It can be understood that the above description is only exemplary, and the embodiment of the present application does not limit this.

[0049] In step S202, if it is determined that the word in the first language corresponds to a word in the second language based on the word correspondence relationship between the first language and the second language, the annotation data of the word in the first language is migrated to the word in the second language to obtain the annotation data of the word in the second language.

[0050] In the embodiment of the present application, if it is determined that the word in the first language corresponds to a word in the second language according to the determination result, the annotation data of the word in the first language is obtained, and the obtained annotation data is used as the annotation data of the word in the second language. It can be understood that the above description is only exemplary, and the embodiment of the present application does not limit this.

[0051] In step S203, the annotation data of the extended word of the word in the second language is determined based on the annotation data of the word in the second language.

[0052] In the embodiment, the annotation data can include part-of-speech annotation data, named entity recognition annotation data, etc. The word in the second language and the extended word corresponding to the word in the second language have the same annotation data. For example, the part-of-speech annotation data of the Spanish word is proper noun, and then it can be determined that the part-of-speech annotation data of the extended word "América" of the Spanish word is also proper noun. It can be understood that the above description is only exemplary, and the embodiment of the present application does not limit this.

[0053] In some optional embodiments, before the annotation data of the extended word of the word in the second language is determined based on the annotation data of the word in the second language, the method further includes: performing a semantic clustering operation on the word in the second language based on semantic representation data of the word in the second language to obtain a semantic clustering result of the word in the second language; and determining the extended word of the word in the second language based on the semantic clustering result of the word in the second language. In this way, the extended word of the word in the second language can be accurately determined. It can be understood that the above description is only exemplary, and the embodiment of the present application does not limit this.

[0054] In one specific example, before performing the semantic clustering operation on the words of the second language based on the semantic representation data of the words of the second language, the method further comprises: training a word vector model to be trained using unlabeled text of the second language; and vectorizing the words of the second language by the trained word vector model to obtain the semantic representation data of the words of the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0055] In one specific example, the semantic representation data of the words of the second language can be a word vector representation of the words of the second language. The word vector representation can be understood as a high-dimensional vector representation of the words obtained by a word vector learning algorithm on unlabeled corpus of the second language, and such representation can learn the semantic information of the words. Specifically, the word vector representation of the words of the second language is trained using a large amount of unlabeled text of the second language, and the specific training method can be word2vec, glove, fasttext, etc. After obtaining the word vector representation of the words of the second language, a semantic clustering operation is performed on the words of the second language based on the word vector representation of the words of the second language to obtain a semantic clustering result of the words of the second language. Specifically, the semantic clustering result of the words of the second language is obtained by calculating the near-neighbor words of the words of the second language using the word vector representation of the words of the second language. More specifically, the semantic clustering result of the words of the second language is obtained by calculating the near-neighbor words of the words of the second language through the cosine distance between the word vector representations of the words of the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0056] In some optional embodiments, the method further comprises: migrating the annotation data of the words in the sentences of the first language to obtain annotation data of the words of the second language; determining annotation data of the extended words of the words of the second language based on the annotation data of the words of the second language; and sampling the extended words of the words of the second language multiple times to obtain annotation data of the words of multiple sentences of the second language. In this way, the annotation data of the words of the second language can be expanded, thereby enriching the annotation of the corpus of the second language. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

[0057] In a specific example, the part-of-speech tagging data of "I" in the Chinese sentence "I like apple" is a pronoun, the part-of-speech tagging data of "like" is a verb, and the part-of-speech tagging data of "apple" is a noun. Then, the part-of-speech tagging data of "I", "like" and "apple" are migrated respectively to obtain the part-of-speech tagging data of English word "I" as a pronoun, the part-of-speech tagging data of English word "like" as a verb, and the part-of-speech tagging data of English word "apple" as a noun. Based on the part-of-speech tagging data of English word "I", the part-of-speech tagging data of the extended words "you, she, he, we, they, them" of English word "I" can be obtained as pronouns. Based on the part-of-speech tagging data of English word "like", the part-of-speech tagging data of the extended words "love, enjoy" of English word "like" can be obtained as verbs. Based on the part-of-speech tagging data of English word "apple", the part-of-speech tagging data of the extended words "pear, orange, banana, watermelon" of English word "apple" can be obtained as nouns. Then, the extended words of English word "I", the extended words of English word "like" and the extended words of English word "apple" are sampled respectively, and the tagging data of the words of 48 English sentences can be obtained. For example, "we love orange.", "they love orange.", "she enjoy watermelon.", etc. It can be understood that the above description is only exemplary, and the embodiments of the present application do not limit this.

[0058] In step S204, if it is determined that the word of the first language does not have a corresponding word of the second language based on the word correspondence relationship between the first language and the second language, it is determined whether the extended word of the word of the first language corresponds to a word of the second language based on the word correspondence relationship.

[0059] Since this step S204 is similar to the above step S103, it will not be described here again.

[0060] In step S205, if it is determined that the extended word of the word of the first language corresponds to a word of the second language based on the word correspondence relationship, the tagging data of the extended word of the word of the first language is migrated to the word of the second language to obtain the tagging data of the word of the second language.

[0061] In the embodiment of the present application, if it is determined according to the judgment result that the extended word of the word in the first language corresponds to the word in the second language, the labeled data of the extended word of the word in the first language is obtained, and the labeled data of the extended word is taken as the labeled data of the word in the second language. It can be understood that the above description is only exemplary, and the embodiment of the present application does not limit this.

[0062] In step S206, if it is determined based on the word correspondence relationship that the extended word of the word in the first language does not have a corresponding word in the second language, it is determined based on a preset probability event whether to perform a discard operation on the word in the first language.

[0063] Since this step S206 is similar to the above-mentioned step S105, it will not be described here again.

[0064] In step S207, if it is determined based on a preset probability event that a discard operation is performed on the word in the first language, a discard operation is performed on the word in the first language.

[0065] Since this step S207 is similar to the above-mentioned step S106, it will not be described here again.

[0066] In step S208, if it is determined based on a preset probability event that a discard operation is not performed on the word in the first language, a keep operation is performed on the word in the first language.

[0067] Since this step S208 is similar to the above-mentioned step S107, it will not be described here again.

[0068] In one specific example, monolingual word vector representation is obtained by training monolingual unlabeled corpus, and then semantic clustering relationship of monolingual words is calculated. A small-scale bilingual dictionary is used as a bridge to transfer the labeled data of the words in the high-resource language to the low-resource language to obtain pseudo-labeled data on the low-resource language. Finally, the model on the low-resource language is directly trained using these pseudo-labeled data to improve the sequence labeling effect of the low-resource language. Taking the transfer of English labeled data to Spanish in the entity recognition task of product review as an example, there is English labeled data, no Spanish labeled data, and a large amount of English and Spanish unlabeled text. The specific transfer process of the labeled data is as follows: Figure 2BThe method comprises the following steps: (1) using a large amount of unlabeled text training of language A and language B (for example, English and Spanish) to obtain monolingual word vector representation; (2) using monolingual word vector representation of language A and language B, semantic clustering results of monolingual vocabulary are obtained by calculating the near neighbor words of each word. After setting a threshold to limit the number of near neighbor words and the cosine distance, the semantic clustering results obtained in this step are similar to the near neighbor of USA, which includes China, Japan, German and the like, and these words belong to the same semantic clustering; (3) performing word-by-word migration on the labeled data of language A, and determining whether the current word is in the bilingual dictionary, if yes, performing (4), otherwise performing (5); (4) the current word of language A is in the bilingual dictionary, and the current word of language A is mapped to the word of language B according to the corresponding relationship of the bilingual dictionary, for example, English Spanish is mapped to Spanish According to the expansion of the semantic clustering relationship of language B, for example America and the like are expanded; (5) if the current word of language A is not in the bilingual dictionary, the current word is first expanded according to the semantic clustering relationship of language A, for example, USA is expanded to China, Japan, German and the like. Then the expanded word is mapped by the bilingual dictionary, if the expanded word is not in the bilingual dictionary, the word of language A is discarded with a probability of p, and is reserved with a probability of 1-p; if the expanded word is in the bilingual dictionary, it is mapped to language B through the bilingual dictionary; (6) only in the case of performing step (4), the migration of each word in a sentence of language A (for example, English) is regarded as an independent event, and a candidate set of each word of language B is randomly sampled, and n times of sampling generates n sentences of language B (for example, Spanish). In summary, for the labeled data of language A (for example, English), the corresponding candidate word set of language B (for example, Spanish) is obtained through steps 3, 4 and 5, and then the pseudo-labeled data of language B (for example, Spanish) is obtained through step 6. Then based on the pseudo-labeled data, an entity recognition model can be trained on language B (for example, Spanish), that is, the entity recognition of Spanish data can be performed. It can be understood that the above description is only exemplary, and the embodiments of the present application do not limit the same.

[0069] Based on the labeled data of the word of the second language, the labeled data of the expanded word of the word of the second language is determined, compared with other existing methods, the labeled data of the word of the second language can be expanded, thereby enriching the labeled data of the corpus of the second language.

[0070] Further, the annotation data of the words in the sentence in the first language is migrated to obtain annotation data of words in the second language, and based on the annotation data of the words in the second language, annotation data of extended words of the words in the second language is determined, and the extended words of the words in the second language are sampled multiple times to obtain annotation data of words in multiple sentences in the second language. Compared with the existing other manners, the annotation data of the words in the second language can be expanded, thereby enriching the annotation of the corpus of the second language.

[0071] The cross-language data migration method of the embodiment can be executed by any appropriate device with data processing capability, including but not limited to a camera, a terminal, a mobile terminal, a PC, a server, a vehicle-mounted device, an entertainment device, an advertising device, a personal digital assistant (PDA), a tablet computer, a notebook computer, a palm game console, smart glasses, a smart watch, a wearable device, a virtual display device or a display enhancement device (such as Google Glass, Oculus Rift, Hololens, Gear VR), and the like.

[0072] Referring to Figure 3 , a structural schematic diagram of a cross-language data migration apparatus in Embodiment Three of the present application is shown.

[0073] The cross-language data migration apparatus of the embodiment comprises: a first determination module 301 configured to determine whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; a first migration module 302 configured to, if the word in the first language corresponds to the word in the second language, migrate annotation data of the word in the first language to the word in the second language to obtain annotation data of the word in the second language; a second determination module 303 configured to, if the word in the first language does not correspond to the word in the second language, determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same annotation data; and a second migration module 304 configured to, if the extended word of the word in the first language corresponds to the word in the second language, migrate annotation data of the extended word of the word in the first language to the word in the second language to obtain annotation data of the word in the second language.

[0074] The cross-language data migration apparatus of the embodiment is configured to implement the corresponding cross-language data migration method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again.

[0075] Referring to Figure 4 , a structural schematic diagram of a cross-language data migration apparatus in Embodiment Four of the present application is shown.

[0076] The cross-language data migration device of the embodiment comprises: a first determination module 401 configured to determine whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; a first migration module 402 configured to, if the word in the first language corresponds to the word in the second language, migrate labeled data of the word in the first language to the word in the second language to obtain labeled data of the word in the second language; a second determination module 405 configured to, if the word in the first language does not correspond to the word in the second language, determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same labeled data; and a second migration module 406 configured to, if the extended word of the word in the first language corresponds to the word in the second language, migrate labeled data of the extended word of the word in the first language to the word in the second language to obtain the labeled data of the word in the second language.

[0077] Optionally, the word correspondence relationship comprises a bilingual dictionary, and the device comprises a judgment module 409 configured to determine a word in a first language to be processed from the bilingual dictionary and determine whether the word in the first language corresponds to a word in a second language according to a language correspondence relationship in the bilingual dictionary.

[0078] Optionally, the device further comprises a fifth determination module 410 configured to, if it is determined according to a judgment result that the word in the first language corresponds to the word in the second language, acquire labeled data of the word in the first language and use the acquired labeled data as labeled data of the word in the second language, and if it is determined according to the judgment result that the extended word of the word in the first language corresponds to the word in the second language, acquire labeled data of the extended word of the word in the first language and use the labeled data of the extended word as the labeled data of the word in the second language.

[0079] Optionally, the device further comprises a training module 411 configured to train a word vector model to be trained using unlabeled text in the first language, and a vectorization module 412 configured to vectorize the word in the first language by using the trained word vector model to obtain semantic representation data of the word in the first language.

[0080] Optionally, the apparatus further comprises a first semantic clustering module 403 configured to perform a semantic clustering operation on the words in the first language based on the semantic representation data of the words in the first language to obtain a semantic clustering result of the words in the first language; and a third determination module 404 configured to determine the extended word of the words in the first language based on the semantic clustering result of the words in the first language.

[0081] Optionally, the apparatus further comprises a fourth determination module 407 configured to determine whether to perform a discard operation on the words in the first language based on a preset probability event if no; and a discard module 408 configured to perform the discard operation on the words in the first language if yes.

[0082] The cross-language data migration apparatus of the embodiment is configured to implement the corresponding cross-language data migration method in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described herein again.

[0083] With reference to Figure 5 , a structural schematic diagram of a cross-language data migration apparatus in Embodiment Five of the present application is shown.

[0084] The cross-language data migration apparatus of the embodiment comprises a first determination module 501 configured to determine whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; a first migration module 502 configured to migrate labeled data of the word in the first language to the word in the second language to obtain labeled data of the word in the second language if the word in the first language corresponds to the word in the second language; a second determination module 503 configured to determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship if the word in the first language does not correspond to the word in the second language, wherein the word in the first language and the extended word have the same labeled data; and a second migration module 504 configured to migrate labeled data of the extended word of the word in the first language to the word in the second language to obtain the labeled data of the word in the second language if the extended word of the word in the first language corresponds to the word in the second language.

[0085] Optionally, the apparatus further comprises a sixth determination module 507 configured to determine labeled data of an extended word of a word in the second language based on the labeled data of the word in the second language.

[0086] Optionally, the apparatus further comprises a second semantic clustering module 505 configured to perform a semantic clustering operation on the words in the second language based on the semantic representation data of the words in the second language to obtain a semantic clustering result of the words in the second language; and a seventh determination module 506 configured to determine the extended word of the words in the second language based on the semantic clustering result of the words in the second language.

[0087] Optionally, the apparatus further comprises a third migration module 508 configured to migrate the annotation data of the words in the sentences in the first language to obtain the annotation data of the words in the second language; an eighth determination module 509 configured to determine the annotation data of the extended word of the words in the second language based on the annotation data of the words in the second language; and a sampling module 510 configured to sample the extended word of the words in the second language multiple times to obtain the annotation data of the words in the multiple sentences in the second language.

[0088] The cross-language data migration apparatus of the embodiment is configured to implement the corresponding cross-language data migration method in the plurality of method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again.

[0089] Figure 6 FIG. 6 is a structural schematic diagram of an electronic device according to an embodiment of the present application; the electronic device can comprise:

[0090] one or more processors 601;

[0091] a computer readable medium 602, which can be configured to store one or more programs,

[0092] when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-language data migration method as described in the above embodiment one or embodiment two.

[0093] Figure 7 FIG. 7 is a hardware structure of an electronic device according to an embodiment of the present application; as shown in the figure, the hardware structure of the electronic device can comprise: a processor 701, a communication interface 702, a computer readable medium 703 and a communication bus 704; Figure 7

[0094] wherein the processor 701, the communication interface 702, the computer readable medium 703 and the communication bus 704 complete mutual communication through the communication bus 704;

[0095] Optionally, the communication interface 702 can be an interface of a communication module, such as an interface of a GSM module;

[0096] ​The processor 701 can be specifically configured to: determine whether a word in the first language corresponds to a word in the second language based on a word correspondence relationship between the first language and the second language; if the word in the first language corresponds to the word in the second language, migrate the label data of the word in the first language to the word in the second language to obtain the label data of the word in the second language; and if the word in the first language does not correspond to the word in the second language, determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same label data; if the extended word of the word in the first language corresponds to the word in the second language, migrate the label data of the extended word of the word in the first language to the word in the second language to obtain the label data of the word in the second language.

[0097] The processor 701 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0098] The computer readable medium 703 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0099] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code configured to perform the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, but not limited to, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire line, optical cable, RF or the like, or any suitable combination of the above.

[0100] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0102] The modules described in the embodiments of the present application can be implemented by software, or can be implemented by hardware. The described modules can also be arranged in a processor, for example, a processor can be described as: a processor includes a first determining module, a first migration module, a second determining module, and a second migration module. Among them, the names of these modules do not constitute a limitation of the modules themselves in some cases, for example, the first determining module can also be described as: a module for determining whether the word of the first language corresponds to the word of the second language based on the correspondence between the words of the first language and the second language.

[0103] As another aspect, the present application also provides a computer readable medium, which stores a computer program, and the computer program is executed by a processor to implement the cross-language data migration method described in the above embodiment one or embodiment two.

[0104] As another aspect, the present application also provides a computer readable medium, which can be included in the apparatus described in the above embodiments, or can exist independently without being assembled into the apparatus. The computer readable medium carries one or more programs, and when the one or more programs are executed by the apparatus, the apparatus is caused to: determine whether a word in a first language corresponds to a word in a second language based on a word correspondence relationship between the first language and the second language; if the word in the first language corresponds to the word in the second language, migrate the annotation data of the word in the first language to the word in the second language to obtain the annotation data of the word in the second language; and if the word in the first language does not correspond to the word in the second language, determine whether an extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same annotation data; and if the extended word of the word in the first language corresponds to the word in the second language, migrate the annotation data of the extended word of the word in the first language to the word in the second language to obtain the annotation data of the word in the second language.

[0105] The expressions “first”, “second”, “the first” or “the second” used in various embodiments of the disclosure can modify various components regardless of order and / or importance, but the expressions do not limit the corresponding components. The above expressions are configured only for the purpose of distinguishing elements from other elements. For example, the first user device and the second user device represent different user devices, although both are user devices. For example, without departing from the scope of the disclosure, a first element can be called a second element, and similarly, a second element can be called a first element.

[0106] When an element (for example, a first element) is referred to as being “operatively or communicatively coupled with / to” or “connected to” another element (for example, a second element), it should be understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via a further element (for example, a third element). In contrast, it can be understood that when an element (for example, a first element) is referred to as being “directly connected” or “directly coupled” to another element (a second element), no element (for example, a third element) is interposed therebetween.

[0107] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be understood by those skilled in the art that the scope of the protection of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. It should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the present application. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed (but not limited to) in the present application.

Claims

1. A method for cross-language data migration, the method comprising: The method comprises: determining whether a word in the first language corresponds to a word in the second language based on a word correspondence relationship between the first language and the second language, wherein the word correspondence relationship comprises a bilingual dictionary of the first language and the second language; if the word in the first language corresponds to the word in the second language, migrating the annotation data of the word in the first language to the word in the second language to obtain the annotation data of the word in the second language; if the word in the first language does not correspond to the word in the second language, determining whether an extended word of the word in the first language corresponds to a word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same annotation data; if the extended word of the word in the first language corresponds to the word in the second language, migrating the annotation data of the extended word of the word in the first language to the word in the second language to obtain the annotation data of the word in the second language; wherein the method further comprises: migrating the annotation data of the word in the first language in the sentence to obtain the annotation data of the word in the second language; determining the annotation data of the extended word of the word in the second language based on the annotation data of the word in the second language, wherein the annotation data of the word in the second language includes the same annotation data as the annotation data of the extended word; sampling the extended word of the word in the second language multiple times to obtain the annotation data of the word in the multiple sentences in the second language, and extending the annotation data of the word in the second language; in response to the bilingual dictionary not including the word in the first language, extending the word in the first language according to a monolingual semantic clustering relationship of the first language; in response to the bilingual dictionary not including the extended word in the first language, performing a discarding operation on the extended word in the first language according to a preset probability event, and performing a retaining operation on the word in the first language according to the preset probability event, wherein the preset probability event at least includes a probability event sampled from a Bernoulli distribution; and in response to the bilingual dictionary including the extended word in the first language, mapping the word in the first language to the word in the second language through the bilingual dictionary.

2. The method of claim 1, wherein, The word correspondence relationship comprises a bilingual dictionary, and the method comprises: determining the word in the first language to be processed from the bilingual dictionary, and judging whether the word in the first language corresponds to a word in the second language according to the language correspondence relationship in the bilingual dictionary.

3. The method of claim 2, wherein, The method further comprises: if it is determined according to the judgment result that the word in the first language corresponds to the word in the second language, obtaining the annotation data of the word in the first language, and taking the obtained annotation data as the annotation data of the word in the second language; If it is determined according to the judgment result that the extended word of the word in the first language corresponds to the word in the second language, the labeled data of the extended word of the word in the first language is obtained, and the labeled data of the extended word is taken as the labeled data of the word in the second language.

4. The method of claim 1, wherein, The method further comprises: performing a semantic clustering operation on the words in the first language based on the semantic representation data of the words in the first language, to obtain a semantic clustering result of the words in the first language; determining the extended word of the word in the first language based on the semantic clustering result of the words in the first language.

5. The method of claim 4, wherein, The method further comprises: training a word vector model to be trained using the unlabeled text in the first language; vectorizing the words in the first language by the trained word vector model, to obtain semantic representation data of the words in the first language.

6. The method of claim 1, wherein, The method further comprises: determining the labeled data of the extended word of the word in the second language based on the labeled data of the word in the second language.

7. The method of claim 6, wherein, The method further comprises: performing a semantic clustering operation on the words in the second language based on the semantic representation data of the words in the second language, to obtain a semantic clustering result of the words in the second language; determining the extended word of the word in the second language based on the semantic clustering result of the words in the second language.

8. A data migration apparatus across languages, characterized by, The apparatus comprises: a first determination module configured to determine whether the word in the first language corresponds to the word in the second language based on a word correspondence relationship between the first language and the second language, wherein the word correspondence relationship comprises a bilingual dictionary of the first language and the second language; a first migration module configured to, if the word in the first language corresponds to the word in the second language, migrate the labeled data of the word in the first language to the word in the second language, to obtain the labeled data of the word in the second language; a second determination module configured to, if the word in the first language does not correspond to the word in the second language, determine whether the extended word of the word in the first language corresponds to the word in the second language based on the word correspondence relationship, wherein the word in the first language and the extended word have the same labeled data; a second migration module configured to, if the extended word of the word in the first language corresponds to the word in the second language, migrate the labeled data of the extended word of the word in the first language to the word in the second language, to obtain the labeled data of the word in the second language; wherein the apparatus is further configured to migrate the labeled data of the words in the sentence in the first language, to obtain the labeled data of the words in the second language; determine the labeled data of the extended word of the word in the second language based on the labeled data of the word in the second language, wherein the labeled data of the word in the second language includes the same labeled data as the labeled data of the extended word; sample the extended word of the word in the second language multiple times, to obtain the labeled data of the words in the multiple sentences in the second language, and extend the labeled data of the word in the second language. In response to the bilingual dictionary not including the word in the first language, expanding the word in the first language according to monolingual semantic clustering relations of the first language; in response to the bilingual dictionary not including the expanded word in the first language, performing a discarding operation on the expanded word in the first language according to a preset probability event, and performing a retaining operation on the word in the first language according to the preset probability event, wherein the preset probability event at least includes a probability event sampled from a Bernoulli distribution; and in response to the bilingual dictionary including the expanded word in the first language, mapping the word in the first language to a word in the second language through the bilingual dictionary.

Images