A bilingual word embedding-based cross-language text similarity assessment technique

Abstract

The invention belongs to the field of language processing, in particular to a cross-language text similarity evaluation technology based on bilingual word embedding. The technical route and workflow of cross-language text similarity evaluation technology based on bilingual word embedding can be divided into three stages: the construction of bilingual word embedding model, the construction of textsimilarity calculation framework based on multi-neural network, and the cross-language similarity calculation. Through this model, a bilingual shared word embedding representation can be generated, which is based on the word vector correlation theory and Skip-Gram model is used to train word vectors on artificially constructed pseudo-bilingual corpus. Secondly, in order to make the generated wordembedding space as complete as possible, monolingual corpus is used as a supplement to learn additional word embedding knowledge. The similarity score of sentences is obtained by combining several neural network structures to learn the semantic representation of sentences. By dividing short text into paragraphs and treating paragraphs as long sentences as sequence input, the similarity iteration on a larger scale can be realized.

Description

technical field [0001] The invention belongs to the field of language processing, and in particular relates to a cross-language text similarity evaluation technology based on bilingual word embedding. Background technique [0002] Methods based on statistical machine learning are currently the mainstream of research in the field of natural language processing. These methods usually automatically or semi-automatically obtain statistical knowledge of language from training data, and can effectively establish language representation models. However, methods based on statistical machine learning depend to a large extent on the scale, representativeness, correctness, and processing depth of training data. The more language data trained and the stronger the domain, the better the fit of the language model . It can be said that the quality of training data determines the effect of statistical machine learning methods to a large extent. Therefore, by expanding the corpus to continu...

AI Technical Summary

Problems solved by technology

These symbolic representations and shallow models do not describe the semantic information contained in the data, so they cannot fully express the semantic relationship between data in different languages. It is difficult to use a unified and effective method for multi-task, multi-label specification, and multi-language learning.

Images