Cross-language topic detection method and system

Abstract

The invention discloses a cross-language topic detection method and system, wherein the method comprises the following steps of building a comparable corpus of a first language and a second language; respectively building a first language topic model and a second language topic model on the basis of the comparable corpus; determining the alignment of a first language topic and a second language topic through similarity judgment on the basis of document-topic probability distribution generated by the first language topic model and the second language topic model so as to realize cross-language topic detection. The system comprises a first generating module, a second generation module and a detection module. The cross-language topic detection method and the cross-language topic detection system provided by the invention have the advantages that the accuracy rate of cross-language document similarity calculation is improved; through the building of the topic models based on LDA (latent dirichlet allocation), the cross-language topic detection is realized by utilizing the cross-language topic alignment.

Description

Technical field [0001] The invention relates to the technical field of cross-language topic detection, in particular to a cross-language topic detection method and system based on a comparable corpus. Background technique [0002] The research of cross-language topic detection helps people of different countries and nationalities to share knowledge, enhance the network information security of various countries and national areas, promote the economic and cultural development of my country’s national areas, promote national unity, and build a "harmonious society" and " The social environment of "scientific development" provides important conditions for support. [0003] Currently, cross-language topic detection mainly has three methods based on machine translation, bilingual dictionary, and bilingual parallel corpus. For cross-language detection methods based on machine translation and dictionary, since each language has its own characteristics, there will be semantic deviations and...

AI Technical Summary

Problems solved by technology

For cross-language detection methods based on machine translation and dictionaries, since each language has its own characteristics, in the process of translation from the source language to the target language, there will be semantic deviations and noises, which will change the source language news. The meaning expressed in the report affects the accuracy of text and topic similarity calculations

Therefore, the translation strategy cannot fundamentally improve the performance of cross-lingual topic detection.

The main difficulty of cross-language topic detection methods based on parallel corpora is that parallel corpora are difficult to obtain and resources are scarce.

Images