Graph-based machine translation data selection method and machine translation data selection system

Abstract

The invention belongs to the technical field of data processing, and discloses a graph-based machine translation data selection method and a machine translation data selection system. The method comprises firstly building a graph, that is building an undirected graph; then performing label propagation; and finally performing data selection according to probability distribution of the field corresponding to each node after the label propagation is performed. An existing machine translation data selection method is improved; only data in one field can be selected by the existing machine translation data selection method, general characters between the fields are ignored; and for the data in the fields to be divided, the probability distribution of all fields can be given according to the given number of fields and the partial labeled field data, and the general characters between the fields are accepted in the considering scope of the data selection method.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a graph-based machine translation data selection method. Background technique [0002] Machine translation is a process of translating one natural language into another natural language using machine learning techniques. As an important branch of computational linguistics, it involves cognitive science, linguistics and other disciplines, and is one of the ultimate goals of artificial intelligence. [0003] Existing machine translation uses data-driven techniques. Therefore, in theory, as the amount of data increases, the performance of the machine translation system can also be improved. However, when the training data and the source of the corpus to be translated are very different, the translation performance will often be seriously degraded. For example, the translation system trained with the corpus in the news field is obviously not suitable for transla...

AI Technical Summary

Problems solved by technology

[0005] The existing data selection technology is to use the data of a specific field to train a model and score the data of the field to be divided; the main defect of this method is to give a specific value to each sentence pair of the field to be divided Represents the probability that the sentence pair belongs to a specific field and ignores the commonality between certain fields; in fact, there are some specific sentence pairs that can be divided into many fields at the same time, such as a sentence description in the news is information about sports, then this sentence can be classified into both the news field and the sports field; since the existing method initially assumes that the output result is the probability of a single field, and cannot be based on the given Given the number of domains and some marked domain data, the probability distribution of all domains is given, and the commonality between domains cannot be taken into consideration in the data selection method.

Images