Balance method of actual scene linguistic data and finite state network linguistic data

Abstract

The invention provides a balance method of actual scene linguistic data and finite state network (FSN) linguistic data. In order to train the language model of a continuous speech recognizer, training linguistic data is produced according to the application field of the speech recognizer. Linguistic data mainly has two resources: one part is an actual scene linguistic data which is obtained by tidying records in actual scene, and the other part is FSN linguistic data generated with a finite state network syntactic rule method. Two linguistic data balance methods are mainly researched in the invention, and the invention provides a method that probability comparison of the keyword shared by the actual scene linguistic data and the FSN linguistic data is taken as basis, and a certain multiple of parts of actual scene linguistic data are used for expanding the FSN linguistic data to obtain the final method of language model training linguistic data. The language model for training linguistic data, which is obtained with the method, greatly improves the recognition performance of the continuous speech recognizer.

Description

technical field [0001] The invention belongs to the field of computer technology, and relates to a language model of a continuous speech recognizer, in particular to the problem of making training corpus for a language model in a voice question answering system in a limited field, in particular to a finite state network that considers actual scene corpus and is designed for the actual scene (Finite StateNetwork, FSN) corpus of balanced methods. Background technique [0002] One of the main difficulties in the training of domain-limited language models is the sparse training data. The research on this problem mainly focuses on two aspects: one is the expansion of the corpus, and the other is the corpus smoothing algorithm. The corpus smoothing algorithm does not solve the problem of data sparsity from the root, but can only solve the problems caused by data sparsity to a certain extent, and some algorithms have some shortcomings, such as the Good-Turning Discounting algorithm...

AI Technical Summary

Problems solved by technology

However, the "Wizard of Oz" experiment is not a real application scenario, and the corpus obtained through this implementation is not close to practical; in addition, when obtaining the original training corpus from the network, due to the huge amount of network data and the limitations of search engines, from This part of the corpus (although as many as 30GB) extracted from the network may not be able to cover the words and syntax of the limited domain

In addition, some simple dialogue systems use the training corpus generated by FSN syntax rules to train the language model, and have achieved good results. However, the training corpus generated by this method has a fixed syntax and incomplete vocabulary, so it is only suitable for simple dialogues with fixed forms. Applications

Images