Compound word processing method and device used for semantic mining and equipment thereof

Abstract

The invention puts forward a compound word processing method and device used for semantic mining and equipment thereof. The method comprises the following steps: determining M segmented words of eachstatement in a training corpus; selecting N segmented words to generate N-dimension compound words according to the order of appearance of the M segmented words, wherein M is larger than or equal to 2and N is larger than or equal to 2 but smaller than or equal to M; putting character strings of N-dimension compound words into K-time hash operations, searching a pre-established random hash dictionary space to obtain positions only corresponding to hash operations each time and generating K-dimension word vectors of the N-dimension compound words according to floating point numbers of K positions corresponding to the K-time hash operation results, wherein K is an integer larger than 1; and screening out N-dimension target compound words meeting the pre-set condition according to K-dimensionword vectors of all the N-dimension compound words and inputting the N-dimension target compound words into a word bag model for semantic mining. Therefore, semantic features with more and larger granularities are introduced into the word bag model so that the effect of the word bag model is further improved.

Description

technical field [0001] The invention relates to the technical field of information processing, in particular to a compound word processing method, device and equipment for semantic mining. Background technique [0002] Artificial Intelligence (Artificial Intelligence), the English abbreviation is AI. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that responds in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems, etc. [0003] At present, in the text semantic relevance matching task, the common bag of words (Bag of Words) model has a wide range of applications. In the related technology...

AI Technical Summary

Problems solved by technology

[0006] For this reason, the first purpose of the present invention is to propose a compound word processing method for semantic mining, which is used to solve the high cost of corpus training in the prior art, and in order to improve the effect of the bag-of-words model, more binary words need to be introduced. Meta-bundled words, which affect memory performance; or simply embed the compound words obtained by bundling two words into the word vector space as new semantic features, and input them to the bag-of-words model, which affects the effect of the bag-of-words model

Images