New word discovering method and system thereof

Abstract

The invention discloses a method to find out new words, and it including the following step: finding the character string which appears more than certain threshold in the sound materials; counting respectively the number of the different characters and words which appear on the right and left of the position that every character string appears in the sound materials,; if the number of the different words or characters on the right or the left of the character string is bigger than certain threshold value set in advance, print the character string as new word; otherwise, searching the character string precisely and ambiguously on the search web station, if the number of the results the precise searching returns is bigger than certain threshold value, or / and the ratio of the number of the results the precise searching returns and the number of the results the ambiguous searching returns is bigger than certain threshold value, and / or the number of the kinds of the characters and words on the left and right the character string in the web page that the precise searching returns, print the character string as new word. The invention can not only find out the new words appearing in the sound materials frequently, but also find out the new words appearing in the sound materials infrequently, and the accuracy to find out new words is high.

Description

technical field [0001] The invention relates to the field of text information processing, in particular to a method and system for using a search engine to assist in finding new words in corpus. Background technique [0002] In natural language processing or computational linguistics, neologisms are words that have never appeared before, or new uses of words that have appeared before. New words are generally not included in dictionaries, so many people equate new words with unregistered words. [0003] With the progress of the times and the development of the economy, a large number of new words are constantly emerging in all aspects of people's daily life, especially with the increasing popularity of the Internet in China, a large number of new words on the Internet emerge in endlessly, and the words in daily life New words are also spread more quickly. According to reports, there are at least 1,000 new Chinese words or new usages in my country every year. The speed at w...

AI Technical Summary

Problems solved by technology

[0008] However, it is difficult to set an appropriate threshold for judging whether a string is a meaningful string or a new word based on the stability, independence, and integrity of the string.

If the threshold is too small, the accuracy of new word discovery is very low, and many meaningless garbage strings may be output; if the threshold is too large, some meaningful strings or new words in the corpus will not be output

The method based on the stability, independence and integrity of strings can only identify the part of the new words that appear frequently in the large-scale corpus. For some new words that have clear semantics and can be used independently, it is likely to be due to the presence of new words in the corpus. The frequency of use in is not very high and cannot be output

Images