A new word discovery method, system, device and medium based on graph embedding

Abstract

The present invention relates to a new word discovery method, system, device and medium based on graph embedding, comprising: using a sliding window to cut N-GRAM character strings of the corpus to be calculated, and calculating the statistics of each character string, according to the statistics: Each character string is scored, and the character string whose score meets the requirements is selected and written into the new word candidate set; the word segmentation is performed on the corpus to be calculated, and a graph network is constructed based on the word segmentation result; the graph network is calculated based on the graph attention network , to obtain the graph embedding of the words of the corpus to be calculated; based on the graph embedding of the words in the general dictionary, the graph embedding of the words in the new word candidate set is screened, and the words corresponding to the screened graph embedding are used as candidate new words . Based on the graph embedding technology, the present invention can effectively filter low-quality candidate new words during the new word discovery process, thereby obtaining higher-quality, more reliable general new words or domain new words.

Description

technical field [0001] The present invention relates to the field of natural language processing, in particular to a new word discovery method, system, device and medium based on graph embedding. Background technique [0002] Graph Embedding (Graph Embedding, also called Network Embedding) is a process of mapping graph data (usually a high-dimensional dense matrix) into a low-density vector. Graph widely exists in various scenarios in the real world, that is, a collection of nodes and edges. For example, the connection between people in social networks, the interaction of proteins in biology, and the communication between IP addresses in communication networks, etc. In addition, our most common picture and sentence can also be abstractly regarded as the structure of a graph model, and the graph structure can be said to be ubiquitous. By analyzing them we can gain insight into social structures, languages ​​and different modes of communication, so Figure 1 It has been a ho...

AI Technical Summary

Problems solved by technology

[0003] In the field of natural language processing, in the new word discovery task, the existing methods usually use statistical learning methods to construct new words. The basic idea is the method of information entropy, but this simple method only uses the shallow semantics in the corpus information, often introducing many low-quality new words

Images