A new concept sememe prediction method based on network embedding and pre-training model

Abstract

The present invention discloses a new concept sememe prediction method based on network embedding and pre-training model, comprising the following steps: based on the constructed HowNet sememe annotation system, a sememe prediction model including a representation learning module, a candidate sememe selection module and a sememe sequence sorting module is constructed through a collaborative filtering mechanism, and sememe sequence sorting is performed; a set of sememes to be recommended is obtained by using concepts corresponding to similar words, a set of similar words corresponding to a query word is obtained by performing a similar word search in HowNet, a word-sense-semme relationship subnetwork is constructed based on all the above words, the concept senses corresponding to the words and their semesmes, and candidate sememe nodes are selected based on a network node importance sorting method, thereby expanding HowNet; the present invention realizes dynamic matching between words and candidate semesmes, and improves the interpretability of representation learning through sememe annotation.

Description

Technical Field

[0001] The present invention relates to the technical field of sememe prediction, and in particular to a new concept sememe prediction method based on network embedding and pre-training model. Background Art

[0002] Since its launch, HowNet has garnered widespread attention in the field of natural language processing. Scholars at home and abroad have explored its significant applications in areas such as lexical semantic disambiguation, similarity calculation, text classification, and information retrieval. Research has found that HowNet directly captures semantic information through a unified semantic annotation system. Each semantic element has a clear and fixed meaning, allowing it to be directly incorporated into machine learning models as a semantic label, making deep learning models for natural language processing more robust and interpretable.

[0003] Sememes are the core components of concept descriptions. Currently, HowNet has built a sophisticated semantic description system containing approximately 2,230 sememes and has annotated approximately 148,000 concepts with sememes.

[0004] However, like other knowledge base systems that rely on manual construction, HowNet has the problems of limited scale and high cost of updating, expanding and maintaining. Related research [9] shows that it is difficult for people without background knowledge of HowNet and without special training to complete the task of semantic annotation. This leads to a contradiction between the huge potential application value of HowNet and its own limited scale, sparse semantic resources and difficulty in expansion. A feasible way to resolve this contradiction is to conduct research on automatic or semi-automatic construction technology of HowNet. One of its core issues is to select appropriate semantic primitives for the description of new concepts.

[0005] New concepts refer to concepts that have emerged with the development of the times or are new uses of old words. With the popularization of Internet applications, a large number of new words continue to appear in text big data, and the meanings of existing words are extended and expanded. Therefore, it is necessary to continuously revise and expand the semantic knowledge base based on semantic primitives. In terms of automatic annotation of lexical semantic primitives, XIE et al.

[23] proposed a semantic primitive prediction task, which is to select a set of semantic primitives suitable for constructing the Def description of new concepts from the HowNet semantic primitive set. For example, in the existing HowNet knowledge base, the Def description of "Xiaomi" contains the semantic primitive set of {material|material, edible|food, crop|crop}, but in the current actual language environment, "Xiaomi" not only has the concept of a crop, but can also be described as a company name or a brand of electronic products. Therefore, its semantic primitive set can also include {InstitutePlace|place} or {SpeBrand|specific brand}.

[0006] Typically, semantically similar words or concepts share the same sememes. Therefore, to solve the sememe selection problem for new concepts, we can draw on the idea of ​​collaborative filtering (CF). We can use the sememe sets in the Def descriptions of existing concepts in HowNet to predict the most likely sememe set for the Def description of the new concept. The key lies in measuring the semantic similarity between the new and existing words or concepts.

[0007] In recent years, word-based distributed representation methods have become the mainstream approach for calculating word similarity. However, the process of learning word representations differs fundamentally from the process of forming word or concept descriptions in HowNet. Comparing randomly sampled words reveals that similarity calculations obtained using word-based distributed representation methods often differ from those obtained using semantic primitive-based similarity measures. This manifests as inconsistent measures of semantic distance between words, a problem referred to as "similarity heterogeneity." Therefore, to better select appropriate semantic primitives for new concepts, it is necessary to model a new similarity calculation method that can produce similarity measurements close to those obtained using HowNet semantic primitives, effectively achieving "similarity isomorphism." Summary of the Invention

[0008] In order to solve the above-mentioned technical problems, the purpose of the present invention is to provide a new concept semantic origin prediction method based on network embedding and pre-trained model. By learning the representation of characters, words, meanings, semantic origins and their relationships in HowNet, and integrating pre-trained language models, dynamic matching of words and candidate semantic origins is achieved.

[0009] In order to achieve the above technical objectives, the present invention provides a new concept semantic primitive prediction method based on network embedding and pre-training model, comprising the following steps:

[0010] Build a HowNet sememe annotation system to describe an infinite set of concepts using a limited number of sememes. Sememes are used to represent the most basic and indivisible semantic units.

[0011] Based on the sememe annotation system, a sememe prediction model is constructed through a collaborative filtering mechanism, which includes a representation learning module, a candidate sememe selection module, and a sememe sequence sorting module to sort the sememe sequences.

[0012] The concepts corresponding to similar words are used to obtain the set of sememes to be recommended. The set of similar words corresponding to the query word is obtained by searching similar words in HowNet. Based on all the above words, the conceptual senses corresponding to the words and their sememes, a word-sense-semme relationship subnetwork is constructed. The candidate sememe nodes are selected based on the network node importance ranking method to expand HowNet.

[0013] Preferably, in the process of constructing a semantic annotation system, semantics are divided into five categories: Entity, Event, Attribute, Value, and SecondaryFeature.

[0014] Preferably, in the process of sorting semantic primitive sequences through the semantic primitive prediction model, in the representation learning module, the vector representations of the words and semantic primitives in Hownet are obtained through the network representation learning method, the representation vector of the input token is obtained through the Sentence-Transformers model, and it is spliced ​​to the vector representation of the word contained in the token. The spliced ​​vector representation is passed through the BiLstm and linear layer to obtain a new vector representation of the input token. The vector representation is used to perform similarity judgment with the candidate semantic primitive sequence vector, thereby realizing semantic primitive sequence sorting.

[0015] Preferably, in the process of constructing the word-meaning-semene relational subnetwork, concepts are constructed upward from sememes, and definition words are constructed upward from concepts, and the word-meaning-semene relational subnetwork is constructed based on the definition of one or more concept senses corresponding to the definition words.

[0016] Preferably, when obtaining candidate sememe nodes, vocabulary words similar to the unregistered words are selected in HowNet through Sentence-Transformers as candidate sememes.

[0017] Preferably, in the process of selecting candidate sememe nodes, a measurement method combining degree centrality and betweenness centrality is used to evaluate the importance of the sememe nodes and calculate the recommendation index of the sememe nodes;

[0018] According to the recommendation index, the candidate semes are combined to form sememe sequences of different lengths. The sequences will be used as candidate results for the new concept sememe annotation and form candidate sememe nodes.

[0019] Preferably, in the process of evaluating the importance of sememe nodes, degree centrality is used to characterize node centrality in network analysis, which is expressed as:

[0020] C d (v i )=∑ j x ij / max(C d (v j )), i≠j∈N

[0021] Among them, x ij =1 means there is a direct connection between node i and node j, otherwise, x ij =0;v i、 vj They represent node i and node j respectively, and N is the set of all nodes in the network.

[0022] Preferably, in the process of evaluating the importance of a sememe node, the betweenness centrality is expressed as:

[0023]

[0024] Among them, σ st represents the number of shortest paths from node s to t, σ st (v i ) represents the path from node s to t through v i The number of shortest paths.

[0025] Preferably, in the process of obtaining the recommendation index, the recommendation index is expressed as:

[0026] R(v i )=C d (v i )×log(C b (v i )+1).

[0027] Preferably, in the process of forming candidate sememe nodes, sememe sequences of different lengths are generated according to the scores of the sememe sequences, wherein the scores of the sememe sequences are expressed as:

[0028] R(t,s)=1-fd(t,s),

[0029]

[0030] Where t is the vector representation of the input token, which is obtained by training the BiLSTM model, and s is the vector representation of the sememe sequence, which is obtained by adding the vectors corresponding to the sememes included in the sequence.

[0031] Compared with the prior art, the present invention has the following technical effects:

[0032] This paper solves the definition sentence dependency problem of semantic primitive prediction in low-resource environments. Based on the training language model, it captures richer semantic information and helps to achieve diversified semantic primitive recommendations for new concepts.

[0033] The character-based word representation learning of the present invention can extend the sememe prediction method to multi-granularity sememe annotation scenarios;

[0034] The present invention realizes the dynamic matching of words and candidate sememes and improves the interpretability of representation learning through sememe annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A classification tree of sememes in HowNet described in the present invention;

[0037] Figure 2 Examples of the relationship between the words, meanings (concepts), and sememes described in the present invention;

[0038] Figure 3 This is the sememe prediction model based on the collaborative recommendation framework described in the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0040] like Figure 1-3 As shown in Figure 2, HowNet is a common sense knowledge base that uses concepts represented by Chinese and English words as description objects and reveals the relationships between concepts and the attributes of concepts as its basic content. Figure 1 As shown in Figure 1, it describes the world through seven dimensions: events, objects, attributes, attribute values, components, space, and time. HowNet is constructed based on the principle of reductionism, which holds that the meaning of all words can be composed of the most basic, indivisible, and smallest semantic units called "semites." HowNet aims to use a limited number of sememes to describe an infinite set of concepts.

[0041] In HowNet, semantic primitives are divided into five categories: Entity, Event, Attribute, Value, and SecondaryFeature.

[0042] In HowNet, concepts are constructed from semantics, and words are defined from concepts. The relationship between words, senses (concepts, Def), and semantics is as follows: Figure 2 As shown in the figure, here we take the word "millet" as an example. The concept sense Def = {material:MaterialOf = {edible},material = {crop}} (senseID: 177381) is composed of the semantic primitives "material", "edible", and "crop" and their relationships. This concept sense then defines the word "millet".

[0043] Due to the phenomenon of polysemy, each word in HowNet corresponds to one or more concept sense definitions (i.e., concept descriptions Def), and each Def contains one or more sememes. Sememes in any concept description are connected by relationships between concepts and between their attributes. Typically, a single sememe may exist in multiple concept sense definitions. When modeling the semantic relationships between multiple words, a "word-sense-semme" relationship network is formed.

[0044] To discover the different senses of a new word, we remove word nodes from the "word-meaning-semene" relationship network to form a "sense-semene" relationship network. Each maximally connected subnetwork of the "sense-semene" relationship network potentially represents a different sense of the word. To reflect the shared sense of synonyms, we can construct a "sense-semene" residual relationship network consisting solely of the connections between a sense and all the semes that constitute that sense Def.

[0045] Network Embedding aims to learn the low-dimensional potential representation of nodes in the network. Its basic idea is to find a mapping function f: V→R d , which transforms each node in the network into a low-dimensional latent representation f(x)∈R d , where x∈V is a node in the network. Currently, among homogeneous network embedding methods, Deep Walk and Node2vec are both based on random walks on the network and use the Skip-gram model for node embedding. This paper uses the Node2vec method at the network representation learning layer to vectorize the representation of words and sememe nodes.

[0046] With the development of deep learning, neural network models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), graph neural networks (GNNs), and attention mechanisms have been widely used to solve natural language processing (NLP) tasks. Compared to non-neural network model methods that rely heavily on discrete handcrafted features, neural network methods typically use low-dimensional dense vectors to implicitly represent the grammatical or semantic features of the language. In recent years, numerous studies have demonstrated that pre-trained models (PTMs) trained on large-scale text corpora can learn approximate universal language representations. After fine-tuning on small datasets for specific tasks, they can significantly reduce the difficulty of individual natural language processing tasks while improving system performance.

[0047] In terms of word similarity calculation, the distributed representation method can capture the association information between words by representing words as low-dimensional dense real number vectors. Therefore, this method can efficiently calculate the semantic association between words in a low-dimensional space and effectively solve the problem of data sparsity. Related research shows that word representation learning models represented by Word2vec have achieved good results in word similarity calculation tasks. However, each word in Word2vec is mapped to a unique dense vector, which cannot handle the problem of polysemy. In addition, the existing representation learning model learns the representation vector of a word based on the context distribution of the word. For words that appear less frequently or are not logged in, it will not be able to learn a good representation. The above problems have become the main obstacles that affect the role of the distributed representation method of words in similarity calculation tasks.

[0048] Compared to distributed representation methods such as Word2vec, a prominent advantage of BERT is the dynamic nature of word representation, which can model polysemy. In the task of recommending semantic primitives for new concepts, there is a widespread need for distributed representation of out-of-vocabulary (OOV) words. To mitigate the impact of OOV, one approach is to expand the dictionary to increase word coverage during model training, but this approach does not fundamentally solve the OOV problem. Another approach is to use a character-based BERT model, using the hidden vector of the top-level character in the BERT encoder to obtain the vector of the current word. However, simply using the output as a sentence vector has proven to be unsatisfactory. In HowNet, the key to measuring concept similarity is to calculate the similarity between the semantic primitive sequences corresponding to the concepts.

[0049] The Sentence-Transformers model is based on the Transformer model (using BERT / XLNet for sentence embedding) and fine-tuned for semantic similarity. Therefore, it demonstrates excellent performance in calculating sequential semantic similarity (e.g., sentence similarity). This paper considers the token (word or sentence) of the semantic primitive to be predicted as a sequence of characters, where the vector representation of each character is concatenated from the Sentence-Transformers encoding of the token in which it resides and the network embedding representation. Simultaneously, Sentence-Transformers is used to select words in HowNet that are similar to the input token to form a candidate word set.

[0050] The present invention intends to utilize the structural characteristics of the "word-sense-semene" relationship network and recommend relevant sememes for words based on a collaborative filtering mechanism.

[0051] Collaborative filtering is one of the most important technologies used in recommendation systems. Its basic principle is that if two users have similar purchasing behaviors, their interest in the same product category will be similar, and therefore the current user is likely to be interested in products favored by the other similar user. Regarding semantic similarity measurement, combined with the definition of concept descriptions in HowNet, it can be assumed that semantically similar concepts should have similar sets of sememe associations. Therefore, employing collaborative filtering is a viable approach for sememe recommendation. The basic principle is to recommend sememes that are untagged but likely to express certain aspects of the current word's semantic information in real-world contexts, based on the sememes contained in similar concept senses.

[0052] The semantic source prediction model based on the collaborative recommendation framework proposed in this invention is as follows: Figure 3 The model consists of a representation learning module, a candidate sememe selection module, and a sememe sequence sorting module. In the representation learning module, vector representations of words and sememes in Hownet are obtained through network representation learning methods. The representation vector of the input token is obtained through the Sentence-Transformers model and concatenated with the vector representation of the word contained in the token. The concatenated vector representation is passed through a BiLstm and a linear layer to obtain a new vector representation of the input token. This vector representation is used to determine similarity with the candidate sememe sequence vectors, thereby achieving sememe sequence sorting.

[0053] The goal of the new concept sememe recommendation task is to recommend appropriate sememes for concepts not registered in HowNet. Therefore, it is necessary to find a model for calculating the similarity between concepts with unknown sememes and concepts with known sememes. Since concepts are represented by words, this problem can be transformed into calculating the similarity between unregistered words and vocabulary words. This paper uses Sentence-Transformers to select vocabulary words similar to unregistered words in HowNet as candidate sememes.

[0054] The concepts corresponding to similar words are used to obtain a set of sememes to be recommended. First, a similar word set corresponding to the query word is obtained by searching for similar words in Hownet. Second, based on all the above words, their corresponding conceptual senses, and their sememes, a "word-sense-semme" relationship subnetwork is constructed, and candidate sememe nodes are selected based on a network node importance ranking method.

[0055] In a network, centrality represents the importance of edges and nodes. Here, two centrality metrics are used to evaluate the importance of sememe nodes.

[0056] Degree centrality is the most direct measure of node centrality in network analysis. The greater the node degree of a node, the higher the degree centrality of the node, and the more important the node is in the network. The standardized degree centrality measurement formula is:

[0057] C d (v i )=∑ j x ij / max(C d (v j )), i≠j∈N

[0058] Where: x ij =1 means there is a direct connection between node i and node j, otherwise, x ij = 0; N is the set of all nodes in the network. Because the calculation of a node's degree doesn't consider the importance of adjacent nodes in the graph, it doesn't adequately reflect the shared semantics between words. Therefore, it's necessary to measure node centrality from the path dimension. Here, we introduce a centrality measure based on betweenness centrality.

[0059] Calculate all the shortest paths between any two nodes in the network. If many of these shortest paths pass through a node, then the betweenness centrality of this node is considered high. The formula for calculating betweenness centrality is:

[0060]

[0061] Where: σ st represents the number of shortest paths from node s to t, σ st (v i ) represents the path from node s to t, passing through v i The number of shortest paths.

[0062] A node's betweenness centrality is an indicator of its importance, using the number of shortest paths passing through it. The higher the betweenness centrality of a sememe node, the more likely it is to be shared by similar words.

[0063] Combining the normalized degree centrality and betweenness centrality of the sememe node, the recommendation index of the sememe node is calculated:

[0064] R(v i )=C d (v i )×log(C b (v i )+1).

[0065] Candidate semesmes are combined to form sememe sequences of different lengths, which will be used as candidate results for new concept sememe annotation. Sememe sequence score:

[0066] R(t,s)=1-f d (t, s),

[0067]

[0068] Where t is the vector representation of the input token, which is obtained by training the BiLSTM model, and s is the vector representation of the sememe sequence, which is obtained by adding the vectors corresponding to the sememes included in the sequence.

[0069] This paper proposes a new concept sememe prediction method based on network embedding and pre-trained models. By learning the representations of words, senses, sememes, and their relationships from HowNet, and integrating it with a pre-trained language model, it dynamically constructs a sense-semme relationship network, enabling dynamic matching between words and candidate sememes. Sememe annotation improves the interpretability of representation learning. Future research will explore sememe prediction for linguistic units of varying granularity, and implement interpretable semantic similarity calculations based on the prediction results.

[0070] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A new concept semantics prediction method based on network embedding and pre-training model, characterized by: The following steps are involved: Build a HowNet sememe annotation system to describe an infinite set of concepts using a limited number of sememes, where the sememes are used to represent the most basic and indivisible smallest semantic units. Based on the sememe annotation system, a sememe prediction model is constructed through a collaborative filtering mechanism, which includes a representation learning module, a candidate sememe selection module, and a sememe sequence sorting module to sort the sememe sequences. The concepts corresponding to similar words are used to obtain a set of sememes to be recommended. A similar word set corresponding to the query word is obtained by searching similar words in HowNet. Based on all the above words, their corresponding conceptual senses and their sememes, a subnetwork of word-sense-semme relationships is constructed. Candidate sememe nodes are selected based on the network node importance ranking method to expand HowNet. In the process of sorting semantic sequences using the semantic prediction model, the representation learning module uses the network representation learning method to obtain the vector representations of the words and semantics in Hownet. The representation vector of the input token is obtained through the Sentence-Transformers model and concatenated with the vector representation of the word contained in the token. The concatenated vector representation passes through the BiLstm and linear layers to obtain the new vector representation of the input token. This vector representation is used to perform similarity judgment with the candidate semantic sequence vectors, thereby achieving semantic sequence sorting. In the process of selecting candidate sememe nodes, the importance of sememe nodes is evaluated using a combination of degree centrality and betweenness centrality, and the recommendation index of the sememe nodes is calculated. According to the recommendation index, the candidate sememes are combined to form sememe sequences of different lengths. The sequences are used as candidate results for new concept sememe annotation to form the candidate sememe nodes.

2. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 1, characterized in that: In the process of building a semantic annotation system, semantics are divided into five categories: Entity, Event, Attribute, Value, and SecondaryFeature.

3. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 1, characterized in that: In the process of constructing the relational sub-network of words, sense items and sememes, concepts are constructed upwards from the sememes, and definition words are constructed upwards from the concepts. Then, based on the definitions of one or more concept sense items corresponding to the definition words, the relational sub-network of words, sense items and sememes is constructed.

4. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 3, characterized in that: When obtaining candidate semantic primitives, Sentence-Transformers is used to select vocabulary words similar to unregistered words in HowNet as candidate semantic primitives.

5. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 1, characterized in that: In the process of evaluating the importance of sememe nodes, the degree centrality is used to characterize the node centrality in network analysis, which is expressed as: C d (v i )=∑ j x ij / max(C d (v j )),i≠j∈N Among them, x ij =1 means there is a direct connection between node i and node j, otherwise, x ij =0;v i 、v j They represent node i and node j respectively, and N is the set of all nodes in the network.

6. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 5, characterized in that: In the process of evaluating the importance of sememe nodes, the betweenness centrality is expressed as: Among them, σ st represents the number of shortest paths from node s to t, Σ st(vi) Represents the path from node s to t through v i The number of shortest paths.

7. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 1, characterized in that: In the process of obtaining the recommendation index, the recommendation index is expressed as: R(v i )=C d (v i )×log(C b (v i )+1)。 8. The method for predicting new concept sememes based on network embedding and pre-training models according to claim 7, characterized in that: In the process of forming the candidate sememe nodes, sememe sequences of different lengths are generated according to the scores of the sememe sequences, wherein the scores of the sememe sequences are expressed as: R(t,s)=1-f d (t,s). Where t is the vector representation of the input token, which is obtained by training the BiLSTM model, and s is the vector representation of the sememe sequence, which is obtained by adding the vectors corresponding to the sememes included in the sequence.

Images