A method and device for generating negative samples based on word forms and semantics in the field of Chinese

Abstract

This invention discloses a method and apparatus for generating negative samples based on lexical and semantic similarity in the Chinese domain. The method includes: extracting entities and all relations within one hop of each entity from a Chinese encyclopedia knowledge graph; concatenating the relations to form descriptive text for the entities; using Word2Vec trained on Chinese corpora to perform vector representation of the entities and descriptive text; performing similarity matching between the target matching entity and all entities, returning entities that meet the criteria as a set of semantic similarity candidate entities; calculating the edit distance between the target matching entity and all entities again, returning entities that meet the criteria as a set of lexical similarity candidate entities; and selecting high-confusion negative samples based on the semantic similarity candidate entity set and the lexical similarity candidate entity set. This invention combines lexical similarity and semantic similarity for negative sample selection, which can accelerate the convergence speed of the model and enable the model to train a good decision boundary.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and apparatus for generating negative samples based on word form and semantics in the Chinese language domain. Background Technology

[0002] To better utilize the rapidly growing global data, more and more researchers are dedicated to studying how to better store and retrieve this information. However, a large amount of data exists in the form of natural language, which is highly ambiguous. To organize ambiguous natural language data into information and formalize it as knowledge, knowledge bases such as FreeBase and DBpedia collect vast amounts of information from the internet and represent it as knowledge using triples. However, accessing these knowledge bases often requires certain professional skills, undoubtedly raising the barrier to entry for users. Intelligent question answering, centered on a knowledge base, answers natural language questions, providing ordinary users with a simple method to access knowledge bases. The goal of intelligent question answering is to transform natural language questions into structured query statements and return predicate relations or tail entities from the knowledge graph as the answer. Entity linking, which connects the unstructured knowledge in the question with the structured knowledge in the knowledge base, is a key component of intelligent question answering.

[0003] Unlike traditional entity linking, entity linking in intelligent question answering is typically based on short text questions lacking contextual information. The mainstream approach to entity linking in intelligent question answering uses a pipeline structure to decompose the task into referential recognition and entity disambiguation. However, since the number of existing entities is generally very large, sequentially calculating the references and entities in the knowledge base leads to a large search space. Therefore, entity disambiguation can be further divided into candidate entity generation and candidate entity ranking modules. It's worth noting that most entity disambiguation models are based on similarity calculation and can be transformed into a binary classification task for training. Specifically, for an <entity-relation> pair consisting of an entity reference in the question and an entity in the knowledge base, if the correspondence is correct, it is considered a positive example; otherwise, if the correspondence is incorrect, it is considered a negative example. Transforming the similarity task into a binary classification task helps researchers train the model more effectively.

[0004] Binary classification is one of the most well-known fundamental tasks in natural language processing. From the initial support vector machine to the later BERT, there are various training strategies specifically for binary classification tasks, and negative samples are a very important part of training binary classification models.

[0005] Generally, positive and negative samples can be distinguished based on their impact on model training. Samples that the model can easily identify as correct or incorrect are considered easy examples, while those that the model struggles to identify are considered hard examples. Hard examples typically have a positive impact on the model because they enable better and faster training. Specifically, they avoid updating global parameters every time, instead performing local updates only around certain parameters, and they can improve the quality of the model's decision boundary while increasing convergence speed. Normally, positive samples are difficult to expand or modify, but negative samples offer significant room for manipulation. Simply put, positive and negative samples can be considered two mutually exclusive sets; any part of the sample space that does not belong to positive samples can be considered a negative sample. Therefore, selecting high-quality hard examples from these negative samples has become a key focus of current research.

[0006] Currently, there are three main methods for negative sampling: random negative sampling, statistical metric-based negative sampling, and model-based negative sampling. Random negative sampling is the simplest but least effective method because it typically ignores most useful information. These meaningless negative samples can lead to rapid model convergence but do not provide substantial performance improvement. Statistical metric-based negative sampling uses similarity across different dimensions as statistical measures to evaluate and score negative samples, selecting those with higher scores to form a negative sample set. Common practices include using TF-IDF or Word2Vec to reduce the dimensionality of text vectors, performing rapid similarity calculations while returning negative samples with a certain level of confusion. Model-based negative sampling utilizes existing models to predict positive and negative sample pairs and evaluates them based on a loss function. If a sample has a large loss, it can be classified as a difficult sample.

[0007] In the training process of entity disambiguation models, it is usually necessary to generate negative samples to improve the training effect. The quality of negative samples often determines the robustness of the model. Traditional negative sampling methods are usually based on random sampling or normal distribution sampling, but these simple statistical methods have poor fitting ability and cannot enable the model to learn useful information. For model-based negative sampling methods, the training cycle is long and the cost is huge. In addition, these poor negative samples may even introduce erroneous information into the model, leading to a decline in the performance of the entity disambiguation model. Summary of the Invention

[0008] This invention addresses the issue of constructing high-quality negative samples in the entity connection subtask of Knowledge Base Question Answering (KBQA), specifically when transforming the entity connection task into a binary classification task, to enable the model to train better and faster. Traditional negative sampling methods are usually based on random sampling or normal distribution sampling, but these simple statistical methods have poor fitting ability and cannot enable the model to learn useful information. This invention proposes a negative sample generation method and device based on word form and semantics in the Chinese domain.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention proposes a negative sample generation method based on word form and semantics in the Chinese language domain, comprising:

[0011] Step 1: Extract the entities in the Chinese encyclopedia knowledge graph and all their relationships within one hop, and concatenate the relationships to form the entity's descriptive text.

[0012] Step 2: Use Word2Vec trained on Chinese corpus to generate vector representations of the entities and descriptive information text;

[0013] Step 3: Perform similarity matching between the target matching entity and all entities, and return the entities that meet the conditions as a set of semantic similarity candidate entities;

[0014] Step 4: Calculate the edit distance between the target matching entity and all other entities again, and return the entities that meet the criteria as the candidate entity set for word similarity;

[0015] Step 5: Select negative samples with high confusion based on the semantic similarity candidate entity set and the word form similarity candidate entity set.

[0016] Further, step 1 includes:

[0017] First, extract the entities from the correct <entity-mention> pairs and treat them as target matching entities; then, treat all entities in the Chinese encyclopedia knowledge graph as entities to be matched; finally, use the relationships around each entity to form the descriptive text of that entity. For both target matching entities and entities to be matched, the entity is represented by "entity node name#disambiguation item#entity description text".

[0018] Further, step 2 includes:

[0019] The entity obtained in step 1, represented by "entity node name#disambiguation item#entity description text", is segmented into words. The word vectors obtained by Word2Vec trained on Chinese corpus are used to represent the segmentation results. The vector representations are then concatenated to form the overall vector representation of the entity.

[0020] Furthermore, in step 3, the semantic similarity between entities is calculated based on the overall vector representation of the entities and using cosine similarity.

[0021] Furthermore, in step 4, the word similarity between entities is measured based on the entity node name and using Levenstein's ratio.

[0022] Furthermore, in step 5, high-confusion negative samples are selected as follows:

[0023]

[0024] in Represents the target matching entity Entity e in the Chinese encyclopedia knowledge graph i The Levinstein-Britt score between them; Represents the target matching entity Entity e in the Chinese encyclopedia knowledge graph i The cosine similarity score between them; α, β, γ, δ are adjustable hyperparameters;

[0025] Finally, the selected entities are concatenated with the mentions to form a negative sample in the form of a <mention-entity> pair.

[0026] Another aspect of this invention proposes a negative sample generation device based on word form and semantics in the Chinese domain, comprising:

[0027] The entity description information text extraction module is used to extract entities in the Chinese encyclopedia knowledge graph and all their relationships within one hop, and then concatenate the relationships together to form the entity description information text.

[0028] The feature vector representation module is used to represent the entity and descriptive information text using Word2Vec trained on Chinese corpus;

[0029] The semantic similarity matching module is used to perform similarity matching between the target matching entity and all entities, and return the entities that meet the conditions as a set of semantic similarity candidate entities;

[0030] The word form similarity matching module is used to calculate the edit distance between the target matching entity and all other entities, and return the entities that meet the conditions as a set of candidate entities for word form similarity.

[0031] The negative sample generation module is used to select high-confusion negative samples based on the semantic similarity candidate entity set and the word form similarity candidate entity set.

[0032] Furthermore, the entity description information text deriving module is specifically used for:

[0033] First, extract the entities from the correct <entity-mention> pairs and treat them as target matching entities; then, treat all entities in the Chinese encyclopedia knowledge graph as entities to be matched; finally, use the relationships around each entity to form the descriptive text of that entity. For both target matching entities and entities to be matched, the entity is represented by "entity node name#disambiguation item#entity description text".

[0034] Furthermore, the feature vector representation module is specifically used for:

[0035] The entity obtained from the entity description information text extraction module, represented by "entity node name#disambiguation item#entity description text", is segmented into words. The word vectors obtained by Word2Vec trained on Chinese corpus are used to represent the segmentation results. The vector representations are then concatenated to form the overall vector representation of the entity.

[0036] Furthermore, in the semantic similarity matching module, the semantic similarity between entities is calculated based on the overall vector representation of the entities and using cosine similarity.

[0037] Furthermore, in the word form similarity matching module, the word form similarity between entities is measured based on the entity node name and using Levin-Stanby.

[0038] Furthermore, in the negative sample deriving module, high-confusion negative samples are selected in the following manner:

[0039]

[0040] in Represents the target matching entity Entity e in the Chinese encyclopedia knowledge graph i The Levinstein-Britt score between them; Represents the target matching entity Entity e in the Chinese encyclopedia knowledge graph i The cosine similarity score between them; α, β, γ, δ are adjustable hyperparameters;

[0041] Finally, the selected entities are concatenated with the mentions to form a negative sample in the form of a <mention-entity> pair.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Traditional negative sampling methods are usually based on random sampling or normal distribution sampling, but this simple statistical method has poor fitting ability and cannot enable the model to learn useful information.

[0044] In response to the current situation, this invention analyzes everyday Chinese-based experimental data and finds that most misclassified samples fall into two categories: high semantic similarity and high word form similarity. Therefore, a negative sample selection method combining word form similarity and semantic similarity is proposed. Verification shows that this invention can accelerate the model's convergence speed and enable the model to train with good decision boundaries. Attached Figure Description

[0045] Figure 1 This is a basic flowchart of a negative sample generation method based on word form and semantics in the Chinese domain, according to an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of a negative sample generation device based on word form and semantics in the Chinese language domain, according to an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0048] Analysis of misclassified mention-entity pairs in routine experiments reveals that some are semantically very similar, such as "CBA" and "CUBA," while others are lexically very similar, such as "Beijing Normal University" and "Capital Normal University." These two types account for the majority of cases. To balance semantic and lexical information, this invention utilizes cosine similarity and Levenshtein distance as selection metrics for high-quality obfuscated entities and replaces entities in correct mention-entity pairs.

[0049] like Figure 1 As shown, a negative sample generation method based on word form and semantics in the Chinese domain mainly includes the following steps:

[0050] Step S101: Extract entities from the Chinese encyclopedia knowledge graph and all their relationships within one hop, and concatenate the relationships to form the entity's descriptive information text; as one possible implementation, the Chinese encyclopedia knowledge graph in this embodiment can be KgCLUE.

[0051] Step S102: Use Word2Vec trained on Chinese corpus to perform vector representation of the entity and description information text;

[0052] Step S103: Perform similarity matching between the target matching entity and all entities, and return the entities that meet the conditions as a set of semantic similarity candidate entities;

[0053] Step S104: Calculate the edit distance between the target matching entity and all entities again, and return the entities that meet the conditions as the candidate entity set for word similarity;

[0054] Step S105: Select negative samples with high confusion based on the semantic similarity candidate entity set and the word form similarity candidate entity set.

[0055] Further, step S101 includes:

[0056] Extract entities from the Chinese encyclopedia knowledge graph and all their relationships within one hop. Concatenate these relationships directly to form the entity's descriptive text. Then, concatenate them together using the format "entity node#disambiguation item#descriptive information" to form a whole, and use it to represent the entity.

[0057] Specifically, firstly, entities in the correct <entity-mention> pairs are extracted and treated as target matching entities. Then, all entities in the knowledge base are treated as entities to be matched. Next, the description text of each entity is constructed using the one-hop relations surrounding it. For both target and unmatched entities, the format is "entity node name#disambiguation term#entity description text". For example, the entity node "Let the Bullets Fly" in the knowledge base has the disambiguation term "2010 film directed by Jiang Wen". The one-hop relations surrounding this entity include "imdb code, release date, Chinese name, lead actors, lead actors, other translations, other translations, production company, production date, producer, production region, production cost, online streaming platform, foreign name, dialogue language, dialogue language, director, runtime, runtime, box office, type, type, screenwriter, screenwriter, color". Therefore, the entity "Let the Bullets Fly" can be represented as "Let the Bullets Fly #2010 Jiang Wen directed film #imdb code, release date, Chinese name, main actors, main actors, other translations, other translations, production company, production date, producer, production region, production cost, online streaming platform, foreign name, dialogue language, dialogue language, director, runtime, runtime, box office, genre, genre, screenwriter, screenwriter, color". This string of text can be called entity text.

[0058] Further, step S102 includes:

[0059] The entity text obtained in step S101 is segmented into words, and the word vectors obtained by Word2Vec trained on a large-scale Chinese corpus are used to represent the segmentation results. These vector representations are then concatenated to form the overall vector representation of the entity text.

[0060] Further, step S103 includes:

[0061] After obtaining the overall vector representation of the entity text, the semantic similarity between entities is calculated by introducing cosine similarity. Given two word vectors, A and B, the cosine similarity is given by the dot product and vector length, as shown in formula (1), where A i and B i This represents the components of A and B in different directions. Specifically, it involves calculating the similarity between the overall vector representation of the target matching entity and the overall vector representation of all entities to be matched, and using this similarity to measure the semantic similarity between the target and unmatched entities. Then, the unmatched entities are ranked according to their cosine similarity scores, forming a set of candidate entities with semantic similarity.

[0062]

[0063] Further, step S104 includes:

[0064] Then, the target matching entity and the entity to be matched are compared in terms of lexical similarity. This invention uses Levin-Stanby (i.e., formula (2)) to measure the lexical similarity between entities. Where sum refers to the sum of the lengths of the two strings str1 and str2, and ldist is the class edit distance. For the target matching entity and the entity to be matched, since the purpose is to calculate the lexical similarity between entities and does not involve deep semantic information, it is only necessary to select the names of the entity nodes for matching. Then, the entities to be matched are sorted according to the Levin-Stanby score as a candidate entity set for lexical similarity.

[0065] r = (sum - ldist) / sum (2)

[0066] Further, step S105 includes:

[0067] Finally, high-confusion negative samples are selected based on the semantic similarity candidate entity set and the word form similarity candidate entity set. Intuitively, the similarity between good negative sample pairs should be between significantly consistent and significantly inconsistent. This ensures that the correct matching object is not selected, while also maintaining the necessary confounding power. Finally, these two evaluation rules are combined as the standard for selecting negative sample pairs:

[0068]

[0069] in This represents the target matching entity (i.e., the entity in the correct <entity-mention> pair). Entity e in the Chinese encyclopedia knowledge graph i The Levenstein Ratio score between them. This means that the target matching entity is... Entity e in the Chinese encyclopedia knowledge graph i The cosine similarity score between them; α, β, γ, δ are adjustable hyperparameters. In the experiment, it was found that when α, β, γ, δ are 0.9, 0.5, 0.8, and 0.6 respectively, entities with high confusion can be selected that best meet the requirements of general research.

[0070] Finally, the selected high-quality, highly confusing entities are concatenated with mentions to form negative samples in the form of <mention-entity> pairs.

[0071] To verify the effectiveness of the proposed method, BERT was used to compare the proposed negative sampling method, which combines edit distance and cosine similarity, with a random negative sampling method. The results in Table 1 show that the random negative sample generation strategy performs significantly worse than our proposed method. This is because randomly generated negative samples are too easily distinguishable, causing the model to fail to learn a good decision boundary.

[0072] Table 1 shows the classification accuracy tested using different negative sampling strategies.

[0073]

[0074] Based on the above embodiments, such as Figure 2 As shown, another aspect of the present invention proposes a negative sample generation device based on word form and semantics in the Chinese domain, comprising:

[0075] The entity description information text extraction module is used to extract entities in the Chinese encyclopedia knowledge graph and all their relationships within one hop, and then concatenate the relationships together to form the entity description information text.

[0076] The feature vector representation module is used to represent the entity and descriptive information text using Word2Vec trained on Chinese corpus;

[0077] The semantic similarity matching module is used to perform similarity matching between the target matching entity and all entities, and return the entities that meet the conditions as a set of semantic similarity candidate entities;

[0078] The word form similarity matching module is used to calculate the edit distance between the target matching entity and all other entities, and return the entities that meet the conditions as a set of candidate entities for word form similarity.

[0079] The negative sample generation module is used to select high-confusion negative samples based on the semantic similarity candidate entity set and the word form similarity candidate entity set.

[0080] Furthermore, the entity description information text deriving module is specifically used for:

[0081] First, extract the entities from the correct <entity-mention> pairs and treat them as target matching entities; then, treat all entities in the Chinese encyclopedia knowledge graph as entities to be matched; finally, use the relationships around each entity to form the descriptive text of that entity. For both target matching entities and entities to be matched, the entity is represented by "entity node name#disambiguation item#entity description text".

[0082] Furthermore, the feature vector representation module is specifically used for:

[0083] The entity obtained from the entity description information text extraction module, represented by "entity node name#disambiguation item#entity description text", is segmented into words. The word vectors obtained by Word2Vec trained on Chinese corpus are used to represent the segmentation results. The vector representations are then concatenated to form the overall vector representation of the entity.

[0084] Furthermore, in the semantic similarity matching module, the semantic similarity between entities is calculated based on the overall vector representation of the entities and using cosine similarity.

[0085] Furthermore, in the word form similarity matching module, the word form similarity between entities is measured based on the entity node name and using Levin-Stanby.

[0086] Furthermore, in the negative sample deriving module, high-confusion negative samples are selected in the following manner:

[0087]

[0088] in Represents the entity in the correct <entity-mention> pair. Entity e in the Chinese encyclopedia knowledge graph i The Levinstein-Britt score between them; Represents the entity in the correct <entity-mention> pair. Entity e in the Chinese encyclopedia knowledge graph i The cosine similarity score between them; α, β, γ, δ are adjustable hyperparameters;

[0089] Finally, the selected entities are concatenated with the mentions to form a negative sample in the form of a <mention-entity> pair.

[0090] In summary, traditional negative sampling methods are typically based on random sampling or normal distribution sampling. However, these simple statistical methods have poor fitting capabilities and cannot enable the model to learn useful information. To address this issue, this invention analyzes everyday Chinese-based experimental data and finds that most misclassified samples fall into two categories: high semantic similarity and high word form similarity. Therefore, a negative sample selection method combining word form similarity and semantic similarity is proposed. Verification shows that this invention can accelerate the model's convergence speed and also enable the model to train with good decision boundaries.

[0091] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating negative samples based on word form and semantics in the Chinese domain, characterized in that, include: Step 1: Extract the entities in the Chinese encyclopedia knowledge graph and all their relationships within one hop, and concatenate the relationships to form the entity's descriptive text; Step 1 includes: First, extract the entities from the correct <entity-mention> pairs and treat them as target matching entities; then, treat all entities in the Chinese encyclopedia knowledge graph as entities to be matched; finally, use the relationships around each entity to form the descriptive text of that entity. For both target matching entities and entities to be matched, the entity is represented by "entity node name#disambiguation item#entity description text". Step 2: Use Word2Vec trained on Chinese corpus to generate vector representations of the entity and description text, and concatenate the vector representation results to obtain the overall vector representation of the entity; Step 3: Based on the overall vector representation of the entity, perform similarity matching between the target matching entity and all entities, and return the entities that meet the conditions as a set of semantic similarity candidate entities; Step 4: Based on the entity node name, calculate the edit distance between the target matching entity and all entities again, and return the entities that meet the conditions as the candidate entity set for word similarity; Step 5: Select negative samples with high confusion based on the semantic similarity candidate entity set and the word form similarity candidate entity set; In step 5, high-confusion negative samples are selected as follows: （3） in Represents the target matching entity Entities in Chinese encyclopedia knowledge graphs The Levinstein-Britt score between them; Represents the target matching entity Entities in Chinese encyclopedia knowledge graphs Cosine similarity score between them; It's a hyperparameter; Finally, the selected entities are concatenated with the mentions to form a negative sample in the form of a <mention-entity> pair.

2. The method for generating negative samples in the Chinese domain based on word form and semantics according to claim 1, characterized in that, Step 2 includes: The entity obtained in step 1, represented by "entity node name#disambiguation item#entity description text", is segmented into words. The word vectors obtained by Word2Vec trained on Chinese corpus are used to represent the segmentation results. The vector representations are then concatenated to form the overall vector representation of the entity.

3. The method for generating negative samples in the Chinese domain based on word form and semantics according to claim 2, characterized in that, In step 3, the semantic similarity between entities is calculated based on the overall vector representation of the entities and using cosine similarity.

4. The method for generating negative samples in the Chinese domain based on word form and semantics according to claim 1, characterized in that, In step 4, the word similarity between entities is measured based on the entity node name and using Levenstein's ratio.

5. A negative sample generation device based on word form and semantics in the Chinese domain, characterized in that, include: The entity description text extraction module is used to extract entities and all their relationships within one hop from the Chinese encyclopedia knowledge graph, and concatenate these relationships to form the entity's description text. Specifically, it is used to: first, extract entities from the correct <entity-mention> pairs and treat them as target matching entities; then, treat all entities in the Chinese encyclopedia knowledge graph as entities to be matched; finally, use the relationships within one hop of each entity to compose the entity's description text. For both target matching entities and entities to be matched, the entity is represented by "entity node name#disambiguation item#entity description text". The feature vector representation module is used to perform vector representation of the entity and the description text using Word2Vec trained on Chinese corpus, and to concatenate the vector representation results as the overall vector representation of the entity; The semantic similarity matching module is used to perform similarity matching between the target matching entity and all entities based on the overall vector representation of the entity, and return the entities that meet the conditions as a set of semantic similarity candidate entities; The word form similarity matching module is used to calculate the edit distance between the target matching entity and all entities based on the entity node name, and return the entities that meet the conditions as a candidate entity set for word form similarity; The negative sample generation module is used to select high-confusion negative samples based on the semantic similarity candidate entity set and the word form similarity candidate entity set; In the negative sample deriving module, high-confusion negative samples are selected in the following manner: （3） in Represents the target matching entity Entities in Chinese encyclopedia knowledge graphs The Levinstein-Britt score between them; Represents the target matching entity Entities in Chinese encyclopedia knowledge graphs Cosine similarity score between them; It's a hyperparameter; Finally, the selected entities are concatenated with the mentions to form a negative sample in the form of a <mention-entity> pair.

Images