Human-machine fusion common sense graph construction method and system

Abstract

The present invention provides a method and system for constructing a common sense graph through human-computer fusion, comprising: Step S1: collecting relevant news data and encyclopedia data using crawler technology and integrating it with existing open-source graph data; Step S2: Based on human-computer fusion, the collected news text is annotated with entities, relationships, and rules, and the annotated data is used to jointly train a relationship extraction network and a rule matching network; Step S3: Based on the annotated data, entities and relationships are extracted from the data and a primary common sense graph is constructed; Step S4: Using crowdsourcing, the primary common sense graph is fine-tuned and modified to obtain a high-quality common sense graph; Step S5: Maintaining and evolving the high-quality common sense graph. This invention fully utilizes humans' cognitive ability for abstract knowledge rules and the rapid computing power of computers to improve the efficiency of common sense extraction and the quality of common sense information.

Description

Technical Field

[0001] The present invention relates to the technical field of common sense information extraction and graph construction, and specifically to a human-machine integrated common sense graph construction method and system. Background Art

[0002] Common sense information refers to a widely held consensus about things. Common sense-based artificial intelligence (AI) can integrate human knowledge and reasoning into machine intelligence, and has become a research hotspot in AI and computational science in recent years. However, common sense has the following significant characteristics:

[0003] Abstractness. Common sense is often expressed in an abstract form. People are often unaware of their own common sense, so common sense rarely appears in various documents created by humans and is easily overlooked.

[0004] Common sense is often ambiguous. People in different fields and regions often have different understandings of common sense. Therefore, effective differentiation is crucial when extracting common sense.

[0005] Evolution. Common sense often changes over time. Therefore, the common sense used to guide intelligent processing also needs to be dynamically updated and evolved during the analysis process.

[0006] Relevance. Common sense is often closely related, and some common sense can depend on other common sense. This also leads to the need for the analysis and representation of common sense to clarify its relevance.

[0007] Therefore, acquiring and applying common sense is a highly challenging task. Furthermore, common sense information is complex and rapidly changing, making it difficult to effectively update related common sense libraries. Organizing and summarizing this information requires significant professional time and effort, a cost currently unaffordable. To address this, the present invention proposes a common sense graph construction method and system based on human-machine fusion, capable of learning from text data and generating a common sense graph.

[0008] Many developed countries and well-known research institutions have successively launched common sense research programs with their own characteristics. At present, research related to the construction of common sense graphs has become a hot topic of research at home and abroad. However, the existing methods are not effective and efficient, and the methods involved cannot achieve an organic integration of manual and computer methods. Yaojie Lu et al. proposed a joint extraction framework for text information (Yaojie Lu, Qing Liu, Dai Dai et al. "Unified Structure Generation for Universal Information Extraction", in ACL 2022), which can perform unified modeling for different information extraction tasks and has good collaborative learning and generalization adaptability for information in different fields. However, this method requires a large amount of data annotation during training, which has high labor costs; and the model of this method is large, and the time and hardware costs during training and prediction are high. The dataset construction methods CommonsenseQA and CommonsenseQA 2.0 proposed by scholars Alon Talmor et al. (CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge, Proceedings of NAACL-HLT2019, pages 4149-4158) (CommonsenseQA2.0: Exposing the Limits of AI through Gamification, neural information processing systems 2021) are both crowdsourcing methods. They construct question pairs with complex semantic information based on ConceptNet, but the constructed data is limited to question pairs.

[0009] Patent document CN113553440A (application number: CN202110709798.7) discloses a medical entity relationship extraction method based on hierarchical reasoning. This method extracts relevant entity relationship triplets using an entity relationship extraction model based on hierarchical reasoning. However, the annotation cost required by this invention is very high.

[0010] In summary, existing methods have the following shortcomings: First, there is a lack of methods for effectively utilizing manual and computer resources. Among them, existing manual methods require a large amount of human resources, are too costly, and cannot form large-scale graphs; the graphs constructed by existing machine methods are of low quality and cannot reasonably extract and organize common sense information; there is currently no human-computer fusion method that can complete graph construction efficiently and with high quality. Second, there is currently no crowdsourcing method for expanding and ensuring the quality of common sense graph information, and it is impossible to achieve efficient collection of crowdsourced common sense information. Third, existing common sense extraction methods require a large amount of manually labeled data and hardware resources, and cannot achieve the goal of reducing costs while ensuring quality. Therefore, existing methods cannot achieve the effective extraction of text common sense information and the subsequent construction of common sense graphs. Summary of the Invention

[0011] In response to the deficiencies in the prior art, the present invention aims to provide a method and system for constructing a common sense graph that integrates human and machine fusion.

[0012] The method for constructing a common sense graph for human-machine fusion provided by the present invention includes:

[0013] Step S1: Collect relevant news data and encyclopedia data through crawler technology and integrate them with existing open source graph data;

[0014] Step S2: Based on human-machine fusion, the collected news text is annotated with entities, relationships, and rules, and the annotated data is used to jointly train the relationship extraction network and the rule matching network;

[0015] Step S3: Based on the labeled data, extract entities and relationships from the data and construct a primary common sense graph;

[0016] Step S4: Use crowdsourcing to fine-tune and modify the primary common sense graph to obtain a high-quality common sense graph;

[0017] Step S5: Maintain and evolve the high-quality common sense graph.

[0018] Preferably, the step S1 includes:

[0019] Step S1.1: Obtain relevant news data through crawler technology;

[0020] Step S1.2: Segment the acquired news data into sentences;

[0021] Step S1.3: Obtain relevant entry names from Wikipedia and Baidu Encyclopedia using crawler technology;

[0022] Step S1.4: Filter the acquired entry names according to preset rules, and use the filtered results as the entity dictionary;

[0023] The step S2 comprises:

[0024] Step S2.1: perform entity labeling on each segmented news sentence according to the entity dictionary;

[0025] Step S2.2: Based on the preset rules, label the entity relationships in each news sentence;

[0026] Step S2.3: Jointly train the relation extraction network and the rule matching network based on the labeled data.

[0027] Preferably, the step S2.3 includes:

[0028] Step S2.3.1: Based on all rules For all sentences Perform hard matching, and the data of successful hard matching is The remaining set of sentences that did not match successfully is

[0029] Step S2.3.2: For the set of sentences that failed to match For each sentence in , calculate the similarity of the rule that is closest to it;

[0030] Step S2.3.3: and rule sets It itself is used as a training data set, and label information is added to each data in the data set;

[0031] Add token information to the sentences. Each sentence s contains two entities e1 and e2. In order for the neural network to capture the position information of the two entities, token symbols need to be inserted at the beginning and end of the two entities. For the first entity e1, insert the token '$' as a marker at the beginning and end, and for the second entity e2, insert the token '#' as a marker at the beginning and end. At the same time, insert the token '[CLS]' at the beginning of each sentence.

[0032] Step S2.3.4: Based on the training dataset with added labeled information, jointly train the relation extraction network RelationExtractor and the rule matching network RuelMatcher;

[0033] For each sentence input into the pre-trained Bert, the hidden layer state H is obtained, where H i To H j The hidden layer state of entity e1, H k To H mThe hidden layer state belongs to entity e2, H0 belongs to the hidden layer state of [CLS], the average of the state layer belonging to the entity is taken as the representation of the entity, and then a fully connected layer projection is connected to obtain the final representation;

[0034] Step S2.3.5: Connect H′0, H′1 and H′2, and after a full connection layer, pass the softmax layer to obtain the final prediction result p;

[0035] Step S2.3.6: Perform cross entropy loss on the predicted results and the actual relationship tags, and then train the neural network. The resulting neural network is named Relation Extractor, which is used to extract the relationship between entities in the text. Depending on the data set, the relationship extraction network parameters θ are obtained. rE Three different losses are related: the loss function L for hard matching success matched (θ RE ), loss function L for unsuccessful hard matching unmatched (θ RE ), relation extraction parameter θ RE The loss function L rule (θ RE );

[0036] Step S2.3.7: For rule p, set the rule set Divide into sets and Represents the relationship label r with p respectively p The same and different rule sets, based on this rule set cooperative comparison loss, rule matching parameter θ RM Conduct training;

[0037] Step S2.3.8: Combine the above four losses and perform joint training of the RelationExtractor and RuelMatcher networks. The specific loss function is as follows:

[0038] L(θ RE ,θ RM )=L matched (θ RE )+αL unmatched (θ RE )+βL rule (θ RE )+γL clus (θ RM )

[0039] Among them, α, β and γ are hyperparameters to balance different losses, L clus (θ RM ) is the rule matching parameter θ RM The loss function of .

[0040] Preferably, step S3 includes:

[0041] Step S3.1: For entities in news sentences, use the relationship extraction network to extract the relationship between entities;

[0042] Step S3.2: Count the number of relationships between each entity pair in the entity dictionary, and filter by the threshold of the number of relationships;

[0043] Step S3.3: Import the entity-relationship triples into the Neo4J graph database to obtain the primary common sense graph;

[0044] The step S4 comprises:

[0045] Step S4.1: Invite relevant experts and scholars to modify the common sense entity dictionary and graph;

[0046] Step S4.2: Verify the modification result;

[0047] Step S4.3: Modify the common sense entity dictionary and graph based on the above results.

[0048] Preferably, step S5 includes:

[0049] Step S5.1: Using a logistic classification network, each triple to be detected returns an error confidence score of [0, 1], and then sets a threshold. Triples above the threshold are regarded as outdated triple results of error checking;

[0050] Step S5.2: Preset logical replacement rules to determine positive / negative examples in the binary classification model. Use mutually exclusive relationships to represent common negative example logic, and use other relationships outside of the mutually exclusive relationship as compatible relationships to represent positive example logic, thereby performing common sense triple replacement.

[0051] Step S5.3: After pre-training the translation model and negative sampling evaluation model, perform common sense information relationship completion on missing triplets of missing tail entities or head and tail entities. Specifically, traverse the entity pool of the current graph in sequence to complete the missing triplets. Use the negative sampling evaluation model to obtain the top-k candidate completion triplets with the highest positive confidence scores. Use the translation model to select the triple with the highest score as the completion triplet to support the knowledge graph completion task.

[0052] Step S5.4: Continuously update common sense relationships based on crowdsourcing.

[0053] The human-machine fusion common sense graph construction system provided by the present invention includes:

[0054] Module M1: Collect relevant news data and encyclopedia data through crawler technology, and integrate it with existing open source graph data;

[0055] Module M2: Based on human-machine fusion, the collected news text is annotated with entities, relationships, and rules, and the annotated data is used to jointly train the relationship extraction network and the rule matching network;

[0056] Module M3: Based on the labeled data, extract entities and relationships from the data and build a primary common sense graph;

[0057] Module M4: Use crowdsourcing to fine-tune and modify the primary common sense graph to obtain a high-quality common sense graph;

[0058] Module M5: Maintain and evolve high-quality common sense graphs.

[0059] Preferably, the module M1 includes:

[0060] Module M1.1: Obtain relevant news data through crawler technology;

[0061] Module M1.2: Segment the acquired news data into sentences;

[0062] Module M1.3: Obtain relevant entry names from Wikipedia and Baidu Encyclopedia through crawler technology;

[0063] Module M1.4: Filter the acquired entry names according to preset rules and use the filtered results as the entity dictionary;

[0064] The module M2 includes:

[0065] Module M2.1: Based on the entity dictionary, perform entity labeling on each segmented news sentence;

[0066] Module M2.2: Label the entity relationships in each news sentence based on preset rules;

[0067] Module M2.3: Jointly train the relation extraction network and rule matching network based on labeled data.

[0068] Preferably, the module M2.3 includes:

[0069] Module M2.3.1: Based on all rules For all sentences Perform hard matching, and the data of successful hard matching is The remaining set of sentences that did not match successfully is

[0070] Module M2.3.2: For a set of sentences that failed to match For each sentence in , calculate the similarity of the rule that is closest to it;

[0071] Module M2.3.3: and rule sets It itself is used as a training data set, and label information is added to each data in the data set;

[0072] Add token information to the sentences. Each sentence s contains two entities e1 and e2. In order for the neural network to capture the position information of the two entities, token symbols need to be inserted at the beginning and end of the two entities. For the first entity e1, insert the token '$' as a marker at the beginning and end, and for the second entity e2, insert the token '#' as a marker at the beginning and end. At the same time, insert the token '[CLS]' at the beginning of each sentence.

[0073] Module M2.3.4: Jointly train the RelationExtractor and the RuelMatcher based on the labeled training dataset.

[0074] For each sentence input into the pre-trained Bert, the hidden layer state H is obtained, where H i To H j The hidden layer state of entity e1, H k To H m The hidden layer state belongs to entity e2, H0 belongs to the hidden layer state of [CLS], the average of the state layer belonging to the entity is taken as the representation of the entity, and then a fully connected layer projection is connected to obtain the final representation;

[0075] Module M2.3.5: Connect H′0, H′1 and H′2, and after a full connection layer, obtain the final prediction result p through the softmax layer;

[0076] Module M2.3.6: Perform cross entropy loss on the predicted results and the actual relationship tags, and then train the neural network. The resulting neural network is named Relation Extractor, which is used to extract the relationship between entities in the text. Depending on the data set, the relationship extraction network parameters θ are obtained. rE Three different losses are related: the loss function L for hard matching success matched (θ RE ), loss function L for unsuccessful hard matching unmatched (θ RE ), relation extraction parameter θ RE The loss function L rule (θ RE);

[0077] Module M2.3.7: For rule p, set the rule Divide into sets and Represents the relationship label r with p respectively p The same and different rule sets, based on this rule set cooperative comparison loss, rule matching parameter θ RM Conduct training;

[0078] Module M2.3.8: Combine the above four losses to perform joint training of the RelationExtractor and RuelMatcher networks. The specific loss function is as follows:

[0079] L(θ RE ,θ RM )=L matched (θ RE )+αL unmatched (θ RE )+βL rule (θ RE )+γL clus (θ RM )

[0080] Among them, α, β and γ are hyperparameters to balance different losses, L clus (θ RM ) is the rule matching parameter θ RM The loss function of .

[0081] Preferably, the module M3 includes:

[0082] Module M3.1: For entities in news sentences, use the relationship extraction network to extract the relationship between entities;

[0083] Module M3.2: Count the number of relationships between each entity pair in the entity dictionary, and use the relationship number threshold to filter.

[0084] Module M3.3: Import entity-relationship triples into the Neo4J graph database to obtain a basic common sense graph;

[0085] The module M4 includes:

[0086] Module M4.1: Invite relevant experts and scholars to revise the common sense entity dictionary and atlas;

[0087] Module M4.2: Verify the modification results;

[0088] Module M4.3: Modify the common sense entity dictionary and graph based on the above results.

[0089] Preferably, the module M5 includes:

[0090] Module M5.1: Using a logistic classification network, each triple to be detected returns an error confidence score of [0, 1], and then sets a threshold. Triples above the threshold are regarded as outdated triples of error checking results.

[0091] Module M5.2: Preset logical replacement rules to determine positive / negative examples in a binary classification model. Use mutually exclusive relationships to represent common negative example logic, and use other relationships other than mutually exclusive relationships as compatible relationships to represent positive example logic, thereby performing common sense triple replacement.

[0092] Module M5.3: After pre-training the translation model and negative sampling evaluation model, perform common sense information relationship completion on missing triplets of missing tail entities or head and tail entities. Specifically, the missing triplets are sequentially traversed in the entity pool of the current graph to complete the missing triplets. The top-k candidate triplets predicted as positive examples by the negative sampling evaluation model are selected as candidate completion triplets. The translation model is used to select the triple with the highest score as the completion triplet to support the knowledge graph completion task.

[0093] Module M5.4: Continuously update common sense relationships based on crowdsourcing.

[0094] Compared with the prior art, the present invention has the following beneficial effects:

[0095] (1) This invention applies human-computer fusion to common sense extraction and graph construction. It extracts common sense information based on human-computer collaboration, fully utilizing the human's cognitive ability for abstract knowledge rules and the computer's fast computing power, thereby improving the efficiency of common sense extraction and the quality of common sense graphs.

[0096] (2) The present invention fine-tunes and modifies the common sense graph based on crowdsourcing, which saves the cost of building the common sense graph and improves the quality of the common sense graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0098] Figure 1 It is the overall process framework diagram of the present invention;

[0099] Figure 2 This is a structural diagram of the relationship extraction model in the present invention;

[0100] Figure 3 A flow chart of the crowdsourcing method of the present invention;

[0101] Figure 4A flowchart for maintaining the common sense graph in the present invention;

[0102] Figure 5 Continuously update flowcharts for crowd-sourced common sense relationships;

[0103] Figure 6 This is a system structure diagram for constructing the common sense graph in the present invention. DETAILED DESCRIPTION

[0104] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0105] Example 1:

[0106] like Figure 1 According to the present invention, a common sense graph construction method based on human-computer fusion is provided, including: step S1: collecting relevant news data and encyclopedia data through crawlers, and integrating them with existing open source graph data; step S2: based on human-computer fusion, annotating the collected news text with entities, relationships and rules, and using the annotated data to jointly train the relation extraction network (Relation Extractor) and the rule matching network (Ruel Matcher); step S3: based on the annotated data, extracting entities and relationships from the data, and constructing a primary common sense graph; step S4: using crowdsourcing to fine-tune and modify the primary common sense graph, improve the quality of common sense, and obtain the final common sense graph; step S5: maintaining and evolving the common sense graph.

[0107] The step S1 adopts: step S1.1: obtaining relevant data through crawlers, specific sources include the National Health Commission, UN News, Good Doctor, Dingxiang Doctor, etc.; step S1.2: dividing the data into sentences; step S1.3: obtaining relevant entry names in relevant Wikipedia and Baidu Encyclopedia through crawlers; step S1.4: filtering the entry names, and using the filtered results as the entity dictionary.

[0108] The step S2 adopts the following steps: Step S2.1: perform entity labeling on each segmented news sentence according to the entity dictionary; perform entity matching query on the news sentence according to the entity dictionary. If two different entities in the entity set happen to appear in the sentence, these two entities will be labeled; otherwise, the sentence will be discarded. This labeling method can effectively eliminate irrelevant information and can automatically implement labeling, reducing the cost of manual labeling. The final data set is<e1,e2,s> , indicating that the sentence s contains the head entity e1 and the tail entity e2.

[0109] Step S2.2: Mark the relationship between entities in each sentence according to the rules. Manual screening and marking are performed on each sentence after segmentation and screening.<e1,e2,s> Filter and select those with semantic information, and mark the relationship between the two entities contained in the sentence; remove those without semantic information. The final dataset is <p,r p > indicates that the relationship corresponding to rule p is r p , where rule p is automatically generated. Specifically, rule p replaces e1 and e2 in s with masks, and retains the two masks and the content between the masks.

[0110] Step S2.3: Jointly train the Relation Extractor and the Rule Matcher based on the labeled data, including: Step S2.3.1: Based on all the rules For all sentences Perform hard matching, and the data of successful hard matching is The remaining set of sentences that did not match successfully is Step S2.3.2: For the set of sentences that failed to match For each sentence in , calculate the similarity of the rule that is closest to it. The input to the rule matching network (Ruel Matcher) obtains sentence s i and rule p j The RuelMatcher model can obtain the sentence-level feature vector from the sentence's word vector, and then calculate the similarity of the sentence feature vector. The specific formula is as follows:

[0111]

[0112] z t =φ(x s )

[0113] z t =φ(x p )

[0114]

[0115] Where n is the number of word vectors in the sentence vector x; W, v, and D are all learnable parameters.

[0116] Based on the similarity matrix obtained by RuelMatcher, the rule with the largest similarity is selected as the rule for each sentence s i Final matching result and weight w i , the specific formula is as follows:

[0117]

[0118] Step S2.3.3: and rule sets It is used as a training data set, and tag information is added to each data in the data set. Add token tag information to the sentence. Each sentence s will contain two entities e1 and e2. In order for the neural network to capture the position information of the two entities, token symbols need to be inserted at the head and tail of the two entities. Specifically, the token '$' is inserted as a marker at the head and tail of the first entity e1, and the token '#' is inserted as a marker at the head and tail of the second entity e2, and the token '[CLS]' is inserted at the head of each sentence. In the following sentence, "mask" and "virus" are the two entities of this sentence. After adding the token symbol, it is in the following form: "[CLS] Wearing $mask$ is still an effective way to block the spread of #virus#".

[0119] Step S2.3.4: Based on the training data set after adding the labeled information, the relation extraction network (RelationExtractor) and the rule matching network (RuelMatcher) are jointly trained, such as Figure 2 For each sentence input into the pre-trained Bert, the hidden layer state H is obtained, where H i To H j The hidden layer state of entity e1, H k To H m The hidden layer state belongs to entity e2, and H0 belongs to the hidden layer state of [CLS]. We take the average of the state layers belonging to the entity as the representation of the entity, and then connect it to a fully connected layer to obtain the final representation. The specific formula is as follows:

[0120]

[0121] H′0=W0[tanh(H0)]+b0

[0122] Step S2.3.5: Connect H′0, H′1 and H′2, and after a full connection layer, pass the softmax layer to obtain the final prediction result p. The specific formula is as follows:

[0123] p=softmax(W3[concact(H′0,H′1,H′2)]+b3)

[0124] Step S2.3.6: Perform cross entropy loss on the predicted results and the actual relationship tags, and then train the neural network. The resulting neural network is named Relation Extractor, which can be used to extract the relationship between entities in the text. Depending on the data set, we can get three types of relationship extraction network parameters θ RE Three different kinds of losses are relevant:

[0125]

[0126] Step S2.3.7: For rule p, the rule set Divide into sets and They represent the relationship labels r with p respectively p The same and different rule sets can be used to compare the loss of the rule sets and the rule matching parameters θ RM For training, the specific losses are as follows:

[0127]

[0128] dist + (p,p i )=max(τ-Ruel Matcher(p,p i ),0) 2

[0129] dist - (p,p j )=1-max(Ruel Matcher(p,p i ),0) 2

[0130] Here, τ is a hyperparameter set as an upper limit.

[0131] Step S2.3.8: Combine the above four losses and perform joint training of the RelationExtractor and RuelMatcher networks. The specific loss function is as follows:

[0132] L(θ RE ,θ RM )=L matched (θRE )+αL unmatched (θ RE )+βL rule (θ RE )+γL clus (θ RM )

[0133] where α, β, and γ are hyperparameters that balance different losses.

[0134] The step S3 adopts: step S3.1: for entities in the sentence, use the relationship extraction network (RelationExtractor) to extract the relationship between entities; step S3.2: count the number of relationships between each entity pair in the entity dictionary, and perform screening based on the relationship number threshold; step S3.3: import the entity relationship triples into the Neo4J graph database to obtain a common sense graph.

[0135] Step S4 involves the following steps: Step S4.1: Invite relevant experts and scholars with relevant knowledge; Step S4.2: Have relevant experts and scholars manually modify the common sense entity dictionary and graph; Each time, the respondent can choose from five question formats: the first is to randomly select a common sense entity from the existing common sense dictionary and determine whether the entity should be in the common sense dictionary; the second is to randomly select a relationship triple from the existing common sense graph and determine whether the relationship of the current triple is correct; the third is to randomly select a common sense pair from the existing common sense dictionary and determine the relationship between the common sense pair; the fourth is to randomly select a common sense entity from the existing common sense dictionary and determine another entity related to the entity that is in the dictionary and the relationship between them; the fifth is to randomly select a common sense entity from the existing common sense dictionary and determine another entity related to the entity that is not in the dictionary and the relationship between them. Step S4.3: Verify the modified results; Each time, the verifier can choose from the five question and answer formats corresponding to step S4.2. Each question will be verified at least twice, and questions 1 to 5 are scored 1 to 5 points respectively. If the two verification results are consistent and both judge the answer to the question to be correct, the respondent and the two verifiers will receive the corresponding points; if the two verification results are consistent and both judge the answer to the question to be incorrect, the respondent will not receive any points and both verifiers will receive the corresponding points; if the two verification results are inconsistent, the question will be handed over to a third verifier. If the third verifier judges the answer to be incorrect, the two verifiers who judged it to be incorrect will receive points, and the verifier who judged it to be correct and the respondent will lose points; if the third verifier judges the answer to be correct, the two verifiers who judged it to be correct and the respondent will receive points, and the verifier who judged it to be incorrect will lose points. Finally, both the respondent and the verifier can redeem rewards based on their accumulated points. Step S4.4: Modify the common sense entity dictionary and atlas based on the above results.

[0136] like Figure 4 , the step S5 adopts:

[0137] Step S5.1: Common sense triplet error checking: Using a logistic classification network, each triplet to be tested returns an error confidence score of [0, 1], and then sets a threshold. Triplet scores above the threshold are considered outdated triplet results for error checking.

[0138] Step S5.2: Common sense triple replacement; for triple t=<s,r,o> , determine whether t is t * To replace, a set of logical replacement rules needs to be formulated in advance, so as to determine the positive / negative examples in the subsequent binary classification model. The present invention uses mutually exclusive relationships to represent the common negative example logic, and regards other relationships outside the mutually exclusive relationship as compatible relationships to represent the positive example logic.

[0139] Step S5.2.1: For and If r and r * If it is a pair of mutually exclusive relations, such as "source of infection" and "treatment method", the triple consisting of mutually exclusive relations will be labeled y(t) = 1, indicating that the triple<s,r,o> will be <s,r * ,o>substitution;

[0140] Step S5.2.2: For and If r and r * is a pair of compatible relations, such as "treatment method" and "treatment means", then the triple consisting of compatible relations is set with label y(t) = 1, indicating that the triple<s,r,o> will be <s,r * ,o> is replaced. In addition, for other compatible relations composed of t and t * , set label y(t) = 0, indicating<s,r,o> Pass error checking;

[0141] Step S5.3: Common sense information relationship completion, including: Step S5.3.1: In the process of constructing the common sense graph, its common sense information is sparse, and many hidden relationships have not been mined. For example, in the schema representing "symptoms, treatment methods, treatment descriptions", there is often a lack of treatment method description information corresponding to many disease entities. The purpose of common sense information relationship completion is to predict the missing parts of the common sense triples, so as to make the knowledge graph more complete. For example, for the symptom of "sore throat", it can be established that it has a high semantic correlation with the symptom of "fever", so as to complete the "treatment description" of the symptom of "sore throat" as the treatment description of "fever". Step S5.3.2: After pre-training the translation model and the negative sampling evaluation model, for the missing triplets of the missing tail entity or the head and tail entity, according to Figure 3 The process of completing common sense triples includes: step S5.3.2.1: traversing the entity pool of the current graph in sequence to complete the missing triples; step S5.3.2.2: using the negative sampling evaluation model to obtain the top-k candidate triples predicted as positive examples with the highest confidence as candidate completion triples; step S5.3.2.3: using the translation model to use the triple with the highest score as the completion triple to support the knowledge graph completion task.

[0142] Step S5.4: Continuous update of common sense relationships based on crowdsourcing; in view of the hidden, diverse and evolving nature of common sense relationships, the system allows users to customize relationship types. Crowdsourcing users can identify and correct errors through their own judgment, and other users can review them again. When the common sense data is judged to be correct data, it can be added to the common sense graph by the system for update and storage, such as Figure 5 .

[0143] Example 2:

[0144] like Figure 6 The present invention also provides a common sense graph construction system for human-machine fusion, which can be implemented by executing the process steps of the common sense graph construction method for human-machine fusion, that is, those skilled in the art can understand the common sense graph construction method for human-machine fusion as a preferred implementation method of the common sense graph construction system for human-machine fusion, and the system includes: module M1: collecting relevant news data and encyclopedia data through crawler technology, and integrating them with existing open source graph data; module M2: based on human-machine fusion, annotating the collected news text with entities, relationships and rules, and using the annotated data to jointly train the relationship extraction network and the rule matching network; module M3: based on the annotated data, extracting entities and relationships from the data, and constructing a primary common sense graph; module M4: using crowdsourcing to fine-tune and modify the primary common sense graph to obtain a high-quality common sense graph; module M5: maintaining and evolving the high-quality common sense graph.

[0145] The module M1 includes: module M1.1: obtaining relevant news data through crawler technology; module M1.2: segmenting the obtained news data into sentences; module M1.3: obtaining relevant entry names from Wikipedia and Baidu Encyclopedia through crawler technology; module M1.4: filtering the obtained entry names according to preset rules, and using the filtered results as an entity dictionary;

[0146] The module M2 includes: module M2.1: performing entity labeling on each segmented news sentence according to the entity dictionary; module M2.2: labeling the entity relationship in each news sentence based on preset rules; module M2.3: jointly training the relationship extraction network and the rule matching network based on the labeled data.

[0147] The module M3 includes: module M3.1: for entities in news sentences, using the relationship extraction network to extract the relationship between entities; module M3.2: counting the number of relationships between each entity pair in the entity dictionary, and screening involving a relationship number threshold; module M3.3: importing the entity relationship triples into the Neo4J graph database to obtain a primary common sense graph.

[0148] The module M4 includes: module M4.1: inviting relevant experts and scholars to modify the common sense entity dictionary and atlas; module M4.2: verifying the modification results; module M4.3: modifying the common sense entity dictionary and atlas based on the above results.

[0149] The module M5 includes: Module M5.1: Using the logistic classification network, each triple to be detected returns an error confidence of [0,1], and then sets a threshold, and regards triples above the threshold as outdated triple results of error checking; Module M5.2: Preset logical replacement rules, determine positive / negative examples in the binary classification model, use mutually exclusive relationships to represent common negative example logic, and use other relationships outside the mutually exclusive relationship as compatible relationships to represent positive example logic, thereby performing common sense triple replacement; Module M5.3: Pre- After training the translation model and negative sampling evaluation model, common sense information relationship completion is performed for missing triplets of missing tail entities or head and tail entities. Specifically, the missing triplets are traversed in sequence in the entity pool of the current graph to complete the missing triplets. The top-k candidate triplets with the highest positive confidence predicted by the negative sampling evaluation model are used as candidate completion triplets. The translation model is used to select the triple with the highest score as the completion triplet to support the knowledge graph completion task. Module M5.4: Continuous update of common sense relationships based on crowdsourcing.

[0150] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0151] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for constructing a common sense graph for human-machine fusion, characterized in that: include: Step S1: Collect relevant news data and encyclopedia data through crawler technology and integrate them with existing open source graph data; Step S2: Based on human-machine fusion, the collected news text is annotated with entities, relationships, and rules, and the annotated data is used to jointly train the relationship extraction network and the rule matching network; Step S3: Based on the labeled data, extract entities and relationships from the data and construct a primary common sense graph; Step S4: Use crowdsourcing to fine-tune and modify the primary common sense graph to obtain a high-quality common sense graph; Step S5: Maintain and evolve the high-quality common sense graph; The step S3 comprises: Step S3.1: For entities in news sentences, use the relationship extraction network to extract the relationship between entities; Step S3.2: Count the number of relationships between each entity pair in the entity dictionary, and filter by the threshold of the number of relationships; Step S3.3: Import the entity-relationship triples into the Neo4J graph database to obtain the primary common sense graph; The step S4 comprises: Step S4.1: Invite relevant experts and scholars to modify the common sense entity dictionary and graph; Step S4.2: Verify the modification result; Step S4.3: Modify the common sense entity dictionary and graph based on the above results; The step S5 comprises: Step S5.1: Using a logistic classification network, each triple to be detected returns an error confidence score of [0, 1], and then sets a threshold. Triples above the threshold are regarded as outdated triple results of error checking; Step S5.2: Preset logical replacement rules to determine positive / negative examples in the binary classification model. Use mutually exclusive relationships to represent common negative example logic, and use other relationships outside of the mutually exclusive relationship as compatible relationships to represent positive example logic, thereby performing common sense triple replacement. Step S5.3: After pre-training the translation model and negative sampling evaluation model, perform common sense information relationship completion on missing triplets of missing tail entities or head and tail entities. Specifically, traverse the entity pool of the current graph in sequence to complete the missing triplets. Use the negative sampling evaluation model to obtain the top-k candidate completion triplets with the highest positive confidence scores. Use the translation model to select the triple with the highest score as the completion triplet to support the knowledge graph completion task. Step S5.4: Continuously update common sense relationships based on crowdsourcing.

2. The method for constructing a common sense graph for human-machine fusion according to claim 1, characterized in that: The step S1 comprises: Step S1.1: Obtain relevant news data through crawler technology; Step S1.2: Segment the acquired news data into sentences; Step S1.3: Obtain relevant entry names from Wikipedia and Baidu Encyclopedia using crawler technology; Step S1.4: Filter the acquired entry names according to preset rules, and use the filtered results as the entity dictionary; The step S2 comprises: Step S2.1: perform entity labeling on each segmented news sentence according to the entity dictionary; Step S2.2: Based on the preset rules, label the entity relationships in each news sentence; Step S2.3: Jointly train the relation extraction network and the rule matching network based on the labeled data.

3. The method for constructing a common sense graph for human-machine fusion according to claim 2, characterized in that: The step S2.3 includes: Step S2.3.1: Based on all rules For all sentences Perform hard matching, and the data of successful hard matching is The remaining set of sentences that did not match successfully is Step S2.3.2: For the set of sentences that failed to match For each sentence in , calculate the similarity of the rule that is closest to it; Step S2.3.3: and rule sets It itself is used as a training data set, and label information is added to each data in the data set; Add token information to the sentences. Each sentence s contains two entities e1 and e2. In order for the neural network to capture the position information of the two entities, token symbols need to be inserted at the beginning and end of the two entities. For the first entity e1, insert the token '$' as a marker at the beginning and end, and for the second entity e2, insert the token '#' as a marker at the beginning and end. At the same time, insert the token '[CLS]' at the beginning of each sentence. Step S2.3.4: Based on the training dataset with added labeled information, jointly train the relation extraction network RelationExtractor and the rule matching network RuelMatcher; For each sentence input into the pre-trained Bert, the hidden layer state H is obtained, where H i To H j The hidden layer state of entity e1, H k To H m The hidden layer state belongs to entity e2, H0 belongs to the hidden layer state of [CLS], the average of the state layer belonging to the entity is taken as the representation of the entity, and then a fully connected layer projection is connected to obtain the final representation; Step S2.3.5: Add H ′ 0,H ′ 1 and H ′ 2. After a full connection layer, the final prediction result p is obtained through the softmax layer. Step S2.3.6: Perform cross entropy loss on the predicted results and the actual relationship tags, and then train the neural network. The resulting neural network is named Relation Extractor, which is used to extract the relationship between entities in the text. Depending on the data set, the relationship extraction network parameters θ are obtained. RE Three different losses are related: the loss function L for hard matching success matched (θ RE ), loss function L for unsuccessful hard matching unmatched (θ RE ), relation extraction parameter θ RE The loss function L rule (θ RE ); Step S2.3.7: For rule p, set the rule set Divide into sets and Represents the relationship label r with p respectively p The same and different rule sets, based on this rule set cooperative comparison loss, rule matching parameter θ RM Conduct training; Step S2.3.8: Combine the above four losses and perform joint training of the RelationExtractor and RuelMatcher networks. The specific loss function is as follows: L(θ RE ,i RM )=L matched (i RE )+αL unmatched (i RE )+βL rule (i RE )+γL clus (i RM ) Among them, α, β and γ are hyperparameters to balance different losses, L clus (θ RM ) is the rule matching parameter θ rM The loss function of .

4. A human-machine fusion common sense graph construction system, characterized by: include: Module M1: Collect relevant news data and encyclopedia data through crawler technology, and integrate it with existing open source graph data; Module M2: Based on human-machine fusion, the collected news text is annotated with entities, relationships, and rules, and the annotated data is used to jointly train the relationship extraction network and the rule matching network; Module M3: Based on the labeled data, extract entities and relationships from the data and build a primary common sense graph; Module M4: Use crowdsourcing to fine-tune and modify the primary common sense graph to obtain a high-quality common sense graph; Module M5: Maintain and evolve high-quality common sense graphs; The module M3 includes: Module M3.1: For entities in news sentences, use the relationship extraction network to extract the relationship between entities; Module M3.2: Count the number of relationships between each entity pair in the entity dictionary, and use the relationship number threshold to filter. Module M3.3: Import entity-relationship triples into the Neo4J graph database to obtain a basic common sense graph; The module M4 includes: Module M4.1: Invite relevant experts and scholars to revise the common sense entity dictionary and atlas; Module M4.2: Verify the modification results; Module M4.3: Modify the common sense entity dictionary and graph based on the above results; The module M5 includes: Module M5.1: Using a logistic classification network, each triple to be detected returns an error confidence score of [0, 1], and then sets a threshold. Triples above the threshold are regarded as outdated triples of error checking results. Module M5.2: Preset logical replacement rules to determine positive / negative examples in a binary classification model. Use mutually exclusive relationships to represent common negative example logic, and use other relationships other than mutually exclusive relationships as compatible relationships to represent positive example logic, thereby performing common sense triple replacement. Module M5.3: After pre-training the translation model and negative sampling evaluation model, perform common sense information relationship completion on missing triplets of missing tail entities or head and tail entities. Specifically, the missing triplets are sequentially traversed in the entity pool of the current graph to complete the missing triplets. The top-k candidate triplets predicted as positive examples by the negative sampling evaluation model are selected as candidate completion triplets. The translation model is used to select the triple with the highest score as the completion triplet to support the knowledge graph completion task. Module M5.4: Continuously update common sense relationships based on crowdsourcing.

5. The common sense graph construction system for human-machine fusion according to claim 4 is characterized in that: The module M1 includes: Module M1.1: Obtain relevant news data through crawler technology; Module M1.2: Segment the acquired news data into sentences; Module M1.3: Obtain relevant entry names from Wikipedia and Baidu Encyclopedia through crawler technology; Module M1.4: Filter the acquired entry names according to preset rules and use the filtered results as the entity dictionary; The module M2 includes: Module M2.1: Based on the entity dictionary, perform entity labeling on each segmented news sentence; Module M2.2: Label the entity relationships in each news sentence based on preset rules; Module M2.3: Jointly train the relation extraction network and rule matching network based on labeled data.

6. The common sense graph construction system for human-machine fusion according to claim 5 is characterized in that: The module M2.3 includes: Module M2.3.1: Based on all rules For all sentences Perform hard matching, and the data of successful hard matching is The remaining set of sentences that did not match successfully is Module M2.3.2: For a set of sentences that failed to match For each sentence in , calculate the similarity of the rule that is closest to it; Module M2.3.3: and rule sets It itself is used as a training data set, and label information is added to each data in the data set; Add token information to the sentences. Each sentence s contains two entities e1 and e2. In order for the neural network to capture the position information of the two entities, token symbols need to be inserted at the beginning and end of the two entities. For the first entity e1, insert the token '$' as a marker at the beginning and end, and for the second entity e2, insert the token '#' as a marker at the beginning and end. At the same time, insert the token '[CLS]' at the beginning of each sentence. Module M2.3.4: Jointly train the RelationExtractor and the RuelMatcher based on the labeled training dataset. For each sentence input into the pre-trained Bert, the hidden layer state H is obtained, where H i To H j The hidden layer state of entity e1, H k To H m The hidden layer state belongs to entity e2, H0 belongs to the hidden layer state of [CLS], the average of the state layer belonging to the entity is taken as the representation of the entity, and then a fully connected layer projection is connected to obtain the final representation; Module M2.3.5: H ′ 0,H ′ 1 and H ′ 2. After a full connection layer, the final prediction result p is obtained through the softmax layer. Module M2.3.6: Perform cross entropy loss on the predicted results and the actual relationship tags, and then train the neural network. The resulting neural network is named Relation Extractor, which is used to extract the relationship between entities in the text. Depending on the data set, the relationship extraction network parameters θ are obtained. RE Three different losses are related: the loss function L for hard matching success matched (θ RE ), loss function L for unsuccessful hard matching unmatched (θ RE ), relation extraction parameter θ RE The loss function L rule (θ RE ); Module M2.3.7: For rule p, set the rule Divide into sets and Represents the relationship label r with p respectively p The same and different rule sets, based on this rule set cooperative comparison loss, rule matching parameter θ RM Conduct training; Module M2.3.8: Combine the above four losses to perform joint training of the RelationExtractor and RuelMatcher networks. The specific loss function is as follows: L(θ RE ,i RM )=L matched (i RE )+αL unmatched (i RE )+βL rule (i RE )+γL clus (i RM ) Among them, α, β and γ are hyperparameters to balance different losses, L clus (θ RM ) is the rule matching parameter θ RM The loss function of .

Images