Man-machine cooperation-based knowledge graph refinement method and related device

Abstract

The invention discloses a knowledge graph refinement method based on man-machine cooperation and a related device, and relates to the technical field of knowledge graph refinement, and the method comprises the steps: carrying out the vectorization processing of each triple in a to-be-refined knowledge graph, calculating the score of each vectorized triple, and obtaining the score of each vectorized triple; according to the method, uncertain triples which possibly need to be refined are determined, the uncertain triples with missing elements are selected as tasks, and the type of each task is determined. The tasks and the experts are matched with the lowest time cost and the highest capability coincidence degree as targets, and the multiple experts completing each task are determined. And for each task, obtaining an answer of each expert in the plurality of experts completing the task, determining a final answer of the task based on the type of the task and all the answers, and complementing the task based on the final answer to complete the refinement of the knowledge graph to be refined. According to the method, more efficient, more accurate and more professional knowledge graph refinement can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of knowledge graph refinement, and in particular to a knowledge graph refinement method based on human-computer collaboration and related devices. Background Art

[0002] Current research shows that knowledge graphs containing rich entities and complex semantic relationships are gradually being applied to the field of engineering design. The refinement of knowledge graphs is the process of completing the missing information in the knowledge graph. How to cover more comprehensive problems with fewer triples and improve the accuracy of knowledge graph refinement is the original intention of this method.

[0003] The existing knowledge graph refinement methods are:

[0004] ① Methods based on algorithmic models. With the development of artificial intelligence, fully automatic refinement methods based on algorithmic models to refine knowledge graphs have been widely used. This method has a high degree of automation and greatly reduces human involvement. However, specific algorithms need to rely on specific data sets and specific environments, and their generalization is general. In addition, some internal parameters of the algorithms are difficult to interpret, which is equivalent to a black box and has poor interpretability.

[0005] ② Hire experienced experts to complete the knowledge graph. This method is the most traditional method with extremely high accuracy, strong professionalism, and high interpretability. However, it is obviously a refined method with high cost, low operating efficiency, and low generalization.

[0006] ③ Human-machine collaboration method based on collective intelligence. This method can gather the knowledge and experience of multiple people and provide a more comprehensive perspective. At the same time, the integration of artificial intelligence can process big data beyond human capabilities and provide deep insights for decision-making. However, its shortcomings are also obvious. It requires a large mass base and the quality is difficult to guarantee. It has little effect on the refinement of highly professional knowledge graphs. Summary of the invention

[0007] The purpose of this application is to provide a knowledge graph refinement method and related devices based on human-computer collaboration, which can achieve more efficient, accurate and professional knowledge graph refinement.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] In a first aspect, the present application provides a knowledge graph refinement method based on human-computer collaboration, and the knowledge graph refinement method based on human-computer collaboration includes:

[0010] Each triple in the refined knowledge graph is vectorized to obtain a vectorized triple, and all the vectorized triples are combined into a vectorized knowledge graph; the elements of the triple include a head entity, a tail entity, and a relationship between the head entity and the tail entity;

[0011] Calculating a score for each of the vectorized triples, and determining uncertain triples that may need to be refined based on the scores of all the vectorized triples;

[0012] An uncertain triple with missing elements is selected as a task. For each of the tasks, if the missing element of the task has only one candidate answer, the type of the task is a single-choice task; if the missing element of the task has multiple candidate answers, the type of the task is a multiple-choice task; if the missing element of the task has no candidate answer, the type of the task is a fill-in-the-blank task, so as to determine the type of the task; the candidate answer is the value of the missing element in the candidate vectorized triple in the vectorized knowledge graph, and the candidate vectorized triple is the same as the vectorized triple of the non-missing element of the task;

[0013] With the goal of minimizing time cost and maximizing competence, the tasks and experts are matched to determine multiple experts who can complete each of the tasks;

[0014] For each of the tasks, the answer of each expert among the multiple experts who completed the task is obtained, and based on the type of the task and all the answers, the final answer of the task is determined, and the task is completed based on the final answer to complete the refinement of the knowledge graph to be refined.

[0015] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned knowledge graph refinement method based on human-computer collaboration.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned knowledge graph refinement method based on human-computer collaboration.

[0017] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned knowledge graph refinement method based on human-computer collaboration.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a knowledge graph refinement method and related device based on human-computer collaboration. After vectorizing each triple in the refined knowledge graph to obtain the vectorized triple, the score of each vectorized triple is further calculated. Based on the scores of all vectorized triples, the uncertain triples that may need to be refined are determined, and the uncertain triples with missing elements are selected as tasks, so that the tasks that need to be filled are determined by vectorization, score calculation and element missing judgment, which can reduce the number of tasks faced by refinement and greatly improve the refinement efficiency. According to the missing elements of the task, the type of each task is determined, and further with the lowest time cost and the highest degree of ability compliance as the goal, the tasks and experts are matched, and multiple experts who complete each task are determined, so as to consider the two factors of time cost and ability compliance, accurately match the tasks and experts, ensure that each expert can be assigned to the task that he is more proficient in, improve the professionalism of refinement, and at the same time reduce the time for experts to answer the tasks, improve the efficiency of answering the tasks, further improve the efficiency of refinement, and also improve the effect of answering the tasks, and further improve the accuracy of refinement. For each task, the answers of each expert among the multiple experts who completed the task are obtained, and the final answer of the task is determined based on the type of task and all the answers, and the task is completed based on the final answer. By reaching a consensus on all the answers and determining a better final answer, the effect of answering the task can be improved, and the accuracy and professionalism of the refinement can be further improved. This application can achieve more efficient, accurate and professional knowledge graph refinement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 An application environment diagram of a knowledge graph refinement method based on human-computer collaboration provided in Example 1 of the present application.

[0022] Figure 2 A flowchart of a knowledge graph refinement method based on human-computer collaboration provided in Example 1 of the present application.

[0023] Figure 3 A detailed flowchart of a knowledge graph refinement method based on human-computer collaboration provided in Example 1 of the present application.

[0024] Figure 4 Schematic diagram of the optimal consensus process provided in Example 1 of the present application.

[0025] Figure 5 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Example 1

[0028] The knowledge graph refinement method based on human-computer collaboration provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the knowledge graph to be processed to the server. After the server receives the knowledge graph to be processed, the server vectorizes each triple in the knowledge graph to be refined to obtain vectorized triples; calculates the score of each vectorized triple, and determines the uncertain triples that may need to be refined based on the scores of all vectorized triples; selects uncertain triples with missing elements as tasks. For each task, if there is only one candidate answer for the missing element of the task, the type of task is a single-choice task. If any If there are multiple candidate answers for the missing elements of the task, the type of the task is a multiple-choice task; if there are no candidate answers for the missing elements of the task, the type of the task is a fill-in-the-blank task, so as to determine the type of the task; with the goal of minimizing time cost and maximizing ability matching, tasks and experts are matched to determine multiple experts who complete each task; for each task, the answer of each expert among the multiple experts who complete the task is obtained, and the final answer of the task is determined based on the type of task and all answers, and the task is completed based on the final answer to complete the refinement of the knowledge graph to be refined and obtain the refined knowledge graph. The server can feed back the refined knowledge graph obtained for the knowledge graph to be refined to the terminal.

[0029] In addition, in some embodiments, the knowledge graph refinement method based on human-computer collaboration can also be implemented independently by a server or a terminal. For example, the terminal can directly refine the knowledge graph to be processed, or the server can obtain the knowledge graph to be processed from the data storage system and refine the knowledge graph to be processed.

[0030] The terminals may be, but are not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server.

[0031] like Figure 2 As shown, a knowledge graph refinement method based on human-computer collaboration is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 Taking the server in as an example, the knowledge graph refinement method based on human-computer collaboration includes the following steps:

[0032] Step S1, vectorize each triple in the refined knowledge graph to obtain a vectorized triple, and combine all the vectorized triples into a vectorized knowledge graph; the elements of the triple include a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0033] Step S2: Calculate the score of each of the vectorized triples, and determine the uncertain triples that may need to be refined based on the scores of all the vectorized triples.

[0034] Step S3, select uncertain triples with missing elements as tasks. For each of the tasks, if the missing element of the task has only one candidate answer, the type of the task is a single-choice task; if the missing element of the task has multiple candidate answers, the type of the task is a multiple-choice task; if the missing element of the task has no candidate answer, the type of the task is a fill-in-the-blank task, so as to determine the type of the task; the candidate answer is the value of the missing element in the candidate vectorized triple in the vectorized knowledge graph, and the candidate vectorized triple is the same as the vectorized triple of the non-missing element of the task.

[0035] Step S4, with the goal of minimizing time cost and maximizing capability matching, matches the tasks and experts, and determines a plurality of experts who can complete each of the tasks.

[0036] Step S5, for each of the tasks, obtain the answer of each expert among the multiple experts who have completed the task, determine the final answer of the task based on the type of the task and all the answers, and complete the task based on the final answer to complete the refinement of the knowledge graph to be refined.

[0037] By implementing the above steps S1 to S5, after vectorizing each triple in the knowledge graph to be refined and obtaining the vectorized triple, this embodiment further calculates the score of each vectorized triple, determines the uncertain triples that may need to be refined based on the scores of all vectorized triples, and selects the uncertain triples with missing elements as tasks, thereby determining the tasks that need to be filled through vectorization, score calculation and element missing judgment, which can reduce the number of tasks faced by refinement and greatly improve the efficiency of refinement. According to the missing elements of the task, the type of each task is determined, and further with the lowest time cost and the highest degree of ability compliance as the goal, tasks and experts are matched, and multiple experts who complete each task are determined, so as to consider the two factors of time cost and ability compliance, accurately match tasks and experts, ensure that each expert can be assigned to the task that he is more proficient in, improve the professionalism of refinement, and at the same time reduce the time for experts to answer tasks, improve the efficiency of answering tasks, further improve the efficiency of refinement, and also improve the effect of answering tasks, and further improve the accuracy of refinement. For each task, the answers of each expert among the multiple experts who have completed the task are obtained, and the final answer of the task is determined based on the type of task and all the answers. The task is completed based on the final answer, so that a better final answer can be determined by reaching a consensus on all the answers, which can improve the effect of answering the task and further improve the accuracy and professionalism of the refinement.

[0038] Knowledge graph refinement is a supplement to knowledge graph. It improves knowledge graph through the process of problem extraction, problem solving and graph updating, so that knowledge graph has a more accurate and comprehensive effect. At present, knowledge graph refinement has been widely used in the field of engineering design, such as scalable distributed knowledge management system, user demand knowledge graph for guiding product innovation, equipment maintenance and support knowledge graph for improving maintenance and support capabilities under information technology conditions, military equipment knowledge graph, etc. In the field of engineering design, data statistics show that many highly professional fields have a situation of numerous tasks and scarce experts. Based on this, the knowledge graph refinement method of this embodiment ensures a relatively complete coverage on the basis of reducing tasks through the three processes of design task generation, expert-task allocation and answer aggregation, gives full play to the benefits of experts, improves the accuracy of knowledge graph refinement, and realizes knowledge graph refinement based on human-computer collaboration.

[0039] This embodiment belongs to a method for refining knowledge graphs through human-computer collaboration, and aims to invent a human-computer collaboration method for a small number of domain experts to achieve more efficient, more accurate, and more professional knowledge graph refinement. This embodiment provides a method for refining knowledge graphs based on human-computer collaboration, including: ① redefining and dividing tasks based on the TransE model to refine tasks; ② establishing models for tasks and experts respectively to achieve expert-task allocation; ③ achieving answer aggregation based on the concept of optimal task consensus, and updating the quality level of experts. This method promotes the research on knowledge graph refinement based on human-computer collaboration when the number of research experts is small, and solves three problems: ① the problem of too many tasks in the process of knowledge graph refinement based on human-computer collaboration; ② the problem of difficult reasonable allocation of tasks in the process of knowledge graph refinement based on human-computer collaboration; ③ the problem of sparse answer data due to the small number of experts in the process of knowledge graph refinement based on human-computer collaboration.

[0040] This embodiment is based on the re-interpretation of the knowledge graph, refines task generation, builds an expert-task assignment model, obtains answers through the answer aggregation method, supplements them to the knowledge graph, and realizes the refinement of the knowledge graph. Figure 3 As shown, this embodiment introduces the knowledge graph refinement method of this embodiment through three steps: task generation, expert-task allocation and answer aggregation.

[0041] (I) Task Generation

[0042] In the task generation stage, the vectorized representation of the knowledge graph is realized based on the TransE model. On this basis, the concept definition of tasks and their type classification method are proposed, and the task generation process is realized. Specifically, task generation is to realize the vectorized representation of the knowledge graph through the TransE model, and then select uncertain triples and package them into three task forms: single choice, multiple choice, and fill-in-the-blank, which corresponds to steps S1, S2, and S3 of this embodiment.

[0043] (1) Vectorized Representation of Knowledge Graph

[0044] This embodiment specifically uses the TransE model. In step S1, each triple in the refined knowledge graph is vectorized to obtain a vectorized triple, which specifically includes: using the trained TransE model to vectorize each triple in the refined knowledge graph to obtain a vectorized triple. Subsequently, all the vectorized triples are combined into a vectorized knowledge graph.

[0045] When training the TransE model, the training data uses correct triples in the sample knowledge graph. The elements of the triples include the head entity, the tail entity, and the relationship between the head entity and the tail entity. The correct triples are triples with correct elements. Then, some correct triples are negatively sampled, specifically by replacing the head entity or the tail entity of the correct triple to obtain incorrect triples. When replacing the head entity or the tail entity, in order to reduce the impact of one-to-many and many-to-one relationships in the knowledge graph on the accuracy, the Bernoulli idea is used in replacement sampling, that is, for a correct triple (h, r, t), h represents the head entity, t represents the tail entity, and r represents the relationship between the head entity and the tail entity, with probability Replace the head entity of the correct triple with the tail entity of the correct triple with probability 1-p, where tph (abbreviation of tail entities per head) represents the average number of tail entities corresponding to each head entity in the sample knowledge graph, and hpt (abbreviation of head entities per tail) represents the average number of head entities corresponding to each tail entity in the sample knowledge graph.

[0046] During the training process of the trained TransE model, the training sample set is generated as follows: obtain multiple correct triples in the sample knowledge graph; randomly select several correct triples as triples to be replaced; for each triple to be replaced, replace the head entity or the tail entity of the triple to be replaced to obtain an incorrect triple; all correct triplets and all incorrect triplets form a training sample set, which is used to train the initial TransE model to obtain a trained TransE model.

[0047] When replacing the head entity of the triple to be replaced, the replacement value used is p, and the calculation formula of p is:

[0048]

[0049] In formula (1), tph is the average number of tail entities corresponding to each head entity in the sample knowledge graph; hpt is the average number of head entities corresponding to each tail entity in the sample knowledge graph.

[0050] When replacing the tail entity of the triple to be replaced, the replacement value used is 1-p.

[0051] When the initial TransE model is trained using the training sample set to obtain the trained TransE model, the loss function used is:

[0052]

[0053] In formula (2), L is the loss function value; S is the set of positive samples, that is, the relationship between h, r, and t is a set of correct triples (also called positive samples), h is the head entity in the correct triple, r is the relationship in the correct triple, and t is the tail entity in the correct triple; S′ is the set of negative samples, that is, the relationship between h, r, and t is a set of incorrect wrong triples (also called negative samples), h′ is the head entity in the wrong triple, r′ is the relationship in the wrong triple, and t′ is the tail entity in the wrong triple; γ is a hyperparameter used to define The boundary between positive and negative samples; d is the distance function, which means the distance between the sum of the vector h / h' corresponding to the head entity and the vector r / r' corresponding to the relationship and the vector t / t' corresponding to the tail entity. Specifically, the Euclidean distance can be used. The vector corresponding to the head entity, the vector corresponding to the relationship and the vector corresponding to the tail entity are determined by the vectorized triple. After the triple is converted into the vectorized triple, the head entity in the triple becomes the vector corresponding to the head entity, the relationship in the triple becomes the vector corresponding to the relationship, and the tail entity in the triple becomes the vector corresponding to the tail entity; [x] + Represents the negative sampling distribution associated with the triple. The role of this loss function is to minimize the distance between positive samples and maximize the distance between negative samples.

[0054] Each parameter in the TransE model needs to be updated according to the following formula:

[0055]

[0056] In formula (3), θ new is the updated parameter in the TransE model; θ old is the parameter to be updated in the TransE model; α is the learning rate, α>0; ▽ θ is the gradient; f r (h, t) is d(h+r, t) in the loss function L; f r' (h',t') is d(h'+r',t') in the loss function L.

[0057] Based on the above content, the TransE model is trained to obtain a trained TransE model to realize the vectorized representation of the knowledge graph.

[0058] (2) Task generation based on triplet scoring

[0059] ①Triple scoring technology based on TransE model

[0060] The scoring formula for vectorized triples can generally be expressed as:

[0061]

[0062] In formula (4), f r (h, t) is the score of the vectorized triplet. The score of the vectorized triplet calculated by formula (4) is as follows: the higher the score, the lower the possibility of the existence of the vectorized triplet.

[0063] In order to remove the influence of erroneous vectorized triples as much as possible, a calculation formula for the association existence of vectorized triples (n1, r, n2) is designed as follows:

[0064] p(n1,r,n2)=a1·p(n1,r)+a2·p(n2,r)+a3·p(n1,n2)(5)

[0065] In formula (5), p(n1, r, n2) is the probability that the head entity n1 and the tail entity n2 exist in a relationship r; a1, a2, and a3 are all constants that can be selected by the user. p(n1, r) is the probability that the head entity n1 and the relation r exist at the same time. Specifically, the sum of the number of vectorized triples containing only the head entity n1 and the number of vectorized triples containing only the relation r in the existing knowledge graph is calculated, and the ratio of the number of vectorized triples containing both the head entity n1 and the relation r in the existing knowledge graph is calculated. The existing knowledge graph is a refined knowledge graph belonging to the same field as the knowledge graph to be refined; p(n2, r) is the probability that the tail entity n2 and the relation r exist at the same time. Specifically, the vectorized triples containing only the tail entity n2 in the existing knowledge graph are calculated. The sum of the number of quantified triples and the number of vectorized triples containing only relation r is calculated, and the ratio of the number of vectorized triples containing both tail entity n2 and relation r in the existing knowledge graph to the sum is calculated; p(n1, n2) is the probability that the head entity n1 and the tail entity n2 exist at the same time. Specifically, the sum of the number of vectorized triples containing only head entity n1 and the number of vectorized triples containing only tail entity n2 in the existing knowledge graph is calculated, and the ratio of the number of vectorized triples containing both head entity n1 and tail entity n2 in the existing knowledge graph is calculated.

[0066] The above formula (5) can be used to calculate the probability that the head entity n1 and the tail entity n2 exist in a relation r, thereby eliminating some incorrect vectorized triples. The value of has little correlation with the existing knowledge graph. In fact, when an element in the vectorized triple appears in the existing knowledge graph, it indicates that the vectorized triple has a certain correlation with the existing knowledge graph. The higher the frequency and the more elements appear, the greater the correlation. Based on this, this embodiment defines the score of the vectorized triple based on the entity set E and the relationship set R of the existing knowledge graph. The scoring formula is as follows:

[0067]

[0068] In formula (6), f(n1, r, n2) is the score of the vectorized triple, n1 is the head entity in the vectorized triple, r is the relationship in the vectorized triple, and n2 is the tail entity in the vectorized triple; p(n1, r, n2) is the probability that the head entity n1 and the tail entity n2 exist in the relationship r, which is calculated using formula (5); k i To represent the binary coefficient of whether the i-th element exists in the relationship set E or the entity set R, if i∈E||i∈R, then k i is 1, otherwise, k i is 0, the relationship set E and entity set R are respectively the set of relationships in the existing knowledge graph and the set of head entities and tail entities; θ i is the influence coefficient of the i-th element, which represents the influence of element i on the vectorized triple, and its value is shown in the following Table 1:

[0069] Table 1 The influence of element i on the vectorized triple θ i

[0070] i <![CDATA[θ i ]]> <![CDATA[n1∈E]]> 0.5 r∈R 0.2 <![CDATA[n2∈E]]> 0.3

[0071] Through the above method, the score of each vectorized triplet integrated with external data resources can be calculated.

[0072] Then in step S2, the score of each vectorized triple is calculated, which specifically includes: using a scoring formula to calculate the score of each vectorized triple, and the scoring formula is the above formula (6).

[0073] ②Task generation strategy

[0074] The concept of a task is to encapsulate a problem to be solved by using uncertain triples identified by existing knowledge graphs and related resources. After calculating the score of each vectorized triple, the score ranges from (0 to 1). The meaning of the score is: for any vectorized triple, the closer the score is to 1, the lower the possibility of its existence, and the closer the score is to 0, the higher the possibility of its existence. The subsequent task generation is based on the score.

[0075] Then in step S2, based on the scores of all vectorized triples, the uncertain triples that may need to be refined are determined, specifically including: for each vectorized triple, if the score of the vectorized triple is greater than or equal to a preset value and less than or equal to 1, the vectorized triple is determined to be an uncertain triple that may need to be refined.

[0076] In step S3, an uncertain triple with missing elements is selected as a task. An uncertain triple with missing elements means that at least one element in the uncertain triple is unknown. For each task, if the missing element of the task has only one candidate answer, the type of the task is a single-choice task. If the missing element of the task has multiple candidate answers, the type of the task is a multiple-choice task. If the missing element of the task has no candidate answer, the type of the task is a fill-in-the-blank task, so as to determine the type of the task. Among them, the candidate answer is the value of the missing element in the candidate vectorized triple in the vectorized knowledge graph, and the candidate vectorized triple is the same vectorized triple as the non-missing element of the task. Finally, the task is divided into three parts: a single-choice task, a multiple-choice task, and a fill-in-the-blank task.

[0077] 2. Expert-Task Allocation

[0078] After completing the task generation, we can design a plan to assign the task to the experts. In the expert-task assignment stage, we established the task model and the expert model, and used the genetic algorithm to solve the multi-task assignment model based on the task model and the expert model, completing the matching process of tasks and experts.

[0079] (1) Extraction of key elements of the task and model construction

[0080] This embodiment uses the classification standard C based on expert experience as the task classification result. The classification standard C includes K knowledge categories. For knowledge graphs in different fields, the K knowledge categories are different. i (i=1,2,…,K), use the Text-Rank algorithm to extract the characteristic keywords of task t and the weight of each characteristic keyword, and randomly select some characteristic keywords, arrange the weights of the selected characteristic keywords in descending order, and obtain the vector c of the i-th knowledge category i =(S1,S2,…,S n ), S i is the weight of the i-th feature keyword, i=1, 2, ..., n.

[0081] For task t, based on the vector of the i-th knowledge category, obtain the vector X of task t on the i-th knowledge category t =(f t (w1),f t (w2),…,f t (w n )), where f t (w n ) is the nth characteristic keyword w of task t in the i-th knowledge category n The eigenvalue of is calculated as follows:

[0082]

[0083] In formula (7), q is whether the vocabulary set representing task t contains the nth feature keyword w n Binarization coefficient, if the vocabulary of task t contains the nth feature keyword w n , then q = 1, if the vocabulary of task t does not contain the nth feature keyword w n , then q = 0, the vocabulary set of task t is generated in advance based on expert experience; is the nth feature keyword w n The weight of the vector c of the i-th knowledge category i =(S1,S2,…,S n ), i.e. the sorting.

[0084] Calculate the vector c of the i-th knowledge category respectively i =(S1,S2,…,S n ) and the vector X of task t on the i-th knowledge category t The cosine value of Then the knowledge category feature vector of task t is expressed as: in, is the characteristic value of task t in the i-th knowledge category.

[0085] Based on this, this embodiment can construct a task model T for task t, which is specifically expressed as follows:

[0086]

[0087] In formula (8), ID is the unique number of task t, which is used to accurately distinguish different tasks.

[0088] (2) Experts’ key factor extraction and model construction

[0089] Cost is an important factor that affects the participation of staff in the knowledge crowdsourcing process. Based on this, this embodiment divides the factors affecting the expert model into two main dimensions: response speed to different tasks and performance level on different tasks, based on the two goals of the expert-task allocation stage and starting from the experts' own experience, ability, and task preference.

[0090] The response speed of the expert is quantified and represented by RV. Since there are K categories of knowledge, the response speed feature vector of expert w can be expressed as: in, is the characteristic value of the response speed of expert w in the i-th knowledge category.

[0091] The performance level of the expert is quantified, and S is used to represent the performance level. Since there are K categories of knowledge, the performance level feature vector of expert w can be expressed as: in, is the characteristic value of the performance level of expert w in the i-th knowledge category.

[0092] By setting a set of golden tasks (i.e., tasks with known true answers), and using the performance of each expert on the golden tasks, we initialize the response speed and performance level of all experts. The response speed can be the average answer time of the golden tasks answered by the experts, and the performance level can be the ratio of the sum of the eigenvalues ​​of task t on the knowledge category in the golden tasks answered correctly by the experts to the sum of the eigenvalues ​​of task t on the knowledge category in the golden tasks answered by the experts.

[0093] Based on this, this embodiment constructs an expert model W of expert w, which is specifically expressed as follows:

[0094]

[0095] In formula (9), ID is the unique number of the expert, which is used to accurately distinguish different experts.

[0096] (3) Expert-task allocation

[0097] After the task model and the expert model are established, the next step is to perform allocation. Based on the above two models, this embodiment establishes a multi-task assignment model based on expert-task allocation.

[0098] This embodiment uses represents the time cost of expert i to complete task j. The time cost is closely related to the expert's response speed. Then we have use represents the degree of conformity of task j to the ability of expert i. The degree of conformity is closely related to the performance level of the expert. Then we have The expert-task allocation problem to be solved is abstracted into the following mathematical model:

[0099] There are n tasks to be assigned to m (m < n) people (experts). Each task can be completed by one expert. Each task needs to be completed t times. Each expert needs to complete multiple different tasks. To ensure the number of times each task is completed, each expert must undertake at least k tasks and at most k+1 tasks. Under the condition of target l, the target attribute value of the i-th expert doing the j-th task is Which task should be assigned to which expert to complete so that all objectives considered when completing all tasks are optimal.

[0100] Take 0-1 variable x ij ,make The mathematical model of the expert-task allocation problem considering multiple objectives is:

[0101]

[0102] In formula (10), max' represents max (taking the minimum value) in the target where the larger the value, the better, and represents min (taking the minimum value) in the target where the smaller the value, the better; C is the objective function value; m is the total number of experts; n is the total number of tasks; is the target attribute value of the lth goal when assigning the jth task to the ith expert; x ij is a binary variable representing the assignment of the jth task to the ith expert. If the jth task is assigned to the ith expert, then x ij =1, otherwise, x ij =0; t is a constant, which can be customized by the user; k is a constant, which can be customized by the user. As an example,

[0103] The target attribute value when the i-th expert completes the j-th task based on the target l Get the attribute value matrix C under the target condition l , where i=1,2,…,m, j=1,2,…,n, and l=1,2.

[0104]

[0105] According to the attribute value matrix C l Find the fuzzy relative membership of each attribute value to the optimal value under the target l condition The targets whose larger values ​​are better and the targets whose smaller values ​​are better are calculated according to the following two formulas, where i=1,2,…,m, j=1,2,…,n, and l=1,2.

[0106]

[0107] In formula (12), is the target attribute value of the lth goal when assigning the jth task to the ith expert; c lmin is the minimum value of the target attribute value of the lth target when the jth task is assigned to the ith expert, that is, the attribute value matrix C l The minimum value among lmax is the maximum value of the target attribute value of the lth target when the jth task is assigned to the ith expert, that is, the attribute value matrix C l The maximum value in .

[0108]

[0109] According to the above formula (12) and formula (13), the attribute value matrix C l It becomes the optimal fuzzy relation matrix R under the condition of target l l ,as follows:

[0110]

[0111] Taking the investigation of the ith expert, that is, the response speed feature vector and performance level feature vector of the ith expert as input, the Delphi method or the analytic hierarchy process (AHP) is used to determine the target weight vector W = (w1, w2). i = 1, 2, ..., m; j = 1, 2, ..., n; l = 1, 2, which can be regarded as the synthetic relative membership of each attribute value to the best after comprehensively considering the two goals. ij Combined into a multi-objective fuzzy relationship synthesis matrix as follows:

[0112]

[0113] Then model 1 can be transformed into model 2:

[0114]

[0115] In formula (16), Z is the objective function value, represents the total benefit of the model; m is the total number of experts; n is the total number of tasks; r ij is the composite relative membership for assigning the jth task to the ith expert; x ij is a binary variable indicating the assignment of the jth task to the ith expert; t is a constant; k is a constant,

[0116] Based on this model, this embodiment uses a genetic algorithm to solve the above model, and finally obtains a better solution to complete the expert-task allocation process. For tasks assigned to experts, if the experts are not sure about the answer or it is an area they are not good at, they can choose not to answer.

[0117] Then in step S4, tasks and experts are matched with the goal of minimizing time cost and maximizing ability matching, and multiple experts who can complete each task are determined, which specifically includes: establishing a matching model with the goal of minimizing time cost and maximizing ability matching; solving the matching model to match tasks and experts, and multiple experts who can complete each task are determined.

[0118] The matching model is the above formula (16), and the calculation formula of the synthetic relative membership is:

[0119]

[0120] In formula (17), w l is the weight of the lth target; is the fuzzy relative membership of the lth target.

[0121] When the goal is to minimize the time cost, the fuzzy relative membership is calculated using formula (13). At this time, Among them, RV i is the response speed feature vector of the i-th expert; C j is the knowledge category feature vector of the jth task.

[0122] When the goal is to achieve the highest degree of capability compliance, the fuzzy relative membership is calculated using formula (12). At this time, Among them, S i is the performance level feature vector of the i-th expert; C j is the knowledge category feature vector of the jth task.

[0123] Among them, the knowledge category feature vector is: Among them, C t is the knowledge category feature vector of task t; is the characteristic value of task t in the first knowledge category; is the characteristic value of task t in the second knowledge category; is the characteristic value of task t in the i-th knowledge category; is the eigenvalue of task t in the Kth knowledge category.

[0124] The characteristic value of task t in the i-th knowledge category is the cosine value of the vector of the i-th knowledge category and the vector of task t in the i-th knowledge category, where the vector of the i-th knowledge category is generated as follows: extract the characteristic keywords of task t in the i-th knowledge category and the weight of each characteristic keyword; select some characteristic keywords, and arrange the weights of the selected some characteristic keywords in descending order to obtain the vector of the i-th knowledge category.

[0125] (III) Answer aggregation

[0126] In the answer aggregation stage, multiple experts will provide answers to the same task. These answers may overlap or conflict with each other. In order to obtain the most credible answer, this embodiment proposes a new concept: optimal consensus. That is, in the answer aggregation stage, the concept of optimal task consensus is proposed based on the fusion of task answer information and expert information. The optimal consensus solution methods for three types of tasks are given respectively. Figure 4 As shown in Figure 2, it shows the optimal consensus process in the answer aggregation phase. The input of the optimal consensus is the task set. and the answer set for each task Expert Collection and for each task t, the set of answers provided by the experts (When expert w does not answer task t, a tw is empty), the output is the best consensus set for the task and updated expert collection

[0127] The specific process is as follows:

[0128] For the tth task (t = 1, 2, ..., T, T is the total number of tasks):

[0129] ① Use represents the answer set of task t, where is the i-th answer to task t, and n is the number of answers obtained for task t.

[0130] ② Use represents the probability consensus of all answers to task t in various knowledge categories, where It represents the probability that the ith answer of task t is the best consensus in the kth knowledge category, and is expressed as represents the best consensus for task t.

[0131] ③Use represents the knowledge category feature vector of task t, The knowledge category to which the task belongs.

[0132] ④ Use 1, 2, and 3 to represent the type of task t, which are single choice, multiple choice, and fill-in-the-blank, respectively. l (l=1, 2, 3) represents the probability distribution of tasks.

[0133] For the w-th expert (w = 1, 2, ..., W, W is the total number of experts):

[0134] ①Use T (w) represents the set of tasks answered by expert w, and represents the set of tasks whose answers are adopted among the tasks answered by expert w.

[0135] ②Use the expression level feature vector represents the performance level of expert w in various knowledge categories, is the characteristic value of task t in the kth knowledge category.

[0136] ③Use credibility feature vector represents the credibility of the performance level of expert w in various knowledge categories, It represents the proportion of tasks answered by expert w in knowledge category k, which represents the credibility of expert w’s performance level in knowledge category k.

[0137] ④Use a tw represents the answer of expert w to task t.

[0138] Based on this, the answer to task t can be The best consensus c on the kth knowledge category t The probability of is expressed as follows:

[0139]

[0140] In formula (18), Indicated in φ l The probability of answer a appearing under the distribution.

[0141] The tasks designed in this embodiment include single-choice, multiple-choice and fill-in-the-blank. Due to the different option settings, the distribution of answers collected for each type of task is φ l There are also large differences, so in the process of solving the best consensus, it is necessary to consider them separately according to the task type, as detailed below:

[0142] ① Single-choice task

[0143] This type of task and answer conforms to the binomial distribution, with "1" and "2" representing the two possible answers, 1 for not choosing and 2 for choosing, which can be expressed as:

[0144]

[0145] In formula (19), c t is the best consensus for task t; The probability of selecting the first answer (“1”) for task t on the kth knowledge category; The probability of selecting the second answer (“2”) for task t in the kth knowledge category; K is the total number of knowledge categories; is the i-th answer to task t under probability distribution φ1 is the probability of the first answer, represents the probability value that the answer to task t is "1" under the φ1 distribution, which is the number of answers to task t that are "1" n ai＝1 The total number of answers n(A (t) ) ratio; is the answer a given by expert w to task t tw is the total number of the first type of answers; is the characteristic value of task t in the kth knowledge category; is the characteristic value of the performance level of expert w in the kth knowledge category; is the credibility characteristic value of expert w in the kth knowledge category; is the i-th answer to task t under probability distribution φ1 is the probability of the second answer, represents the probability value that the answer to task t is "2" under the φ1 distribution, which is the number of answers to task t that are "2" The total number of answers n(A (t) ) ratio; is the answer a given by expert w to task t tw is the total number of the second type of answers.

[0146] ②Multiple choice tasks

[0147] After analysis, it is found that for each option (i.e., candidate answer) of a task, there are two results: selected or not selected. If not selected is regarded as "1" and selected is regarded as "2", "1" and "2" are also mutually exclusive events, and each expert's operation on this option is independent of each other. Therefore, each option of this type of task obeys the binomial distribution. represents the best consensus of the rth option of task t, then It can be expressed as follows:

[0148]

[0149] In formula (20), is the best consensus for the rth option of task t; is the i-th answer to task t under probability distribution φ2 is the probability of the first answer, represents the probability value that the answer to the rth option is "1" under the φ2 distribution; is the i-th answer to task t under probability distribution φ2 is the probability of the second answer, It represents the probability value that the answer to the rth option is "2" under the φ2 distribution.

[0150] ③Fill-in-the-blank task

[0151] This type of answer is in text form. Since each expert has his or her own usage habits, the distribution of the answers is unknown. However, in theory, the more times an answer appears, the greater the probability that it is the best consensus. Therefore, this embodiment uses the frequency of words to express the probability of each answer appearing in the unknown distribution. For this type of task t, assuming that there are n answers, represented by "1", "2", ..., "n" respectively, the best consensus can be expressed as:

[0152]

[0153] In formula (21), is the probability of selecting the i-th answer for task t in the k-th knowledge category; K is the total number of knowledge categories; n is the total number of answer types; is the jth answer to task t under probability distribution φ3 is the probability of the i-th answer, represents the probability value of the answer to task t being “i” (i=1, 2, …, n) under the φ3 distribution, which is the number of answers to task t that are “i” The total number of answers n(A (t) ) ratio; is the answer a given by expert w to task t tw is the total number of the i-th answer.

[0154] After obtaining the answer, the expert's quality level is updated. This embodiment designs an iterative method to update the best consensus of the task and the expert's quality level parameters in a human-machine collaboration process until convergence is reached. The input of the iterative process is the expert set (including the expert's performance level feature vector S w , credibility feature vector U w ) and a collection of answers provided by experts Task set {t}1 T (including knowledge category feature vector C t ) and the answer set for each task The output is the updated expert's quality level (i.e., performance level feature vector and credibility feature vector) and the best consensus of the task answer. The iteration process mainly includes two steps: in step 1, the best consensus for each task is inferred based on the expert quality level; in step 2, the quality level of each expert is reversely inferred based on the best consensus of the task. Iterations are continued until convergence, and the convergence condition can be that the probability of the best consensus is greater than the preset value, or the maximum number of iterations is reached.

[0155] After obtaining the best consensus, the answers provided by the experts are updated to the knowledge graph, thus completing the refinement of the knowledge graph.

[0156] Then in step S5, based on the type of task and all answers, the final answer of the task is determined, which specifically includes:

[0157] (1) Based on the type of task, all answers, and the performance level feature vector and credibility feature vector of each expert among multiple experts who completed the task, calculate the intermediate answer of the task.

[0158] (2) Determine whether the iteration termination condition is met.

[0159] (3) If yes, then the intermediate answer to the task is taken as the final answer to the task.

[0160] (4) If not, then the performance level feature vector and credibility feature vector of each expert among the multiple experts who have completed the task are updated, and the updated performance level feature vector and credibility feature vector are used as the performance level feature vector and credibility feature vector of the next iteration, and the step of "calculating the intermediate answer of the task based on the type of task, all answers, and the performance level feature vector and credibility feature vector of each expert among the multiple experts who have completed the task" is returned.

[0161] When the task type is a single-choice task, the calculation formula for the intermediate answer is the above formula (19); when the task type is a multiple-choice task, the method for determining the intermediate answer is: for each candidate answer, calculate the answer of the candidate answer, and determine the probability of the answer of the candidate answer, and the answers of the candidate answers with probabilities greater than the probability threshold form the intermediate answer, and the calculation formula for the answers of the candidate answers is the above formula (20); when the task type is a multiple-choice task, the calculation formula for the intermediate answer is the above formula (21).

[0162] The advantages of the knowledge graph refinement method of this embodiment are: compared with the algorithm model, the human-computer collaboration method can complete a large number of knowledge graph refinement tasks with less computational effort, and the human-computer collaboration method can take into account multiple goals of knowledge graph refinement, such as entity completion, relationship completion, etc.; compared with relying solely on manual methods, human-computer collaboration can achieve knowledge graph refinement at a lower cost. This method not only ensures the retention of key information in the knowledge graph refinement task in the engineering design field where there are fewer research experts, but also greatly reduces the workload of experts and optimizes the expert-task allocation scheme, so that the knowledge graph can be well applied in highly professional fields. Compared with the general knowledge graph construction, this embodiment proposes a task generation method based on the vectorized representation of the knowledge graph; constructs a detailed model of tasks and experts; and proposes an answer aggregation method based on the best consensus discovery, so as to achieve the refinement of the knowledge graph. The advantages are: ① Based on the TransE model, the knowledge graph is vectorized, and then the task concept and type division are defined. The task generation strategy is designed based on the vectorized knowledge graph, which has the advantage of generating as few tasks as possible to cover as many problem triples in the subgraph as possible. ② By constructing detailed models of tasks and experts, a benefit matrix that meets the two goals of short time consumption and high matching between the capabilities required by the task and the capabilities possessed by the experts can be derived. Based on this, a two-objective task allocation model is established, and the optimal allocation plan under limited conditions is obtained to achieve full and reasonable utilization of the capabilities of domain experts. ③ Based on the concept of the proposed best consensus, information on expert capabilities and preferences and the probability information of the answer itself are comprehensively integrated into the calculation, and an iterative method is designed to update the expert's capabilities, so as to fully utilize sparse data resources and improve accuracy. From task generation to answer aggregation, the entire process achieves full utilization of sparse data on the basis of achieving a balance between the number of tasks and the coverage of questions, and improves the accuracy of the final results, laying a good foundation for the further development of knowledge graphs in highly professional fields in the future.

[0163] The embodiment of the present application also provides an application scenario, which applies the above-mentioned knowledge graph refinement method based on human-computer collaboration. Specifically, the knowledge graph refinement method based on human-computer collaboration provided in this embodiment can be applied in the knowledge graph refinement scenario. The knowledge graph refinement scenario includes an acquisition link, a refinement link, and a display link. The acquisition link is used to obtain the knowledge graph to be refined, the refinement link is used to refine the knowledge graph to be refined to obtain the refined knowledge graph, and the display link is used to display the refined knowledge graph to the user. The knowledge graph refinement method based on human-computer collaboration provided in this embodiment belongs to the refinement link.

[0164] Example 2

[0165] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the knowledge graph to be refined. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a knowledge graph refinement method based on human-computer collaboration is implemented.

[0166] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge graph refinement method based on human-computer collaboration described in Example 1.

[0168] Example 3

[0169] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the knowledge graph refinement method based on human-computer collaboration described in Example 1 is implemented.

[0170] Example 4

[0171] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the knowledge graph refinement method based on human-computer collaboration described in Example 1.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A knowledge graph refinement method based on human-computer collaboration, characterized in that: The knowledge graph refinement method based on human-computer collaboration includes: Each triple in the refined knowledge graph is vectorized to obtain a vectorized triple, and all the vectorized triples are combined into a vectorized knowledge graph; the elements of the triple include a head entity, a tail entity, and a relationship between the head entity and the tail entity; Calculating a score for each of the vectorized triples, and determining uncertain triples that may need to be refined based on the scores of all the vectorized triples; An uncertain triple with missing elements is selected as a task. For each of the tasks, if the missing element of the task has only one candidate answer, the type of the task is a single-choice task; if the missing element of the task has multiple candidate answers, the type of the task is a multiple-choice task; if the missing element of the task has no candidate answer, the type of the task is a fill-in-the-blank task, so as to determine the type of the task; the candidate answer is the value of the missing element in the candidate vectorized triple in the vectorized knowledge graph, and the candidate vectorized triple is the same as the vectorized triple of the non-missing element of the task; With the goal of minimizing time cost and maximizing competence, the tasks and experts are matched to determine multiple experts who can complete each of the tasks; For each of the tasks, the answer of each expert among the multiple experts who completed the task is obtained, and based on the type of the task and all the answers, the final answer of the task is determined, and the task is completed based on the final answer to complete the refinement of the knowledge graph to be refined.

2. The knowledge graph refinement method based on human-computer collaboration according to claim 1 is characterized in that: Performing vectorization processing on each triple in the refined knowledge graph to obtain a vectorized triple, specifically including: using the trained TransE model to perform vectorization processing on each triple in the refined knowledge graph to obtain a vectorized triple; Among them, during the training process of the trained TransE model, the training sample set is generated in the following manner: obtain multiple correct triples in the sample knowledge graph; randomly select several of the correct triples as triples to be replaced; for each of the triples to be replaced, replace the head entity or the tail entity of the triple to be replaced to obtain an incorrect triple; all the correct triples and all the incorrect triples form a training sample set; the correct triple is a triple with correct elements; the training sample set is used to train the initial TransE model to obtain a trained TransE model; When replacing the head entity of the triple to be replaced, the replacement value used is p, and the calculation formula of p is: Among them, tph is the average number of tail entities corresponding to each head entity in the sample knowledge graph; hpt is the average number of head entities corresponding to each tail entity in the sample knowledge graph; When replacing the tail entity of the triple to be replaced, the replacement value used is 1-p.

3. The knowledge graph refinement method based on human-computer collaboration according to claim 1 is characterized in that: Calculating the score of each of the vectorized triples, specifically comprising: calculating the score of each of the vectorized triples using a scoring formula; The scoring formula is: Where f(n1, r, n2) is the score of the vectorized triple, n1 is the head entity in the vectorized triple, r is the relationship in the vectorized triple, and n2 is the tail entity in the vectorized triple; p(n1, r, n2) is the probability that the relationship r exists between the head entity n1 and the tail entity n2; k i is the binary coefficient that indicates whether the i-th element exists in the relationship set or entity set; θ i is the influence coefficient of the i-th element; p(n1,r,n2)=a1·p(n1,r)+a2·p(n2,r)+a3·p(n1,n2); Among them, a1, a2, and a3 are all constants. p(n1, r) is the probability that the head entity n1 and the relation r exist at the same time; p(n2, r) is the probability that the tail entity n2 and the relation r exist at the same time; p(n1, n2) is the probability that the head entity n1 and the tail entity n2 exist at the same time.

4. The knowledge graph refinement method based on human-computer collaboration according to claim 1 is characterized in that: Based on the scores of all the vectorized triples, determining uncertain triples that may need to be refined includes: For each of the vectorized triples, if the score of the vectorized triple is greater than or equal to a preset value and less than or equal to 1, the vectorized triple is determined to be an uncertain triple that may need to be refined.

5. The knowledge graph refinement method based on human-computer collaboration according to claim 1 is characterized in that: The tasks and experts are matched with the goal of minimizing time cost and maximizing ability matching, and a plurality of experts who can complete each of the tasks are determined, specifically comprising: establishing a matching model with the goal of minimizing time cost and maximizing ability matching; solving the matching model to match the tasks and experts, and a plurality of experts who can complete each of the tasks are determined; The matching model is: Among them, Z is the objective function value; m is the total number of experts; n is the total number of tasks; r ij is the composite relative membership for assigning the jth task to the ith expert; ij is a binary variable indicating the assignment of the jth task to the ith expert; t is a constant; k is a constant; Among them, w l is the weight of the lth target; is the fuzzy relative membership of the lth target; When the goal is to minimize time cost, Among them, c lmax is the maximum value of the target attribute of the lth goal when the jth task is assigned to the i-th expert; is the target attribute value of the lth goal when assigning the jth task to the ith expert; c lmin is the minimum value of the target attribute of the lth goal when the jth task is assigned to the i-th expert; Among them, RV i is the response speed feature vector of the i-th expert; C j is the knowledge category feature vector of the jth task; When the goal is the highest level of ability fit, Among them, S i is the performance level feature vector of the i-th expert; C j is the knowledge category feature vector of the jth task.

6. The knowledge graph refinement method based on human-computer collaboration according to claim 5 is characterized in that: The knowledge category feature vector is: Among them, C t is the knowledge category feature vector of task t; is the characteristic value of task t in the first knowledge category; is the characteristic value of task t in the second knowledge category; is the characteristic value of task t in the i-th knowledge category; is the feature value of task t in the Kth knowledge category; The characteristic value of task t in the i-th knowledge category is the cosine value of the vector of the i-th knowledge category and the vector of task t in the i-th knowledge category; wherein, the vector of the i-th knowledge category is generated as follows: extract the characteristic keywords of task t in the i-th knowledge category and the weight of each of the characteristic keywords; select some of the characteristic keywords, and arrange the weights of the selected some of the characteristic keywords in descending order to obtain the vector of the i-th knowledge category.

7. The knowledge graph refinement method based on human-computer collaboration according to claim 1 is characterized in that: Based on the type of the task and all the answers, determine the final answer to the task, specifically including: Calculate an intermediate answer to the task based on the type of the task, all the answers, and a performance level feature vector and a credibility feature vector of each of the multiple experts who complete the task; Determine whether the iteration termination condition is reached; If yes, the intermediate answer to the task is used as the final answer to the task; If not, then the performance level feature vector and the credibility feature vector of each expert among the multiple experts who have completed the task are updated, and the updated performance level feature vector and the updated credibility feature vector are used as the performance level feature vector and the credibility feature vector of the next iteration, and the step of "calculating the intermediate answer of the task based on the type of the task, all the answers, and the performance level feature vector and the credibility feature vector of each expert among the multiple experts who have completed the task" is returned; When the task type is a single-choice task, the calculation formula for the intermediate answer is: Among them, c t is the intermediate answer to task t; The probability of choosing the first answer for task t in the kth knowledge category; is the probability of selecting the second answer for task t in the kth knowledge category; K is the total number of knowledge categories; is the i-th answer to task t under probability distribution φ1 is the probability of the first answer; is the answer a given by expert w to task t tw is the total number of the first type of answers; is the characteristic value of task t in the kth knowledge category; is the characteristic value of the performance level of expert w in the kth knowledge category; is the credibility characteristic value of expert w in the kth knowledge category; is the i-th answer to task t under probability distribution φ1 is the probability of the second answer; is the answer a given by expert w to task t tw is the total number of the second type of answers; When the task type is a multiple-choice task, the intermediate answer is determined as follows: for each candidate answer, the answer of the candidate answer is calculated, and the probability of the answer of the candidate answer is determined, and the answers of the candidate answers whose probability is greater than the probability threshold are combined into the intermediate answer; the calculation formula of the answer of the candidate answer is: in, is the answer to the rth candidate answer for task t; is the i-th answer to task t under probability distribution φ2 is the probability of the first answer; is the i-th answer to task t under probability distribution φ2 is the probability of the second answer; When the task type is a multiple-choice task, the calculation formula for the intermediate answer is: in, is the probability of selecting the i-th answer for task t in the k-th knowledge category; K is the total number of knowledge categories; n is the total number of answer types; is the jth answer to task t under probability distribution φ3 is the probability of occurrence of the i-th answer; is the answer a given by expert w to task t tw is the total number of the i-th answer.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge graph refinement method based on human-computer collaboration as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the knowledge graph refinement method based on human-computer collaboration as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the knowledge graph refinement method based on human-computer collaboration as described in any one of claims 1 to 7.

Images