A system evaluation method based on knowledge graph and network analysis

Abstract

The present invention discloses a system evaluation method based on knowledge graph and network analysis method, comprising the following steps: establishing a knowledge graph according to text data related to the evaluation target; determining the control layer of the network analysis method, and then according to the control layer criteria, using the knowledge graph query and keyword vector, retrieving the specific indicators controlled by the criteria, constructing the network layer, and modifying or supplementing it by experts; for the expert group to give the relative importance scale between the indicators, the scale is corrected according to the similarity between the indicators and the criteria, a judgment matrix is ​​constructed, the limit relative ranking vector of the network analysis method is calculated, and the weight of each indicator is determined. The present invention improves the efficiency of the evaluation process; at the same time, the relative importance scale between the indicators is corrected based on the similarity, which increases the objectivity of the evaluation decision-making process.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a system evaluation method based on knowledge graph and network analysis. Background Art

[0002] The concept of knowledge graph was first proposed by Google in 2012 and was initially used to improve the quality of its search engine. Knowledge graph is essentially a large-scale semantic network with rich nodes and relationships. The nodes that make up the knowledge graph represent entities or concepts, and the edges represent various relationships between entities or concepts. The technical process of knowledge graph is generally as follows: first determine the knowledge representation model, then select asynchronous knowledge acquisition methods to import knowledge based on the data source, then comprehensively use knowledge reasoning, knowledge fusion, knowledge mining and other technologies to improve the quality of the constructed knowledge graph, and finally design different knowledge access or presentation methods according to needs, such as semantic search, question-answering interaction, graph visualization analysis, etc. As an application-oriented technology, knowledge graph has specific applications in many industries, such as information retrieval, question-answering systems, recommendation systems, e-commerce, education and medical care, industrial production, etc.

[0003] The Analytical Network Process (ANP) is a decision-making method proposed by Satty that is adapted to complex structures. It is a practical decision-making method developed on the basis of the Analytical Hierarchy Process (AHP). The Analytical Network Process divides system factors into two parts: the control factor layer and the network layer. The control factor layer includes objectives and control criteria. The weight of each control criterion can be obtained by the Analytical Hierarchy Process. The network layer consists of all element groups controlled by the control layer. The element groups are network structures that influence each other. Each element group has elements with the same properties. The network layer uses a supermatrix to conduct a comprehensive analysis of the various factors that influence each other to obtain mixed weights. Compared with the Analytical Hierarchy Process, the Analytical Network Process takes into account the cross-effects of factors at different levels and at the same level, and the system structure constructed by the Analytical Network Process does not require a strict hierarchical relationship like the Analytical Hierarchy Process. The Analytical Network Process is more in line with the actual situation of decision-making problems, has a wider application scenario than the Analytical Hierarchy Process, and can solve more complex practical problems.

[0004] At present, most of the research on system evaluation methods tends to be combined with specific applications. The theoretical basis is often the improvement of a single system evaluation method and the integration of multiple existing system evaluation methods, and research and development have encountered bottlenecks. Traditional system evaluation methods often rely too much on the opinions of experts, which leads to an incomplete indicator system and difficulty in reflecting the potential connection between indicators; the allocation of weights of various factors is highly subjective; the entire system evaluation process is long and inefficient; and the existing data cannot be used quickly and fully. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a system evaluation method based on knowledge graph and network analysis method, which can realize the innovative combination of system evaluation theory and artificial intelligence, make full use of existing data as a supplement to expert knowledge, improve the efficiency and intelligence of the evaluation process, and reduce the subjectivity in the evaluation decision-making process.

[0006] In order to solve the above technical problems, the present invention provides a system evaluation method based on knowledge graph and network analysis method, comprising the following steps:

[0007] S1. According to the evaluation objectives, collect the control layer criteria and related text data initially set by the expert group, embed the data into word vector library; finally build the text data into knowledge graph through text classification, entity recognition and relationship extraction technology, and the knowledge graph is used as an indicator library for the network layer in the network analysis method;

[0008] S2. For the initial control layer of the network analysis method constructed by the expert group, according to the criteria of the control layer, based on the knowledge graph query and keyword word vector information, search for relevant nodes in the knowledge graph as indicators to establish the network layer, and submit it to the expert group for modification or supplementation;

[0009] S3. For the relative importance scales between the indicators given by the expert group, according to the similarities between the indicators, the relative importance scales between the indicators are modified, and the supermatrix, weighted supermatrix and extreme relative ranking vector of the network analysis method are calculated to determine the weights of each indicator.

[0010] Preferably, in step S1, when word embedding is performed on the text data, if the control layer criterion initially given by the expert in step S1 does not exist in the text data, the expert is required to give a supplementary description of the criterion and put it into the text data for word embedding to obtain the word vector of the control layer criterion. It is necessary to set a custom dictionary according to the text data to ensure that important proprietary words are not destroyed, and set a disabled dictionary to filter useless words;

[0011] After building the knowledge graph, if there is a node entity that is a long text message rather than a single word, it is necessary to extract keywords from the node to prepare for the query according to the control layer criteria in step S2 and the correction of the relative importance scale between indicators in step S3. The algorithm for extracting keywords is to calculate the TF-INF value of each word in the node:

[0012] TF-INF=TF*INF

[0013] Among them, TF is defined as the frequency of a word appearing in the node, and INF is defined as the inverse node frequency;

[0014] Then sort in descending order, taking the highest ranked word or words;

[0015]

[0016] Among them, N represents the total number of nodes in the knowledge graph, and m represents the number of nodes containing the word.

[0017] Preferably, in step S2, if the control layer criterion has the same node in the knowledge graph, that is, the criterion has a direct relationship with the network layer, it can be regarded as a "connection" criterion in the network analysis method. When querying, only the adjacent nodes with specific labels need to be returned as candidate related indicators of the criterion;

[0018] If the control layer criterion has no identical node in the knowledge graph, that is, the criterion has no direct relationship with the network layer, it can be regarded as an "inducing" criterion in the network analysis method. Before querying, you need to find the node with the highest similarity to the criterion:

[0019] Define criteria and similarity between nodes The similarity of its keyword vector is calculated as follows:

[0020]

[0021] in, Denotes the control layer criterion P S The word vector of Represents the word vector of the i-th keyword of node Node;

[0022] Calculate the correlation between each node and the criterion If there are multiple nodes with similarity to the criterion In the same situation, compare This process is repeated until the most similar nodes or keywords are selected and compared. Finally, the node is found in the knowledge graph through query and its adjacent nodes with specific labels are returned as candidate indicators.

[0023] When constructing the network layer using the indicators searched in the above two cases, their original relationships in the knowledge graph are retained.

[0024] Preferably, in step S3, the network analysis method structure has been constructed, assuming that the control layer criterion P S Control element group C i and C j , with element group C j Element C jk As the secondary criterion, compare the element group C i The element c in il and c im The relative importance of the elements needs to be modified according to the similarity of the elements. Define the element c il and c imWith subcriterion c jk The similarity is:

[0025]

[0026] in, For element c jk 、c il 、c im The highest ranked keyword word vector.

[0027] Similarly, we can also directly use P S As a criterion, element group C i The element c in il and c im the relative importance of

[0028] The relative importance scale uses a 1-9 scale method. Assuming that the expert group gives jk Under the second criterion, c il C im The relative importance scale is a il_im , (a il_im =1,2,…,9), then after correction, the final relative importance scale is expressed as:

[0029]

[0030] Its specific meaning is that if the element c il Than element c im In an objective sense, the meaning is closer to sub-criterion c jk , that is, c il With c jk The similarity is greater than c im With c jk If the similarity is high, then add a scale to the relative importance scale given by the expert, otherwise, subtract a scale, so as to make an objective correction to the relative importance scale given by the expert group, and the correction will not make the relative importance scale exceed the upper and lower limits.

[0031] The beneficial effects of the present invention are as follows: (1) The present invention successfully combines knowledge graph-related technologies with system evaluation methods, provides a new research direction, breaks the thinking limitations of traditional system evaluation methods, makes the entire system evaluation process more efficient and intelligent, and is also in line with the current era of big data, with good application prospects; (2) The present invention uses knowledge graph technology to improve the construction process of the evaluation index system in the traditional network analysis method. Considering the consistency of the network layer structure in the knowledge graph and the network analysis method, the text knowledge and data related to the evaluation target are used to establish a knowledge graph as an evaluation index library, so that the use of knowledge graph query and other technologies can be realized to find relevant indicators in the library and establish a network layer. This method can be used as an effective supplement to experts in establishing an index system. (3) The present invention uses word embedding in natural language processing technology to correct the relative importance scale in network analysis. In the process of using the system evaluation method to establish a knowledge graph, the text will be embedded and the vocabulary will be mapped to a real vector. This word representation method takes into account the text context information, so that the cosine similarity between word vectors can reflect the objective similarity of entities. When the network analysis method determines the relative importance scale between two indicators, the word vectors contained in the two indicators and their corresponding criteria are used as objective basis. By comparing the cosine similarity between the word vectors of the two indicators and the criteria, the relative importance scale given by the expert group is corrected. This relative importance scale that combines the subjective and objective increases the objectivity of the evaluation decision process. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the evaluation method of the present invention.

[0033] Figure 2 Schematic diagram of the knowledge graph established for the present invention.

[0034] Figure 3 This is a control layer diagram given by the expert group of the present invention.

[0035] Figure 4 This is a schematic diagram of the structure of the network analysis method constructed based on the knowledge graph of the present invention. DETAILED DESCRIPTION

[0036] like Figure 1 As shown in FIG, a system evaluation method based on knowledge graph and network analysis method includes the following steps:

[0037] S1. According to the evaluation objectives, collect the control layer criteria and related text data initially set by the expert group, embed the data into word vector library; finally build the text data into knowledge graph through text classification, entity recognition and relationship extraction technology, and the knowledge graph is used as an indicator library for the network layer in the network analysis method;

[0038] S2. For the initial control layer of the network analysis method constructed by the expert group, according to the criteria of the control layer, based on the knowledge graph query and keyword word vector information, search for relevant nodes in the knowledge graph as indicators to establish the network layer, and submit it to the expert group for modification or supplementation;

[0039] S3. For the relative importance scales between the indicators given by the expert group, according to the similarities between the indicators, the relative importance scales between the indicators are modified, and the supermatrix, weighted supermatrix and extreme relative ranking vector of the network analysis method are calculated to determine the weights of each indicator.

[0040] In this embodiment, the evaluation goal is to determine the fault phenomenon that has the greatest impact on the machine tool. The relevant steps of constructing and querying the knowledge graph are performed in the pycharm software and the neo4j graph database, and the relevant steps of calculation in the network analysis method are performed in the yaanp software.

[0041] The specific steps of implementing the technical solution of the present invention into this embodiment are as follows:

[0042] Step 1: According to the evaluation objectives, collect relevant text data and embed the data into word vector libraries; through text classification, entity recognition, relationship extraction and other technologies, finally build the text data into a knowledge graph.

[0043] This example selects the PDF document "500 Examples of Practical Fault Diagnosis and Repair Technology for CNC Machine Tools" as the main text data source;

[0044] Use the jieba Chinese word segmentation component, set a custom dictionary and a stop dictionary to segment text data, and use the word2vec algorithm to obtain the word vector library;

[0045] Combined with the evaluation objectives, two triple forms of (entity, relationship, entity) in the document are summarized, namely: (fault phenomenon, related, fault phenomenon) and (fault phenomenon, fault location, specific fault location), which are used as templates for classification through the CNN model;

[0046] After being classified according to the template, the triples are classified and stored, and then imported into the neo4j graph database to form a knowledge graph. The established knowledge graph is partially displayed in the neo4j interface as follows: Figure 2 shown.

[0047] Step 2: The expert group constructs the control layer of the network analysis method. According to the criteria of the control layer, based on the knowledge graph query and keyword word vector information, the relevant nodes in the knowledge graph are searched as indicators to establish the network layer. Finally, the expert group reviews, modifies and supplements it.

[0048] For the sake of convenience, only three criteria are listed here: "Power", "System", and "Display". The meanings of these three criteria are the relationship between (fault phenomenon, fault location, and specific fault location). There are corresponding nodes and relationships in the knowledge graph. Therefore, when querying, only the adjacent nodes need to be returned as candidate related indicators of the criterion. The query language is Cypher. Taking the criterion "Power" as an example, the query instruction is: MATCH(a:TroubleLocation{title:"Power"})<-->(b:Phenomenon)return b. The returned nodes are as follows: Figure 3 shown.

[0049] Finally, the nodes returned by the above criteria are summed up, and after review, modification and supplementation by the expert group, the final network analysis method structure diagram is as follows Figure 4 For the purpose of explanation and presentation, only three elements are listed in each criterion-controlled part, and subsequent calculations are also based on the listed parts.

[0050] The specific names of the elements in the network layer are shown in Table 1.

[0051] Table 1 Specific names of network layer elements

[0052] C11 The power cannot be turned on. C12 The CNC power supply cannot be turned on when starting up C13 The servo power cannot be turned on C21 No display when booting the system C22 The system suddenly loses power C23 System freeze C31 No display on the monitor C32 The display is unstable C33 The display suddenly turns into a horizontal bright line

[0053] Step 3: The expert group gives the relative importance scale of each indicator, and modifies the relative importance scale according to the similarity between the indicators. The supermatrix, weighted supermatrix and extreme relative ranking vector of the network analysis method are calculated to determine the weight of each indicator.

[0054] According to the similarity between the defined indicators, the relative importance scales given by the expert group between the elements in the element groups under the control of the three criteria of "power supply", "system" and "display" are respectively corrected. Taking the elements C31 and C32 under the criterion "display" as examples, the similarity between C31 and the criterion is calculated to be 0.632 and the similarity between C32 and the criterion is 0.535 according to the element word vector and the formula. The expert group gives the relative importance scale of C31 to C32 as 4, and the relative importance scale corrected according to the formula is 5. After determining the relative importance scales of each indicator in this way, the judgment matrix is ​​constructed, and the limit relative ranking vector is calculated according to the steps of the network analysis method. Finally, the weight of each fault phenomenon to the evaluation target is determined as shown in Table 2.

[0055] Table 2 Final weight of each element in the network layer on the evaluation target

[0056] index Final weight of evaluation target C11 0.205 C12 0.205 C13 0.205 C21 0.049 C22 0.216 C23 0.027 C31 0.054 C32 0.010 C33 0.029

Claims

1. A system evaluation method based on knowledge graph and network analysis method, characterized in that: The steps include: S1. According to the evaluation objectives, collect the control layer criteria and related text data initially set by the expert group, embed the data into word vector library; finally build the text data into knowledge graph through text classification, entity recognition and relationship extraction technology, and the knowledge graph is used as an indicator library for the network layer in the network analysis method; S2. For the initial control layer of the network analysis method constructed by the expert group, according to the criteria of the control layer, based on the knowledge graph query and keyword word vector information, search for relevant nodes in the knowledge graph as indicators to establish the network layer, and submit it to the expert group for modification or supplementation; if the control layer criteria have the same nodes in the knowledge graph, that is, the criteria have a direct relationship with the network layer, it is regarded as the "connection" criteria in the network analysis method, and when querying, only the adjacent nodes with specific labels need to be returned as candidate related indicators of the criteria; If the control layer criterion has no identical node in the knowledge graph, that is, the criterion has no direct relationship with the network layer, it is regarded as an "inducing" criterion in the network analysis method. Before querying, you need to find the node with the highest similarity to the criterion: Define criteria and similarity between nodes The cosine similarity of its keyword vector is calculated as follows: in, Denotes the control layer criterion P S The word vector of Represents the word vector of the i-th keyword of node Node; Calculate the correlation between each node and the criterion If there are multiple nodes with similarity to the criterion In the same situation, compare This process is repeated until the most similar nodes or keywords are selected and compared. Finally, the node is found in the knowledge graph through query and its adjacent nodes with specific labels are returned as candidate indicators. When constructing the network layer using the indicators searched in the above two cases, the original relationship between them in the knowledge graph is retained; S3. For the relative importance scales of the indicators given by the expert group, according to the similarity between the indicators, the relative importance scales of the indicators are modified, the supermatrix, weighted supermatrix and limit relative ranking vector of the network analysis method are calculated, and the weights of the indicators are determined; the network analysis method structure has been constructed, assuming that the control layer criterion P S Control element group C i and C j , with element group C j Element C jk As the secondary criterion, compare the element group C i The element c in il and c im The relative importance of the elements needs to be modified according to the similarity of the elements. Define the element c il and c im With subcriterion c jk The similarity is: in, For element c jk 、c il 、c im The highest ranked keyword vector; Similarly, it is more direct to use P S As a criterion, element group C i The element c in il and c im the relative importance of The relative importance scale uses a 1-9 scale method. Assuming that the expert group gives jk Under the second criterion, c il C im The relative importance scale is a il_im , (a il_im =1,2,…,9), then after correction, the final relative importance scale is expressed as: Its specific meaning is that if the element c il Than element c im In an objective sense, the meaning is closer to sub-criterion c jk , that is, c il With c jk The similarity is greater than c im With c jk If the similarity is high, then add a scale to the relative importance scale given by the expert, otherwise, subtract a scale, so as to make an objective correction to the relative importance scale given by the expert group, and the correction will not make the relative importance scale exceed the upper and lower limits.

2. The system evaluation method based on knowledge graph and network analysis method according to claim 1, characterized in that: In step S1, when embedding text data, if the control layer criteria initially given by the expert in step S1 does not exist in the text data, the expert needs to give a supplementary description of the criteria and put it into the text data for word embedding to obtain the word vector of the control layer criteria. It is necessary to set a custom dictionary based on the text data to ensure that important proprietary words are not destroyed, and set a stop dictionary to filter useless words; After building the knowledge graph, if there is a node entity that is a long text message rather than a single word, it is necessary to extract keywords from the node to prepare for the query according to the control layer criteria in step S2 and the correction of the relative importance scale between indicators in step S3. The algorithm for extracting keywords is to calculate the TF-INF value of each word in the node: TF-INF=TF*INF Among them, TF is defined as the frequency of a word appearing in the node, and INF is defined as the inverse node frequency; then sort in descending order, and take the highest ranked word or several higher ranked words; Among them, N represents the total number of nodes in the knowledge graph, and m represents the number of nodes containing the word.

Images