E-government project evaluation expert group recommendation method based on multi-dimensional feature balance

Abstract

E-government projects have more administrative management connotations, which makes the knowledge structure of the e-government project evaluation expert group play a very important role in project management. Based on cognitive science theory and psychology theory, the differences in various characteristics of individual experts are the root causes of cognitive differences and emotional polarity differences. As a group decision-making method that balances such differences, the question of whether the expert group is more effective should actually be based on the composition method of the expert group with multiple characteristics balanced. Based on this, characterizing expert characteristics and realizing the expert group recommendation method based on characteristic balance has important basic value for the consistency of e-government project evaluation and improving the overall level of project evaluation. This achievement is used in the expert extraction stage before project evaluation, which can greatly improve the conformity between the expert group and the evaluated project, the balance of multi-dimensional characteristics, and thus improve the consistency level of project evaluation results.

Description

1. Technical field

[0001] Computer Science and Technology (Artificial Intelligence, Knowledge Engineering), Science and Technology Management and Information Science, Knowledge Metrics, Knowledge Management, Text Processing Technology 2. Background technology

[0002] (1) Science and Technology Management and Information Science

[0003] The study of science of science pays more attention to the consistency between the expert's research expertise and the project being evaluated, the expert's experience and authority, etc., with the aim of achieving peer evaluation as much as possible and solving the cognitive differences caused by complex knowledge boundaries. The study of information science focuses on the method of discovering experts with specific knowledge types, which is one of the methodologies of the study of science of science.

[0004] In science and technology management, meta-evaluation theory provides a method to understand the quality deviation of expert evaluation. Meta-evaluation indicators mainly include objective indicators such as coefficient of deviation and coefficient of variation based on scoring data.

[0005] (2) Knowledge metrics

[0006] Knowledge metrics is the basic theory of expert expertise identification. Knowledge metrics defines knowledge elements as the basic unit of knowledge measurement. Expert opinions, as a knowledge unit, are analogies composed of different knowledge elements in different structures. Scientific and technological literature is actually different knowledge units, and knowledge elements can be obtained from scientific and technological literature through text mining technology.

[0007] On the basis of the definition of expert characteristics oriented by evaluation goals, the expert opinions accumulated in the process of e-government project review are used as the main data source of meta-evaluation. The basic methods of knowledge mining and knowledge measurement are used to characterize the multidimensional characteristics of experts, and then the expert group recommendation method based on the balance of multidimensional characteristics is completed, which is problem-oriented and reasonable.

[0008] (3) Knowledge Management

[0009] The project evaluation score sheet and expert opinion short text with time continuity of e-government project review experts are used as data sources to conduct text mining on expert opinions and complete knowledge acquisition. The comprehensive score of the expert group formed at the same time and the expert group opinion text are used as analogy to evaluate the level of experts, etc. It includes the following contents:

[0010] ① Establish a project knowledge concept tree: The project knowledge concept tree is actually a hierarchical knowledge representation method with standard concepts for evaluation objectives. The role of the project knowledge concept tree here is to support hierarchical semantic recognition of expert review opinion entities. It can support the structured processing of expert opinion texts by dependency syntactic analysis based on the SAO structure, and realize SAO structure semantic mapping based on the concept tree, thereby realizing semantic recognition of entities in the expert opinion texts at different conceptual levels.

[0011] ② Establish e-government project knowledge ontology: Project knowledge ontology can fully represent project background knowledge and is also the semantic specification of knowledge fusion. The concepts and relationships between concepts of e-government project knowledge ontology come from the relevant standards and regulations of e-government project management.

[0012] ③ Opinion mining technology: Opinion mining is an effective means of acquiring knowledge from expert opinion texts. The difficulty of short text opinion mining lies in the fact that the expression of opinion elements in short texts is diverse or non-explicit, which is even more difficult to understand in the Chinese context. Automated short text understanding requires additional knowledge that can help the machine fully explore the connection between words in short texts, such as semantic relevance.

[0013] ④ Knowledge aggregation technology: From the perspective of knowledge management, the knowledge structure of the expert group can be characterized by the aggregation of the knowledge structures of multiple experts in the group. The emotional polarity and evaluation experience of the expert group can be presented through scientific measurement, and then the multi-dimensional characteristics of the expert group can be calculated.

[0014] (4) Text processing technology

[0015] Conventional text processing technology: including the compilation of the project evaluation index scoring table, the use of the concept word library of the e-government project standards and management specifications to train the project, supplemented by the basic word library consisting of synonyms and near synonyms. The open source tool Hanlp is used to segment the expert opinion text into sentences, words, part-of-speech tags, remove meaningless stop words, use dependency syntax analysis to identify the sentence SAO (subject-verb-object) structure, split the opinion sentences with multiple subjects or multiple objects, and refine the complex sentences containing multiple evaluation objects into multiple simple sentences with SAO structures containing only unique evaluation objects.

[0016] (5) Sentiment Analysis Technology

[0017] A deep learning model is used to judge the sentiment tendency of expert opinion sentences, and four classifiers, RNN, LSTM, BIRNN, and BILSTM, are used to compare sentiment classification to obtain the optimal opinion set with semantic determination and sentiment tendency determination. 3. Summary of the invention

[0018] This achievement adopts the definition of expert characteristics oriented to the evaluation goal, takes the expert opinions accumulated in the e-government project review process as the main meta-evaluation content, and uses the basic methods of knowledge mining and knowledge measurement to achieve targeted multi-dimensional feature characterization of experts, and then completes the expert group recommendation based on multi-dimensional feature balance. The specific steps and methods are described as follows:

[0019] (1) Knowledge Supplementation and Knowledge Representation

[0020] In order to achieve the acquisition and classification of knowledge elements in expert opinions, the project knowledge concept tree at the micro level is defined to hierarchically express the semantic hierarchy between project concept knowledge and concepts; the domain knowledge ontology is introduced at the macro level to represent project knowledge, support knowledge expansion and association, knowledge transformation to achieve implicit knowledge discovery; thereby supporting the goal of knowledge element acquisition and classification. The knowledge that constitutes the concept tree and ontology comes from national standards and relevant management policies for e-government projects.

[0021] (2) Knowledge acquisition based on hierarchical semantic recognition

[0022] Expert opinion knowledge acquisition aims to obtain the knowledge elements in expert opinions, providing a basis for expert knowledge structure characterization and sentiment polarity analysis. This strategy conducts knowledge mining and acquisition at the lexical level (word segmentation and part-of-speech tagging), syntactic level (named entity recognition and grammatical analysis), and semantic level (semantic recognition).

[0023] In view of the short text features of the review opinions, SAO (subject-action-object) extraction of expert opinions is performed on the basis of grammatical analysis to achieve the acquisition of structured knowledge, and then the concept tree is used to support hierarchical semantic recognition, the evaluation features of expert opinions are mined, and the sentiment classifier is constructed using the deep learning model to obtain the sentiment polarity. This part specifically includes the following contents:

[0024] ① Named Entity Recognition

[0025] By integrating the four named entity recognition methods of HMM, CRF, BILSTM, and BILSTM-CRF through the voting method, we can obtain better results in the extraction of government function entities. Here, we use the expert review opinion text as data, first use the open source tool Hanlp to preliminarily segment the expert opinion text into sentences, words, part-of-speech tags, and remove stop words, and then integrate its word segmentation features Xi and part-of-speech features POSi into named entity annotation to improve the effect of the integrated model. At the same time as the extraction, a domain entity vocabulary is formed.

[0026] ②Intrinsic semantic analysis and structured knowledge acquisition

[0027] Using dependency syntactic analysis, opinion sentences with multiple subjects or multiple objects are split up and fine-grained SAO structured knowledge is extracted. It is used as the basic semantic unit to represent expert opinions. Subject S and object O mainly represent the evaluated objects, which are generally nouns, gerunds, etc. The subjects or objects composed of these words or phrases are closely related to the semantics of the project concepts.

[0028] ③ Evaluation feature hierarchical semantic recognition

[0029] The review opinions after SAO structure need to perform hierarchical semantic recognition of evaluation features for the subject and object to obtain highly targeted fine-grained evaluation features. Due to the differences in concept expression among different experts, the precise matching and fuzzy matching methods based on the maximum similarity of text are used to comprehensively realize the concept semantic mapping based on the concept tree to obtain better hierarchical semantic recognition effect. The method is as follows:

[0030] 1) Exact matching method

[0031] For SAO structure text SAO i , its subject S i and object O i Respectively with the concept c in the concept set C of the concept tree j Perform a two-way maximum match and take the union of the two successfully matched concept sets to obtain the set R i If R i Not empty, compare R i The hierarchical depth of each concept in , the concept with the largest depth is taken as the semantic recognition result of the sentence; if R i If it is empty, fuzzy matching is performed.

[0032] 2) Fuzzy matching method

[0033] In the process of structuring the opinion text, in order not to destroy the integrity of the expert opinion, modifying words such as adjectives, adverbs, prepositions, conjunctions, etc. are retained. However, in the fuzzy matching using text similarity calculation, it is necessary to use word segmentation and remove modifying words through part of speech screening to improve accuracy, and retain the main word set S i '={s i1 ,s i2 , ..., s im} and O i '={o i1 , o i2 , ..., o in}(m, n are the number of words).

[0034] Therefore, concept c in concept set C j Perform word segmentation to obtain c j ={c j1 , c j2, ..., c jt}(t is the number of words). Similarity Sim(S i ',c j )、Sim(O i ',c j ) is calculated as shown in formula (1)(2).

[0035]

[0036]

[0037] Where sim(s ik , c jr ) is the cosine similarity calculation of word2vec word vectors. Through fine-grained word segmentation, the average similarity is calculated in the form of Cartesian product to obtain a more accurate result. Loop through the concept set C and calculate Sim(S i ',c j )、Sim(O i ',c j ), take the maximum similarity greater than the threshold, and its corresponding concept is the SAO i The semantics of the identified concepts.

[0038] ④ Definition of unstructured knowledge acquisition

[0039] Due to the irregularities in the language writing in short texts, the grammatical analysis of some opinions is not good, which inevitably leads to errors in the SAO structured knowledge extraction and the inability to identify the intrinsic semantic relationships in the text. The three main problems in the opinions are summarized and relevant rules are customized to relocate the knowledge, as shown in Table 1.

[0040] Table 1 Semantic analysis of non-SAO structured knowledge of expert opinions

[0041]

[0042] (3) Expert multi-dimensional feature selection and calculation

[0043] The selection of expert features has clear target constraints - knowledge constraints for e-government project evaluation. Experts evaluate projects based on their own knowledge structure, which is essentially a knowledge exchange activity between the evaluation subject and the object. As an information-based project for the construction of government functions, e-government projects have more requirements on the breadth of expert knowledge structure. Based on this, with the domain concept tree and domain knowledge ontology as reference, the four components of the expert knowledge structure are defined for knowledge classification, as shown in Table 2.

[0044] Table 2 Expert knowledge structure definition table

[0045]

[0046] Drawing on the basic theories of meta-evaluation theory and objective meta-evaluation indicators, we define knowledge level, review depth, emotional style, and domain expertise as indicators for describing the multidimensional characteristics of experts. This definition comprehensively considers the expert's knowledge and ability, the psychological factors that affect the expert's review, and the availability of relevant information. The internal logic of the expert's characteristics is shown in Table 3.

[0047] Table 3 Expert multi-dimensional feature evaluation indicators

[0048]

[0049] ① Knowledge level

[0050] According to the knowledge level definition in Table 2, the indicator measurement design is as follows:

[0051] t∈{Government knowledge, Technical knowledge, Management knowledge, Budget knowledge} (3)

[0052]

[0053] in m represents the number of experts in this review activity.

[0054] Level t is the expert's knowledge level indicator in the knowledge structure t element, through the weight α l and β l The importance of controlling scoring bias and opinion bias. n represents the number of project elements corresponding to the knowledge element. j is the scoring deviation coefficient, which means that for item factor j, the expert score x j Share equally with all experts The relative deviation of j is the opinion deviation coefficient, which indicates the relative deviation between the average sentiment intensity of individual experts’ opinions and the overall average sentiment intensity of the expert group for item element j, sk 1j ,sk 2j They represent the sentiment intensity of opinions of individual experts and expert groups on project element j, respectively. K1 and K2 are the number of opinions of individual experts and expert groups belonging to project element j, respectively.

[0055] ②Profundity of review

[0056] This paper believes that the more profound the expert's evaluation opinions are, the more they can grasp the essential knowledge of the information used in the construction of e-government projects. Therefore, this paper proposes an expert review profundity index and its measurement method based on the semantic hierarchy of the domain concept tree to characterize the depth of expert knowledge.

[0057] The expert review profundity metric captures the information described in the opinions in the following ways:

[0058] The number of characteristic vocabulary in the field of e-government involved in expert opinions is the same as the length of the opinions. If the vocabulary is large, the content of the opinions may be richer and more profound, and the breadth and depth of the expert's knowledge is greater.

[0059] If certain feature words appear more frequently, the concepts involved in the feature are clearer and more targeted.

[0060] The level, distribution path and concentration (out-degree of the node) of the characteristic words in the concept tree determine the specificity and clarity of the semantic content of the reviewed characteristics, the pertinence and prominence of the knowledge structure elements involved, or the emphasis of the knowledge structure.

[0061] Among them, RF t and RF t -count are the evaluation feature set and the number of evaluation features in the opinion that belong to a certain knowledge structure, rf-count i is the number of the i-th evaluation feature in the evaluation opinion.

[0062] Definition 1: The frequency of evaluation feature k belonging to element t of knowledge structure in expert review opinions is as shown in formula (6):

[0063] t∈{Government knowledge, Technical knowledge, Management knowledge, Budget knowledge} (6)

[0064] Definition 2: The concentration of evaluation feature c is the ratio of the number of times the hyponym c-son of evaluation feature c appears in the evaluation opinion to the number of hyponym set elements. Therefore, the concentration of evaluation feature k in expert opinion is given as shown in formula (7):

[0065]

[0066] Definition 3: Review profundity is the sum of the number of occurrences and concentrations of all evaluation feature words in the opinion in the domain feature concept tree. The calculation formula (8) is as follows:

[0067]

[0068] Set two bias weights α l , β lThe knowledge level of each expert review is 0.5, and the overall knowledge structure level of the expert is obtained by comprehensively averaging the knowledge level of each expert review. The depth of all expert review opinions is calculated through concept tree semantic hierarchy recognition to obtain the expert review depth value. For the upper and lower parent-child relationships, the weight of the child concept node is determined to be 1.2 times the weight of its corresponding parent concept node, and two depth measurement weights α are set. d , β d In order to clearly understand the breadth and depth of knowledge of the same expert, the knowledge structure distribution of the two is presented in the same radar chart.

[0069] ③Emotional style

[0070] Expert emotions are transmitted through evaluation opinions. According to the definition of emotional style in Table 2, the calculation of the total emotional intensity REmo of expert opinions is shown in formula (9), where K is the total number of opinions of the expert reviewed projects, and sk represents the emotional intensity of expert opinion k. This work visualizes the emotional style of expert review in a bar chart.

[0071]

[0072] ④ Field expertise

[0073] This achievement uses the LDA method, takes the titles and abstracts of all review projects as data sources, and conducts content theme analysis on expert reviews to reflect the project areas involved by the experts. The main steps are as follows: 1) Perform data cleaning, remove punctuation marks, numbers, and filter stop words; remove commonly used software description words to improve LDA's theme representation of government affairs, project functions, and functions (such as: modules, businesses, platforms, systems, etc.); complete the bag of words construction. 2) Use perplexity to determine the number of topics in the data set to improve the model effect, and use the LDA model to obtain a matrix containing the relationship between topics and words and between documents and topics. 3) By parsing the matrix data, obtain the distribution of words under each topic, as well as the topic of each document. Finally, count the probability of each expert's reviewed project belonging to the topic, and analyze the expert's past review content topics.

[0074] (4) Expert group recommendation method with balanced multi-dimensional features

[0075] The expert group recommendation method based on multi-dimensional feature balance is based on the measurement results of multi-dimensional features between expert groups, and its goal is to achieve consistency in evaluation between different expert groups. On the basis of random and fair extraction, the recommendation method must not only meet the requirements that the domain expertise of the candidate expert group is better aligned with the content of the project to be reviewed, but also that the breadth and depth of the candidate expert group's knowledge is not lower than the average level of the knowledge structure configuration of all experts in the expert database.

[0076] ① Field relevance assessment

[0077] The topic probability of the project to be reviewed is obtained through the LDA model, and the topic similarity is calculated with the expert field to obtain the relevance of the expert and the project to be reviewed. The larger the value, the higher the relevance between the expert and the project to be reviewed, as shown in formula (10). Indicates the project to be reviewed P a The topic feature vector, Indicates expert E i The topic feature vector The average probability of the tth topic is calculated as shown in (11), where M represents the expert E i Number of projects reviewed.

[0078]

[0079]

[0080] ②Knowledge balanced random extraction

[0081] The balance of knowledge levels among expert groups can actually be achieved through the knowledge complementarity of experts in each group. To this end, we first calculate the mean of the knowledge level and review depth of each knowledge structure in the expert database, denoted as and As the threshold, i∈{government knowledge, technical knowledge, management knowledge, budget knowledge}; secondly, the field relevance obtained in the previous stage is incorporated into the random extraction as the weight ω; finally, m (3≤m<N) experts are randomly selected from the expert database (N experts) to form a candidate expert group, and the knowledge level and evaluation depth mean of each knowledge structure of the candidate expert group are calculated respectively. If both exceed the threshold and Keep the result of this draw, otherwise re-randomly draw. 4. Description of the drawings

[0082] Figure 1 :E-government project expert review opinion form: shows the data source of this achievement;

[0083] Figure 2 :Schematic diagram of the concept tree of the e-government project domain knowledge: It shows the e-government domain concept tree (part) created by this achievement;

[0084] Figure 3 : E-government project domain knowledge ontology diagram: shows the e-government project domain knowledge ontology created by this achievement;

[0085] Figure 4 : Framework diagram for achievement realization: shows the complete process of achievement realization;

[0086] Figure 5: Expert knowledge level and review depth characterization results Figure: shows the characterization results of a single expert;

[0087] Figure 6 : Expert emotional style measurement results: The figure shows the emotional style in the opinions of multiple experts;

[0088] Figure 7 : Visualization result diagram of expert domain expertise: shows the evaluation domain experience of a single expert;

[0089] Figure 8 : The comparison chart of the results of this method and the random sampling method shows the comparison of the results of the two methods. 5. Specific implementation methods

[0090] The purpose of this achievement is to realize the e-government project expert group recommendation method based on multi-dimensional feature balance. The expert group review opinions of 214 provincial e-government projects in Tianjin from 2017 to 2018 were used as the data source, and 50 experts with long-term evaluation experience in the expert database were used to characterize the characteristics and conduct expert group recommendation experiments based on 3 and 5 experts. This method is used in the expert extraction stage before project evaluation. Through the method of this achievement, the knowledge measurement results of the expert group extraction results effectively prove that it is better than the random expert extraction method. It can greatly improve the conformity between the expert group and the evaluated project, the balance of multi-dimensional features, and thus improve the consistency level of project evaluation results.

[0091] The domain ontology of this achievement is stored in Neo4j based on the Cypher language; the experiment is based on the Python language and is implemented in a Windows environment using the pyhanlp package provided by the open source tool Hanlp, Google's open source deep learning framework Tensorflow and its advanced API - keras.

Claims

1. A project evaluation expert group recommendation method for the field of e-government, characterized in that: include: S1. Based on the definition of expert characteristics oriented by evaluation goals, the expert opinions accumulated in the process of e-government project review are used as meta-evaluation content. The multi-dimensional characteristics of experts are characterized by using knowledge mining and knowledge measurement methods, and then the expert group recommendation based on multi-dimensional feature balance is completed. In order to achieve the acquisition and classification of knowledge meta-information in expert opinions, the project knowledge concept tree at the micro level is defined to hierarchically express the semantic hierarchy between project concept knowledge and concepts; at the macro level, domain knowledge ontology is introduced to represent project knowledge, supporting knowledge expansion and association, knowledge transformation to achieve implicit knowledge discovery; thereby supporting the goal of knowledge meta-information acquisition and classification; S2. Conduct knowledge mining at the lexical, syntactic and semantic levels of the evaluation opinions of the participating experts, extract the knowledge elements in the expert opinions, and use the deep learning model to build a sentiment classifier to obtain the sentiment polarity of the evaluation opinions; S3. With the concept tree and knowledge ontology as reference, we define government affairs knowledge, technical knowledge, management knowledge, and budget knowledge as the four components of the expert knowledge structure to classify expert knowledge. We use the meta-evaluation theory and the theory of objective meta-evaluation indicators to define knowledge level, review depth, emotional style, and domain expertise as indicators for describing the multi-dimensional characteristics of experts: Knowledge Level: in m represents the number of experts in this review activity. Level t is the expert's knowledge level indicator in the knowledge structure t element, through the weight ɑ l and β l The importance of controlling scoring bias and opinion bias, n represents the number of project elements corresponding to the knowledge element, D j is the scoring deviation coefficient, which means that for item factor j, the expert score x j Share equally with all experts The relative deviation of O j is the opinion deviation coefficient, which indicates the relative deviation between the average sentiment intensity of individual experts’ opinions and the average sentiment intensity of the expert group as a whole for item element j. k1j , S k2j They represent the sentiment intensity of opinions of individual experts and expert groups on project element j, respectively. K1 and K2 represent the number of opinions of individual experts and expert groups belonging to project element j, respectively. Review profundity: Review profundity is the sum of the number of times all evaluation feature words in the opinion appear in the domain feature concept tree and the profundity reflected by their concentration. The calculation formula is as follows: α d +β d ＝1 in, is the frequency of evaluation feature k belonging to element t of knowledge structure in the expert review opinion, is the concentration of evaluation feature k in expert opinions; Emotional style: refers to the total emotional intensity REmo of expert opinions. Where K is the total number of opinions of the experts on the project, S k represents the sentiment intensity of expert opinion k; Field expertise: The LDA method was used to conduct content thematic analysis of expert reviews using all review project titles and abstracts as data sources to reflect the project areas covered by the experts; S4. The expert group recommendation is completed through two steps of domain relevance evaluation and knowledge balanced random sampling. The domain relevance evaluation method is: The topic probability of the project to be reviewed is obtained through the LDA model, and the topic similarity is calculated to obtain the relevance between the expert and the project to be reviewed. The larger the value of field relevance is, the higher the field relevance between the expert and the project to be reviewed is. The calculation formula is: in Indicates the project to be reviewed P a The topic feature vector, Indicates expert E i The topic feature vector The average probability of the tth topic, Among them, M represents the expert E i Number of projects reviewed; The knowledge balanced random extraction method is as follows: first, the knowledge level and the mean of the depth of the evaluation of each knowledge structure in the expert database are calculated, which is recorded as and And as the threshold, i∈{government knowledge, technical knowledge, management knowledge, budget knowledge}, secondly, the field relevance is incorporated into the random extraction as the weight ε, and then m experts are randomly selected from the N experts in the expert database to form a candidate expert group. The knowledge level and the mean of the depth of the evaluation of each knowledge structure of the candidate expert group are calculated respectively. If both exceed the threshold and Keep the result of this draw, otherwise re-randomly draw.

2. The method according to claim 1, characterized in that Aiming at the short text features of evaluation opinions, SA0 extraction of expert opinions is carried out on the basis of grammatical analysis to achieve the acquisition of structured knowledge. Then, the concept tree is used to support hierarchical semantic recognition and the evaluation features of expert opinions are mined. The concept semantic mapping based on concept tree is comprehensively realized by using exact matching and fuzzy matching methods based on maximum text similarity to obtain better hierarchical semantic recognition effect.

3. The method according to claim 1, characterized in that The topic probability of the project to be reviewed is obtained through the LDA model, and the topic similarity is calculated with the expert field to obtain the relevance between the expert and the project to be reviewed field. The final result is further obtained using the knowledge balanced random extraction method.

Images