Project review method for selecting experts based on pre-training statement representation model

Abstract

The invention discloses a project review method for selecting experts based on a pre-training statement representation model, and the method comprises the following steps: S1) collecting a review expert list and corresponding expert data, and building a review expert database; s2) obtaining a declaration of a to-be-reviewed project, performing form auditing, accepting the project passing the form auditing, collecting project data, and establishing a to-be-reviewed project library; s3) for each to-be-reviewed project, performing similarity matching on the project data and the expert data based on a pre-training statement representation model to obtain a selected expert candidate list set; s4) sending an invitation to the selected candidate experts according to the number of experts required by the preset project review; if the candidate experts refer to review or receive invitations, review opinions are not submitted on time; and if yes, continuing to automatically supplement from the expert candidate list set for invitation until the number of people completing project review is met. According to the method, the extraction result is more accurate and efficient, the review organization efficiency is greatly improved, and the review fairness is improved.

Description

Technical Field

[0001] The present invention relates to a project review method, and in particular to a project review method for selecting experts based on a pre-trained sentence representation model. Background Art

[0002] The most important part of the project review process is the review of the project proposal by experts. Review experts are people with rich professional knowledge and experience who play the role of reviewers in specific fields or areas. Review experts can participate in the evaluation of project applications, research plans, papers, design plans, financial statements, etc. Review experts play an important role in the review process. Their professional opinions and suggestions help improve the quality and feasibility of the project and ensure that the project can receive appropriate funding and support. Therefore, the role of review experts is crucial, and their professional ability and quality play an important role in the fairness and accuracy of the review process.

[0003] With the substantial increase in the number of applications for science and technology projects, the corresponding review tasks have become increasingly arduous and onerous. Since different fields have their own unique knowledge and skills requirements, the professional background and expertise of the review experts need to match the corresponding fields. This ensures that the review experts can have an in-depth understanding of the problems and challenges in the relevant fields and share effective opinions and suggestions related to the field. Only when the technology and areas of expertise mastered by the experts match the content of the project application can a more accurate review result be obtained, that is, the reliability of the review results is directly linked to the degree of matching.

[0004] As can be seen from the above, expert selection is a key link in project approval and review. The traditional approach is to select experts in subdivided fields based on the degree of match between the experts and the fields of the project. The matching result of this method is that the fields of the project and the experts are completely matched, but the actual matching degree between the project and the expert is still not accurate enough. The main reasons are: First, the diversity of experts' fields, including scientific fields, technical fields, and industry fields. Second, for several experts in the same field, what kind of weight is used to distinguish the matching degree of several experts.

[0005] Therefore, in the face of the ever-increasing number and scale of scientific and technological projects, expert databases, and expert academic achievements with constantly updated information, it is particularly important to quickly and accurately select suitable experts to conduct manual reviews of projects and improve the efficiency and fairness of review organizations. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a project review method for selecting experts based on a pre-trained sentence representation model, and by continuously evaluating and optimizing the extraction results, the extraction results are made more accurate and efficient, thereby greatly improving the efficiency of the review organization and improving the fairness of the review.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is to provide a project review method for selecting experts based on a pre-trained sentence representation model, comprising the following steps: S1) collecting a list of review experts and corresponding expert data, and establishing a review expert database; S2) obtaining the application form of the project to be reviewed for formal review, accepting the projects that have passed the formal review, collecting project data and establishing a database of projects to be reviewed; S3) for each project to be reviewed, using a pre-trained sentence representation model to perform similarity matching on the project data and the expert data to obtain a set of selected expert candidate lists; S4) based on the preset number of experts required for the project review, sending invitations to the selected candidate experts; if the candidate expert refuses to review or fails to submit the review opinion on time after receiving the invitation; then continue to automatically supplement the invitations from the expert candidate list until the number of people who have completed the project review is met.

[0008] Furthermore, the expert data collected in step S1 includes the expert's scientific field, technical field, industry field, papers, research results and award data; the research results include monographs, invention patents, software copyrights, standards, new plant variety rights, integrated circuit layout design rights, new drug certificates, medical device registration certificates and conference reports.

[0009] Furthermore, the step S3 includes: S31: using text fields to construct natural language expressions for project data and expert data respectively; S32: inputting the text expressions into a pre-trained sentence representation model and converting them into continuous vector representations; S33: calculating the similarity of the continuous vector representations of project data and expert data, and selecting a set of expert candidate lists through a sorting algorithm.

[0010] Furthermore, the step S32 analyzes the project data and expert data using statistical features and semantic features of text information; and regards different values ​​within the same field of the same object in the project data and expert data as having similar semantics, and values ​​of different objects as having different semantics.

[0011] Furthermore, the step S33 first selects candidate words in the scientific field for matching, and then selects keyword fields for matching; for candidate words in the scientific field, only leaf nodes in the predefined dictionary are used to obtain values; for keyword fields, IDF values ​​are used to filter them; the precision ranking score of each expert is the sum of the similarities of all keywords hit by him; the expert candidate list set selected in the step S33 is sorted from high to low according to the expert's precision ranking score.

[0012] Furthermore, when the step S33 calculates the refined ranking score for each expert, it is first assumed that the weights of the candidate words in the scientific field and all keywords are 1, and then the cosine similarity between the project and the expert is calculated and the first expert candidate list set is selected through the sorting algorithm, and then the Jaccard similarity coefficient between the project and the expert is calculated and the second expert candidate list set is selected through the sorting algorithm; if the ranking results calculated twice are inconsistent, the weights of the candidate words in the scientific field are adjusted until the rankings of the top multiple candidate experts are consistent.

[0013] Furthermore, the keyword field includes the expert's technical field, industry field, research results and review experience, and step S33 also includes reducing the weight of the expert's review experience when the number of projects a certain expert has participated in annually reaches a preset threshold; the threshold for the number of experts in the same group and the same unit does not exceed 50%.

[0014] Furthermore, step S4 also includes collecting the expert rejection rate and the invitation acceptance incomplete rate, and using them as adjustment factors for their review experience weights; when the threshold of the number of experts in the same group and the same unit after adjustment is still greater than 50%, the weights of all expert industry fields are reduced to continue calculating the refined ranking scores and sorting them until the threshold of the number of experts in the same group and the same unit is less than 50%.

[0015] Furthermore, the step S3 uses a recall algorithm to dynamically adjust the threshold so that the number of experts recalled for each project is greater than the preset number of experts required for project review.

[0016] Furthermore, the experts automatically added in step S4 determine whether their review deadline is greater than or equal to the review group deadline according to the invitation time, review cycle and review group deadline; when the day before the review group deadline is reached, the total number of unfilled persons in all review groups is determined, and when it is greater than a preset ratio, the review cycle is extended and experts continue to be invited.

[0017] Compared with the prior art, the present invention has the following beneficial effects: the project review method provided by the present invention for selecting experts based on a pre-trained sentence representation model makes the extraction results more accurate and efficient by continuously evaluating and optimizing the extraction results, thereby greatly improving the review organization efficiency and improving the fairness of the review. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a project review process for selecting experts based on a pre-trained sentence representation model in the present invention; Figure 2 Keyword map for experts and projects of this invention; Figure 3 The present invention is a flowchart of selecting and inviting review experts. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] Figure 1 This is a flowchart of the project review process of selecting experts based on the pre-trained sentence representation model in the present invention.

[0021] See also Figure 1 The project review method for selecting experts based on a pre-trained sentence representation model provided by the present invention comprises the following steps: S1) Collect the list of review experts and the corresponding expert data to establish a review expert database; S2) Obtain the application form of the project to be reviewed for formal review, accept the projects that pass the formal review, collect project data and establish a database of projects to be reviewed; S3) For each project to be reviewed, similarity matching is performed between project data and expert data based on a pre-trained sentence representation model to obtain a selected expert candidate list set; S4) Invitations are sent to the selected candidate experts based on the preset number of experts required for project review; if the candidate expert refuses to review or fails to submit the review opinion on time after receiving the invitation, invitations will continue to be automatically added from the expert candidate list until the number of experts required to complete the project review is met.

[0022] The present invention uses the similarity of keywords to calculate the similarity between the project and the expert, such as Figure 2 As shown. According to the weight of similarity, a priority ranking of candidate experts is formed. According to the result of similarity matching, repeated multiple combination tests of keywords are performed. The present invention selects the scientific field, technical field, industry field, papers, research results, and award-winning data of experts to extract keywords, and then determine the optimal parameters. The present invention can process the extraction process in combination with external data, add expert refusal, leave, and untimely review, and adjust the expert ranking weight to meet the actual project review organization application.

[0023] Please continue to see Figure 3 The selection and invitation process of the review experts of the present invention is as follows: 1) Keyword selection of design experts and projects Expert review database: three factors: scientific field, technical field, and industry field. Labels can be set for candidate experts to improve the selection and avoidance of people with specific labels. Personal achievements (papers, monographs, invention patents, software copyrights, standards, new plant variety rights, integrated circuit layout design rights, new drug certificates, medical device registration certificates, conference reports); Project proposal: project field, project results.

[0024] 2) By using an association matching algorithm between the expert information in the expert list and the project information, a candidate list of experts matching the project is formed.

[0025] 3) Evaluate the results and adjust the input parameters.

[0026] Using scientific fields and technological fields as keywords will result in a more accurate match of experts than selecting experts based on three factors: scientific fields, technological fields, and industry fields.

[0027] 4) Integrate external influencing factors and continue to optimize the algorithm model Collect information about experts who do not participate due to incompatible fields after the invitation is sent, and fine-tune the model algorithm for keyword similarity in the subdivided fields to correct the deviation.

[0028] The expert refusal rate and the invitation acceptance incomplete rate are collected to influence the ranking of experts in the candidate list to meet the needs of the review organization.

[0029] 5) Expert screening based on label conditions Experts can be screened based on the following: age (age range), professional title (senior, senior), awards; the selection process includes the following: 5.1 Screen experts according to the extraction setting rules: 5.2 Screening of experts to be avoided 6) Sorting interventions based on expert labels For example, experts from institutes and national award winners are prioritized. Experts are ranked by score: the ranking priority from high to low is: experts with labels -> experts with high matching scores; 7) Extract and sort the results Protection mechanism intervention: the number of projects an expert can participate in each year is limited. For experts who have reviewed too many times, their weight can be reduced; the threshold for the number of experts in the same group and the same unit cannot exceed 50%.

[0030] 8) Re-evaluation of results Design a matching model between the expert candidate set and the project to verify the matching between the expert and the project.

[0031] In order to accurately and efficiently assign review experts to projects, the project / expert data in Chinese is translated into English and matched at the nodes of a well-constructed English knowledge graph; at the same time, project expert recommendations are adopted based on a pre-trained sentence representation model and semantic similarity matching.

[0032] As a preferred implementation, step S3 includes: S31: Use text fields to construct natural language representations for project data and expert data respectively; for example: {“Project 1”: “Key technologies and applications for improving the quality of Shanghai rice wine, food brewing, Shanghai rice wine: green brewing; intelligent brewing: aging: microbiome; manufacturing industry”} {“Expert 1”: “Digestive system tumors, Endocrine system tumors, Tumor development, Pathology, Oncology”} {“Expert 2”: “Intersection of computer and mathematics, intersection of electronic communication and mathematics, cryptography, authentication and authorization technology, security and confidentiality technology, application security technology, information security blockchain, cryptography, authentication, privacy protection”} S32: Input the text representation into the pre-trained sentence representation model and convert it into a continuous vector representation; P1: [0.1,0.6,0.8,…,1.3] E1: [0.2,0.4,1.8,…,0.3] E2: [0.1,0.06,10.8,…,11.3] S33: Calculate the similarity between the continuous vector representations of the project data P1 and the expert data E1, E2, E3, E4, ..., and select a set of expert candidate lists through a sorting algorithm: sim(P1,E1): 0.856 sim(P1,E2): 0.100 sim(P1,E3): 0.009 sim(P1,E4): 0.902 sim(P1,E5): 0.832 sim() means finding the cosine similarity between two vectors. After sorting, the top three expert candidate lists are selected as {P1:[E1,E4,E5]}.

[0033] As a preferred implementation, the step S32 uses the statistical features and semantic features of text information to analyze the project data and expert data; and different values ​​within the same field of the same object in the project data and expert data are considered to be semantically similar, and values ​​of different objects are considered to be semantically different. For example: (Internet of Things Communication, Sensor Network Theory and Technology, 1), (Internet of Things Communication, Self-organizing Network, 1), (Internet of Things Communication, New Coronavirus, 0), etc.; The step S33 first selects the candidate words in the scientific field for matching, and then selects the keyword field for matching; this is because according to observation and testing, the values ​​of the two fields in the industry field and the technical field have weak professional relevance, which may interfere with the results; For candidate words in the scientific field, only leaf nodes in the predefined dictionary are used to obtain values; this is because non-leaf nodes such as "artificial intelligence" and "big data" usually cover a wide range of data objects and are not discriminative; Since words that appear too frequently in general corpora usually have no discrimination, for keyword fields, IDF values ​​are used to filter them; the present invention uses TF-IDF (Term Frequency-Inverse Document Frequency) to evaluate the importance of a word to a document set or a document in a corpus. TF (term frequency) refers to the frequency of a word appearing in a document. The more times a word appears in a document, the more important it is to the document.

[0034] IDF (Inverse Document Frequency) refers to the general importance of a keyword in the entire document set or corpus. IDF is calculated by measuring the inverse of the number of documents in the document set that contain the keyword. If a keyword appears in many documents, its IDF value will be low, indicating that it is less important in distinguishing documents.

[0035] The present invention multiplies the TF value of a keyword by its IDF value to obtain a comprehensive importance score. By performing TF-IDF calculation on each keyword in the document set, the importance score of each keyword for each document can be obtained, which can then be used for tasks such as data filtering, text classification, and information retrieval.

[0036] The ranking score of each expert is the sum of the similarities of all the keywords it hits. The experts are sorted from high to low according to their ranking scores to obtain a candidate list of experts to be selected. At the same time, the present invention uses a recall algorithm to dynamically adjust the threshold to ensure that the number of experts recalled for each project is greater than the number of experts required for the preset project review.

[0037] When calculating the precise ranking score for each expert, the present invention first assumes that the weights of the candidate words in the scientific field and all keywords are 1, then calculates the cosine similarity between the project and the expert and selects the first expert candidate list set through the sorting algorithm, and then calculates the Jaccard similarity coefficient between the project and the expert and selects the second expert candidate list set through the sorting algorithm; if the sorting results calculated twice are inconsistent, the weights of the candidate words in the scientific field are adjusted until the rankings of the top multiple candidate experts are consistent, so as to reduce the occurrence of inaccurate matching.

[0038] The present invention uses the Jaccard similarity coefficient to compare the similarities and differences between finite sample sets; the larger the Jaccard coefficient value, the higher the sample similarity. Given two sets A and B, the Jaccard coefficient is defined as the ratio of the size of the intersection of A and B to the size of the union of A and B, and is defined as follows: , when both sets A and B are empty, J(A,B) is defined as 1.

[0039] The expert who is automatically replaced in step S4 of the present invention determines whether his review deadline is greater than or equal to the review group deadline according to the invitation time, review cycle and review group deadline. The expert's review deadline can be greater than the review group deadline. When the day before the review group deadline is reached, it is necessary to determine the total number of unfilled people in all review groups. When it is greater than a certain proportion, the system appropriately postpones the invitation of experts or enters the manual intervention mode to ensure that the review cycle will not be excessively extended.

[0040] Finally, step S4 of the present invention also includes collecting the expert rejection rate and the uncompleted rate of accepting invitations, and using them as adjustment factors for the expert ranking weight. The number of projects that an expert can participate in annually can be limited. For experts who have reviewed too many times, the weight can be reduced; the threshold of the number of experts in the same group and the same unit cannot exceed 50%, so as to further ensure the fairness of the review and avoid multiple experts from the same unit from reviewing together. Specifically, the keyword field includes the technical field, industry field, research results and review experience of the expert, and the step S33 also includes reducing the weight of his review experience when the number of projects that a certain expert participates in annually reaches a preset threshold; it can ensure that top authoritative experts are selected as much as possible, and avoid overusing certain experts to affect the review efficiency. The step S4 also includes collecting the expert rejection rate and the uncompleted rate of accepting invitations, and using them as adjustment factors for the weight of his review experience; when the threshold of the number of experts in the same group and the same unit still exceeds 50% after the above adjustment measures are adopted, it is considered to reduce the weight of all experts' industry fields to continue to calculate the refined ranking score and sort until the threshold of the number of experts in the same group and the same unit is less than 50%. By precisely fine-tuning different weights, the extraction results are continuously evaluated and optimized, making the extraction results more accurate and efficient, greatly improving the efficiency of the review organization.

[0041] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A project review method for selecting experts based on a pre-trained sentence representation model, comprising the following steps: S1) Collect the list of review experts and the corresponding expert data to establish a review expert database; S2) Obtain the application form of the project to be reviewed for formal review, accept the projects that pass the formal review, collect project data and establish a database of projects to be reviewed; S3) For each project to be reviewed, similarity matching is performed between project data and expert data based on a pre-trained sentence representation model to obtain a selected expert candidate list set; S4) Invitations are sent to the selected candidate experts based on the preset number of experts required for project review; if the candidate expert refuses to review or fails to submit the review opinion on time after receiving the invitation, invitations will continue to be automatically added from the expert candidate list until the number of experts required to complete the project review is met.

2. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 1, characterized in that: The expert data collected in step S1 includes the expert's scientific field, technical field, industry field, papers, research results and award data; the research results include monographs, invention patents, software copyrights, standards, new plant variety rights, integrated circuit layout design rights, new drug certificates, medical device registration certificates and conference reports.

3. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 1, characterized in that: The step S3 comprises: S31: Use text fields to construct natural language representations for project data and expert data respectively; S32: Input the text representation into the pre-trained sentence representation model and convert it into a continuous vector representation; S33: Calculate the similarity of the continuous vector representations of the project data and the expert data, and select a set of expert candidate lists through a sorting algorithm.

4. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 3, characterized in that: The step S32 analyzes the project data and expert data using the statistical features and semantic features of the text information; and regards different values ​​within the same field of the same object in the project data and the expert data as having similar semantics, and values ​​of different objects as having different semantics.

5. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 3, characterized in that: The step S33 first selects candidate words in the scientific field for matching, and then selects keyword fields for matching; for candidate words in the scientific field, only leaf nodes in the predefined dictionary are used to obtain values; for keyword fields, IDF values ​​are used to filter them; the precision ranking score of each expert is the sum of the similarities of all keywords he / she hits; the expert candidate list set selected in the step S33 is sorted from high to low according to the expert's precision ranking score.

6. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 5, characterized in that: When calculating the refined ranking score for each expert in step S33, it is first assumed that the weights of the candidate words in the scientific field and all keywords are 1, and then the cosine similarity between the project and the expert is calculated and the first expert candidate list set is selected through the sorting algorithm, and then the Jaccard similarity coefficient between the project and the expert is calculated and the second expert candidate list set is selected through the sorting algorithm; if the ranking results calculated twice are inconsistent, the weights of the candidate words in the scientific field are adjusted until the rankings of the top multiple candidate experts are consistent.

7. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 6, characterized in that: The keyword field includes the expert's technical field, industry field, research results and review experience. Step S33 also includes reducing the weight of the expert's review experience when the number of projects a certain expert has participated in a year reaches a preset threshold; the threshold for the number of experts in the same group and the same unit does not exceed 50%.

8. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 7, characterized in that: The step S4 also includes collecting the expert rejection rate and the invitation acceptance incomplete rate, and using them as adjustment factors for the weight of their review experience; when the threshold of the number of experts in the same group and the same unit after adjustment is still greater than 50%, the weights of all experts' industry fields are reduced to continue calculating the refined ranking scores and sorting them until the threshold of the number of experts in the same group and the same unit is less than 50%.

9. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 1, characterized in that: The step S3 uses a recall algorithm to dynamically adjust the threshold so that the number of experts recalled for each project is greater than the preset number of experts required for project review.

10. The project review method for selecting experts based on a pre-trained sentence representation model according to claim 1, characterized in that: The experts automatically added in step S4 determine whether their review deadline is greater than or equal to the review group deadline according to the invitation time, review cycle and review group deadline; when the day before the review group deadline is reached, the total number of unfilled persons in all review groups is determined. When it is greater than the preset ratio, the review cycle is extended and experts continue to be invited.

Images