Scientific research scheme evaluation method and system based on multi-feature fusion

Abstract

The application is based on a scientific research scheme evaluation method and system based on multi-feature fusion, comprising: constructing a data set; the data set comprises personal information, ability and research idea of the evaluated person; the data set is encoded to obtain a personal information feature vector, an ability feature vector and a research idea feature vector; the extracted personal information feature vector, ability feature vector and research idea feature vector are fused, and after fusion, evaluation prediction is carried out to obtain the ability representation of the evaluated person; the data set covering personal information, ability and research idea is systematically constructed, high-dimensional feature vectors are extracted through deep feature coding technology, and an advanced fusion mechanism is adopted to realize effective interaction of multi-source features, thereby improving the accuracy, objectivity and automation level of scientific research idea evaluation, and the quality of the final work or the evaluation result in the preliminary idea stage is predicted, thereby reducing unnecessary resource investment.

Description

Technical Field

[0001] This invention relates to the field of capability prediction technology, specifically to a research program evaluation method based on multi-feature fusion.

[0002] The present invention also relates to a scientific research scheme evaluation system based on multi-feature fusion. Background Technology

[0003] In the field of research evaluation, existing methods typically focus only on the content analysis of the research ideas themselves, such as using text mining or natural language processing techniques to assess the originality or relevance of the ideas. However, these methods often overlook key factors such as the individual's background and capabilities, resource conditions (e.g., computing resources, time budget), and past research achievements, leading to biased evaluation results that fail to accurately reflect the actual feasibility and success potential of the ideas. Furthermore, traditional evaluations often rely on expert experience or simple scoring systems, lacking objective, data-driven comprehensive analysis and struggling to handle large-scale, multi-dimensional evaluation data, thus limiting their efficiency and accuracy.

[0004] To address these issues, existing technologies have developed AI-based assessment systems. For example, these systems utilize pre-trained language models to analyze the textual content of research ideas or identify relevant domain knowledge through database matching to aid judgment. These methods improve efficiency and reduce subjective bias by automating parts of the assessment task. Meanwhile, some systems attempt to incorporate personal resumes or publication data to enrich the assessment dimensions, but this information is typically processed as an independent module, undergoing only superficial integration such as rule-based combination or simple weighting. However, existing technologies still have significant shortcomings: First, the extraction of ability and research idea data often relies on general models, lacking structured construction tailored to research scenarios, such as multi-layered logical language partitioning, resulting in imprecise feature representations. Second, feature encoding often employs single models or traditional machine learning methods, failing to fully utilize deep learning methods such as the Transformer encoder for fine-grained feature extraction. Finally, feature fusion strategies are relatively simple, such as direct concatenation or average pooling, without considering the interaction relationships between different features, thus limiting the comprehensiveness and reliability of the assessment predictions.

[0005] In summary, there is an urgent need to develop an evaluation method that can overcome the shortcomings of existing technologies: systematically construct a dataset covering personal information, capabilities, and research ideas; extract high-dimensional feature vectors through deep feature encoding technology; and adopt an advanced fusion mechanism to achieve effective interaction of multi-source features, thereby improving the accuracy, objectivity, and automation of research idea evaluation. Summary of the Invention

[0006] The purpose of this invention is to provide a research proposal evaluation method based on multi-feature fusion. By extracting personal information, abilities and research ideas from existing public databases, extracting corresponding features through feature encoding, and performing fusion calculations through a self-attention mechanism, a predicted score for the final evaluated person is obtained. This score can be used to predict the quality of the final work or the review results in the preliminary conception stage, thereby reducing unnecessary resource investment.

[0007] The technical solution adopted in this invention is as follows: a scientific research scheme evaluation method based on multi-feature fusion, the method comprising: Construct a dataset; the dataset includes the personal information, abilities, and research ideas of the subjects being evaluated; Feature encoding involves encoding the dataset to obtain personal information feature vectors, capability feature vectors, and research idea feature vectors. Fusion assessment involves fusing extracted personal information feature vectors, ability feature vectors, and research idea feature vectors, and then using the fused feature vectors for assessment and prediction to obtain the ability representation of the assessed individual.

[0008] Furthermore, the construction of the capability data specifically includes: constructing prompt words and extracting them through a large language model. The prompt words include the research questions of the evaluated papers in existing public databases, the methods used to solve the problems in the papers, and the innovative points involved in the methods. The construction of the research concept data specifically includes: constructing a multi-layered logical language in the database related to the research of the evaluated individuals, and extracting it through a large language model.

[0009] Furthermore, the multi-layered logic language specifically includes: mathematical derivation / proof, dataset collection, model / architecture design, experimental design and paper writing / presentation; professional domain knowledge possessed by multiple assessed domains; and computing resources, financial and time budgets in GPU type / hour.

[0010] Furthermore, the feature encoding specifically includes: The personal information, abilities, and research ideas of the evaluated individuals in the dataset are encoded using three independent Transformer encoders to obtain corresponding feature vectors for personal information, abilities, and research ideas.

[0011] Furthermore, the fusion evaluation specifically includes: performing feature fusion calculation using a gated feature fusion method to obtain the final prediction score.

[0012] Furthermore, the gated feature fusion method specifically includes: The aforementioned personal information feature vector, capability feature vector, and research idea feature vector are linearly transformed as follows: (1) In the formula, , and These are respectively the personal information feature vector, the ability feature vector, and the research idea feature vector. , , For learnable parameters, , , As an intermediate variable; The gating weights are calculated using the exponential activation function and softmax normalization, as shown in the following formula: (2) In the formula, This represents element-wise multiplication. , , , As intermediate variables, the final output is obtained through weighted combination and summation, as shown in the following formula: (3) in , , Output intermediate variables for formula (1), To represent the vector dimension, formula (3) sums the vector dimensions, and out is the model output used for prediction. This prediction score can represent the quality or value score of a paper or research project.

[0013] The second technical solution adopted in this invention is a research proposal evaluation system based on multi-feature fusion, which employs a research proposal evaluation method based on multi-feature fusion. The system includes: The data acquisition module constructs prompt word templates and extracts the personal information, abilities, and research ideas of the current assessee through a large language model; The feature encoding module includes three independent Transformer encoders, which are used to encode the personal information, abilities and research ideas of the evaluated subjects in the dataset to obtain corresponding personal information feature vectors, ability feature vectors and research idea feature vectors; The fusion assessment module is used to fuse extracted personal information feature vectors, ability feature vectors and research idea feature vectors through a feature fusion network, and then perform assessment and prediction to obtain the ability representation of the assessed person.

[0014] Furthermore, the feature fusion network includes self-attention mechanisms, gated feature fusion, or residual linear fusion.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of this application are: By constructing a comprehensive dataset including the personal information, abilities, and research ideas of the evaluated individuals, and utilizing an AI large language model based on structured prompts and multi-layered logical language to extract ability and research idea data, the accuracy and relevance of the evaluation data were ensured. After extraction, three independent Transformer encoders were used to perform deep feature encoding on personal information, abilities, and research ideas respectively, obtaining their respective independent high-dimensional feature vectors, thus enhancing the richness of feature representation. Finally, a self-attention mechanism was used to stack the personal information feature vector, ability feature vector, and research idea feature vector into a sequence matrix for fusion evaluation, achieving effective interaction and integration among multi-source features to obtain the final predicted score. This predicted score represents the probability of success of the paper or research project, thereby significantly improving the comprehensiveness, accuracy, and automation level of scientific research idea evaluation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the review result prediction model of the method of the present invention; Figure 3 This is a diagram illustrating the AI-generated suggestions used in the method of this invention. Figure 4 This is a diagram illustrating the data collected in the verification example of the method of the present invention; Figure 5 This is a schematic diagram of the prompt words used to extract the abilities of the person being assessed in the method of the present invention; Figure 6 This is a summary diagram illustrating the performance of the fine-tuned BERT model under different input settings in the verification examples of the method of this invention; Figure 7 This is a summary of the performance of fine-tuning the BERT model for acceptance rate prediction under different settings in the verification examples of the method of this invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings.

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Example This embodiment provides a research proposal evaluation method based on multi-feature fusion, such as... Figure 1 As shown, this is used to predict the quality of papers; This embodiment extracts the personal information of the person being evaluated through an AI model, and then constructs prompt words to extract ability information and research ideas through a large language model. The prompt words include the research questions of the papers of the person being evaluated in existing public databases, the methods used to solve the problems in the papers, and the innovative points involved in the methods. The construction of research concept data specifically includes building a multi-layered logical language in the database related to the research of the evaluated researchers. The multi-layered logical language specifically includes: mathematical derivation / proof, dataset collection, model / architecture design, experimental design and paper writing / presentation; professional domain knowledge of multiple fields of the evaluated researchers; computing resources, financial and time budgets in GPU type / hour, and relevant information is extracted through a large language model.

[0020] This embodiment encodes author information, abilities, and research ideas into independent representations, and then fuses them to predict review results. Unlike the single-encoder Transformer, this embodiment's method better captures the interactions between these different text components, similar to user-item embedding interactions in a recommendation system. To predict review results based on author information, ability representations, and research ideas, this embodiment employs a gated feature fusion method for fusion evaluation, specifically including: The personal information feature vector, ability feature vector, and research idea feature vector are linearly transformed as follows: (1) In the formula, , and These are respectively the personal information feature vector, the ability feature vector, and the research idea feature vector. , , The learnable parameter matrix; The gating weights are calculated using the exponential activation function and softmax normalization, as shown in the following formula: (2) In the formula, This represents element-wise multiplication. , , , As intermediate variables, the final output is obtained through weighted combination and summation, as shown in the following formula: (3) in , , Output intermediate variables for formula (1), To represent the vector dimension, formula (3) sums the vector dimensions, and out is the model output used for prediction. This prediction score can represent the quality or value score of a paper or research project.

[0021] Example 2 In this embodiment, the fusion evaluation uses a self-attention mechanism to model the interaction between feature vectors. The specific process is as follows: After constructing the dataset using AI big data and constructing prompt words, such as... Figure 2 As shown, feature encoding is performed using three independent Transformer encoders to obtain corresponding personal information feature vectors, capability feature vectors, and research idea feature vectors. Given three feature representations , and First, stack them into a sequence matrix, as shown below: (4) In the formula, , and These are respectively the personal information feature vector, the ability feature vector, and the research idea feature vector. Given the length of each eigenvector, concatenate the three eigenvectors side by side to form... 3D matrix; Then, the self-attention mechanism is calculated: (5) In the formula, , , For a trainable parameter matrix, The calculation results are aggregated into the output representation sequence corresponding to the three input representations as follows: (6) In the formula, The output of equation (2) The three output representation vectors correspond to the components of each input representation. The aggregation results Project the results to obtain the final evaluation score. As shown in the following formula: (7) In the formula, For trainable parameters, in regression problems, To predict evaluation scores, training is performed using mean squared loss (MSE) given the true rating labels. In binary classification problems, The sigmoid function provides the probabilities of different classifications. Given the true evaluation decision labels, binary cross-entropy loss is used for training, and the classification result can be given by the sign or a threshold. Furthermore, training parameters can be selected. At this time, you can The relative magnitudes of different components are used as the classification criterion, and the probabilities of different categories are obtained using softmax. Cross-entropy loss is used for model training.

[0022] Example 3 In this embodiment, the residual linear fusion process is selected for the fusion evaluation process. The residual linear fusion process is a simple and effective linear transformation method. It combines element-wise addition for feature fusion. Each feature is independently projected and combined through element-wise addition, as shown in the following formula: (8) In the formula, , , As learnable parameters, the final output is calculated using the following formula: (9) In the formula, For learnable parameters, To predict the assessment score.

[0023] Verification Example In constructing the dataset for this invention, the design of AI prompts specifically includes: This invention uses prompts in a standardized format to collect author capabilities and research ideas, excluding information from completed research. Prompts for extracting research ideas are as follows: Figure 3 As shown, the research question, the method applied to solving the problem in the paper, and the innovative aspects of that method are extracted from it. This invention does not include experimental results in its research concept, which ensures that it can be evaluated at an early stage or during the research design phase. Capability extraction prompts include... Figure 5 As shown, this invention uses a template to extract the authors' skills and proficiency, considering six general skills, including mathematical derivation, theoretical analysis / proof, model / architecture design, data collection, and experimental design, as well as 5-10 areas of expertise specific to the author group. Simultaneously, computational resources, financial and time budgets in GPU type / hour were extracted, as computational power and financial budgets can also be considered as the author group's capabilities, which can also be planned in the research design.

[0024] I downloaded 25,889 PDF submissions from Openreview, including submissions from ICLR 2024 / 2025 and NeurIPS 2023 / 2024. Statistics are as follows: Figure 4As shown; although most of the rejected papers from NeurIPS were not open-source, their papers were merged because the acceptance criteria for these two conferences are similar; since the method in this application involves training author competence, this application screened approximately 16,712 papers whose first authors appeared twice in all papers. This application used Grobid to parse the downloaded PDF files and used the DeepseekR1 large language model to extract research ideas and author competence according to the aforementioned predefined cue word format. In addition, this application directly collected author information, titles, abstracts, scores, and acceptance decisions from the Openreview API; for average score prediction, this application also used conference information such as year and conference name; for acceptance prediction, conference information was not used because most of the available open-source NeurIPS papers are already accepted papers, such as... Figure 4 As shown, the merged dataset was shuffled using a random seed of 42, and then divided into training, validation, and test sets in an 8:1:1 ratio. That is, the first 80% of the shuffled dataset was used as the training set, the next 10% as the validation set, and the last 10% as the test set. For each experiment, five trials were conducted using different random seeds (42, 0, 1, 2, 3) to initialize model parameters and the data loader, select the model checkpoint with the minimum validation loss, and report the median or mean and standard deviation of the corresponding model performance on the test set. The experiments were conducted in a PyTorch 2.5.1 + Transformers 4.54.1 environment using two GeForce RTX 4090 GPUs.

[0025] The Transformer encoder's encoding process involves fine-tuning the BERT-base and BERT-large models on different input combinations to predict paper acceptance and average scores: (a) title, (b) title + abstract, (c) author information, (d) ability, (e) ideas, (f) author + ideas, (g) ability + ideas, and (h) author + ability + ideas (ACI); to account for differences in score distribution across different conferences and years, conference information is included in the average score prediction across all settings; this application employs a discriminative fine-tuning strategy: the backbone network parameters use... The learning rate is updated, while the prediction head uses... The learning rate is optimized using AdamW (weight decay of 0.01) and a linear warm-up scheduler; such as Figure 6 The report presents the rating prediction performance (MSE / MAE). Models using capability-aware inputs or combinations thereof, such as ACI, achieved slightly lower errors than models using authors or ideas alone. The BERT-large model using ACI showed modest gains. Although all models performed similarly, this suggests that capability-aware inputs offer a subtle benefit. Figure 7The results summarize the recruitment predictions. Capability-based settings, such as ACI, yielded higher accuracy and F1 scores, with BERT-large / ACI achieving 64.55% accuracy on non-text inputs. Text-based inputs, including summaries, generally performed best, with BERT-base achieving 68.48% accuracy, highlighting the complementary role of full-text information and capability features. Statistical Analysis: This application collected the predicted values ​​of all Transformer encoder models trained on (a), (b), (c), (d), and (e), evaluated their correlation matrices, and established a linear regression to fit the average score. To avoid overfitting, the regression model was fitted on the test set. For the fitted models, their predicted values ​​on the test set were collected. The outputs of different models trained on different text information were used as independent variables, and the average score was selected as the dependent variable. After controlling for other independent variables, ability was always a significant factor. Author identity, ideas, and ability all showed significant effects, ensuring that they should all be included in the joint three-way model. Compared with author identity and ideas, ability was shown to have a higher slope and was therefore more significant. Furthermore, the ability output predicted based on author information was also a significant factor, regardless of whether the ability model output existed.

[0026] This article uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A research proposal evaluation method based on multi-feature fusion, characterized in that, The method includes: Construct a dataset; the dataset includes the personal information, abilities, and research ideas of the subjects being evaluated; Feature encoding involves encoding the dataset to obtain personal information feature vectors, capability feature vectors, and research idea feature vectors. Fusion assessment involves fusing extracted personal information feature vectors, ability feature vectors, and research idea feature vectors, and then using the fused feature vectors for assessment and prediction to obtain the ability representation of the assessed individual.

2. The research scheme evaluation method based on multi-feature fusion according to claim 1, characterized in that, The construction of the capability data specifically includes: constructing prompt words and extracting them through a large language model. The prompt words include the research questions of the papers being evaluated in existing public databases, the methods used in the papers to solve the problems, and the innovative points involved in the methods. The construction of the research concept data specifically includes: constructing a multi-layered logical language in the database related to the research of the evaluated individuals, and extracting it through a large language model.

3. The research scheme evaluation method based on multi-feature fusion according to claim 2, characterized in that, The multi-layered logic language specifically includes: mathematical derivation / proof, dataset collection, model / architecture design, experimental design and paper writing / presentation; professional domain knowledge in multiple fields of the evaluated entity; and computing resources, financial and time budgets in GPU type / hour.

4. The research scheme evaluation method based on multi-feature fusion according to claim 1, characterized in that, The feature encoding specifically includes: The personal information, abilities, and research ideas of the evaluated individuals in the dataset are encoded using three independent Transformer encoders to obtain corresponding feature vectors for personal information, abilities, and research ideas.

5. The research scheme evaluation method based on multi-feature fusion according to claim 4, characterized in that, The fusion evaluation specifically includes: using a gated feature fusion method to perform feature fusion calculation and obtain the final prediction score.

6. The research scheme evaluation method based on multi-feature fusion according to claim 5, characterized in that, The gated feature fusion method specifically includes: The aforementioned personal information feature vector, capability feature vector, and research idea feature vector are linearly transformed as follows: （1） In the formula, , and These are respectively the personal information feature vector, the ability feature vector, and the research idea feature vector. , , These are learnable parameters; The gating weights are calculated using the exponential activation function and softmax normalization, as shown in the following formula: （2） In the formula, This represents element-wise multiplication. , , , As intermediate variables, the final output is obtained through weighted combination and summation, as shown in the following formula: (3) in , , Output intermediate variables for formula (1), To represent the vector dimension, formula (3) sums the vector dimensions, and out is the model output used for prediction. This prediction score can represent the quality or value score of a paper or research project.

7. A research proposal evaluation system based on multi-feature fusion, employing the research proposal evaluation method based on multi-feature fusion as described in any one of claims 1 to 6, characterized in that, The system includes: The data acquisition module constructs prompt word templates and extracts the personal information, abilities, and research ideas of the current assessee through a large language model; The feature encoding module includes three independent Transformer encoders, which are used to encode the personal information, abilities and research ideas of the evaluated subjects in the dataset to obtain corresponding personal information feature vectors, ability feature vectors and research idea feature vectors; The fusion assessment module is used to fuse extracted personal information feature vectors, ability feature vectors and research idea feature vectors through a feature fusion network, and then perform assessment and prediction to obtain the ability representation of the assessed person.

8. The scientific research scheme evaluation system based on multi-feature fusion according to claim 7, characterized in that, The feature fusion network includes self-attention mechanisms, gated feature fusion, or residual linear fusion.

Images