Scientific and technological attack effectiveness evaluation method and device based on large language model

Abstract

The invention provides a scientific and technological attack effectiveness evaluation method and device based on a large language model, and is suitable for multi-dimensional comprehensive evaluation of scientific research projects under the background of a novel power system. The method comprises the following steps: firstly, constructing an evaluation index system covering dimensions such as technical innovation, technical feasibility, economic benefit and environmental influence, and collecting expert text evaluation of each project under each index; text semantic features are extracted through a BERT model, sentiment analysis is carried out in combination with a large language model, satisfaction scores are quantified, and an evaluation decision matrix is constructed; and calculating the index weight by using a CRITIC method, and finally carrying out comprehensive scoring and sorting on the projects in combination with a grey relational degree-ideal point method. According to the scheme, the problems that a traditional expert scoring method is high in subjectivity and low in information utilization rate are effectively solved, the objectivity, accuracy and scientificity of an evaluation result are improved, and intelligent and data-driven decision support is provided for scientific research project management and resource allocation.

Description

Technical Field

[0001] The present invention belongs to the technical field of scientific research project evaluation, and specifically relates to a method and device for evaluating the effectiveness of scientific and technological research based on a large language model. Background Art

[0002] As new power systems rapidly develop toward green, low-carbon, and intelligent approaches, scientific and technological research is playing an increasingly prominent role in driving breakthroughs in key power technologies, improving system efficiency, and ensuring energy security. To rationally allocate scientific research resources, evaluate project effectiveness, and optimize scientific and technological management systems, evaluating the effectiveness of scientific research projects has become a key topic in current research and practice.

[0003] Currently, the effectiveness evaluation of key scientific and technological projects often relies on expert scoring and questionnaires, combined with analytical methods such as the analytic hierarchy process, fuzzy comprehensive evaluation, and grey theory. While these methods have a solid theoretical foundation, they still face several challenges in practical application. First, the expert scoring process is significantly influenced by subjective judgment, resulting in bias and difficulty ensuring the objectivity and consistency of evaluation results. Second, a large amount of review information is presented in text form, and traditional methods often overlook the underlying semantic information and sentiment, failing to fully leverage the value of expert opinion. Furthermore, most methods rely heavily on empirically determined indicator weights and lack dynamic adjustment mechanisms based on data variability and information volume, making them difficult to adapt to complex and changing evaluation scenarios. Fourth, when faced with multiple research projects to be evaluated, existing methods lack comprehensive ranking and quantitative analysis capabilities, making it difficult to support scientific and rational resource allocation and decision-making.

[0004] In the prior art, Chinese patent CN114282806A discloses a method for evaluating power grid enterprises in the context of a new type of power system, including: obtaining basic data of evaluation indicators of the enterprise to be evaluated based on a pre-constructed evaluation index system for power grid enterprises; using set value statistics to perform valuation processing on the basic data of evaluation indicators to obtain the attribute values ​​of the evaluation indicators; using the attribute hierarchy model method to calculate the weight values ​​of the evaluation indicators based on the attribute values ​​of the evaluation indicators; using the grey correlation-ideal point method to solve the relative proximity between the evaluation indicators of the enterprise to be evaluated and the ideal indicators based on the weight values ​​of the evaluation indicators, so as to obtain the evaluation results based on the relative proximity. This solution comprehensively considers the evaluation indicators of multiple dimensions such as the power supply side, the power grid side, and the load side. By weighting the evaluation index system and solving the degree of closeness between each evaluation indicator and the ideal indicator, the rationality and effectiveness of the evaluation results are improved, so that the evaluation indicators of power grid enterprises can better adapt to the requirements of the development of the new type of power system.

[0005] However, this method is mainly applicable to the evaluation scenarios of power enterprise operating efficiency and system level. The evaluation object is biased towards organizational entities and lacks a precise description of the characteristics of scientific and technological research projects themselves. Its indicator system and calculation logic focus more on operating status and static performance indicators, which makes it difficult to reflect the comprehensive effectiveness of scientific and technological projects in terms of technological advancement, output, application prospects, etc. At the same time, this method fails to effectively integrate expert text evaluation opinions, lacks the ability to extract and model unstructured information, and cannot fully utilize the semantic information implicit in expert reviews, resulting in the need to improve the intelligence level and explanatory power of the evaluation conclusions. In addition, its indicator weighting and scoring method does not fully consider the correlation and dynamics between multi-source heterogeneous data, lacks a response mechanism for indicator sensitivity and changing trends, and cannot meet the higher requirements for flexibility, pertinence, and transparency in scientific research project evaluation.

[0006] Therefore, how to build a more scientific, objective and intelligent evaluation mechanism for the effectiveness of scientific and technological research in the context of the new power system has become an important issue that needs to be solved urgently. Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for evaluating the effectiveness of scientific and technological research based on a large language model.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] In one aspect, the present invention provides a method for evaluating the effectiveness of scientific and technological research based on a large language model, comprising the following steps:

[0010] Establish an evaluation index system for the effectiveness of scientific and technological research based on the goals and key influencing factors of scientific and technological research on new power systems;

[0011] Obtaining a set of expert comments on each evaluation indicator of a plurality of scientific research projects to be evaluated, wherein the set of comments includes a plurality of expert textual evaluations corresponding to each evaluation indicator;

[0012] Using the pre-trained BERT model to extract text semantic features from the review set, and combining it with a large language model to perform sentiment analysis and extract emotional tendencies;

[0013] Convert the extracted sentiment tendency into a satisfaction level and construct an evaluation decision matrix, wherein the evaluation decision matrix includes scores of multiple expert textual evaluations corresponding to various evaluation indicators of multiple scientific research projects to be evaluated;

[0014] The weight of each evaluation indicator is calculated using the CRITIC method based on the evaluation decision matrix;

[0015] Based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated.

[0016] Furthermore, the evaluation index system for the effectiveness of scientific and technological research includes:

[0017] Technological innovation indicators are used to measure the originality and degree of technological breakthroughs in scientific and technological research, including technological advancement, novelty, number of patents, and uniqueness of technical solutions;

[0018] Technical feasibility indicators are used to evaluate the feasibility of scientific and technological research plans, including technical maturity, engineering feasibility, experimental verification, and completion of key technology research;

[0019] Economic benefit indicators are used to measure the economic value and market competitiveness of scientific and technological projects, including cost control, return on investment, industrialization prospects and market adaptability;

[0020] Environmental impact indicators are used to assess the impact of scientific and technological research on the environment, including contributions to energy conservation and emission reduction, resource utilization efficiency, environmental friendliness and sustainable development potential.

[0021] Furthermore, the comment set is:

[0022]

[0023] Among them, T ij It represents the expert textual evaluation of the i-th scientific research project to be evaluated under the j-th evaluation indicator, m represents the number of scientific research projects to be evaluated, and n represents the number of evaluation indicators.

[0024] Furthermore, the pre-trained BERT model is used to extract text semantic features from the review set, specifically including:

[0025] Before the text is input into the BERT model, the expert textual evaluations in the review set are preprocessed to remove irrelevant information, perform text cleaning, and unify the text format;

[0026] Expert textual evaluation of each pre-processed Perform word segmentation and use BERT's vocabulary to convert words into Token IDs to obtain Token sequences :

[0027]

[0028] in, A token sequence representing the textual evaluation of the i-th research project under the j-th evaluation indicator by the k-th expert;

[0029] The token ID sequence after word segmentation Input into the pre-trained BERT model for encoding to obtain the semantic feature vector of each expert's textual evaluation :

[0030]

[0031] in, It represents the semantic information of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0032] The semantic feature vector Input into the large language model for further processing.

[0033] Furthermore, the sentiment analysis is performed in conjunction with a large language model to extract sentiment tendencies, specifically including:

[0034] The semantic feature vector h of each expert's textual evaluation ij Input into the large language model, use its natural language processing capabilities to perform sentiment classification or sentiment tendency analysis, and obtain the sentiment tendency value of the evaluation indicators of each scientific research project to be evaluated Includes extremely positive, positive, neutral, negative, and extremely negative.

[0035] Furthermore, the extracted sentiment tendency is converted into a satisfaction level and an evaluation decision matrix is ​​constructed, which specifically includes:

[0036] Map the sentiment analysis results of the large language model of the kth expert on the i-th scientific research project to be evaluated under the j-th evaluation index into the sentiment tendency value The mapping relationship is as follows:

[0037]

[0038] in, represents the sentiment tendency value of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0039] The sentiment tendency value Mapped to satisfaction score The conversion formula is as follows:

[0040]

[0041] in, represents the satisfaction score of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0042] Take the average of all experts’ satisfaction scores under each evaluation indicator and construct the evaluation decision matrix D = [d ij ],in:

[0043]

[0044] Among them, K is the number of experts participating in the scoring, d ij It represents the average satisfaction score of the i-th scientific research project to be evaluated under the j-th evaluation indicator.

[0045] Furthermore, the weight of each evaluation indicator is calculated by the CRITIC method based on the evaluation decision matrix, specifically including:

[0046] Calculate the standard deviation σ of each evaluation indicator j , the standard deviation reflects the degree of variation of each evaluation index score, and the formula is as follows:

[0047]

[0048] Among them, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation index, is the average value of the jth evaluation index, and m is the number of scientific research projects to be evaluated;

[0049] Calculate the correlation between the evaluation indicators and use the Pearson correlation coefficient to calculate the information redundancy between the evaluation indicators. The correlation calculation formula is as follows:

[0050]

[0051] Among them, r jk represents the linear correlation between the jth evaluation index and the kth evaluation index, d ij d ik are the scores of the i-th scientific research project under the j-th and k-th evaluation indicators respectively;

[0052] According to the standard deviation σ j The correlation between each indicator r jk Calculate the indicator weight w for each evaluation indicator j , the calculation formula is as follows:

[0053]

[0054] Among them, w j is the indicator weight of evaluation indicator j, and n is the number of evaluation indicators.

[0055] Furthermore, based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated, specifically including:

[0056] Based on the evaluation decision matrix, the ideal point and negative ideal point of each evaluation indicator are determined. The formula is:

[0057]

[0058] in, are the ideal point and negative ideal point of the jth evaluation index, respectively, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation indicator, and n is the number of evaluation indicators;

[0059] Based on the determined ideal points and negative ideal points, the grey correlation degree between each scientific research project to be evaluated and the ideal points and negative ideal points is calculated. The formula is:

[0060]

[0061] in, It represents the grey correlation between the i-th scientific research project to be evaluated and the ideal point and the negative ideal point under the j-th evaluation index, and ρ is the resolution coefficient;

[0062] Based on the grey correlation degree and the calculated weights of each evaluation index, the comprehensive evaluation score of each scientific research project to be evaluated is calculated using the following formula:

[0063]

[0064] Among them, S i is the comprehensive evaluation score of the i-th scientific research project to be evaluated, w j is the indicator weight of evaluation indicator j.

[0065] On the other hand, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the program, it implements a method for evaluating the effectiveness of scientific and technological research based on a large language model as described above.

[0066] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for evaluating the effectiveness of scientific and technological research based on a large language model as described above.

[0067] Compared with the prior art, the present invention has the following advantages:

[0068] (1) This paper uses a large language model to extract semantics and analyze sentiment from expert textual evaluations. This method structures and quantifies subjective textual opinions into calculable satisfaction levels. It then constructs an evaluation decision matrix, combines it with the CRITIC method to calculate objective weights, and ultimately uses the grey correlation-ideal point method for comprehensive scoring and ranking. This technical approach significantly enhances the intelligence level of the scientific research project evaluation process, the objectivity of the evaluation results, and the quantitative analysis capabilities.

[0069] (2) The present invention systematically introduces the BERT model and the large language model in the evaluation of scientific research projects, and deeply processes the unstructured text information of expert comments. It not only retains the fine-grained content of expert opinions, but also realizes quantitative transformation through sentiment analysis, breaking through the information loss bottleneck of traditional qualitative scoring.

[0070] (3) Traditional evaluation methods such as the analytic hierarchy process (AHP) and the fuzzy comprehensive evaluation method often rely on manual weighting, which can easily lead to weight bias due to differences in expert experience. This paper uses the CRITIC method to automatically calculate the weights of each evaluation indicator, comprehensively considering the data differences and information redundancy of the indicators, and implements an objective weighting mechanism based on data drive, effectively improving the rationality and stability of indicator weight setting.

[0071] (4) When faced with multiple scientific research projects to be evaluated, traditional methods often have difficulty balancing the integration and ranking of multi-dimensional indicators. This invention constructs a satisfaction rating matrix and combines it with the grey correlation-ideal point method to achieve comprehensive evaluation and scientific ranking of each scientific research project under multiple indicator dimensions, providing more powerful support for the rational allocation of scientific research resources and project decision-making.

[0072] (5) In view of the characteristics of the scientific and technological tasks in the development of new power systems involving multi-dimensional and complex requirements such as technological innovation, feasibility, economy, and environmental friendliness, the present invention establishes an indicator system with a clear structure and comprehensive content, combines natural language processing and intelligent computing methods, and realizes effective modeling and dynamic evaluation of complex indicators, which has good industry adaptability and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a flow chart of the method of the present invention;

[0074] Figure 2 This is a system model diagram of the present invention. DETAILED DESCRIPTION

[0075] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0076] Example 1:

[0077] This embodiment provides a method for evaluating the effectiveness of scientific and technological research based on a large language model. Figure 1 As shown, the following steps are included:

[0078] Step S1: Establish an evaluation index system for the effectiveness of scientific and technological research based on the goals and key influencing factors of the new power system scientific and technological research;

[0079] Step S2: Obtain a set of expert comments for each evaluation indicator of multiple scientific research projects to be evaluated, wherein the comment set includes multiple expert textual evaluations corresponding to each evaluation indicator;

[0080] Step S3: Use the pre-trained BERT model to extract text semantic features from the review set, and combine it with the large language model to perform sentiment analysis and extract emotional tendencies;

[0081] Step S4: converting the extracted sentiment tendency into a satisfaction level and constructing an evaluation decision matrix, which includes the scores of multiple expert textual evaluations corresponding to various evaluation indicators of multiple scientific research projects to be evaluated;

[0082] Step S5: Calculate the weight of each evaluation indicator using the CRITIC method based on the evaluation decision matrix;

[0083] Step S6: Based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated.

[0084] The scientific and technological research effectiveness evaluation method based on a large language model provided in this embodiment is designed to address the problems of strong subjectivity, low utilization of text information, arbitrary weight setting, and insufficient sorting support in the current scientific and technological research project evaluation process. A bottom-up intelligent evaluation mechanism is designed. Each step is closely linked in the overall solution to collaboratively achieve the scientificity and objectivity of the evaluation results.

[0085] First, establishing a scientifically sound evaluation index system based on the objectives and key influencing factors of new power system scientific and technological research is the foundation of the entire method. By covering multi-dimensional indicators such as technological innovation, technical feasibility, economic benefits, and environmental impact, it ensures that the evaluation content is aligned with the actual needs of the new power system, provides a clear structure and semantic basis for the subsequent collection and processing of multi-source data, and effectively avoids the evaluation bias caused by vague indicator settings in traditional methods.

[0086] With the established indicator system in place, obtaining expert reviews of multiple research projects across various indicator dimensions helps incorporate the professional judgments of experts from diverse fields, thereby comprehensively reflecting the projects' performance across all evaluation dimensions. Preserving these reviews in textual form lays the data foundation for further exploration of the deeper insights from expert opinions. This design not only preserves the flexibility of expert scoring but also avoids the severe information loss associated with traditional numerical scoring methods, providing a richer and more authentic basis for evaluation.

[0087] To address the inherent difficulty of quantifying and utilizing text data, this embodiment introduces the BERT model to extract semantic features from review sets. Combined with a large language model for sentiment analysis, this model is able to mine subjective emotional tendencies within text and form a structured emotional representation. This design overcomes the limitation of existing techniques, which rely solely on qualitative references for review comments, by transforming subjective text into system-recognizable and calculable evaluation information. This improves the efficiency of expert knowledge utilization and provides quantifiable support for subsequent satisfaction modeling and matrix construction.

[0088] Mapping sentiment results into satisfaction ratings and constructing an evaluation decision matrix is ​​a crucial step in transitioning from unstructured data to structured computing. This matrix, in a unified format, presents the comprehensive evaluations of multiple projects across various metrics by multiple experts, providing the system with the foundation for weighted learning and comprehensive analysis. Compared to traditional manual scoring methods, this approach not only draws on a wider range of data sources and provides more nuanced expression, but also effectively reduces the noise introduced by subjective emotional fluctuations, improving data comparability and the stability of subsequent processing.

[0089] To overcome the previous problem of over-reliance on expert experience and high subjectivity in weighting evaluation indicators, this implementation introduces the CRITIC method. This method uses an evaluation matrix to calculate the standard deviation and correlation of each indicator and dynamically generates indicator weights. This objective weighting mechanism fully leverages the diversity and information content of expert evaluations, improving the objectivity, precision, and adaptability of weight assignment. This helps improve the stability and fairness of the evaluation model and is particularly suitable for scenarios with complex, multi-dimensional, and highly uncertain scientific research projects.

[0090] After the weights are determined, the comprehensive evaluation score for each research project is calculated using the grey correlation-ideal point method. This not only reflects the degree to which each project approaches the "ideal project" in each evaluation dimension, but also enables multi-project ranking analysis and decision support. This method takes fuzziness and nonlinear factors into account, more realistically reflecting the relationship between the advantages and disadvantages of projects, enhancing the system's discriminative and explanatory capabilities, and addressing the issues of traditional evaluation methods with insufficient precision in ranking judgments and weak ability to express differences.

[0091] In summary, this embodiment, by constructing a complete set of intelligent evaluation mechanisms from semantic understanding to quantitative scoring to comprehensive decision-making, not only solves the problems that have long plagued the evaluation of scientific and technological projects, such as strong subjectivity, poor structure, and arbitrary empowerment, but also improves the system's ability to analyze and sort multiple projects, with higher scientificity, objectivity and practicality, and is particularly suitable for key industry scenarios such as new power systems that have higher requirements for scientific and technological resource allocation and project screening.

[0092] The evaluation index system for the effectiveness of scientific and technological research includes:

[0093] Technological innovation indicators are used to measure the originality and degree of technological breakthroughs in scientific and technological research, including technological advancement, novelty, number of patents, and uniqueness of technical solutions;

[0094] Technical feasibility indicators are used to evaluate the feasibility of scientific and technological research plans, including technical maturity, engineering feasibility, experimental verification, and completion of key technology research;

[0095] Economic benefit indicators are used to measure the economic value and market competitiveness of scientific and technological projects, including cost control, return on investment, industrialization prospects and market adaptability;

[0096] Environmental impact indicators are used to assess the impact of scientific and technological research on the environment, including contributions to energy conservation and emission reduction, resource utilization efficiency, environmental friendliness and sustainable development potential.

[0097] The comment collection is:

[0098]

[0099] Among them, T ij It represents the expert textual evaluation of the i-th scientific research project to be evaluated under the j-th evaluation indicator, m represents the number of scientific research projects to be evaluated, and n represents the number of evaluation indicators.

[0100] Use the pre-trained BERT model to extract text semantic features from the review set, including:

[0101] Before the text is input into the BERT model, the expert textual reviews in the review set are preprocessed to remove irrelevant information, perform text cleaning, and unify the text format;

[0102] Expert textual evaluation of each pre-processed Perform word segmentation and use BERT's vocabulary to

[0103] Convert words into Token IDs and get Token sequences :

[0104]

[0105] in, A token sequence representing the textual evaluation of the i-th research project under the j-th evaluation indicator by the k-th expert;

[0106] The token ID sequence after word segmentation Input into the pre-trained BERT model for encoding to obtain the semantic feature vector of each expert's textual evaluation :

[0107]

[0108] in, It represents the semantic information of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0109] The semantic feature vector Input into the large language model for further processing.

[0110] This embodiment utilizes a pre-trained BERT model to extract textual semantic features from a collection of expert reviews, efficiently converting expert textual evaluations into machine-processable structured data and extracting deep semantic information based on this. By preprocessing the expert text, removing irrelevant information, and unifying the format, the text input to the BERT model is clear and standardized, thereby avoiding interference from irrelevant data and improving processing efficiency and accuracy. The BERT model converts the preprocessed text into a sequence of token IDs and performs semantic encoding. BERT is able to capture the contextual information of each word, allowing the deep meaning of the expert evaluation to be accurately extracted. This not only preserves the details of the text but also understands the expert's emotional tendencies and evaluation attitudes. By further inputting into a large language model for sentiment analysis, this embodiment can identify the emotional tendencies in the expert evaluations, such as positive or negative, thereby providing a scientific basis for subsequent scoring. Compared to traditional expert scoring or questionnaire surveys, this evaluation method based on a large language model can more accurately understand the expert's evaluation intentions, reduce subjective bias, and improve the objectivity and consistency of the evaluation results. At the same time, this method can automatically process a large number of expert comments, greatly improving the efficiency of evaluation and providing strong support for the rational allocation and decision-making of scientific research resources.

[0111] Combined with a large language model, sentiment analysis is performed to extract emotional tendencies, including:

[0112] The semantic feature vector h of each expert's textual evaluation ij Input into the large language model, use its natural language processing capabilities to perform sentiment classification or sentiment tendency analysis, and obtain the sentiment tendency value of the evaluation indicators of each scientific research project to be evaluated Includes extremely positive, positive, neutral, negative, and extremely negative.

[0113] The extracted sentiment tendency is converted into satisfaction level and the evaluation decision matrix is ​​constructed, which includes:

[0114] Map the sentiment analysis results of the large language model of the kth expert on the i-th scientific research project to be evaluated under the j-th evaluation index into the sentiment tendency value The mapping relationship is as follows:

[0115]

[0116] in, represents the sentiment tendency value of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0117] The sentiment tendency value Mapped to satisfaction score The conversion formula is as follows:

[0118]

[0119] in, represents the satisfaction score of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator;

[0120] Take the average of all experts’ satisfaction scores under each evaluation indicator and construct the evaluation decision matrix D = [d ij ],in:

[0121]

[0122] Among them, K is the number of experts participating in the scoring, d ij It represents the average satisfaction score of the i-th scientific research project to be evaluated under the j-th evaluation indicator.

[0123] The weight of each evaluation indicator is calculated using the CRITIC method based on the evaluation decision matrix, including:

[0124] Calculate the standard deviation σ of each evaluation indicator j , the standard deviation reflects the degree of variation of each evaluation index score, and the formula is as follows:

[0125]

[0126] Among them, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation index, is the average value of the jth evaluation index, and m is the number of scientific research projects to be evaluated;

[0127] Calculate the correlation between the evaluation indicators and use the Pearson correlation coefficient to calculate the information redundancy between the evaluation indicators. The correlation calculation formula is as follows:

[0128]

[0129] Among them, r jk represents the linear correlation between the jth evaluation index and the kth evaluation index, d ij d ik are the scores of the i-th scientific research project under the j-th and k-th evaluation indicators respectively;

[0130] According to the standard deviation σ j The correlation between each indicator r jk Calculate the indicator weight w for each evaluation indicator j , the calculation formula is as follows:

[0131]

[0132] Among them, w j is the indicator weight of evaluation indicator j, and n is the number of evaluation indicators.

[0133] This example further converts the sentiment values ​​derived from large language model analysis into satisfaction ratings and constructs an evaluation decision matrix. Its core purpose is to transform subjective, unstructured expert text evaluations into quantifiable and comparable rating data, laying the foundation for subsequent multi-item quantitative analysis. By mapping sentiment polarity from "extremely negative" to "extremely positive" into a numerical range and normalizing it into a satisfaction score, it unifies the scales of different expert evaluations, reduces errors caused by the ambiguity of subjective descriptions, and thus improves the objectivity and consistency of the entire evaluation system.

[0134] Based on the constructed evaluation decision matrix, the CRITIC method is introduced to calculate the weights of each indicator, further strengthening the scientific and data-driven capabilities of this embodiment. By calculating the standard deviation of each evaluation indicator score, its ability to distinguish evaluation results can be measured; the introduction of the Pearson correlation coefficient can reflect the redundancy between indicators. Combined, these two methods can dynamically adjust the weights of each evaluation indicator, making the weight distribution more consistent with the actual data characteristics and avoiding the irrationality or subjective bias that may be caused by empirical weighting in traditional methods.

[0135] This design realizes a closed loop of the entire process from text evaluation to quantitative scoring and then to weight optimization. By jointly driving the indicator weight distribution through the amount of information and differences, it not only improves the accuracy and discrimination of multi-project evaluation, but also provides a more scientific and reliable basis for resource allocation and scientific and technological management decision-making, thereby significantly enhancing the practicality and advancement of the entire scheme in the effectiveness evaluation of new power system scientific and technological research projects.

[0136] Based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated, including:

[0137] Based on the evaluation decision matrix, the ideal point and negative ideal point of each evaluation indicator are determined. The formula is:

[0138]

[0139]

[0140] in, are the ideal point and negative ideal point of the jth evaluation index, respectively, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation indicator, and n is the number of evaluation indicators;

[0141] Based on the determined ideal points and negative ideal points, the grey correlation degree between each scientific research project to be evaluated and the ideal points and negative ideal points is calculated. The formula is:

[0142]

[0143] in, It represents the grey correlation between the i-th scientific research project to be evaluated and the ideal point and the negative ideal point under the j-th evaluation index, and ρ is the resolution coefficient;

[0144] Based on the grey correlation degree and the calculated weights of each evaluation index, the comprehensive evaluation score of each scientific research project to be evaluated is calculated using the following formula:

[0145]

[0146] Among them, S i is the comprehensive evaluation score of the i-th scientific research project to be evaluated, w j is the indicator weight of evaluation indicator j.

[0147] Example 2:

[0148] This embodiment also provides a scientific and technological research effectiveness evaluation system based on a large language model, such as Figure 2 Shown, including:

[0149] An evaluation index construction module is used to establish an evaluation index system for the effectiveness of scientific and technological research covering dimensions such as technological innovation, technical feasibility, economic benefits, and environmental impact based on the goals and key influencing factors of scientific and technological research on new power systems;

[0150] The data acquisition and preprocessing module is used to collect expert comments on multiple research projects under various evaluation indicators, and perform preprocessing operations such as cleaning, denoising, and formatting the comments;

[0151] The semantic feature extraction module is used to segment and encode the pre-processed expert textual evaluation through the BERT model to extract semantic feature vectors;

[0152] Sentiment analysis module, which is used to input semantic feature vectors into the large language model for processing, extract the sentiment tendency of expert comments, and quantify it into sentiment tendency values;

[0153] The rating conversion module is used to map the sentiment tendency value into a satisfaction score, average the scores of all experts, and construct an evaluation decision matrix;

[0154] The weight calculation module is used to calculate the objective weight of each evaluation indicator based on the evaluation decision matrix using the CRITIC method, taking into account the variability and redundancy of the indicators;

[0155] The comprehensive evaluation module is used to combine the calculated evaluation index weights and evaluation decision matrix, calculate the comprehensive evaluation scores of each scientific research project to be evaluated based on the grey correlation-ideal point method, perform ranking analysis, and output the final evaluation results.

[0156] On the other hand, this embodiment provides an electronic device including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, a method for evaluating the effectiveness of scientific and technological research based on a large language model as described above is implemented.

[0157] On the other hand, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for evaluating the effectiveness of scientific and technological research based on a large language model, such as any one of the above.

[0158] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for evaluating the effectiveness of scientific and technological research based on a large language model, characterized in that: The following steps are involved: Establish an evaluation index system for the effectiveness of scientific and technological research based on the goals and key influencing factors of scientific and technological research on new power systems; Obtaining a set of expert comments on each evaluation indicator of a plurality of scientific research projects to be evaluated, wherein the set of comments includes a plurality of expert textual evaluations corresponding to each evaluation indicator; Using the pre-trained BERT model to extract text semantic features from the review set, and combining it with a large language model to perform sentiment analysis and extract emotional tendencies; Convert the extracted sentiment tendency into a satisfaction level and construct an evaluation decision matrix, wherein the evaluation decision matrix includes scores of multiple expert textual evaluations corresponding to various evaluation indicators of multiple scientific research projects to be evaluated; The weight of each evaluation indicator is calculated using the CRITIC method based on the evaluation decision matrix; Based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated.

2. A method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The evaluation index system for the effectiveness of scientific and technological research includes: Technological innovation indicators are used to measure the originality and degree of technological breakthroughs in scientific and technological research, including technological advancement, novelty, number of patents, and uniqueness of technical solutions; Technical feasibility indicators are used to evaluate the feasibility of scientific and technological research plans, including technical maturity, engineering feasibility, experimental verification, and completion of key technology research; Economic benefit indicators are used to measure the economic value and market competitiveness of scientific and technological projects, including cost control, return on investment, industrialization prospects and market adaptability; Environmental impact indicators are used to assess the impact of scientific and technological research on the environment, including contributions to energy conservation and emission reduction, resource utilization efficiency, environmental friendliness and sustainable development potential.

3. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The review set is: Among them, T ij It represents the expert textual evaluation of the i-th scientific research project to be evaluated under the j-th evaluation indicator, m represents the number of scientific research projects to be evaluated, and n represents the number of evaluation indicators.

4. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The method of extracting text semantic features from the review set using the pre-trained BERT model specifically includes: Before the text is input into the BERT model, the expert textual evaluations in the review set are preprocessed to remove irrelevant information, perform text cleaning, and unify the text format; Expert textual evaluation of each pre-processed Perform word segmentation and use BERT's vocabulary to convert words into Token IDs to obtain Token sequences in, A token sequence representing the textual evaluation of the i-th research project under the j-th evaluation indicator by the k-th expert; The token ID sequence after word segmentation Input into the pre-trained BERT model for encoding to obtain the semantic feature vector of each expert's textual evaluation in, It represents the semantic information of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator; The semantic feature vector Input into the large language model for further processing.

5. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The sentiment analysis and emotional tendency extraction based on the large language model specifically include: The semantic feature vector h of each expert textual evaluation ij Input into the large language model, use its natural language processing capabilities to perform sentiment classification or sentiment tendency analysis, and obtain the sentiment tendency value of the evaluation indicators of each scientific research project to be evaluated Includes extremely positive, positive, neutral, negative, and extremely negative.

6. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The extracted emotional tendency is converted into a satisfaction level and an evaluation decision matrix is ​​constructed, specifically including: Map the sentiment analysis results of the large language model of the kth expert on the i-th scientific research project to be evaluated under the j-th evaluation index into the sentiment tendency value The mapping relationship is as follows: in, represents the sentiment tendency value of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator; The sentiment tendency value Mapped to satisfaction score The conversion formula is as follows: in, represents the satisfaction score of the textual evaluation given by the kth expert for the i-th scientific research project under the j-th evaluation indicator; Take the average of all experts’ satisfaction scores under each evaluation indicator and construct the evaluation decision matrix D = [d ij ],in: Among them, K is the number of experts participating in the scoring, d ij It represents the average satisfaction score of the i-th scientific research project to be evaluated under the j-th evaluation indicator.

7. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: The weight of each evaluation indicator is calculated by the CRITIC method based on the evaluation decision matrix, specifically including: Calculate the standard deviation σ of each evaluation indicator j , the standard deviation reflects the degree of variation of each evaluation index score, and the formula is as follows: Among them, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation index, is the average value of the jth evaluation index, and m is the number of scientific research projects to be evaluated; Calculate the correlation between the evaluation indicators and use the Pearson correlation coefficient to calculate the information redundancy between the evaluation indicators. The correlation calculation formula is as follows: Among them, r jk represents the linear correlation between the jth evaluation index and the kth evaluation index, d ij d ik are the scores of the i-th scientific research project under the j-th and k-th evaluation indicators respectively; According to the standard deviation σ j The correlation between each indicator r jk Calculate the indicator weight w for each evaluation indicator j , the calculation formula is as follows: Among them, w j is the indicator weight of evaluation indicator j, and n is the number of evaluation indicators.

8. The method for evaluating the effectiveness of scientific and technological research based on a large language model according to claim 1, characterized in that: Based on the calculated indicator weights and evaluation decision matrix, the grey correlation-ideal point method is used to calculate the comprehensive evaluation score of each scientific research project to be evaluated, specifically including: Based on the evaluation decision matrix, the ideal point and negative ideal point of each evaluation indicator are determined. The formula is: in, are the ideal point and negative ideal point of the jth evaluation index, respectively, d ij It represents the score of the i-th scientific research project to be evaluated under the j-th evaluation indicator, and n is the number of evaluation indicators; Based on the determined ideal points and negative ideal points, the grey correlation degree between each scientific research project to be evaluated and the ideal points and negative ideal points is calculated. The formula is: in, It represents the grey correlation between the i-th scientific research project to be evaluated and the ideal point and the negative ideal point under the j-th evaluation index, and ρ is the resolution coefficient; Based on the grey correlation degree and the calculated weights of each evaluation index, the comprehensive evaluation score of each scientific research project to be evaluated is calculated using the following formula: Among them, S i is the comprehensive evaluation score of the i-th scientific research project to be evaluated, w j is the indicator weight of evaluation indicator j.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images