A method for constructing an impact analysis and evaluation model

Abstract

The present invention discloses a method for constructing an impact analysis and evaluation model, comprising collecting literature input variables; performing modeling and analysis using Altmetrics, an international influence proxy indicator, and academic citations as output variables; selecting the algorithm with the highest benchmark coefficient as the input algorithm for modeling grid parameter adjustment; using a feature importance evaluation model and comparing the contribution of each variable to the impact of the results; using a partial correlation graph to display the trend of the impact of each variable on the results, and combining the feature importance and partial correlation graph to comprehensively evaluate the impact of the results and output a result. By comparing the factors that have the greatest impact on academic citations with the factors that have the greatest impact on Almetrics, the present invention can effectively analyze the main sources of the international impact of research results and their influence methods, providing a more scientific and traceable analysis tool for the international impact assessment of research results.

Description

Technical Field

[0001] The present invention relates to the field of industrial 5G, and in particular to a method for constructing an impact analysis and evaluation model. Background Art

[0002] Evaluation technologies are increasingly being used in the industrial 5G sector, enabling timely and efficient network evaluation and analysis for interactive systems. Currently, due to the vast volume, diversity, and high density of 5G information, evaluation methods are subject to numerous uncertainties, leading to significant uncertainty. While some methods for constructing 5G network evaluation models have been developed, these remain ineffective in addressing this uncertainty. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for constructing an impact analysis and evaluation model.

[0004] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0005] The present invention comprises the following steps:

[0006] A. Collect literature input variables;

[0007] B uses Altmetrics (a proxy indicator of international influence) and academic citations as output variables for modeling and analysis;

[0008] C adds academic application volume and international influence proxy Altmetrics to the dataset, performs 10-fold cross-validation based on the inherited algorithm pool of the decision tree, evaluates the important performance of the algorithm on the dataset in turn, and uses 10-fold cross-validation to evaluate the benchmark performance of each algorithm on the dataset in turn;

[0009] D. Select the algorithm with the highest benchmark coefficient as the input algorithm for modeling and adjust the grid parameters for modeling;

[0010] E uses the feature importance assessment model and compares the contribution of each variable to the impact of the results;

[0011] F uses partial correlation diagrams to show the trend of the impact of each variable on the results, and combines the use of feature importance and partial correlation diagrams to comprehensively evaluate the impact of the results and output the results.

[0012] Furthermore, the preprocessing method includes missing value processing, outlier detection and processing, and data type conversion.

[0013] Furthermore, the literature input variables include the publication year, author, research institution, research field, abstract content, and keywords of the paper.

[0014] Furthermore, the inherited decision tree-based algorithms include Adaptive Boosting, Extremely Randomized Trees, Random Forest, and Gradient Boosting Machine.

[0015] A method for constructing an impact analysis and evaluation model, characterized by performing 10-fold cross-validation to evaluate the performance of an algorithm on a dataset, including accuracy, recall, and F1 score indicators, to estimate whether the algorithm performs best among all algorithms. If not, returning to select another algorithm and repeating the steps.

[0016] Furthermore, the specific normalization equation of the partial dependence graph is calculated as follows:

[0017]

[0018] Where f is the algorithm, i is the sample instance, i is the variable, n is the number of samples, and j is the sample number.

[0019] Furthermore, the partial dependence graph is calculated as follows:

[0020]

[0021] Where X is the input data set, Xs is the variable data set to be analyzed, and Xc is the other variable data set. xs is a single instance in Xs, and xc is a single instance in Xc. The analysis of a single variable includes the following:

[0022]

[0023] Where n is the sample size.

[0024] In a second aspect, an embodiment of the present application further provides an electronic device, including:

[0025] processor; and

[0026] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method steps of the first aspect.

[0027] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs.

[0028] The beneficial effects of the present invention are:

[0029] The present invention is a method for constructing an impact analysis and evaluation model. Compared with the prior art, the present invention has the following technical effects:

[0030] By comparing the factors that have the greatest impact on academic citations with those that have the greatest impact on Almetrics, this paper can effectively analyze the main sources of the international impact of research results and their impact methods, providing a more scientific and traceable analysis tool for the assessment of the international impact of research results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of the steps of a method for constructing an impact analysis and evaluation model according to the present invention;

[0032] Figure 2 It is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present invention will be further described below through specific examples. The illustrative examples and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0034] The present invention provides a method for constructing an impact analysis and evaluation model, comprising the following steps:

[0035] like Figure 1 As shown, in this embodiment, the following steps are included:

[0036] This paper uses the research results funded by the Second Regional Comprehensive Scientific Survey Research Project as an example to demonstrate the model establishment and evaluation process. The research data comes from the Dimensions database (https: / / app.dimensions.ai / discover / publication).

[0037] Based on the STEP project funding numbers as of November 11, 2022, a total of 4,679 publication records were retrieved. During the modeling process, 3,221 records with matching journal names found on Resurchify were selected. The entry dimensions used for modeling include: research topics, number of research citations, altmetrics indicators of the research, years since the research was published, number of authors, professional science communication actions, and the impact factor, h-index, SCImago Journal Ranking (SJR) and overall ranking of the journal in which the research was published. Among them, professional science communication actions refer to promotional press releases issued by professional science communication personnel of scientific research institutions on international science communication platforms. If press releases of related research can be found, they are professional science communication actions. Except for professional science communication actions, all data were obtained from the Resurchify website.

[0038] Collect literature input variables; use Altmetrics, a proxy indicator of international influence, and academic citations as output variables for modeling analysis; add academic citations and Altmetrics to the data set, perform 10-fold cross-validation based on the inherited algorithm pool of the decision tree, evaluate the important performance of the algorithm on the data set in turn, and use 10-fold cross-validation to evaluate the benchmark performance of each algorithm on the data set in turn; select the algorithm with the highest benchmark coefficient as the input algorithm for modeling grid parameter adjustment; use the feature importance evaluation model and compare the contribution of each variable to the impact of the results. Use PDP to show the trend of the impact of each variable on the results, and finally combine FIS and PDP to comprehensively evaluate the impact of the results.

[0039] Among them, Gradient Boosting Machine (GBM):

[0040] GBM is a popular gradient boosting algorithm that improves the performance of a model by iteratively training weak learners (usually decision trees), each time trying to correct the errors of the previous model.

[0041] It starts with an initial model and then adds new models to correct the residual errors of the previous model. This process is repeated until a predetermined number of iterations is reached or the model performance no longer improves significantly.

[0042] Random Forest(RF):Gradient Boosting Machine

[0043] RF is an ensemble learning method that builds multiple decision trees and combines their prediction results to improve the accuracy and robustness of the model.

[0044] Each decision tree is trained using a randomly selected subset of features, which increases diversity among trees and reduces the risk of overfitting.

[0045] Extremely Randomized Trees(ET):Random Forest(RF):Gradient BoostingMachine

[0046] ET, also known as Extra-Trees, is an ensemble method similar to random forest, but it uses extreme randomness when building decision trees.

[0047] In ET, the splitting of each node is not based on the best split point, but on randomly selected features and randomly selected thresholds, which helps to increase the diversity of the model.

[0048] AB, also known as AdaBoost, is an adaptive ensemble method that iteratively trains weak learners, with each new learner focusing on the samples misclassified by the previous learner.

[0049] AdaBoost works by assigning weights to each training example, with each new model placing more emphasis on examples that were misclassified by the previous model.

[0050] The core innovation of this invention lies in the establishment of two models for the same set of data and the comparative interpretation between these models. This interpretation is achieved through analysis of feature importance scores (FIS) and partial dependency plots (PDPs). FIS primarily identifies variables important for impact assessment, while PDPs reflect the impact of these variables. The comparison focuses on the FIS and key variable PDPs of the academic citation model and the international impact model. FIS incorporates standardization during the calculation process, enabling cross-model comparisons, while PDPs primarily compare within a model, with cross-model trend comparisons.

[0051] Eliminate features that appear flat or jitter irregularly in the PDP analysis, as these features contribute little to the model. Features with very steep PDP curves indicate a significant impact on model predictions and should be retained for further analysis. Use PDP to demonstrate the average impact of individual features on model predictions, and determine whether the relationship between features and predictions is approximately linear, monotonic, or more complex.

[0052] For categorical features, the PDP can be calculated for each category to estimate the partial dependence.

[0053] Comparative analysis of the FIS and key variable PDP of the academic citation model and the international impact model to identify the key features of impact prediction in different models and their influence modes;

[0054] By standardizing FIS, cross-model comparison is achieved, while PDP is mainly used for comparison within a model and trend comparison across models.

[0055] Based on the analysis results of FIS and PDP, the model is continuously optimized iteratively to eliminate useless features, enhance the model's ability to capture important features, and improve the model's accuracy and interpretability.

[0056] Through these steps, the FIS and PDP analysis results can be effectively used to optimize model performance and improve the model's prediction accuracy and interpretability.

[0057] The specific standardization methods are as follows:

[0058]

[0059] PDP is calculated as follows:

[0060]

[0061] Where X is the input data set, Xs is the variable data set to be analyzed, and Xc is the other variable data set, xs is a single instance in Xs, and xc is a single instance in Xc.

[0062] Analyses of individual variables included the following:

[0063]

[0064] Where n is the sample size.

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 2 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0066] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0067] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0068] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming an information security risk early warning device at the logical level. The processor executes the program stored in the memory and is specifically used to implement any of the aforementioned impact analysis and assessment model construction methods.

[0069] The above application Figure 1 The method for constructing an impact analysis and assessment model disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0070] The electronic device may also perform Figure 1 A method for constructing an impact analysis and evaluation model is proposed and implemented. Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0071] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple applications, they execute any of the aforementioned methods for constructing an impact analysis and evaluation model.

[0072] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0078] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0080] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing an impact analysis and evaluation model, characterized in that: The following steps are involved: A. Collect literature input variables; B uses Altmetrics (a proxy indicator of international influence) and academic citations as output variables for modeling and analysis; C adds Altmetrics, a proxy for academic application volume and international influence, to the dataset, performs 10-fold cross-validation on the inherited algorithm pool based on the decision tree, and evaluates the important performance of the algorithms on the dataset in turn. 10-fold cross-validation is used to evaluate the benchmark performance of each algorithm on the dataset in turn; D. Select the algorithm with the highest benchmark coefficient as the input algorithm for modeling and adjust the grid parameters for modeling; E uses the feature importance assessment model and compares the contribution of each variable to the impact of the results; F uses partial correlation diagrams to show the trend of the impact of each variable on the results, and combines the use of feature importance and partial correlation diagrams to comprehensively evaluate the impact of the results and output the results; The specific normalization equation for the partial dependence graph is calculated as follows: ； Where f is the algorithm, i is the sample instance, i is the variable, n is the number of samples, and j is the sample number; The partial dependence plot is calculated as follows: ； Where X is the input data set, Xs is the variable data set to be analyzed, and Xc is the other variable data set. xs is a single instance in Xs, and xc is a single instance in Xc. The analysis of a single variable includes the following: ； Where n is the sample size.

2. The method for constructing an impact analysis and evaluation model according to claim 1, characterized in that: The preprocessing methods include missing value processing, outlier detection and processing, and data type conversion.

3. The method for constructing an impact analysis and evaluation model according to claim 1, characterized in that: The literature input variables include the publication year, author, research institution, research field, abstract content, and keywords of the paper.

4. The method for constructing an impact analysis and evaluation model according to claim 1, wherein: The succession of decision tree-based algorithms include Adaptive Boosting, Extremely Randomized Trees, Random Forest, and Gradient Boosting Machine.

5. The method for constructing an impact analysis and evaluation model according to claim 1, wherein: Perform 10-fold cross-validation to evaluate the performance of the algorithm on the dataset, including accuracy, recall, and F1 score indicators, to estimate whether the algorithm performs best among all algorithms. If not, return and select another algorithm to repeat the steps.

6. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 5.

7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute the method according to any one of claims 1 to 5.

Images