Automatic judgment method for innovativeness of academic papers based on fusion of big language model feedback and expert comments
By constructing an automatic thesis innovation assessment model that integrates a large language model with expert comments, the problems of subjectivity and insufficient accuracy in identifying innovation in manual reviews are solved. This achieves efficient and accurate automatic assessment of thesis innovation, supporting the intelligent and scientific nature of academic evaluation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-10
AI Technical Summary
Existing evaluations of the innovation of dissertations mainly rely on subjective judgments by human experts, which suffers from long review cycles, large differences in standards, and poor consistency of results. Furthermore, research based on text mining or machine learning has failed to effectively utilize the deep semantic features of expert comments, resulting in insufficient accuracy in identifying innovation.
This paper adopts a method that integrates feedback from a large language model with expert comments. It constructs an automatic evaluation model for the innovation of dissertations through an encoding module, a self-attention module, a sparse attention module, a feedforward neural network module, and a classifier module. Combined with PDF parsing and retrieval enhancement generation technology, it generates evaluation feedback text and performs cross-text semantic feature fusion and feature dimensionality reduction to ultimately achieve automatic evaluation of innovation.
It reduces the subjectivity and repetitive work of manual judgment, improves review efficiency, enhances the accuracy and stability of innovation judgment, and enables efficient automated innovation assessment on large-scale paper data, supporting the scientific and intelligent nature of academic evaluation.
Smart Images
Figure CN121833959A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automated academic review technology, and in particular to an automatic method for judging the innovation of dissertations based on the fusion of large language model feedback and expert comments. Background Technology
[0003] The continuous expansion of the graduate student population has resulted in a massive number of dissertations and corresponding expert review comments. Although these comments contain high-quality evaluation information on the value of research topics, innovative methods, research depth, and academic standards, in the current traditional manual review system, these expert comments are usually used only once and are not subsequently systematically managed or analyzed in depth, resulting in low resource utilization and insufficient knowledge mining.
[0004] Current methods for evaluating the innovativeness of dissertations primarily rely on subjective judgment by human experts, resulting in problems such as long review cycles, significant differences in standards, and poor consistency of results. Meanwhile, while some studies based on text mining or machine learning attempt to automate the evaluation of dissertation innovativeness, most only extract features from the main text, neglecting the deep semantic features and cross-document semantic relationships contained in expert comments, leading to insufficient accuracy in innovativeness identification. Furthermore, traditional natural language processing models have limited ability to capture the novelty of research questions and the innovativeness of methods, making it difficult to form a systematic understanding of the research logic and innovative contributions of a dissertation. Summary of the Invention
[0005] This invention provides an automatic method for determining the innovativeness of dissertations based on the fusion of feedback from a large language model and expert comments, in order to overcome the above-mentioned technical problems.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows: An automatic method for determining the innovativeness of dissertations based on the fusion of large language model feedback and expert comments, characterized by the following specific steps: S1. Obtain the thesis and expert review results; preprocess the thesis using a large language model and preprocess the expert review results using a manual review method to obtain the evaluation feedback text and review comments; the thesis includes the thesis title, keywords, abstract, and full text file, and the expert review results include expert comments. S2. Establish an automatic evaluation model for the innovation of dissertations, and use the evaluation feedback text and review comments as a training set to train the automatic evaluation model for the innovation of dissertations, so as to obtain the trained automatic evaluation model for the innovation of dissertations. The automatic model for determining the originality of dissertations includes: The module includes an encoding module, a self-attention module, a sparse attention module, a feedforward neural network module, an attention feature dimensionality reduction module, and a classifier module. The encoding module is used to encode the evaluation feedback text and the evaluation annotation results respectively to obtain the first context semantic features and the second context semantic features; The self-attention module is used to capture global semantic features in the first context semantic features through a multi-head self-attention mechanism, obtain a self-attention feature matrix, and transmit it to the sparse attention module. The sparse attention module is used to obtain sparse attention weights based on the interaction relationship between the second context semantic features and the self-attention feature matrix, so as to perform feature fusion on the second context semantic features and the self-attention feature matrix, obtain the fused feature vector, and transmit it to the feedforward neural network module. The feedforward neural network module is used to perform nonlinear feature enhancement and dimensionality transformation on the input data to obtain enhanced features, which are then transmitted to the attention feature dimensionality reduction module. The attention feature dimensionality reduction module is used to extract global semantic information from the enhanced features, obtain a compressed semantic representation vector, and transmit it to the classifier module; The classifier module is used to perform feature mapping and classification based on the compressed semantic representation vector, and output the prediction result of the paper's innovativeness category; S3. Based on the trained automatic evaluation model for the innovation of dissertations, the level of innovation of dissertations is automatically evaluated.
[0007] Furthermore, the specific steps for preprocessing the thesis include: The thesis was parsed using a PDF parsing tool to obtain the original plain text of the thesis. Using the original plain text as the retrieval corpus, and employing retrieval enhancement generation technology, text fragments related to innovation are retrieved and extracted from the original plain text. Based on the original plain text, key text content is located and extracted using a semantic matching algorithm to obtain key paragraph text. The key text content is text paragraphs describing research theories, technical routes, and experimental designs. By integrating the aforementioned text fragments, key paragraphs, and the paper title, a prompt word template was designed and constructed. The prompt word template is input into the selected general language model, and an output character limit is set to generate evaluation feedback text from the perspective of methodological innovation.
[0008] Furthermore, the encoding module includes a word embedding matrix and a pre-trained Chinese BERT model; Furthermore, the specific steps of the encoding module to encode the evaluation feedback text and the evaluation annotation results to obtain the first contextual semantic features and the second contextual semantic features include: The evaluation feedback text and the evaluation annotation results are mapped to a first word vector and a second word vector respectively using a word embedding matrix, as follows: , in, , For vocabulary size, The dimension is the word vector. To evaluate the feedback text, ; For the review and approval results, ; The pre-trained Chinese BERT model includes a multi-layered sequentially connected Transformer encoder. The first word vector and the second word vector are processed by the multi-layered sequentially connected Transformer encoder to obtain first contextual semantic features and second contextual semantic features, represented as follows: , in, The semantic feature matrix representing the evaluation and annotation results is the second context semantic feature. ; The semantic feature matrix representing the evaluation feedback text is the first context semantic feature. ; Among them, the first in the pre-trained Chinese BERT model The process of processing input data by a layer Transformer encoder includes: Self-attention is calculated on the input data to obtain the output of single-head attention, as shown in the formula: , , in, These are linear mapping matrices representing queries, keys, and values, respectively. , Dimensions for each attention head; These represent the query, key, and value, respectively. Indicates input data; The outputs of multiple attention heads are concatenated, and a linear transformation is applied to the concatenated multi-head attention outputs to obtain the fused context representation. : , in, For the number of attention heads, To output the transformation matrix, ; The fused context representation is nonlinearly transformed using the feedforward neural network module in the Transformer encoder to obtain the encoded context representation, as shown in the formula: , in, For learnable weight matrix, This is a bias term.
[0009] Furthermore, the self-attention module includes a linear mapping layer and a scaled dot product attention mechanism; Furthermore, the self-attention module captures global semantic features from the first context semantic features through a multi-head self-attention mechanism and obtains a self-attention feature matrix. Specific steps include: The first context semantic features are processed by a linear mapping layer to generate a query matrix. Key matrix Sum matrix , represented as: , in, For learnable weight matrix, , For single-head attention dimension; Calculate the query matrix using the scaled dot product attention mechanism Key matrix Sum matrix The correlation weight between them is given by the formula: , in, This is the scaling factor; Extended based on the scaled dot product attention mechanism A parallel attention head, represented as: , The outputs of multiple attention heads are concatenated, and a linear transformation is applied to the concatenated multi-head attention output to obtain the self-attention representation, which is expressed as: , in, To output the mapping matrix, ; Perform residual connection and layer normalization operations on the self-attention representation to obtain the self-attention feature matrix, which is expressed as: , in, .
[0010] Further, the sparse attention module obtains sparse attention weights based on the interaction relationship between the second contextual semantic features and the self-attention feature matrix, and obtains the fused feature vector. Specific steps include: Perform linear transformations on the second context semantic feature matrix and the self-attention feature matrix respectively to obtain the query, key, and value matrix, represented as follows: , in, For a trainable parameter matrix, , For attention dimension; Based on the query and key matrix, and using a sparse masking function to define the local attention neighborhood to calculate the sparse attention weights, the sparse masking function is defined as follows: , in, Indicates the first The set of sparse neighborhood indexes corresponding to each query vector; Based on the sparse attention weight , value matrix We perform a weighted summation to obtain the fusion feature, which is represented as: , in, This represents the fused cross-textual semantic features. ; For all By splicing Regarding the splicing results After performing linear layer mapping and normalization, the fused feature vector is obtained, represented as follows: , in, To output the transformation matrix, , This is the layer normalization function. Furthermore, the feedforward neural network module performs nonlinear feature enhancement and dimensionality transformation on the input data to obtain enhanced features. Specific steps include: The input data is mapped position-by-position using a combination of two linear transformations and a nonlinear activation function. The calculation process is as follows: , in, , For trainable parameter matrix, , for, , For bias terms, , , For the hidden dimensions of the feedforward layer, It is a linear rectified function; right By performing residual connections and layer normalization operations, the enhanced features are obtained, represented as follows: .
[0011] Furthermore, the specific steps of the attention feature dimensionality reduction module to extract global semantic information from the enhanced features and obtain the compressed semantic representation vector include: Enhanced features A linear transformation is performed to generate a learnable attention weight vector, represented as: , in, For trainable weight vectors, , For bias terms, ; Based on the learnable attention weight vector, the enhanced features are weighted and summed and then normalized layer by layer to generate a compressed semantic representation vector, as follows: , , in, Indicating the enhancement feature, the first Feature vectors at each position , This is the compressed semantic representation vector. Furthermore, the specific steps of the classifier module performing feature mapping and classification based on the compressed semantic representation vector, and outputting the prediction result of the paper's innovativeness category, include: The compressed semantic representation vector is feature-mapped using a fully connected linear layer in the classifier module to generate a classification score vector, as follows: , in, For a trainable weight matrix, , For bias terms, , This represents the total number of papers categorized by their innovativeness. The classification score vector is obtained by using the Softmax function. The mapping to a class probability distribution is expressed as: , in, This represents the predicted probability for each category. ,satisfy ; The final prediction results for the paper's innovation category are as follows: , in, The label indicates the predicted category of paper innovation. .
[0012] Beneficial effects: This invention encodes the evaluation feedback text and the reviewer's annotation results separately through the encoding module in the automatic determination model of thesis innovation; captures global semantic features in the first context semantic features through the self-attention module; fuses the input features through the sparse attention module and the feedforward neural network module to obtain enhanced features; obtains the compressed semantic representation vector through the attention feature dimensionality reduction module; and realizes the prediction of thesis innovation category through the classifier.
[0013] This invention reduces the subjectivity and repetitive work of manual judgment, saves review time, and improves work efficiency. Simultaneously, by fusing cross-textual semantic features through a sparse attention module and a feedforward neural network module, it ensures the sufficiency and expressive power of the features required for innovativeness judgment, thereby improving the accuracy and stability of the automatic innovativeness judgment model for dissertations. Furthermore, this invention eliminates the need for manually constructing complex rules or relying on syntactic analysis, reducing data processing difficulty and enabling efficient automated innovativeness evaluation on large-scale dissertation data. Through systematic modeling and semantic analysis of historical dissertation and expert comment data, this invention not only improves the scientific and intelligent level of dissertation innovativeness evaluation but also provides data support and decision-making references for monitoring postgraduate training quality, optimizing academic directions, and enhancing scientific research innovation capabilities, thus providing technical support for the construction of a higher education quality assurance and scientific research innovation system. Attached Figure Description
[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 This is a flowchart of an automatic method for determining the innovativeness of dissertations based on the fusion of large language model feedback and expert comments, as described in this invention. Figure 2 This is a schematic diagram of the evaluation feedback text generated in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of the automatic thesis innovation determination model in an embodiment of the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] This embodiment provides an automatic method for determining the innovativeness of dissertations based on the fusion of large language model feedback and expert comments, such as... Figure 1 and Figure 3 As shown, the specific steps include: S1. Obtain the thesis and expert review results, preprocess the thesis using a large language model and preprocess the expert review results using a manual review method to obtain the evaluation feedback text and review annotation results. In a specific embodiment, the thesis includes a title, keywords, abstract, and full text file, and the expert review results include expert comments.
[0018] Specifically, large language models (such as ChatGPT, GPT series, BERT, ERNIE, etc.) demonstrate powerful language understanding and knowledge representation capabilities in tasks such as semantic understanding, knowledge reasoning, and text generation. This embodiment combines expert comments with the feedback mechanism of large language models to form a human-machine collaborative intelligent framework for judging the innovativeness of papers. This framework can improve the efficiency and objectivity of automated analysis, and simulate the logical reasoning and innovativeness recognition process of experts at the semantic level.
[0019] S2. Establish an automatic evaluation model for the innovation of dissertations, and use the evaluation feedback text and review comments as a training set to train the automatic evaluation model for the innovation of dissertations, so as to obtain the trained automatic evaluation model for the innovation of dissertations. The automatic model for determining the originality of dissertations includes: The module includes an encoding module, a self-attention module, a sparse attention module, a feedforward neural network module, an attention feature dimensionality reduction module, and a classifier module. The encoding module is used to encode the evaluation feedback text and the evaluation annotation results respectively to obtain the first context semantic features and the second context semantic features; The self-attention module is used to capture global semantic features in the first context semantic features through a multi-head self-attention mechanism, obtain a self-attention feature matrix, and transmit it to the sparse attention module.
[0020] The sparse attention module is used to obtain sparse attention weights based on the interaction relationship between the second context semantic features and the self-attention feature matrix, so as to perform feature fusion on the second context semantic features and the self-attention feature matrix, obtain the fused feature vector, and transmit it to the feedforward neural network module. The feedforward neural network module is used to perform nonlinear feature enhancement and dimensionality transformation on the input data to obtain enhanced features, which are then transmitted to the attention feature dimensionality reduction module. The attention feature dimensionality reduction module is used to extract global semantic information from the enhanced features, obtain a compressed semantic representation vector, and transmit it to the classifier module; The classifier module is used to perform feature mapping and classification based on the compressed semantic representation vector, and output the prediction result of the paper's innovativeness category; S3. Based on the trained automatic evaluation model for the innovation of dissertations, the level of innovation of dissertations is automatically evaluated.
[0021] In a specific embodiment, the preprocessing steps for the thesis include: The thesis was parsed using a PDF parsing tool (PDF Plumber) to obtain the original plain text of the thesis; Using the original plain text as the retrieval corpus, the Retrieval Enhanced Generation (RAG) technique is applied to retrieve and extract text fragments related to innovation from the original plain text; Based on the original plain text, key text content is located and extracted using a semantic matching algorithm to obtain key paragraph text. The key text content is text paragraphs describing research theories, technical routes, and experimental designs. By integrating the aforementioned text fragments, key paragraphs, and the paper title, a prompt word template was designed and constructed. like Figure 2 As shown, the prompt word template is input into the selected General Language Model (GLM), and an output word limit is set, such as 200 words, to generate evaluation feedback text from the perspective of methodological innovation. In a specific embodiment, the specific steps for preprocessing the expert review results using a manual review method include: Identify and extract all statements related to research methodology, theory, or technological innovation from the expert review results; The extracted statements are categorized according to a preset innovation category, which includes low innovation and high innovation, wherein: The term "low innovation" refers to the research methods used in a dissertation that are simple repetitions or direct applications of existing technologies, or that are merely non-essential adjustments, extensions, or combinations based on existing methods, without achieving substantial breakthroughs in research methods, theories, or technical paths. The high level of innovation refers to the fact that the research methods used in the thesis have effectively improved, optimized, or innovatively reconstructed the existing research system or technical path, and proposed new ideas, methods, or solutions with significant academic value or application potential; or proposed a completely new theoretical system, model framework, or technical solution, which has solved representative or key problems in the field. The extracted innovative statements are categorized as either low or high innovativeness and labeled as 0 or 1, respectively.
[0022] In a specific embodiment, the encoding module includes a word embedding matrix and a pre-trained Chinese BERT model; Furthermore, the specific steps of the encoding module to encode the evaluation feedback text and the evaluation annotation results to obtain the first contextual semantic features and the second contextual semantic features include: The evaluation feedback text and the evaluation annotation results are mapped to a first word vector and a second word vector respectively using a word embedding matrix, as follows: , in, , For vocabulary size, The dimension is the word vector. To evaluate the feedback text, ; For the review and approval results, ; The pre-trained Chinese BERT model includes a multi-layered sequentially connected Transformer encoder. The first word vector and the second word vector are processed by the multi-layered sequentially connected Transformer encoder to obtain first contextual semantic features and second contextual semantic features, represented as follows: , in, The semantic feature matrix representing the evaluation and annotation results is the second context semantic feature. ; The semantic feature matrix representing the evaluation feedback text is the first context semantic feature. ; Among them, the first in the pre-trained Chinese BERT model The process of processing input data by a layer Transformer encoder includes: Self-attention is calculated on the input data to obtain the output of single-head attention, as shown in the formula: , , in, These are linear mapping matrices representing queries, keys, and values, respectively. , Dimensions for each attention head; These represent the query, key, and value, respectively. Indicates input data; The outputs of multiple attention heads are concatenated, and a linear transformation is applied to the concatenated multi-head attention outputs to obtain the fused context representation. : , in, For the number of attention heads, To output the transformation matrix, ; The fused context representation is nonlinearly transformed using the feedforward neural network module in the Transformer encoder to obtain the encoded context representation, as shown in the formula: , in, For learnable weight matrix, This is a bias term.
[0023] In a specific embodiment, the self-attention module includes a linear mapping layer and a scaled dot product attention mechanism; Furthermore, the self-attention module captures global semantic features in the first context semantic features through a multi-head self-attention mechanism to enhance the completeness and accuracy of semantic feature representation and obtain a self-attention feature matrix. Specific steps include: The first context semantic features are processed by a linear mapping layer to generate a query matrix. Key matrix Sum matrix , represented as: , in, For learnable weight matrix, , For single-head attention dimension; Calculate the query matrix using the scaled dot product attention mechanism Key matrix Sum matrix The correlation weight between them is given by the formula: , The Softmax function is used to normalize the attention scores so that the sum of the weights at each position is 1. This is a scaling factor used to prevent gradient vanishing and excessive smoothing of the attention distribution; Extended based on the scaled dot product attention mechanism Multiple parallel attention heads are used to learn information in different semantic subspaces, as shown below: , The outputs of multiple attention heads are concatenated, and a linear transformation is applied to the concatenated multi-head attention output to obtain the self-attention representation, which is expressed as: , in, To output the mapping matrix, ; To maintain gradient stability and feature consistency, residual connections and layer normalization operations are performed on the self-attention representation to obtain the self-attention feature matrix, as follows: , in, .
[0024] Specifically, the self-attention feature matrix is used as the input to the subsequent sparse attention module for cross-feature fusion with the annotation results of human expert review.
[0025] In a specific embodiment, the sparse attention module obtains sparse attention weights based on the interaction relationship between the second contextual semantic features and the self-attention feature matrix, calculates the sparse attention weights within a limited feature dimension range, filters key information regions, and reduces redundant feature interference, thereby achieving efficient feature fusion of the second contextual semantic features and the self-attention feature matrix, and obtaining the fused feature vector. Specific steps include: Perform linear transformations on the second context semantic feature matrix and the self-attention feature matrix respectively to obtain the query, key, and value matrix, represented as follows: , in, For a trainable parameter matrix, , For attention dimension; Based on the query and key matrix, and using a sparse mask function to define the local attention neighborhood to calculate sparse attention weights, redundant computation is reduced while maintaining the main semantic relevance. The sparse mask function is defined as follows: , in, Indicates the first The sparse neighborhood index set corresponding to each query vector contains only those... Previous with high semantic similarity A key vector index is used to achieve local sparsity computation; Based on the sparse attention weight , value matrix We perform a weighted summation to obtain the fusion feature, which is represented as: , in, This represents the fused cross-textual semantic features. This is used to reflect the semantic interaction strength between the results of manual review and annotation and the feedback from the automatic judgment model for the innovation of dissertations; For all By splicing Regarding the splicing results After performing linear layer mapping and normalization, the fused feature vector is obtained, represented as follows: , in, To output the transformation matrix, , This is the layer normalization function, used to stabilize training and improve the consistency of the fused feature distribution. In a specific embodiment, the feedforward neural network module performs nonlinear feature enhancement and dimensionality transformation on the input data to obtain enhanced features. The specific steps include: A combination of two-layer linear transformation and nonlinear activation function is used to perform position-wise mapping of the input data. The calculation process is as follows: , in, , For trainable parameter matrix, , for, , For bias terms, , , For the hidden dimensions of the feedforward layer, It is a linear rectifier function used to introduce nonlinear mapping capability; To maintain model stability and consistency of inter-layer feature distribution, By performing residual connections and layer normalization operations, the enhanced features are obtained, represented as follows: , Residual connections are used to mitigate the vanishing gradient problem. Used to improve training convergence and feature scale consistency.
[0026] In a specific embodiment, the attention feature dimensionality reduction module extracts global semantic information from the enhanced features to obtain a compressed semantic representation vector, thereby achieving dimensionality reduction and information aggregation of high-dimensional semantic features. Specific steps include: Enhanced features A linear transformation is performed to generate a learnable attention weight vector, represented as: , in, For trainable weight vectors, , For bias terms, , This is used to normalize the attention score so that the sum of the weights at each position is 1; Based on the learnable attention weight vector, the enhanced features are weighted and summed and then normalized layer by layer to generate a compressed semantic representation vector, as follows: , , in, Indicating the enhancement feature, the first Feature vectors at each position , This is the compressed semantic representation vector, which contains the overall semantic information of the sequence. Layer normalization is used to standardize the feature scale, improve the model convergence speed and feature expression consistency, and finally obtain the dimensionality-reduced semantic vector. It can be used as input for subsequent innovation judgment modules to achieve effective compression and information aggregation from high-dimensional sequence feature vectors to global semantic representation. In a specific embodiment, the classifier module performs feature mapping and classification based on the compressed semantic representation vector, and outputs the prediction result of the paper's innovativeness category. The specific steps include: The compressed semantic representation vector is feature-mapped using a fully connected linear layer in the classifier module to generate a classification score vector, as follows: , in, For a trainable weight matrix, , For bias terms, , This represents the total number of papers categorized by their innovativeness. The classification score vector is obtained by using the Softmax function. The mapping to a class probability distribution is expressed as: , in, This represents the predicted probability for each category. ,satisfy ; The final prediction results for the paper's innovation category are as follows: , in, The label indicates the predicted category of paper innovation. .
[0027] In a specific embodiment, during the training phase of the automatic thesis innovation assessment model, based on the actual thesis tags... The cross-entropy loss function is used for optimization. The loss function is: , in, One-hot representation of the true label. This is the training loss for the classifier, used to guide model parameter updates.
[0028] Specifically, in this embodiment, the input dimension is set. Set to 768, output dimension The value is set to 2, corresponding to the paper's innovation categories of "low innovation" and "high innovation." During forward propagation, the compressed semantic representation vector is passed through a trainable weight matrix. and bias terms Mapped to classification score vector Then, it is converted into a class probability distribution by the Softmax function. The final predicted category is determined by the sum of the predicted probabilities for each category being 1. The novelty assessment model for dissertations is determined by the index with the highest probability. During the training phase, it uses the cross-entropy loss function. Optimize the difference between the true label and the predicted probability, and dynamically update the weights. and bias The parameters enable effective mapping and determination of semantic features to the paper's innovation category.
[0029] In this embodiment, the text of the dissertation to be predicted and the corresponding expert comments are input into the trained automatic dissertation innovation assessment model, and the model outputs the innovation classification result of the dissertation, thereby realizing the automatic assessment of the innovation level of the dissertation. Specifically, this embodiment selects the dissertation "Path Planning in a Multi-Pedestrian Dynamic Environment Based on SAC" indexed by CNKI as the experimental sample. The dissertation abstract, expert comments, and innovation feedback generated by the large language model are shown in Table 1. Based on the evaluation feedback text of the expert comments and the feedback from the large language model, the automatic dissertation innovation assessment model extracts semantic features, performs attention fusion, and feature dimensionality reduction, and finally outputs the innovation classification result of the dissertation as highly innovative.
[0030] Table 1:
[0031] Specifically, Table 1 shows the main content of the paper, expert comments, and model analysis results. It can be seen that the model in this embodiment can accurately identify the research innovations and methodological breakthroughs involved in the text. The judgment results are consistent with the expert evaluation, which verifies the feasibility and reliability of the model in real paper scenarios.
[0032] Furthermore, the paper innovation assessment model proposed in this embodiment is compared with existing BERT-Base-Cased, BERT-Base-Chinese, Chinese-BERT-WWM, Chinese-MACBERT-Base, ERNIE-1.0, and NEZHA-CN-Base models, and the results are shown in Table 2. It can be seen that the model proposed in this embodiment outperforms other baseline models in both F1 score and accuracy, with both F1 and accuracy reaching 0.89, significantly higher than other BERT series and pre-trained Chinese models (F1 score between 0.76 and 0.84, accuracy between 0.76 and 0.84). This indicates that the model in this embodiment can more effectively capture the semantic features in the paper text and expert comments, achieving deep fusion of cross-textual information, thereby improving the accuracy and robustness of innovation assessment, while achieving a significant performance improvement while maintaining reasonable computational efficiency.
[0033] Table 2:
[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for automatically determining the innovation of a dissertation based on the fusion of large language model feedback and expert comments, characterized in that, The specific steps include: S1, obtaining a dissertation and expert review results, preprocessing the dissertation using a large language model and preprocessing the expert review results using an artificial review method, obtaining evaluation feedback text and review annotation results; the dissertation includes a paper title, keywords, an abstract and a paper full text file, and the expert review results include expert comment content; S2, establishing an automatic innovation judgment model for dissertations, and using the evaluation feedback text and review annotation results as a training set to train the automatic innovation judgment model for dissertations, to obtain a trained automatic innovation judgment model for dissertations; The automatic innovation judgment model for dissertations includes: An encoding module, a self-attention module, a sparse attention module, a feedforward neural network module, an attention feature dimension reduction module and a classifier module; The encoding module is used to encode the evaluation feedback text and review annotation results respectively to obtain first and second context semantic features; The self-attention module is used to capture global semantic features in the first context semantic features through a multi-head self-attention mechanism to obtain a self-attention feature matrix, which is transmitted to the sparse attention module; The sparse attention module is used to obtain sparse attention weights based on the interaction between the second context semantic features and the self-attention feature matrix, to perform feature fusion on the second context semantic features and the self-attention feature matrix, to obtain a fused feature vector which is transmitted to the feedforward neural network module; The feedforward neural network module is used to perform nonlinear feature enhancement and dimension transformation processing on the input data to obtain enhanced features, which are transmitted to the attention feature dimension reduction module; The attention feature dimension reduction module is used to extract global semantic information from the enhanced features to obtain a compressed semantic representation vector, which is transmitted to the classifier module; The classifier module is used to perform feature mapping and classification based on the compressed semantic representation vector, and output a prediction result of the innovation category of the paper; S3, based on the trained automatic innovation judgment model for dissertations, to realize automatic judgment of the innovation level of dissertations.
2. The method of claim 1, wherein the method further comprises: The specific steps of preprocessing the dissertation include: Using a PDF analysis tool to analyze the dissertation to obtain the original pure text of the paper; Using the original pure text as a retrieval corpus, using retrieval enhancement generation technology to retrieve and extract text segments related to innovation from the original pure text; Based on the original pure text, locate and extract key text content through a semantic matching algorithm to obtain key paragraph text, the key text content being text paragraphs describing research theory, technical route and experimental design content; Integrate the text segments and key paragraph text and the paper title to design and build a prompt word template; Input the prompt word template into the selected general language model and set the output word limit to generate the evaluation feedback text of the method innovation perspective.
3. The method of claim 2, wherein the method further comprises: The encoding module includes a word embedding matrix and a pre-trained Chinese BERT model; And, the encoding module respectively encodes the evaluation feedback text and the review comment result to obtain the first context semantic feature and the second context semantic feature, and the specific steps include: The evaluation feedback text and the review comment result are respectively mapped into a first word vector and a second word vector through a word embedding matrix, and the formula is: ， wherein, , is a vocabulary size, is a word vector dimension; is an evaluation feedback text, ; is a review comment result, ; The pre-trained Chinese BERT model includes a plurality of sequentially connected Transformer encoders, and the first word vector and the second word vector are respectively processed through the plurality of sequentially connected Transformer encoders to obtain the first context semantic feature and the second context semantic feature, which is represented as: ， wherein, a semantic feature matrix representing the result of the review comment, i.e., a second context semantic feature, ; a semantic feature matrix representing the result of the review comment, i.e., a second context semantic feature, ; wherein the first The processing procedure of the input data by the layer Transformer encoder comprises: Self-attention calculation is performed on the input data to obtain the output of single-head attention, and the formula is: ， ， wherein, are linear mapping matrices for queries, keys, values, respectively, , is the dimension of each attention head; denote queries, keys, values, respectively; denotes input data; The output results of the plurality of attention heads are spliced, and a linear transformation is performed on the spliced multi-head attention output to obtain a fused context representation : ， wherein is the number of attention heads, is the output transformation matrix, ; The fused context representation is subjected to nonlinear transformation through the feedforward neural network module in the Transformer encoder to obtain the encoded context representation, and the formula is: ， wherein, is a learnable weight matrix, is a bias term.
4. The method of claim 3, wherein the method further comprises: The self-attention module includes a linear mapping layer and a scaled dot-product attention mechanism; And the self-attention module captures the global semantic feature in the first context semantic feature through a multi-head self-attention mechanism and obtains a self-attention feature matrix, and the specific steps include: The first context semantic features are processed through a linear mapping layer to generate a query matrix , a key matrix , and a value matrix , represented as: ， wherein, is a learnable weight matrix, , is a single-head attention dimension; Calculate the query matrix using the scaled dot product attention mechanism Key matrix Sum matrix The correlation weight between them is given by the formula: ， wherein is a scaling factor; Based on the scaling dot-product attention mechanism An individual parallel attention head is extended, denoted as: ， The output results of a plurality of attention heads are spliced, and the spliced multi-head attention output is subjected to linear transformation to obtain a self-attention representation, which is represented as: ， wherein is an output mapping matrix, ; The self-attention representation is subjected to residual connection and layer normalization operation to obtain the self-attention feature matrix, which is represented as: ， wherein .
5. The method of claim 4, wherein the method further comprises: The sparse attention module obtains sparse attention weights based on the interaction relationship between the second context semantic feature and the self-attention feature matrix, and obtains a fused feature vector, and the specific steps include: The second context semantic feature and the self-attention feature matrix are respectively subjected to linear transformation to obtain query, key and value matrices, which are represented as: ， wherein, is a trainable parameter matrix, , is an attention dimension; Based on the query and key matrices, a local attention neighborhood is defined using a sparse mask function to calculate the sparse attention weights, and the sparse mask function is defined as follows: ， wherein, represents the sparse neighborhood index set corresponding to the i-th query vector; and represents the sparse neighborhood index set corresponding to the i-th query vector; and According to the sparse attention weight performing weighted sum on the value matrix to obtain the fusion feature, denoted as: ， wherein, is the fused cross-textual semantic feature representation, ; For all The splicing is performed to obtain Linear layer mapping and normalization processing are performed on the splicing result to obtain the fused feature vector, denoted as: ， wherein is an output transform matrix, , is a layer normalization function.
6. The method of claim 5, wherein the method further comprises: The feedforward neural network module performs nonlinear feature enhancement and dimension transformation processing on the input data, and the specific steps of obtaining the enhanced feature include: A two-layer linear transformation and a nonlinear activation function combination are used to map the input data position by position, and the calculation process is as follows: ， wherein, , is a trainable parameter matrix, , is a bias term, , is a bias term, , , is a hidden dimension of the feed-forward layer, is a linear rectifier function; right By performing residual connections and layer normalization operations, the enhanced features are obtained, represented as follows: 。 7. The method of claim 6, wherein the method further comprises: The attention feature dimension reduction module extracts the global semantic information in the enhanced feature to obtain a compressed semantic representation vector, and the specific steps include: On enhancing features A linear transformation is performed to generate a learnable attention weight vector, denoted as: ， wherein, is a trainable weight vector, , is a bias term, ; The enhanced feature is weighted and summed based on a learnable attention weight vector and subjected to layer normalization processing to generate a compressed semantic representation vector, which is represented as: ， ， wherein, represents a feature vector of the th position in the enhanced feature, , is the compressed semantic representation vector.
8. The method of claim 7, wherein the method further comprises: The classifier module performs feature mapping and classification based on the compressed semantic representation vector and outputs the predicted result of the paper innovation category, and the specific steps include: The compressed semantic representation vector is subjected to feature mapping through a fully connected linear layer in the classifier module to generate a classification score vector, which is represented as: ， wherein, is a trainable weight matrix, , is a bias term, , is the total number of innovation categories for the paper. The classification score vector is obtained by using the Softmax function. The mapping to a class probability distribution is expressed as: ， wherein, denotes the predicted probability for each class, satisfying ; The final predicted result of the paper innovation category is: ， wherein, denotes the predicted paper novelty class label, .