LLM-based graduation design thesis intelligent evaluation system

Abstract

The invention discloses an LLM-based graduation design thesis intelligent evaluation system. The system comprises a data input module, a preprocessing module, an LLM evaluation module and an evaluation result output module. Text data of graduation papers are input into the data input module, the data input module is connected with the preprocessing module, the text data, processed by the preprocessing module, of the graduation papers are input into the LLM evaluation module, and the evaluation result output module is connected with the LLM evaluation module. According to the system, a large language model is utilized to perform multi-dimensional evaluation on graduation theses submitted by students, including but not limited to content quality, logic structure, language expression, innovation, academic specification and the like. According to the LLM-based undergraduate thesis evaluation system, the LLM can adapt to thesis evaluation requirements in different subject fields through a fine adjustment mode.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing (NLP) and educational assessment technology, and belongs to the field of artificial intelligence (AI) technology, and in particular to an intelligent evaluation system for graduation thesis based on a large language model (LLM). Background Art

[0002] With the rapid development of artificial intelligence technology, especially the emergence of Large Language Models (LLM), text processing and natural language understanding capabilities have been significantly improved. In the field of higher education, the quality evaluation of undergraduate theses has always been a complex and time-consuming process. Traditional manual review not only consumes a lot of human resources, but is also easily affected by subjective factors. The undergraduate thesis evaluation system based on LLM aims to use advanced natural language processing technology to achieve automated evaluation of graduation theses and improve review efficiency and objectivity. Traditional undergraduate thesis evaluation systems often rely on manual review, which requires a lot of human resources and is easily affected by subjective factors; while the expert system-based evaluation system relies on traditional technologies such as feature extraction of thesis texts, and cannot effectively automatically extract, identify and evaluate thesis.

[0003] Natural language processing is an important direction in the field of artificial intelligence, which aims to enable computers to understand and generate human natural language. It covers multiple tasks such as text classification, sentiment analysis, machine translation, question-answering systems, etc.

[0004] As a cutting-edge technology in the field of NLP, LLM can understand and generate complex natural language texts, providing strong technical support for the automatic evaluation of undergraduate theses. Through LLM, the system can automatically analyze the content, structure, language quality and other aspects of the thesis and give a comprehensive and objective evaluation.

[0005] Educational evaluation is a value judgment on the educational process and its results. It aims to make an objective and scientific evaluation of the quality and effectiveness of education by collecting information and analyzing data.

[0006] The undergraduate thesis evaluation system based on LLM is essentially an innovative application of educational evaluation tools. It uses the powerful capabilities of LLM to achieve automated and intelligent evaluation of graduation theses, greatly improving the evaluation efficiency and accuracy and reducing the burden of manual evaluation. Summary of the invention

[0007] The technical purpose of the present invention is to propose an undergraduate graduation thesis evaluation system based on LLM, which uses a large language model to evaluate the graduation thesis submitted by students in multiple dimensions, including but not limited to content quality, logical structure, language expression, innovation and academic norms. The undergraduate graduation thesis evaluation system based on LLM can adapt to the thesis evaluation needs of different disciplines through fine-tuning.

[0008] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is an intelligent evaluation system for graduation thesis based on LLM, which includes a data input module, a preprocessing module, an LLM evaluation module, and an evaluation result output module; the text data of the graduation thesis is input into the data input module, the data input module is connected with the preprocessing module, the text data of the graduation thesis processed by the preprocessing module is input into the LLM evaluation module, and the evaluation result output module is connected with the LLM evaluation module; Furthermore, the data input module is used to process the original manuscript of the graduation thesis into chapter-by-chapter plain text that can be input into the LLM. This module mainly uses technologies such as PDF file parsing and Word file parsing. The preprocessing module is one of the key modules of this system. It reads the results (section text) processed by the data input module and outputs a multi-angle summary that is conducive to the final LLM evaluation. The preprocessing module is a key component of the system. It is responsible for processing the section text processed by the data input module to generate a multi-angle summary to support the evaluation of the large language model (LLM). The module first reads the input data, cleans and standardizes it, then extracts the content of each chapter, and identifies key information such as the problem that the paper attempts to solve, related research and experimental design. Through structured summarization, the module integrates the extracted information into fluent paragraphs and outputs it in a format suitable for subsequent processing. Finally, the module undergoes functional testing and performance optimization to ensure its accuracy and efficiency, thereby providing a solid foundation for the final research results. The preprocessing module has a built-in general question library for academic papers and a classification question library suitable for graduation theses in various disciplines; the question-answering framework calls a general pre-trained large model (such as Qianwen, etc.), inputs the paper, and asks questions about the above questions separately to form a multi-angle summary and output it to the next module.

[0009] The LLM evaluation module is the core module of this system. In the training stage, based on the expert evaluation database of graduation theses accumulated over several years, previous graduation theses are input into the pre-training module. The technical implementation process of the evaluation module includes first receiving the multi-angle summary and original paper text output by the pre-processing module to generate professional evaluation. Through the instruction fine-tuning (SFT) technology, the multi-angle summary is combined with the expert evaluation to train the large language model (LLM) to improve its evaluation ability. The fine-tuned expert evaluation LLM is deployed as an inference service, which receives input in real time and generates structured evaluation data (JSON format). In the inference stage, a prompt containing a multi-angle summary and the paper is constructed, requiring the model to focus on content accuracy, research innovation, methodological rigor, literature review completeness and logical coherence for evaluation and scoring. Finally, the evaluation data output by the module provides users with comprehensive feedback on the quality of the paper to ensure the high quality and reliability of the evaluation results.

[0010] Finally, the evaluation data is processed through structured analysis by the output module. The specific form includes converting the evaluation data generated by LLM into a standardized format, such as JSON or XML, for subsequent processing and analysis. This process uses data analysis technology to classify and organize information such as evaluation dimensions, scores, and comments to ensure a clear structure and easy understanding and use. On this basis, rule verification is performed to ensure that the output data conforms to the preset logic and format specifications, such as checking whether the scoring range is reasonable and whether the evaluation content is complete. After rule verification, the evaluation results will be presented in a graphical form, including the use of data visualization tools to generate charts, such as radar charts, bar charts, or pie charts, to intuitively display the scores of each evaluation dimension and the overall evaluation results. This graphical output not only improves the readability of the data, but also facilitates users to quickly identify the advantages and disadvantages of the paper, providing effective guidance for subsequent improvements.

[0011] Compared with the prior art, the present invention utilizes the powerful text processing capability of LLM to significantly improve the efficiency of graduation thesis evaluation and reduce the workload of manual review. Through unified evaluation standards and algorithms, the influence of subjective factors on thesis evaluation results is reduced, ensuring the fairness of the evaluation. The system can comprehensively evaluate the thesis from multiple dimensions, provide detailed feedback and improvement suggestions, and help students improve the quality of their thesis. Through pre-training and fine-tuning, the system can adapt to the evaluation needs of different disciplines and has a wide range of applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flow chart for implementing the present invention. DETAILED DESCRIPTION

[0013] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0014] Data input module: responsible for receiving the text data of graduation thesis submitted by students.

[0015] Preprocessing module: perform preliminary processing on the input papers; LLM evaluation module: Use the pre-trained large-scale language model to perform multi-dimensional evaluation on the pre-processed text.

[0016] Evaluation result output module: Generates a detailed evaluation report, including scores for each dimension and suggestions for improvement.

[0017] Pre-training and fine-tuning Pre-training: Omitted. You should directly use a Chinese pre-trained language model with a higher score, such as Qianwen.

[0018] Fine-tuning: Based on the accumulated expert database, a high-quality instruction fine-tuning dataset is formed to accurately fine-tune the language model.

[0019] Evaluation Dimensions Content quality: The richness and accuracy of the content of the paper are assessed by analyzing aspects such as the relevance of the topic, the amount of information, and the depth of the content.

[0020] Logical structure: Check the overall structure of the paper, the logical relationship between paragraphs, and the coherence of the argument.

[0021] Language: The paper is assessed for its language fluency, use of technical terminology, and grammatical correctness.

[0022] Innovation: Evaluate the originality and innovation of the paper by comparing it with existing literature.

[0023] Academic standards: Check citation format, completeness of references, academic integrity, etc.

[0024] User Interaction Student side: Students can submit graduation theses through the system and receive a detailed evaluation report within the specified time.

[0025] Teacher side: Teachers can view the evaluation report generated by the system and conduct manual review and supplementary evaluation as needed.

[0026] Management side: School administrators can use the system to conduct statistical analysis of overall evaluation data to understand the overall quality of student papers.

[0027] Example: This system uses multi-level processing (preprocessing) to conduct preliminary multi-angle parsing and analysis of the thesis, and through LLM fine-tuning, it makes full use of expert scoring in previous years to train knowledge into a dedicated evaluation model, and ultimately can more accurately evaluate graduation thesis.

[0028] For example, if we input a paper, "Vision LLaMA: A Unified LLaMA Backbone for Vision Tasks", the preprocessing module will process it into a multi-angle problem: Based on the above preprocessed content and the original text of the paper chapters, the LLM evaluation module conducts multi-angle expert evaluation and scoring of the paper, and finally outputs it to the user interface.

Claims

1. The intelligent evaluation system for graduation thesis based on LLM is characterized by: It includes a data input module, a preprocessing module, an LLM evaluation module, and an evaluation result output module; the text data of the graduation thesis is input into the data input module, the data input module is connected to the preprocessing module, the text data of the graduation thesis processed by the preprocessing module is input into the LLM evaluation module, and the evaluation result output module is connected to the LLM evaluation module.

2. According to the LLM-based graduation thesis intelligent evaluation system of claim 1, it is characterized by: The data input module is used to process the original manuscript of the graduation thesis into chapter-by-chapter plain text that can be input into LLM, using PDF file parsing and Word file parsing methods.

3. The intelligent evaluation system for graduation thesis based on LLM according to claim 1 is characterized in that: The pre-processing module reads the results processed by the data input module and outputs them into a multi-angle summary that is conducive to the final LLM evaluation; The preprocessing module processes the chapter text processed by the data input module to generate a multi-angle summary to support the evaluation of the large language model LLM; The preprocessing module first reads the input data, cleans and standardizes it, then extracts the content of each chapter and identifies the problems that the paper attempts to solve, related research, and experimental design; it integrates the extracted information into fluent paragraphs and outputs it in a format suitable for subsequent processing; finally, it undergoes functional testing and performance optimization; The preprocessing module has a built-in general question library for academic papers and a classification question library suitable for graduation theses in various disciplines. The question-answering framework calls a general pre-trained large model, inputs the paper, and asks the above questions separately to form a multi-angle summary and output.

4. The intelligent evaluation system for graduation thesis based on LLM according to claim 1 is characterized in that: During the training phase, the LLM evaluation module inputs past graduation theses into the pre-training module based on the expert evaluation database of graduation theses accumulated over several years. The technical implementation process of the evaluation module includes first receiving the multi-angle summary and original thesis text output by the pre-processing module to generate professional evaluations; Fine-tune the SFT technology through instructions, combine multi-angle summarization with expert evaluation, and train the large language model LLM to improve its evaluation ability; The fine-tuned expert evaluation LLM is deployed as an inference service, which receives input in real time and generates structured evaluation data. In the inference stage, a prompt containing a multi-angle summary and thesis is constructed, requiring the model to focus on content accuracy, research innovation, methodological rigor, literature review completeness, and logical coherence for evaluation and scoring. Finally, the evaluation data output by the module provides users with comprehensive feedback on paper quality, ensuring the high quality and reliability of the evaluation results.

5. The intelligent evaluation system for graduation thesis based on LLM according to claim 1 is characterized in that: The evaluation data is processed through structured analysis through the output module. The specific forms include converting the evaluation data generated by LLM into a standardized format; using data analysis technology to classify and organize the evaluation dimensions, scores, and comment information, and perform rule verification to ensure that the output data conforms to the preset logic and format specifications; after rule verification, the evaluation results will be presented in a graphical form, including the use of data visualization tools to generate charts.

Images