A process evaluation system for academic paper quality based on a large language model

Abstract

The present invention relates to the field of academic paper evaluation technology, and in particular to an academic paper quality process evaluation system based on a large language model, comprising an evaluation index collection module, a paper feature extraction unit, an evaluation generation module, an evaluation system establishment module, a writing guidance scoring module and an evaluation system optimization module. The writing guidance scoring module is used to analyze paper features in the writing process, and to guide the author through the correlation mined by the evaluation generation module, and to monitor newly added paper features according to the paper feature extraction unit to judge the importance of the added paper features. After the paper scoring system is established, the writing guidance unit analyzes the paper features that are meaningful in the writing process, and at this time, guides the paper according to the association rules in the dynamic evaluation unit, and then monitors the newly added paper features through the paper feature extraction unit, and analyzes the importance of the newly added paper features, so as to provide the author with writing direction.

Description

Technical Field

[0001] The present invention relates to the technical field of academic paper evaluation, and in particular to a process evaluation system for academic paper quality based on a large language model. Background Art

[0002] Research frontiers are the most advanced, latest and most promising research topics or research fields in scientific and technological research, representing the difficulties, hot spots and development trends of scientific and technological development. At present, domestic and foreign scholars have conducted a lot of research on the evaluation of cutting-edge technologies and constructed different types of evaluation and method models. Among them, peer review and bibliometrics are two widely used methods in academic competitiveness assessment. The evaluation of a scientific frontier technology is generally based on peer review, and some metrological indicators are used for evaluation, such as awards, number of published papers, number of citations, H index, etc.

[0003] The process of paper writing involves multiple links, each of which has an important impact on the quality of the final paper. At present, the existing systems can only provide authors with some grammatical hints for typos or paragraph defects during the process of paper writing. As a result, there is a lack of a systematic and standardized model to comprehensively evaluate the paper writing process, especially in the process of topic selection, title, abstract, body paragraphs and manuscript revision. In view of this, we propose a procedural evaluation system for academic paper quality based on a large language model. Summary of the invention

[0004] The purpose of the present invention is to provide a process evaluation system for academic paper quality based on a large language model to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides an academic paper quality process evaluation system based on a large language model, including an evaluation index collection module, a paper feature extraction unit, an evaluation generation module, an evaluation system establishment module, a writing guidance scoring module and an evaluation system optimization module.

[0006] The evaluation index collection module includes an evaluation index collection unit and a paper arrangement unit;

[0007] The evaluation index collection unit is used to use a web crawler to send an HTTP request to a server to request to obtain the paper evaluation score and paper text in a web page. The server responds to the request and returns the web page content. At this time, the crawler parses the returned HTML page and collects the paper evaluation score and paper text;

[0008] Among them, the paper evaluation result indicators include citation rate and download rate;

[0009] Citation rate refers to the ratio of the number of times a paper is cited by other research papers in a certain period to the total number of times the paper is cited in the same period. It reflects the influence and importance of the paper in the academic field. Papers that are widely cited are considered to have higher academic value and reference significance.

[0010] The download rate refers to the ratio of the number of times a paper is downloaded by the author in a specific period of time to the total number of times the paper is displayed or available for download in that period of time. The download rate reflects the degree of attention and demand for the paper.

[0011] The paper sorting unit is used to read two similar paper text files in the historical paper database, analyze the read paper content line by line, compare the different characters and paragraph structures between the two paper texts, and sort out the added, deleted and modified contents;

[0012] The origin of the paper text in the historical paper database: During the project development or research process, team members modify and iterate the paper many times. After each modification, the historical records are retained, and then the modified paper text is saved in the historical database.

[0013] The paper feature extraction unit is used to segment the paper text into sentences, further segment the sentences into words, mark the part of speech of each word, and then extract paper features according to the part of speech and the paper content;

[0014] And according to the changes of the paper features before and after the paper text is modified and the changes of the paper scoring results, the importance of the paper features in the paper text is judged;

[0015] Paper features include sentence-level features, paragraph-level features, and global features;

[0016] in:

[0017] Sentence-level features: extract sentence length, complexity, grammatical structure and other features;

[0018] Paragraph-level features: analyzing the coherence, logical structure, etc. of a paragraph;

[0019] Global features: consider the overall structure, format, citations, etc. of the paper;

[0020] The method for extracting paper features is as follows:

[0021] Split the paper text into several paper features, identify the part of speech of each paper feature, remove meaningless words, and calculate the frequency of paper features. Inverse document frequency with paper features Multiply them together to assess the importance of each paper feature in the paper text;

[0022] The sentence-level feature calculation formula is as follows:

[0023] ;

[0024] in, It is The characteristics of the word in the sentence, is the inverse document frequency of the paper feature;

[0025] The paragraph-level feature calculation formula is as follows:

[0026] ;

[0027] in, is the number of sentences in the paragraph;

[0028] The global feature calculation formula is as follows:

[0029] ;

[0030] in, is the number of paragraphs in the paper;

[0031] The calculation formula for the importance of paper features in the paper text is as follows:

[0032] Suppose we have a paper text , which contains both the pre- and post-modification versions and ;

[0033] For each feature , calculate the eigenvalues ​​before and after the modification:

[0034] ;

[0035] Calculate feature changes:

[0036] For each feature , calculate its change:

[0037] ;

[0038] Evaluating feature importance:

[0039] ;

[0040] in: It is a feature The importance score of is the change in the paper scoring results, It is the sum of all feature changes.

[0041] The evaluation generation module includes an evaluation relationship generation unit and a dynamic evaluation unit;

[0042] The evaluation relationship generating unit is used to receive the paper features in the paper feature extracting unit, and calculate the correlation coefficient between the plurality of paper features extracted by the paper feature extracting unit and the paper evaluation scores collected in the evaluation index collecting unit;

[0043] According to the calculated correlation coefficient value, the relationship between the two is judged. If the correlation coefficient is close to 1, it means that there is a positive correlation between the paper feature and the paper score, that is, the paper feature improves the overall paper score result; if it is close to -1, it means that there is a negative correlation, that is, the paper feature reduces the overall paper score result; if it is close to 0, it means that there is no significant linear relationship, that is, the paper feature has nothing to do with the overall paper score;

[0044] The calculation formula of the correlation coefficient is as follows:

[0045] ;

[0046] in, and They are paper characteristics and paper evaluation scores. Observations, and are the average values ​​of paper characteristics and paper evaluation scores, respectively;

[0047] The dynamic evaluation unit is used to mine the association rules between paper features and correlation coefficients at different stages;

[0048] In the process of writing a paper, the evaluation relationship generation unit scores the paper at the current stage as 10. When the new paper features appear in the process of continuing writing, the paper score increases, which means that the paper features appearing at this time are meaningful, thereby mining the association rules between paper features and correlation coefficients at different stages;

[0049] If feature A appears during the writing process, then the score B changes at the same time. The calculation formula is:

[0050] ;

[0051] The above formula is expressed as the ratio of the probability of paper score B appearing when paper feature A appears to the probability of paper score B appearing in all data. If the lift is greater than 1, it means that paper feature A has a positive meaning.

[0052] The evaluation system establishment module is used to establish a standardized paper scoring system according to the correlation coefficient between the paper characteristics and the paper scores calculated by the evaluation relationship generation unit;

[0053] The calculation formula for establishing the essay scoring system is as follows:

[0054] ;

[0055] in, For the The weight of the feature, For the The quantitative value of the feature;

[0056] Every The weight of the feature is determined by the correlation coefficient between the paper feature and the paper score calculated by the evaluation relationship generation unit. The higher the correlation coefficient, the stronger the linear relationship between the paper feature and the paper score, and therefore the higher the assigned weight.

[0057] The writing guidance unit is used to analyze the meaningful paper features in the writing process and provide writing guidance for the paper being written according to the association rules in the dynamic evaluation unit;

[0058] During the writing process, the paper scoring system established by the evaluation system establishment module continuously monitors the newly added paper features through the paper feature extraction unit, and analyzes the importance of the newly added paper features in the paper text;

[0059] When the correlation coefficient of the added paper feature in the evaluation relationship generation unit is greater than 0, it means that the newly added paper feature can effectively improve the overall quality of the paper. At this time, writing guidance is provided to the author according to the meaning of the newly added paper feature in the paper;

[0060] The following is a sample writing guide:

[0061] If the correlation coefficient > 0;

[0062] Guidance content: Emphasize the importance of experimental design, and suggest that authors describe the experimental methods, experimental steps, experimental conditions, etc. in detail to ensure the reliability and repeatability of the experiment;

[0063] If the correlation coefficient = 0;

[0064] Guidance content: Point out that the "experimental design" feature is not significantly correlated with the paper quality score, and suggest that authors consider whether they need to adjust the experimental design or find other relevant features;

[0065] Specific suggestions:

[0066] In the introduction, clearly state the purpose and importance of the experiment;

[0067] In the methods section, describe the experimental design in detail, including experimental materials, experimental methods, and data analysis methods;

[0068] Present the experimental data in the Results section and discuss the significance of the experimental results;

[0069] In the discussion section, the experimental results are analyzed and compared with existing studies;

[0070] The author feedback unit is used to submit the completed paper to the scoring system established by the evaluation system establishment module. The evaluation system establishment module scores the paper. After scoring, a user-friendly interface is established to receive author feedback. At this time, the paper scoring reason is generated according to the change of the correlation coefficient of the evaluation relationship generation unit. The paper scoring reason is fed back to the author through the user-friendly interface;

[0071] Establishing a corresponding evaluation of the correlation coefficient change of the evaluation relationship generating unit;

[0072] If the correlation coefficient is When the content of the paper is highly relevant to the evaluation criteria, the arguments are clear, the evidence is sufficient, and the structure is reasonable;

[0073] The feedback was: the paper was of high quality and the author was advised to maintain the existing writing style and structure;

[0074] If the correlation coefficient is If the paper is between 1 and 2, it means that the relevance between the content and the evaluation criteria is average, and there are problems such as unclear arguments, insufficient evidence, or unreasonable structure.

[0075] Feedback: The paper is of average quality. The author is advised to further optimize the arguments, evidence and structure.

[0076] If the correlation coefficient is When the number is between 0 and 1, it indicates that there is a certain degree of negative correlation between the content of the paper and the evaluation criteria, because the argument is contrary to the evaluation criteria, or the evidence is inconsistent with the facts;

[0077] Feedback: The paper is of low quality. The author is advised to review the arguments and evidence to ensure consistency with the evaluation criteria.

[0078] If the correlation coefficient is If the number is between 0 and 1, it means that the content of the paper is highly negatively correlated with the evaluation criteria, the argument is completely opposite to the evaluation criteria, and there are serious errors or logical loopholes in the argument;

[0079] Feedback: The quality of the paper is extremely low. It is recommended that the author rewrite the paper to ensure that the content of the paper is consistent with the evaluation criteria.

[0080] The text feature similarity calculation unit is used to output the author's feedback content on the scoring result of the evaluation system establishment module in the author feedback unit to the paper feature extraction unit, and extract the text features of the author's feedback content through the paper feature extraction unit;

[0081] Then, the similarity between the author feedback text feature and the paper text feature extracted by the paper feature extraction unit is calculated;

[0082] The method for comparing the extracted feedback text features with the paper text features extracted by the paper feature extraction unit is as follows:

[0083] ;

[0084] in, is the extracted feedback text features, is the paper text feature extracted by the paper feature extraction unit, It is The weight of the feature, It is The minimum matching degree of a feature in two feature sets;

[0085] The scoring system optimization unit is used to compare the similarity calculated by the text feature similarity calculation unit with a set similarity threshold;

[0086] If the similarity is lower than the similarity threshold, the correlation coefficient in the evaluation relationship generation unit is optimized, thereby optimizing the paper scoring system in the evaluation system establishment module;

[0087] The similarity comparison method is as follows:

[0088] like ,in, and are similarity and similarity threshold, respectively, which means that the similarity is lower than the similarity threshold, and at this time, the correlation coefficient in the evaluation relationship generation unit is optimized;

[0089] The optimization method is as follows:

[0090] ;

[0091] in, are the updated model parameters, are the old model parameters, is the parameter update amount, determined by the gradient descent algorithm.

[0092] The parameter update amount is determined by the gradient descent algorithm, and the method of calculating the parameter update amount specifically includes:

[0093] The formula for calculating the parameter update amount in the gradient descent algorithm is as follows:

[0094] ;

[0095] in, Representation parameters The update amount, is the learning rate, which determines the size of the parameter update, is the loss function Parameters The gradient of the loss function is the gradient of the parameter rate of change in direction;

[0096] The steps for calculating the above parameter updates by the gradient descent algorithm are as follows:

[0097] Step 1: First, set an initial value for the parameters in the model;

[0098] Step 2: For a given parameter value, calculate the loss function , the loss function is used to measure the difference between the model prediction value and the true value;

[0099] Step 3: Calculate the gradient of the loss function for each parameter , which means that under the current parameter value, the loss function is in the parameter Trends in direction;

[0100] Step 4: According to the gradient descent formula, use negative gradient to update the parameters. Negative gradient means moving in the opposite direction of the gradient, which is the direction to reduce the loss function.

[0101] Step 5: Repeat steps 2 to 4 until the stopping condition is met. The stopping condition includes any of the following:

[0102] The value of the loss function is below a certain threshold;

[0103] If the parameter update amount is less than a certain threshold, it means that the parameter is close to the optimal value;

[0104] Reach the preset number of iterations.

[0105] The evaluation index collection module collects the paper text and paper scores through a web crawler, the paper text includes a first draft paper and a final draft paper, the paper scores are the scores corresponding to the first draft and the final draft respectively, and analyzes the differences between the first draft paper and the final draft paper, the differences are the different characters and paragraph structures between the first draft paper and the final draft paper;

[0106] The evaluation generation module includes an evaluation relationship generation unit and a dynamic evaluation unit, wherein:

[0107] The paper feature extraction unit uses the TF-IDF method to extract paper features in the paper text. The paper features are extracted according to the part of speech and the paper content, and the importance of the paper features is judged according to the differences in the paper and the changes in the paper score before and after the modification.

[0108] The evaluation relationship generation unit calculates the correlation coefficient between the paper features and the difference points of the paper scores through a phase relationship algorithm. The dynamic evaluation unit is used to explore the correlation between the paper features and the correlation coefficients in papers at different stages. The evaluation system establishment module establishes a paper scoring system through the correlation coefficient calculated by the evaluation generation module. After the paper evaluation system is generated, the correlation explored by the dynamic evaluation unit provides guidance for the paper being written.

[0109] The writing guidance and scoring module includes a writing guidance unit and an author feedback unit, and the evaluation system optimization module includes a text feature similarity calculation unit and a scoring system optimization unit;

[0110] The writing guidance unit is used to analyze the paper features during the writing process, and guide the author through the correlation mined by the evaluation generation module, and monitor the newly added paper features according to the paper feature extraction unit to determine the importance of the added paper features; the author feedback unit is used to establish a user-friendly interface, feedback the reasons for the paper scoring to the paper author, and receive the author's feedback;

[0111] The text feature similarity calculation unit is used to extract text features with the same content as the author's feedback through the paper feature extraction unit, and calculate the similarity between the feedback text features and the original text features; the scoring system optimization unit is used to compare the text features of the author's feedback with the set similarity threshold. If the similarity is lower than the similarity threshold, the correlation coefficient in the evaluation relationship generation unit is optimized through the similarity to achieve the purpose of optimizing the paper scoring system.

[0112] Compared with the prior art, the present invention has the following beneficial effects:

[0113] In the academic paper quality process evaluation system based on the large language model, the evaluation index collection module collects the existing paper text, paper score, paper text before and after the paper modification, and then judges the importance of paper features in the paper text according to the changes in paper features before and after the paper text modification and the changes in paper scoring results. The evaluation relationship generation unit calculates the correlation coefficient between paper features and paper scores, and the dynamic evaluation unit mines the correlation according to the correlation coefficient and paper features at different stages, and thus the evaluation system establishment module establishes a paper scoring system.

[0114] After the paper scoring system is established, the writing guidance unit analyzes the paper features that are meaningful in the writing process. At this time, the paper is guided according to the association rules in the dynamic evaluation unit. Then, the paper feature extraction unit monitors the newly added paper features and analyzes the importance of the newly added paper features, so as to provide the author with writing direction.

[0115] After the paper scoring system is established through the evaluation system establishment module to score the paper, the author feedback unit scores the paper through the evaluation system establishment module. After the scoring, a user-friendly interface is established to receive author feedback, and the scoring content is determined by the correlation coefficient of the evaluation relationship generation unit;

[0116] The text feature similarity calculation unit calculates the similarity between the paper text features and the user feedback text features based on the text features in the user feedback extracted by the paper feature extraction unit, and optimizes the paper scoring system established by the evaluation system establishment module by comparing the similarity with the set similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 It is a flow chart of the working principle of the present invention;

[0118] Figure 2 This is a workflow diagram of the writing guidance scoring module and the evaluation system optimization module of the present invention;

[0119] Figure 3 This is a flow chart of the working principle of the writing guidance scoring module of the present invention;

[0120] Figure 4 The figure is a flow chart of the working principle of the scoring system optimization unit of the present invention.

[0121] The meaning of each number in the figure is:

[0122] 100. Evaluation index collection module; 110. Evaluation index collection unit; 120. Paper arrangement unit; 200. Paper feature extraction unit; 300. Evaluation generation module; 310. Evaluation relationship generation unit; 320. Dynamic evaluation unit; 400. Evaluation system establishment module; 500. Writing guidance and scoring module; 510. Writing guidance unit; 520. Author feedback unit; 600. Evaluation system optimization module; 610. Text feature similarity calculation unit; 620. Scoring system optimization unit. DETAILED DESCRIPTION

[0123] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0124] Research frontiers are the most advanced, latest, and most promising research topics or research fields in scientific and technological research. They represent the difficulties, hot spots, and development trends of scientific and technological development. At present, domestic and foreign scholars have conducted a lot of research on the evaluation of cutting-edge technologies and constructed different types of evaluation and method models. Among them, peer review and bibliometrics are two widely used methods in academic competitiveness assessment. The evaluation of a scientific frontier technology is generally based on peer review, and some metrological indicators are used for evaluation, such as awards, number of published papers, number of citations, H index, etc. Therefore, please refer to Figure 1-Figure 4 As shown, the academic paper quality process evaluation system based on the large language model includes an evaluation index collection module 100, a paper feature extraction unit 200, an evaluation generation module 300, an evaluation system establishment module 400, a writing guidance scoring module 500 and an evaluation system optimization module 600.

[0125] The evaluation index collection module 100 includes an evaluation index collection unit 110 and a paper arrangement unit 120;

[0126] The evaluation index collection unit 110 is used to use a web crawler to send an HTTP request to a server to request to obtain the paper evaluation score and paper text in a web page. The server responds to the request and returns the web page content. At this time, the crawler parses the returned HTML page and collects the paper evaluation score and paper text.

[0127] Among them, the paper evaluation result indicators include citation rate and download rate;

[0128] Citation rate refers to the ratio of the number of times a paper is cited by other research papers in a certain period to the total number of times the paper is cited in the same period. It reflects the influence and importance of the paper in the academic field. Papers that are widely cited are considered to have higher academic value and reference significance.

[0129] The download rate refers to the ratio of the number of times a paper is downloaded by the author in a specific period of time to the total number of times the paper is displayed or available for download in that period of time. The download rate reflects the degree of attention and demand for the paper.

[0130] The paper sorting unit 120 is used to read two similar paper text files in the historical paper database, analyze the read paper content line by line, compare the different characters and paragraph structures between the two paper texts, and sort out the added, deleted and modified contents;

[0131] The origin of the paper text in the historical paper database: During the project development or research process, team members modify and iterate the paper many times. After each modification, the historical records are retained, and then the modified paper text is saved in the historical database.

[0132] The paper feature extraction unit 200 is used to segment the paper text into sentences, further segment the sentences into words, mark the part of speech of each word, and then extract paper features according to the part of speech and the paper content;

[0133] And according to the changes of the paper features before and after the paper text is modified and the changes of the paper scoring results, the importance of the paper features in the paper text is judged;

[0134] Paper features include sentence-level features, paragraph-level features, and global features;

[0135] in:

[0136] Sentence-level features: extract sentence length, complexity, grammatical structure and other features;

[0137] Paragraph-level features: analyzing the coherence, logical structure, etc. of a paragraph;

[0138] Global features: consider the overall structure, format, citations, etc. of the paper;

[0139] The method for extracting paper features is as follows:

[0140] Split the paper text into several paper features, identify the part of speech of each paper feature, remove meaningless words, and calculate the frequency of paper features. Inverse document frequency with paper features Multiply them together to assess the importance of each paper feature in the paper text;

[0141] The sentence-level feature calculation formula is as follows:

[0142] ;

[0143] in, It is The characteristics of the word in the sentence, is the inverse document frequency of the paper feature;

[0144] The paragraph-level feature calculation formula is as follows:

[0145] ;

[0146] in, is the number of sentences in the paragraph;

[0147] The global feature calculation formula is as follows:

[0148] ;

[0149] in, is the number of paragraphs in the paper;

[0150] The calculation formula for the importance of paper features in the paper text is as follows:

[0151] Suppose we have a paper text , which contains both the pre- and post-modification versions and ;

[0152] For each feature , calculate the eigenvalues ​​before and after the modification:

[0153] ;

[0154] Calculate feature changes:

[0155] For each feature , calculate its change:

[0156] ;

[0157] Evaluating feature importance:

[0158] ;

[0159] in:

[0160] It is a feature The importance score of is the change in the paper scoring results, It is the sum of all feature changes.

[0161] The evaluation generation module 300 includes an evaluation relationship generation unit 310 and a dynamic evaluation unit 320;

[0162] The evaluation relationship generating unit 310 is used to receive the paper features in the paper feature extracting unit 200, and calculate the correlation coefficient between the several paper features extracted by the paper feature extracting unit 200 and the paper evaluation scores collected by the evaluation index collecting unit 110;

[0163] According to the calculated correlation coefficient value, the relationship between the two is judged. If the correlation coefficient is close to 1, it means that there is a positive correlation between the paper feature and the paper score, that is, the paper feature improves the overall paper score result; if it is close to -1, it means that there is a negative correlation, that is, the paper feature reduces the overall paper score result; if it is close to 0, it means that there is no significant linear relationship, that is, the paper feature has nothing to do with the overall paper score;

[0164] The calculation formula of the correlation coefficient is as follows:

[0165] ;

[0166] in, and They are paper characteristics and paper evaluation scores. Observations, and are the average values ​​of paper characteristics and paper evaluation scores, respectively;

[0167] The dynamic evaluation unit 320 is used to mine the association rules between the paper features and correlation coefficients at different stages;

[0168] In the process of writing a paper, the current stage paper is scored as 10 by the evaluation relationship generation unit 310. When the paper score increases during the writing process, the newly appeared paper features increase the paper score, which means that the paper features appearing at this time are meaningful, thereby mining the association rules between paper features and correlation coefficients at different stages;

[0169] If feature A appears during the writing process, then the score B changes at the same time. The calculation formula is:

[0170] ;

[0171] The above formula is expressed as the ratio of the probability of paper score B appearing when paper feature A appears to the probability of paper score B appearing in all data. If the lift is greater than 1, it means that paper feature A has a positive meaning.

[0172] The evaluation system establishment module 400 is used to establish a standardized paper scoring system according to the correlation coefficient between the paper features and the paper scores calculated by the evaluation relationship generation unit 310;

[0173] The calculation formula for establishing the essay scoring system is as follows:

[0174] ;

[0175] in, For the The weight of the feature, For the The quantitative value of the feature;

[0176] Every The weight of the feature is determined by the correlation coefficient between the paper feature and the paper score calculated by the evaluation relationship generation unit 310. The higher the correlation coefficient, the stronger the linear relationship between the paper feature and the paper score, and thus the higher the assigned weight.

[0177] The writing guidance unit 510 is used to analyze the meaningful paper features in the writing process and provide writing guidance for the paper being written according to the association rules in the dynamic evaluation unit 320;

[0178] During the writing process, the paper scoring system established by the evaluation system establishment module 400 continuously monitors the newly added paper features through the paper feature extraction unit 200, and analyzes the importance of the newly added paper features in the paper text;

[0179] When the correlation coefficient of the added paper feature in the evaluation relationship generation unit 310 is greater than 0, it means that the newly added paper feature can effectively improve the overall quality of the paper. At this time, writing guidance is provided to the author according to the meaning of the newly added paper feature in the paper.

[0180] The following is a sample writing guide:

[0181] If the correlation coefficient > 0;

[0182] Guidance content: Emphasize the importance of experimental design, and suggest that authors describe the experimental methods, experimental steps, experimental conditions, etc. in detail to ensure the reliability and repeatability of the experiment;

[0183] If the correlation coefficient = 0;

[0184] Guidance content: Point out that the "experimental design" feature is not significantly correlated with the paper quality score, and suggest that authors consider whether they need to adjust the experimental design or find other relevant features;

[0185] Specific suggestions:

[0186] In the introduction, clearly state the purpose and importance of the experiment;

[0187] In the methods section, describe the experimental design in detail, including experimental materials, experimental methods, and data analysis methods;

[0188] Present the experimental data in the Results section and discuss the significance of the experimental results;

[0189] In the discussion section, the experimental results are analyzed and compared with existing studies;

[0190] The author feedback unit 520 is used to submit the completed paper to the scoring system established by the evaluation system establishment module 400. The evaluation system establishment module 400 scores the paper. After the scoring, a user-friendly interface is established to receive author feedback. At this time, the paper scoring reason is generated according to the change of the correlation coefficient of the evaluation relationship generation unit 310. The paper scoring reason is fed back to the author through the user-friendly interface;

[0191] Establishing a corresponding evaluation of the correlation coefficient change of the evaluation relationship generating unit 310;

[0192] If the correlation coefficient is When the content of the paper is highly relevant to the evaluation criteria, the arguments are clear, the evidence is sufficient, and the structure is reasonable;

[0193] The feedback was: the paper was of high quality and the author was advised to maintain the existing writing style and structure;

[0194] If the correlation coefficient is If the paper is between 1 and 2, it means that the relevance between the content and the evaluation criteria is average, and there are problems such as unclear arguments, insufficient evidence, or unreasonable structure.

[0195] Feedback: The paper is of average quality. The author is advised to further optimize the arguments, evidence and structure.

[0196] If the correlation coefficient is When the number is between 0 and 1, it indicates that there is a certain degree of negative correlation between the content of the paper and the evaluation criteria, because the argument is contrary to the evaluation criteria, or the evidence is inconsistent with the facts;

[0197] Feedback: The paper is of low quality. The author is advised to review the arguments and evidence to ensure consistency with the evaluation criteria.

[0198] If the correlation coefficient is If the number is between 0 and 1, it means that the content of the paper is highly negatively correlated with the evaluation criteria, the argument is completely opposite to the evaluation criteria, and there are serious errors or logical loopholes in the argument;

[0199] Feedback: The quality of the paper is extremely low. It is recommended that the author rewrite the paper to ensure that the content of the paper is consistent with the evaluation criteria.

[0200] The text feature similarity calculation unit 610 is used to output the author's feedback content on the scoring result of the evaluation system establishment module 400 in the author feedback unit 520 to the paper feature extraction unit 200, and extract the text features of the author's feedback content through the paper feature extraction unit 200;

[0201] Then, the similarity between the author feedback text feature and the paper text feature extracted by the paper feature extraction unit 200 is calculated;

[0202] The method for comparing the extracted feedback text features with the paper text features extracted by the paper feature extraction unit 200 is as follows:

[0203] ;

[0204] in, is the extracted feedback text features, is the paper text feature extracted by the paper feature extraction unit 200, It is The weight of the feature, It is The minimum matching degree of a feature in two feature sets;

[0205] The scoring system optimization unit 620 is used to compare the similarity calculated by the text feature similarity calculation unit 610 with a set similarity threshold;

[0206] If the similarity is lower than the similarity threshold, the correlation coefficient in the evaluation relationship generation unit 310 is optimized, thereby optimizing the paper scoring system in the evaluation system establishment module 400;

[0207] The similarity comparison method is as follows:

[0208] like ,in, and are similarity and similarity threshold, respectively, which means that the similarity is lower than the similarity threshold, and the correlation coefficient in the evaluation relationship generating unit 310 is optimized at this time;

[0209] The optimization method is as follows:

[0210] ;

[0211] in, are the updated model parameters, are the old model parameters, is the parameter update amount, determined by the gradient descent algorithm.

[0212] The parameter update amount is determined by the gradient descent algorithm. The method of calculating the parameter update amount includes:

[0213] The formula for calculating the parameter update amount in the gradient descent algorithm is as follows:

[0214] ;

[0215] in, Representation parameters The update amount, is the learning rate, which determines the size of the parameter update, is the loss function Parameters The gradient of the loss function is the gradient of the parameter rate of change in direction;

[0216] The steps for calculating the above parameter updates by the gradient descent algorithm are as follows:

[0217] Step 1: First, set an initial value for the parameters in the model;

[0218] Step 2: For a given parameter value, calculate the loss function , the loss function is used to measure the difference between the model prediction value and the true value;

[0219] Step 3: Calculate the gradient of the loss function for each parameter , which means that under the current parameter value, the loss function is in the parameter Trends in direction;

[0220] Step 4: According to the gradient descent formula, use negative gradient to update the parameters. Negative gradient means moving in the opposite direction of the gradient, which is the direction to reduce the loss function.

[0221] Step 5: Repeat steps 2 to 4 until the stopping condition is met. The stopping condition includes any of the following:

[0222] The value of the loss function is below a certain threshold;

[0223] If the parameter update amount is less than a certain threshold, it means that the parameter is close to the optimal value;

[0224] Reach the preset number of iterations.

[0225] The evaluation index collection module 100 collects the paper text and paper scores through a web crawler, the paper text includes a first draft paper and a final draft paper, the paper scores are scores corresponding to the first draft and the final draft, and analyzes the differences between the first draft paper and the final draft paper, the differences are different characters and paragraph structures between the first draft paper and the final draft paper;

[0226] The evaluation generation module 300 includes an evaluation relationship generation unit 310 and a dynamic evaluation unit 320, wherein:

[0227] The paper feature extraction unit 200 uses the TF-IDF method to extract paper features from the paper text. The paper features are extracted based on the part of speech and the paper content, and the importance of the paper features is judged based on the differences in the paper and the changes in the paper scores before and after modification.

[0228] The evaluation relationship generation unit 310 calculates the correlation coefficient between the paper features and the difference points of the paper scores through a phase relationship algorithm. The dynamic evaluation unit 320 is used to explore the correlation between the paper features and the correlation coefficients in the papers at different stages. The evaluation system establishment module 400 establishes a paper scoring system through the correlation coefficient calculated by the evaluation generation module 300. After the paper evaluation system is generated, the correlation mined by the dynamic evaluation unit 320 provides guidance for the paper being written.

[0229] The writing guidance and scoring module 500 includes a writing guidance unit 510 and an author feedback unit 520, and the evaluation system optimization module 600 includes a text feature similarity calculation unit 610 and a scoring system optimization unit 620;

[0230] The writing guidance unit 510 is used to analyze the paper features in the writing process, and guide the author through the correlation mined by the evaluation generation module 300, and monitor the newly added paper features according to the paper feature extraction unit 200 to determine the importance of the added paper features; the author feedback unit 520 is used to establish a user-friendly interface, feedback the reasons for the paper rating to the paper author, and receive the author's feedback;

[0231] The text feature similarity calculation unit 610 is used to extract text features with the same content as the author's feedback through the paper feature extraction unit 200, and calculate the similarity between the feedback text features and the original text features; the scoring system optimization unit 620 is used to compare the text features of the author's feedback with the set similarity threshold. If the similarity is lower than the similarity threshold, the correlation coefficient in the evaluation relationship generation unit 310 is optimized through the similarity to achieve the purpose of optimizing the paper scoring system.

[0232] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The academic paper quality process evaluation system based on the large language model is characterized by: It includes an evaluation index collection module (100), a paper feature extraction unit (200), an evaluation generation module (300), an evaluation system establishment module (400), a writing guidance scoring module (500) and an evaluation system optimization module (600); The evaluation index collection module (100) collects the paper text and paper scores through a web crawler, the paper text includes a first draft paper and a final draft paper, the paper scores are scores corresponding to the first draft and the final draft respectively, and analyzes the differences between the first draft paper and the final draft paper, the differences being different characters and paragraph structures between the first draft paper and the final draft paper; The evaluation generation module (300) comprises an evaluation relationship generation unit (310) and a dynamic evaluation unit (320), wherein: The paper feature extraction unit (200) extracts paper features from the paper text using the TF-IDF method, wherein the paper features are extracted based on the part of speech and the paper content, and the importance of the paper features is judged based on the paper differences and the changes in the paper scores before and after modification; The evaluation relationship generation unit (310) calculates the correlation coefficient between the paper characteristics and the paper score difference points through a phase relationship algorithm, the dynamic evaluation unit (320) is used to mine the correlation between the paper characteristics and the correlation coefficient in the papers at different stages, the evaluation system establishment module (400) establishes the paper scoring system through the correlation coefficient calculated by the evaluation generation module (300), and after the paper evaluation system is generated, the correlation mined by the dynamic evaluation unit (320) provides guidance for the paper being written; The evaluation relationship generation unit (310) is used to receive the paper features in the paper feature extraction unit (200), and calculate the correlation coefficient between the plurality of paper features extracted by the paper feature extraction unit (200) and the paper evaluation scores collected in the evaluation index collection module (100); According to the calculated correlation coefficient value, the relationship between the two is judged. If the correlation coefficient is close to , indicating that there is a positive correlation between the paper characteristics and the paper score, that is, the paper characteristics improve the overall paper score result; if it is close to , indicating that there is a negative correlation, that is, the paper feature reduces the score of the overall paper; if it is close to , indicating that there is no significant linear relationship, that is, the paper feature has nothing to do with the overall paper score; The dynamic evaluation unit (320) is used to mine association rules between paper features and correlation coefficients at different stages; In the process of writing a paper, the evaluation relationship generation unit (310) scores the paper at the current stage as 10. When the paper continues to be written, if new paper features increase the paper score, it means that the paper features appearing at this time are meaningful, thereby mining the association rules between paper features and correlation coefficients at different stages; The writing guidance scoring module (500) includes a writing guidance unit (510) and an author feedback unit (520); The writing guidance unit (510) is used to analyze the paper features that are meaningful during the writing process, and provide writing guidance for the paper being written according to the association rules in the dynamic evaluation unit (320); During the writing process, the paper scoring system established by the evaluation system establishment module (400) continuously monitors the newly added paper features through the paper feature extraction unit (200) and analyzes the importance of the newly added paper features in the paper text; The author feedback unit (520) is used to submit the completed paper to the scoring system established by the evaluation system establishment module (400). The evaluation system establishment module (400) scores the paper. After scoring, a user-friendly interface is established to receive author feedback. At this time, the paper scoring reason is generated according to the change of the correlation coefficient of the evaluation relationship generation unit (310), and the paper scoring reason is fed back to the author through the user-friendly interface.

2. The academic paper quality process evaluation system based on a large language model according to claim 1 is characterized by: The evaluation index collection module (100) comprises an evaluation index collection unit (110) and a paper sorting unit (120); The evaluation index collection unit (110) is used to use a web crawler to send an HTTP request to a server, requesting to obtain the paper evaluation score and paper text in a web page, the server responds to the request and returns the web page content, and the crawler parses the returned HTML page to collect the paper evaluation score and paper text, the collected paper text includes the first draft paper and the final draft paper, and the evaluation score includes the scores corresponding to the first draft paper and the final draft paper; The paper sorting unit (120) is used to read two similar paper text files in the historical paper database, analyze the read paper content line by line, compare the different characters and paragraph structures between the two paper texts, and sort out the added, deleted and modified contents.

3. The academic paper quality process evaluation system based on a large language model according to claim 1 is characterized by: The paper feature extraction unit (200) is used to segment the paper text into sentences, further segment the sentences into words, mark the part of speech of each word, and then extract paper features according to the part of speech and the paper content; And based on the changes in the paper features before and after the paper text is modified and the changes in the paper scoring results, the importance of the paper features in the paper text is judged.

4. The academic paper quality process evaluation system based on a large language model according to claim 1 is characterized by: The evaluation system establishment module (400) is used to establish a standardized paper scoring system based on the correlation coefficient between the paper characteristics and the paper scores calculated by the evaluation relationship generation unit (310).

5. The academic paper quality process evaluation system based on a large language model according to claim 1 is characterized by: The evaluation system optimization module (600) comprises a text feature similarity calculation unit (610) and a scoring system optimization unit (620); The text feature similarity calculation unit (610) is used to output the feedback content of the author in the writing guidance scoring module (500) on the scoring result of the evaluation system establishment module (400) to the paper feature extraction unit (200), and extract the text features of the author's feedback content through the paper feature extraction unit (200); Then, the similarity between the author's feedback text features and the paper text features extracted by the paper feature extraction unit (200) is calculated; The scoring system optimization unit (620) is used to compare the similarity calculated by the text feature similarity calculation unit (610) with a set similarity threshold; If the similarity is lower than the similarity threshold, the correlation coefficient in the evaluation relationship generation unit (310) is optimized, thereby optimizing the paper scoring system in the evaluation system establishment module (400); The similarity comparison method is as follows: like ,in, and are similarity and similarity threshold respectively, which means that the similarity is lower than the similarity threshold, and at this time, the correlation coefficient in the evaluation relationship generation unit (310) is optimized; The optimization method is as follows: ； in, are the updated model parameters, are the old model parameters, is the parameter update amount, determined by the gradient descent algorithm.

6. The academic paper quality process evaluation system based on large language model according to claim 5 is characterized by: The parameter update amount in the scoring system optimization unit (620) specifically includes: The formula for calculating the parameter update amount in the gradient descent algorithm is as follows: ； in, Representation parameters The update amount, is the learning rate, which determines the size of the parameter update, is the loss function Parameters The gradient of the loss function is the gradient of the parameter rate of change in direction; The steps for calculating the above parameter updates by the gradient descent algorithm are as follows: Step 1: First, set an initial value for the parameters in the model; Step 2: For a given parameter value, calculate the loss function , the loss function is used to measure the difference between the model prediction value and the true value; Step 3: Calculate the gradient of the loss function for each parameter , which means that under the current parameter value, the loss function is in the parameter Trends in direction; Step 4: According to the gradient descent formula, use negative gradient to update the parameters. Negative gradient means moving in the opposite direction of the gradient, which is the direction to reduce the loss function. Step 5: Repeat steps 2 to 4 until the stopping condition is met. The stopping condition includes any of the following: The value of the loss function is below a certain threshold; If the parameter update amount is less than a certain threshold, it means that the parameter is close to the optimal value; Reach the preset number of iterations.

Images