Composition scoring cue word optimization method based on large language model

Abstract

The invention provides a composition scoring cue word optimization method based on a large language model. The method comprises the following steps: obtaining composition topic information and a labeled composition sample set; constructing an initial scoring cue word according to the composition topic information, and evaluating the performance of the scoring cue word; using the current scoring cue word to guide the large language model to score each composition in the composition sample set; calculating a difference value between the large language model score and the manual annotation score; guiding the large language model to analyze the composition with the maximum difference value and obtain a score difference reason, so as to guide the large language model to modify the score cue words and generate candidate score cue words; evaluating the performance of the candidate scoring cue word, and comparing the performance of the candidate scoring cue word with the performance of the current scoring cue word to judge whether the current scoring cue word is updated; and carrying out loop iteration optimization on the scoring cue word step until an iteration termination condition is reached, and outputting a final scoring cue word. According to the method, the scoring cue word is optimized, so that the composition scoring accuracy and consistency of the large language model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a composition scoring prompt word optimization method based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the breakthrough progress of large language models in the field of natural language processing, the application of automated scoring systems in the field of education is becoming more and more widespread. As a complex language understanding task, essay scoring mainly relies on preset scoring rules and feature extraction in traditional methods. This method is often unable to cope with diverse writing styles and content. In recent years, although the automatic essay scoring system based on machine learning has improved the accuracy of scoring to a certain extent, there are still problems such as opaque scoring standards and difficulty in interpretation, and it performs poorly when dealing with innovative and deep thinking essays.

[0003] The emergence of large language models has brought new opportunities for automatic essay scoring. These models have acquired extensive knowledge and strong language understanding capabilities through pre-training, and can better capture the semantics, structure, and logical relationships of essays. However, the key to effectively using large language models for essay scoring lies in designing appropriate prompt words. Prompt words, as instructions for inputting large language models, directly affect the output quality of the model and the accuracy of scoring. Existing essay scoring methods based on large language models often use pre-designed fixed prompt words and lack specific optimization for different essay topics and scoring criteria. This method ignores the particularity and diversity of essay scoring tasks, which may lead to deviations or inconsistencies in scoring results, especially when faced with essay topics of different types and difficulty levels. Therefore, how to effectively optimize the prompt words for essay scoring has become a key issue in improving the accuracy and consistency of essay scoring with large language models.

[0004] At present, some methods for optimizing prompt words have been proposed. For example, a method for optimizing prompt words for online question and answer based on a large language model, which constructs a consulting question database, forms a keyword set, vectorizes prompt words, and uses a genetic algorithm for optimization. This method is mainly aimed at online question and answer scenarios, and generates and optimizes prompt words by calculating vector similarity and using a genetic algorithm. However, this method mainly focuses on question and answer tasks and cannot be directly applied to tasks such as essay scoring that require complex language understanding and multi-dimensional evaluation. Essay scoring tasks are essentially different from general question and answer tasks. Essay scoring needs to consider multiple dimensions such as content relevance, structural rationality, and language expression ability, and also needs to adapt to different essay topics and scoring standards. Therefore, the optimization of prompt words for essay scoring requires a method that can iteratively optimize based on actual scoring results for specific essay topics. In view of the shortcomings of the prior art, the present invention proposes an iterative optimization method for essay scoring prompt words based on a large language model. This method is different from a fixed prompt word design or an optimization method based on vector similarity, but guides the large language model itself to optimize prompt words by analyzing the difference between the large language model score and the manual score, thereby achieving dynamic optimization for a specific essay topic. Summary of the invention

[0005] The purpose of the present invention is to propose a composition scoring prompt word optimization method based on a large language model to improve the accuracy and consistency of automatic composition scoring.

[0006] To achieve the above object, the technical solution of the present invention is: a composition scoring prompt word optimization method based on a large language model, comprising the following steps:

[0007] Step S1: Obtain composition title information and annotated composition sample set;

[0008] Step S2: constructing an initial scoring prompt word according to the composition title information as the current scoring prompt word, and evaluating the performance of the current scoring prompt word;

[0009] Step S3: Use the current scoring prompt words to guide the large language model to score each composition in the composition sample set one by one, calculate the difference between the large language model score and the manual annotation score, and select the composition with the largest difference as the object of subsequent in-depth analysis;

[0010] Step S4: guide the large language model to conduct in-depth analysis on the essays identified in the previous step to explore the specific reasons leading to the difference in scores;

[0011] Step S5: guiding the large language model to modify the current scoring prompt word according to the reason for the scoring difference to generate a candidate scoring prompt word;

[0012] Step S6: Evaluate the performance of the candidate scoring prompt word using the same method as step S2; if the performance of the candidate scoring prompt word is better than that of the current scoring prompt word, update the current scoring prompt word with the candidate scoring prompt word; otherwise, keep the current scoring prompt word unchanged;

[0013] Step S7: Determine whether the preset number of iterations or target performance threshold is reached; if any condition is met, terminate the iteration process and output the final optimized scoring prompt word; otherwise, return to step S3 and enter the next round of iterative optimization.

[0014] Furthermore, the composition title information in step S1 includes a title description and a total score, and each sample in the labeled composition sample set includes the full text of the composition and its corresponding manual scoring score.

[0015] Furthermore, the initial scoring prompt words constructed in step S2 should include the following three parts: a brief description of the essay scoring task, a detailed description of the scoring steps, and the required scoring result output format.

[0016] Furthermore, the performance of the current scoring prompt word is evaluated in step S2, and the specific steps are as follows: use the current scoring prompt word to guide the large language model to score each composition in the composition sample set, and calculate the quadratic weighted Kappa coefficient between the large language model score and the manual score, and use the quadratic weighted Kappa coefficient as the performance evaluation indicator of the current scoring prompt word.

[0017] Furthermore, step S4 is performed by constructing specific analysis prompt words; the analysis prompt words should include the following parts: a task description for analyzing score differences, current score prompt words, the composition to be analyzed and its large language model score and manual score, and a specific format description for outputting the reasons for the score differences.

[0018] Furthermore, step S5 is performed by constructing specific optimization prompt words; the optimization prompt words should include the following parts: a task description for generating candidate scoring prompt words, the current scoring prompt words, the reasons for the scoring differences analyzed in step S4, and a specific format description for outputting candidate scoring prompt words.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The invention first constructs the initial scoring prompt words, then scores the composition samples through the large language model, and compares them with the manual scores to identify the compositions with the most significant scoring differences. Subsequently, the large language model is guided to deeply analyze the reasons for the scoring differences, and the scoring prompt words are modified accordingly to generate candidate prompt words. By evaluating the performance of the candidate prompt words, it is decided whether to update the current prompt words, thereby achieving continuous optimization of the prompt words. This solution realizes the automatic iterative optimization of the scoring prompt words, thereby improving the accuracy and consistency of the large language model composition scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flow chart of the composition scoring prompt word optimization method based on the large language model. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without paying creative work all belong to the scope of protection of the present invention. It should also be noted that in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0023] Before introducing the composition scoring prompt word optimization method based on the large language model provided in the embodiment of the present invention, some concepts involved in the embodiment of the present invention are first introduced: the large language model is a natural language processing model based on deep learning technology, which is usually trained using massive text data. These models can understand and generate human language and have powerful text understanding, generation and reasoning capabilities. The large language model learns the statistical laws and knowledge of language from large-scale corpora through self-supervised learning methods, and can perform various complex natural language tasks such as text generation, question answering, summarization, translation, etc. Typical large language models include the GPT series, Tongyi Qianwen, etc. Prompt words are input texts used to guide the model to generate specific outputs when using a large language model. It is a key interface for interacting with the model, and usually contains content such as task descriptions, contextual information, examples or instructions. Well-designed prompt words can significantly improve the performance of the model, enabling it to more accurately understand user intent and produce more accurate and effective outputs.

[0024] Step S1: Obtain composition title information and a set of annotated composition samples.

[0025] In this step, first obtain the composition title information, including the title description and the total score. For example, the composition title is "Please write an essay on "My Ideal Career"", and the total score is set to 100 points. Then obtain the annotated composition sample set, each sample includes the complete composition text and its corresponding manual scoring score. To ensure the representativeness of the sample, it is recommended to collect no less than 50 composition samples covering different score ranges, and to cover as many different writing styles and content quality as possible.

[0026] Here is a sample example:

[0027] Essay text: "My ideal career is to become a doctor. Doctors can not only help patients recover their health, but also improve their skills through continuous learning... (the rest of the essay is omitted)"

[0028] Manual score: 85 points

[0029] Step S2: constructing initial scoring prompt words according to the composition title information, and evaluating the performance of the scoring prompt words.

[0030] In this step, you need to construct an initial scoring prompt word, which will serve as a preliminary guide for the large language model to score the composition. The scoring prompt word contains three key parts:

[0031] 1) A brief description of the essay scoring task: This part provides the scoring task to the large language model. For example, "You are an experienced Chinese teacher and now need to score a high school student's essay."

[0032] 2) Detailed Scoring Procedure: This section lists in detail the factors to be considered and the scoring process. For example, "First, read the entire essay, and then evaluate it from three aspects: content, structure, and language. For content, focus on the relevance of the topic and the depth of thought; for structure, evaluate the logic and coherence of the essay; for language, examine the richness of vocabulary and the accuracy of expression."

[0033] 3) Required scoring result output format: This part specifies how the scoring results are presented. For example, "Please give an integer score between 0 and 100.\nOutput format:\nTotal score: [integer between 0 and 100]" The output format can also be set to XML, TXT, or JSON, etc., so as to extract the desired information from the output content.

[0034] For example, an initial scoring prompt may be as follows: "You are an experienced Chinese teacher and need to score a high school student's essay "My Ideal Career".

[0035] Scoring steps:

[0036] 1. Read the entire essay carefully.

[0037] 2. Evaluate from the following three aspects:

[0038] Content (40 points): relevance of topic, depth of thought, sufficiency of arguments

[0039] Structure (30 points): The logic and coherence of the article structure

[0040] Language (30 points): vocabulary richness, expression accuracy, grammatical correctness

[0041] 3. Give a total score based on the above assessments (maximum 100 points).

[0042] Output format:

[0043] Total score: [integer from 0 to 100]

[0044] Comment: [Briefly explain the reason for the rating]”

[0045] Next, we need to evaluate the performance of the scoring prompt words constructed in this step.

[0046] When implementing this method, it is necessary to select an appropriate large language model as a tool for scoring and subsequent optimization. When selecting a model, its capabilities in text understanding, generation, and complex reasoning, as well as its compatibility with this method, should be considered. The selected model should be able to effectively handle essay scoring tasks and support the analysis and optimization of scoring prompts. The specific selection criteria and methods can be adjusted according to the actual application scenario and available resources.

[0047] The scoring prompt words are then combined with each essay in the sample set to form a complete input. They are then input into the large language model in sequence to obtain the model's output scoring results. This process may require multiple API calls or local model inference, depending on the large language model selected and the deployment method.

[0048] For example, suppose we combine the scoring prompt word and an essay in the sample set:

[0049] "You are an experienced Chinese teacher and need to grade a high school student's essay titled "My Ideal Career".

[0050] Scoring steps:

[0051] 1. Read the entire essay carefully.

[0052] 2. Evaluate from the following three aspects:

[0053] Content (40 points): relevance of topic, depth of thought, sufficiency of arguments

[0054] Structure (30 points): The logic and coherence of the article structure

[0055] Language (30 points): vocabulary richness, expression accuracy, grammatical correctness

[0056] 3. Give a total score based on the above assessments (maximum 100 points).

[0057] Output format:

[0058] Total score: [integer from 0 to 100]

[0059] Comments: [Briefly explain the reason for the rating]

[0060] The following are the essays to be graded:

[0061] Everyone has their own ideal career, and my ideal is to become an environmental engineer. This dream is not a whim, but stems from my deep concern for environmental protection and my love for scientific and technological innovation. \nIn recent years, we have witnessed the challenges brought by climate change: frequent extreme weather events, sharp decline in biodiversity, and increasingly severe resource shortages. These phenomena not only affect our daily lives, but also threaten the future of the earth. As an environmental engineer, I will have the opportunity to directly participate in the process of solving these problems and contribute to creating a more sustainable world. \nThe work of environmental engineers is very broad, including designing pollution control systems, developing renewable energy technologies, and optimizing resource utilization efficiency. This diversity is what attracts me. I can use multidisciplinary knowledge such as physics, chemistry, and biology, combined with engineering technology, to deal with complex environmental problems. For example, I may be involved in designing more efficient solar cells or developing new water treatment technologies, which can have a substantial impact on environmental protection. \nIn addition, environmental engineering is also a field full of innovative opportunities. With the advancement of technology, we have more tools to monitor and analyze environmental data. Emerging technologies such as artificial intelligence and big data analysis are changing the way we understand and manage the environment. As a future environmental engineer, I look forward to using these advanced technologies to develop smarter and more efficient environmental protection solutions. \nOf course, I also clearly realize that becoming an excellent environmental engineer is not an easy task. It requires solid scientific knowledge, innovative thinking ability, and engineering skills to solve practical problems. More importantly, it requires us to have a strong sense of social responsibility and perseverance. In the face of possible setbacks and challenges, I am ready to rise to the challenge. \nIn short, I chose environmental engineer as my ideal career not only because it can satisfy my interests and hobbies, but also because it allows me to make meaningful contributions to society. I believe that through unremitting efforts and continuous learning, I will be able to make a difference in this field and contribute my own strength to creating a cleaner and more sustainable future. ”

[0062] After inputting the above combination of content into the big language model, the following output may be obtained: "Total score: 90\nComment: This essay is very relevant in content, showing the author's deep understanding of environmental protection and scientific and technological innovation, with sufficient and in-depth arguments (content score: 38 / 40). The article has a reasonable structure, clear levels, and logical coherence, which can effectively unfold ideas and guide readers (structure score: 27 / 30). In terms of language, the author uses a rich vocabulary, accurate expression, and basically no grammatical errors, but there are some places that can be more concise (language score: 25 / 30). Overall, the article shows a high level of writing, especially in terms of ideas and expression. "Through this output, it can be extracted that the score given by the big language model is 90. It is worth noting that in actual applications, the score of the big language model can be automatically extracted through regular expressions. This process is performed on each essay in the sample set in turn to obtain the big language model score of each essay.

[0063] Finally, the quadratic weighted Kappa coefficient between the large language model score and the manual score is calculated and used as a performance evaluation indicator for the scoring prompt words. The quadratic weighted Kappa coefficient is a scoring consistency indicator that takes into account the order of levels, and is particularly suitable for ordinal scoring tasks such as essay scoring. The coefficient ranges from -1 to 1, where 1 indicates complete consistency, 0 indicates that the consistency does not exceed the random level, and a negative value indicates that the consistency is worse than random. In practical applications, the coefficient can be calculated using statistical software or related functions in programming languages. When calculating, it is necessary to input the scoring results of the large language model and the corresponding manual scoring results, and the coefficient obtained is the performance indicator of the scoring prompt words.

[0064] Step S3: Use the current scoring prompt words to guide the large language model to score each essay in the essay sample set one by one. Calculate the difference between the large language model score and the manual annotation score, and select the essay with the largest difference as the object of subsequent in-depth analysis.

[0065] Specifically, the scoring prompt words are used to guide the large language model to score each composition in the composition sample set one by one, and the implementation method is consistent with the composition scoring method used when evaluating the performance of the scoring prompt words in step S2. After completing the scoring of all the compositions, the large language model scores are compared with the manual scores one by one, the difference between the two is calculated, and the composition with the largest difference is screened out. This composition reflects the most deviations between the large language model model and the manual scoring process, which may be because the model fails to fully understand the details of some scoring criteria, or has deviations in some language expressions.

[0066] Step S4: Guide the large language model to conduct an in-depth analysis of the essays identified in the previous step to explore the specific reasons that lead to the difference in scores.

[0067] Specifically, a specific analysis prompt word needs to be constructed to guide the large language model to analyze the reasons for the score difference. This prompt word usually contains the following key parts:

[0068] 1) Task description: Requires the large language model to analyze the reasons for the difference in scores. It can also include more detailed instructions, such as requiring the large language model to analyze the reasons for the difference in scores from multiple perspectives, such as understanding the scoring criteria, identifying essay features, and understanding language.

[0069] 2) Background information: Provides the current scoring prompt words, the essay to be analyzed, and the large language model score and manual score;

[0070] 3) Output format: Specify the output format of the analysis results for subsequent processing.

[0071] For example, the following prompt word template can be used to guide the large language model to analyze the reasons for the score difference:

[0072] “As an experienced Chinese language education expert, your task is to analyze the reasons for the gap between the manual scoring and the large language model scoring of the following essays.

[0073] Current scoring prompt: [insert the current scoring prompt]

[0074] Content of the composition: [Insert the full text of the composition to be analyzed]

[0075] Large language model score: [insert score of large language model]

[0076] Manual scoring: [insert manually marked score]

[0077] Please analyze the possible reasons for the rating gap in detail from the following perspectives and provide specific examples for each perspective:

[0078] 1. Scoring criteria understanding: Is it possible that the model misunderstands certain scoring criteria? For example, is the understanding of “innovation” consistent with that of human raters?

[0079] 2. Composition feature identification: Does the model ignore or over-focus on certain composition features? For example, does it focus too much on vocabulary diversity and ignore the depth of content?

[0080] 3. Language understanding: Is the model biased in understanding the content or expression of the essay? For example, is the author’s metaphor or sarcasm correctly understood?

[0081] 4. Prompt word design: Are the original scoring prompt words unclear or misleading? For example, are the weights of certain scoring dimensions not clearly explained?

[0082] After the analysis is complete, please summarize the most likely main causes and output them in the following format:

[0083] Main reason: [Please explain the main reason in detail here and provide specific suggestions for improvement]"

[0084] The constructed prompt words are input into the large language model. The large language model performs in-depth analysis based on the given information and outputs the analysis results.

[0085] For example, suppose a composition is scored 95 points by humans, while the large language model scores 90 points. The large language model may output the following analysis results: "Main reason: The model's understanding of the "innovation" standard is different from that of the human scorer. The model may pay more attention to the diversity of vocabulary and ignore the overall logical structure and thought depth of the article, resulting in a lower score than the manual score. At the same time, the weight of the "innovation" standard in the prompt word is not clear enough, which may also affect the model's judgment."

[0086] Step S5: guiding the large language model to modify the scoring prompt words according to the reasons for the scoring differences, thereby generating candidate scoring prompt words.

[0087] Specifically, an optimization prompt word needs to be constructed to guide the large language model on how to modify the scoring prompt word to generate candidate scoring prompt words. This prompt word usually contains the following key parts:

[0088] 1) Task description: The large language model is required to modify the current scoring prompt word based on the analysis results;

[0089] 2) Background information: providing the current scoring prompt words and the analysis results of step S4;

[0090] 3) Output format: Specify the output format of the improved prompt words for subsequent processing.

[0091] Exemplarily, the following prompt word template can be used to guide the large language model to modify the scoring prompt words:

[0092] “As an experienced language education expert, your task is to modify the scoring prompts based on the current scoring prompts and the reasons for the scoring differences you have discovered.

[0093] Current rating prompt: [insert current rating prompt]

[0094] Reason for score difference: [insert reason for analysis in step S4]

[0095] Please provide the revised scoring prompts to ensure that the revised scoring prompts can solve the problems found and improve the accuracy and consistency of the scoring. Finally, output in the following format:

[0096] Modified scoring prompt word: [modified prompt word content]"

[0097] Input the constructed prompt word into the large language model. The large language model will output an improved scoring prompt word based on the given information. This scoring prompt word will be used as a candidate scoring prompt word.

[0098] For example, after receiving the complete prompt word, the large language model may output the following candidate scoring prompt word: "Task Description: You are an experienced Chinese teacher and need to score a high school student's essay "My Ideal Career".

[0099] Scoring steps:

[0100] 1. Read the entire essay carefully.

[0101] 2. Evaluate from the following three aspects:

[0102] Content (30 points): relevance of topic, sufficiency of arguments

[0103] Structure (20 points): The logic and coherence of the article structure

[0104] Language (20 points): vocabulary richness, expression accuracy, grammatical correctness

[0105] Innovation (15 points): evaluates the unique ideas or perspectives in the essay

[0106] Deep Thinking (15 points): evaluates the author's in-depth analysis and reflection on the topic.

[0107] 3. Give a total score based on the above assessments (maximum 100 points).

[0108] Output format:

[0109] Total score: [integer from 0 to 100]

[0110] Comment: [Briefly explain the reason for the rating]”

[0111] S6: Evaluate the performance of the candidate scoring prompt word using the same method as step S2. If its performance is better than the current scoring prompt word, update the current scoring prompt word with the candidate scoring prompt word; otherwise, keep the current scoring prompt word unchanged.

[0112] Specifically, when comparing the performance of the current scoring cue word and the candidate scoring cue word, we directly compare their performance indicators. For example, suppose the performance indicator of the current scoring cue word (i.e., the quadratic weighted Kappa coefficient) is 0.75, while the performance indicator of the candidate scoring cue word is 0.78. In this case, since 0.78>0.75, the performance of the candidate scoring cue word is better, so it will become the scoring cue word in the next round of iterations. On the contrary, if the performance indicator of the candidate scoring cue word is 0.73, since 0.73<0.75, it means that the candidate cue word has failed to improve the performance, so the current scoring cue word will remain unchanged. This direct comparison method ensures that each iteration moves in the direction of improving performance while maintaining the stability of the scoring process.

[0113] Step S7: Determine whether the preset number of iterations or target performance threshold is reached. If any condition is met, the iteration process is terminated and the final optimized scoring prompt word is output; otherwise, return to step S3 and enter the next round of iterative optimization.

[0114] For example, suppose the preset termination condition is: the number of iterations reaches 5 times or the performance of the scoring prompt word reaches 0.9. If it is currently in the third iteration and the performance of the scoring prompt word is 0.85. Determine whether the termination condition is met: the number of iterations is 3, which has not reached the upper limit of 5 times, and the performance is 0.85, which has not reached the target of 0.9. After judgment, both termination conditions are not met, so return to step S3 and enter the fourth iteration. If the performance of the scoring prompt word reaches 0.92 in the third iteration, the performance has reached the preset target (0.9), then the iteration is stopped and the scoring prompt word is output. For example, the final output of the iteratively optimized scoring prompt word may be: "Task Description: You are an experienced Chinese teacher, and now you need to score a high school composition "My Ideal Career".

[0115] Scoring steps:

[0116] 1. Please read the essay carefully and rate it according to the following dimensions:

[0117] 1) Content and theme (0-25 points): Evaluate whether the essay is closely related to the theme, whether the content is substantial, and whether the arguments are reasonable.

[0118] 2) Language expression (0-20 points): evaluate whether the language is fluent, the words are accurate, and the sentences are diverse.

[0119] 3) Structural organization (0-20 points): Evaluate whether the article has a clear structure, distinct levels, and natural transitions.

[0120] 4) Innovative thinking (0-20 points): evaluates whether there are unique insights and whether novel ideas can be put forward.

[0121] 5) Writing skills (0-15 points): evaluate whether there are beautiful sentences and whether rhetoric techniques are used.

[0122] 2. Please give a total score (out of 100) and provide a brief reason for your score. Please be objective and fair when scoring and avoid subjective bias.

[0123] Output format:

[0124] Total score: [integer from 0 to 100]

[0125] Comment: [Briefly explain the reason for the rating]”

[0126] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A composition scoring prompt word optimization method based on a large language model, characterized in that: The method comprises the following steps: Step S1: Obtain composition title information and annotated composition sample set; Step S2: constructing an initial scoring prompt word according to the composition title information as the current scoring prompt word, and evaluating the performance of the current scoring prompt word; Step S3: Use the current scoring prompt words to guide the large language model to score each composition in the composition sample set one by one, calculate the difference between the large language model score and the manual annotation score, and select the composition with the largest difference as the object of subsequent in-depth analysis; Step S4: guide the large language model to conduct in-depth analysis on the essays identified in the previous step to explore the specific reasons leading to the difference in scores; Step S5: guiding the large language model to modify the current scoring prompt word according to the reason for the scoring difference to generate a candidate scoring prompt word; Step S6: Evaluate the performance of the candidate scoring prompt word using the same method as step S2; if the performance of the candidate scoring prompt word is better than that of the current scoring prompt word, update the current scoring prompt word with the candidate scoring prompt word; otherwise, keep the current scoring prompt word unchanged; Step S7: Determine whether the preset number of iterations or target performance threshold is reached; if any condition is met, terminate the iteration process and output the final optimized scoring prompt word; otherwise, return to step S3 and enter the next round of iterative optimization.

2. According to the method for optimizing essay scoring prompt words based on a large language model in claim 1, it is characterized in that: The composition title information in step S1 includes a title description and a total score, and each sample in the labeled composition sample set includes the full text of the composition and its corresponding manual scoring score.

3. The composition scoring prompt word optimization method based on a large language model according to claim 1, characterized in that: The initial scoring prompt words constructed in step S2 should include the following three parts: a brief description of the essay scoring task, a detailed description of the scoring steps, and the required scoring result output format.

4. The composition scoring prompt word optimization method based on a large language model according to claim 1, characterized in that: The specific steps of evaluating the performance of the current scoring prompt word in step S2 are as follows: using the current scoring prompt word to guide the large language model to score each composition in the composition sample set, and calculating the quadratic weighted Kappa coefficient between the large language model score and the manual score, and using the quadratic weighted Kappa coefficient as the performance evaluation indicator of the current scoring prompt word.

5. The composition scoring prompt word optimization method based on a large language model according to claim 1, characterized in that: Step S4 guides the large language model to conduct an in-depth analysis of the composition identified in the previous step to explore the specific reasons for the score differences, which is executed by constructing specific analysis prompt words; the analysis prompt words include the following parts: a task description for analyzing the score differences, a current score prompt word, the composition to be analyzed and its large language model score, manual scoring, and a specific format description for outputting the reasons for the score differences.

6. The composition scoring prompt word optimization method based on a large language model according to claim 1, characterized in that: The step S5 of guiding the large language model to modify the current scoring prompt word to generate a candidate scoring prompt word according to the reason for the scoring difference is performed by constructing a specific optimization prompt word; the optimization prompt word should include the following parts: a task description for generating a candidate scoring prompt word, the current scoring prompt word, the reason for the scoring difference analyzed in step S4, and a specific format description for outputting the candidate scoring prompt word.

Images