Large language model marking method, system and equipment based on iterative knowledge enhancement and medium

Abstract

The embodiment of the invention provides a paper marking method, system and equipment of a large language model based on iterative knowledge enhancement and a medium, and belongs to the field of examination evaluation. The method comprises the steps of simulating expert paper marking logic by using a large language model based on a reference answer of a specified question, a scoring description and a plurality of expert scoring samples, and generating an initial scoring knowledge base; iteratively enhancing the structured initial score knowledge base; after each round of iteration, calculating an inter-scorer reliability coefficient between the model score and the expert score, and if the coefficient is higher than a preset threshold value in multiple iterations and a core rule of the score knowledge base is not changed in the iterations, obtaining a target score knowledge base; and scoring the answer sheet of the examinee to be evaluated. The model continuously absorbs expert experience through iteration, so that a paper marking standard more accurately adapts to changes of different examinations and questions, finally all answer sheets are judged by a unified and stabilized knowledge base, and the problem of inconsistency caused by scale floating of human paper markers is effectively avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of examination evaluation, in particular to an evaluation method and system based on an iterative knowledge enhanced large language model, a device and a medium. BACKGROUND

[0002] Automatic evaluation technology is a long-term research focus in the field of education evaluation, and its development process mainly goes through two stages: First, the early method based on rule engine. This method relies on artificially preset structured scoring rules, although it is simple to implement and has strong interpretability, but its flexibility is poor, and it is difficult to effectively understand and evaluate the diversified expression methods and innovative ideas in open questions (such as discussion questions and composition questions), and it is difficult to cope with the complexity and fuzziness commonly existing in natural language.

[0003] Subsequently, methods based on machine learning models, especially deep learning models, gradually become the mainstream. This kind of method can learn the scoring mode to a certain extent through a large amount of labeled answer data for end-to-end training. However, they have two inherent defects: first, they rely heavily on large-scale and high-quality labeled data, which has high data collection costs; second, once the model is trained, its parameters and evaluation criteria are "fixed", and it is difficult to incorporate new evaluation criteria or expert knowledge for adjustment and update, that is, there is a "model fixation" problem. When the examination outline or evaluation criteria change, it is often necessary to collect data and train the model again, which is costly.

[0004] In recent years, large language models have shown strong natural language understanding and generation capabilities, providing new possibilities for directly understanding and evaluating open answer sheets. However, directly applying general large language models to serious evaluation scenarios faces serious challenges: (1) Inconsistent evaluation criteria: general large language models contain general knowledge, not specific evaluation rules required by specific exams, and their evaluation results often deviate from expert criteria, lacking standardization.

[0005] (2) Poor domain adaptability: different disciplines, different exams, and even different test questions have different focuses and evaluation logic, and test question evaluation often has some subjectivity and fuzziness. General large language models cannot adaptively grasp these subtle differences.

[0006] (3) High customization cost: although large language models can be fine-tuned or reinforced to adapt to specific evaluation tasks, this requires huge computing resources and time costs, and the process is complex, making it difficult to meet the requirements of timeliness and economy in actual applications.

[0007] To this end, the industry often uses retrieval enhancement generation technology to provide auxiliary information for large language models by introducing external knowledge bases. However, the knowledge bases in existing methods are usually static and pre-constructed, and cannot be dynamically optimized and evolved according to expert feedback during the evaluation process, resulting in an upper limit to the scoring ability of the system and difficulty in continuous improvement. Therefore, there is a lack of an intelligent evaluation scheme in the prior art that can balance scoring accuracy, domain adaptability, low cost, and continuous evolution capability. SUMMARY

[0008] The purpose of the embodiments of the present application is to provide a large language model evaluation method, system, device and medium based on iterative knowledge enhancement, which continuously absorbs expert experience through iteration, so that the model evaluation standard is more accurate and more adaptable to changes in different exams and questions. Ultimately, the answers are judged by the stable knowledge base, effectively avoiding the inconsistency problem caused by the scale floating of human evaluators.

[0009] To achieve the above purpose, the embodiments of the present application provide a large language model evaluation method based on iterative knowledge enhancement, comprising: Based on the reference answer, scoring instructions and multiple expert scoring samples of the specified question, a large language model is used to simulate expert evaluation logic to generate a structured initial scoring knowledge base; Iteratively enhance the structured initial scoring knowledge base, and repeat the following steps until the scoring knowledge base is stable: Use the structured initial scoring knowledge base to score the unseen examinee's answers to obtain model scoring results; Obtain the artificial scoring results of the expert on the unseen examinee's answers; Based on the model scoring results and the artificial scoring results, guide the large language model to reflect on itself to update the structured initial scoring knowledge base and form an enhanced scoring knowledge base; Use the enhanced scoring knowledge base for a new round of scoring and reflection, and after each iteration, calculate the inter-rater reliability coefficient between the model scoring and the expert scoring. When the inter-rater reliability coefficient is higher than the preset threshold value in multiple iterations, and the core rules of the scoring knowledge base do not change in continuous iterations, it is determined that the scoring knowledge base reaches a stable state, and a target scoring knowledge base is obtained; Use the target scoring knowledge base to score the examinee's answers to be evaluated.

[0010] Optionally, the generated structured initial scoring knowledge base is represented as follows: ; In the formula, represents a large language model pre-trained with professional knowledge of the target subject field, is an instruction for generating a structured knowledge base, is a test question title, is a reference answer corresponding to the test question title, is a scoring explanation, is at least one expert scoring sample containing detailed scoring reasons, is a structured initial scoring knowledge base.

[0011] Optionally, the structured initial scoring knowledge base is used to score the unseen examinee's answer sheet to obtain a model scoring result as follows: ; In the formula, is an instruction for scoring according to the scoring knowledge base, is a test question of the unseen examinee's answer sheet, contains at least one unseen examinee's answer sheet sample, is the version of the scoring knowledge base, is the large language model scoring result according to the version of the scoring knowledge base.

[0012] Optionally, the enhanced scoring knowledge base formed is as follows: ; In the formula, is an instruction for updating the scoring knowledge base, is a test question of the unseen examinee's answer sheet, contains at least one unseen examinee's answer sheet sample, is the enhanced scoring knowledge base formed after the large language model has been self-reflecting for the time.

[0013] Optionally, the formula for calculating the inter-rater reliability coefficient between the model scoring and the expert scoring is as follows: ; In the formula, is the consistency rate of the model scoring result and the expert scoring result, is the random consistency rate, which is the calculated value of the expected consistency rate under the assumption that the model scoring and the expert scoring occur randomly.

[0014] Optionally, whether the core rules of the scoring knowledge base change in continuous iteration is determined in the following way: ; In the formula, is an instruction for judging whether there is a substantial change in the scoring knowledge base, is a test question of the unseen examinee's answer sheet, is a binary judgment result, For the first version of the scoring knowledge base, For the enhanced scoring knowledge base formed by the large language model after the first self-reflection.

[0015] Optionally, the formula for scoring the examinee's answer sheet using the target scoring knowledge base is as follows: ; In the formula, is the examinee's answer sheet, is the scoring result of the large language model according to the target scoring knowledge base , is the question of the examinee's answer sheet, is the instruction for scoring according to the scoring knowledge base, and the scoring result includes the scoring points and the scoring reasons.

[0016] In a second aspect, the present application also provides a marking system based on an iteratively knowledge-enhanced large language model, comprising: An initial scoring knowledge base generation unit for generating a structured initial scoring knowledge base by simulating expert marking logic using a large language model based on the reference answers, scoring instructions and multiple expert scoring samples of specified questions; A scoring knowledge base iterative enhancement unit for iteratively enhancing the structured initial scoring knowledge base and repeating the following steps until the scoring knowledge base is stable: Scoring unseen examinee's answer sheets using the structured initial scoring knowledge base to obtain model scoring results; Obtaining expert manual scoring results of the unseen examinee's answer sheets; Guiding the large language model to perform self-reflection based on the model scoring results and the manual scoring results to update the structured initial scoring knowledge base and form an enhanced scoring knowledge base; Using the enhanced scoring knowledge base for a new round of scoring and reflection, and after each iteration, calculating the inter-rater reliability coefficient between the model scoring and the expert scoring, when the inter-rater reliability coefficient is higher than a preset threshold in multiple iterations, and the core rules of the scoring knowledge base do not change in continuous iterations, determining that the scoring knowledge base reaches a stable state, and obtaining a target scoring knowledge base; A marking unit for scoring examinee's answer sheets using the target scoring knowledge base.

[0017] In a third aspect, the present application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the marking method based on the iteratively knowledge-enhanced large language model as described above when executing the program.

[0018] In a fourth aspect, the present application also provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the marking method of the large language model based on iterative knowledge enhancement.

[0019] Through the above technical solution, the model continuously absorbs expert experience through iteration, so that the marking standard becomes more and more accurate, and adapts to the changes of different examinations and questions. Moreover, the scoring knowledge base is a white box, and the scoring rules and basis are clear and visible, avoiding the "black box" problem of neural network model, and facilitating subsequent dispute processing. Finally, all answer sheets are judged by the unified and stable knowledge base, effectively avoiding the inconsistency problem caused by fatigue or scale floating of human marking personnel.

[0020] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used together with the following specific implementation to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the drawings: Figure 1 is an implementation flowchart of the marking method of the large language model based on iterative knowledge enhancement provided by the embodiments of the present application; Figure 2 is a detailed flowchart of the marking method of the large language model based on iterative knowledge enhancement provided by the embodiments of the present application; Figure 3 is a structural schematic diagram of the marking system of the large language model based on iterative knowledge enhancement provided by the embodiments of the present application; Figure 4 is a hardware structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0022] In the following detailed description, various embodiments of the present disclosure will be described more fully. The present disclosure can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives falling within the spirit and scope of various embodiments of the present disclosure.

[0023] Hereinafter, the term "include" or "may include" used in various embodiments of the disclosure indicates the presence of the disclosed function or operation and does not limit one or more functions or operations to be added. Also, as used in various embodiments of the disclosure, the terms "include", "have", and their conjugates merely indicate that specific features, numbers, steps, operations, or combinations thereof are present, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, or combinations thereof.

[0024] In various embodiments of the disclosure, the expression "or" or "at least one of A or / and B" includes any combination of the listed terms or all combinations thereof. For example, the expression "A or B" or "at least one of A or / and B" can include A, can include B, or can include both A and B.

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0026] Referring to Figure 1 The flowchart shown is a flowchart of an evaluation method for a large language model based on iterative knowledge enhancement in a specific embodiment, including the following execution steps: Step 100: Based on the reference answer, scoring instructions, and multiple expert scoring samples of a specified question, a large language model is used to simulate expert evaluation logic to generate a structured initial scoring knowledge base.

[0027] Specifically, the reference answer, scoring instructions, and M expert scoring samples with detailed scoring instructions of a specific subjective question are input into a large language model. The large language model is driven by structured instructions to simulate expert evaluation logic, and a structured initial scoring knowledge base containing scoring rules for the question is summarized and induced.

[0028] ; wherein, refers to a large language model pre-trained with professional knowledge in the target subject field; is an instruction for generating a structured knowledge base; is a test question; is a reference answer; is a scoring instruction; For a structured initial scoring knowledge base, the scoring knowledge base at least includes: scoring rules, scoring points, boundary examples, and scoring explanations. For a structured initial scoring knowledge base, the scoring knowledge base at least includes: scoring rules, scoring points, boundary examples, and scoring explanations.

[0029] For example, the intelligent scoring of the subjective question "Discuss the historical significance of the Silk Road" is performed. For the illustrative example, the following operations are performed: Instructions: Please summarize the scoring rules for the "Discuss the historical significance of the Silk Road" question based on the provided reference answer and expert scoring samples. Question: Discuss the historical significance of the Silk Road; Reference answer: Reference answer provided by the subject group (covering economic, cultural, historical, and other aspects of significance); Scoring criteria: Scoring criteria provided by the subject group; Scoring samples: 10 expert-scored sample answer sheets (including high and low score sheets).

[0030] Large language model outputs initial knowledge base: Scoring rules: Economic exchange: Mention the westward transmission of goods such as silk, porcelain, and tea, and the eastward transmission of goods from the Western Regions. (5 points).

[0031] Cultural exchange: Mention the spread of religious sects, technology (such as papermaking), and art along the Silk Road. (5 points).

[0032] Historical exchange: Mention the role of the Han and Tang dynasties in consolidating the border and promoting cross-cultural exchange. (5 points).

[0033] Long-term impact: Mention the contribution to the development of world civilization or the ancient and modern connection with "One Belt, One Road". (5 points).

[0034] Deduction items: Historical errors, logical confusion, and only listing knowledge points without discussion.

[0035] Scoring points: Check if the examinee's answer contains economic exchange, cultural exchange, historical exchange, and long-term development.

[0036] Boundary examples and scoring explanations: Full score sheet: full score sheet summary and scoring explanation; zero score sheet: zero score sheet summary and scoring explanation.

[0037] Step 101: Iteratively enhance the structured initial scoring knowledge base, and repeat the following steps until the scoring knowledge base is stable.

[0038] S1010: Use the structured initial scoring knowledge base to score unseen examinee answer sheets, obtaining model scoring results.

[0039] Specifically, a batch of new N unseen examinee answer samples is input into the large language model, and it is instructed to score according to the scoring knowledge base to generate a model scoring result.

[0040] ; Among them, is the instruction for scoring according to the scoring knowledge base, contains at least N unseen examinee answer samples, N≥5, is the scoring knowledge base of the first version, when it is the first time , is the large language model scoring result according to the scoring knowledge base of the first version, and the scoring result contains the score point score and the score point scoring basis.

[0041] S1011: Obtain the artificial scoring result of the expert on the unseen examinee answer.

[0042] Specifically, the artificial scoring result of the expert on the same batch of unseen answer samples is obtained , and the scoring result contains the score point score and the score point scoring basis given by the expert.

[0043] For example, for the illustrative example, the following operations are performed: Input: 10 new, ungraded examinee answer sheets are input into the large language model, and it is instructed to score according to the initial knowledge base.

[0044] Output model score: The large language model gives scores and scoring explanations for the 10 answer sheets.

[0045] Obtain expert feedback: The expert independently scores the 10 answer sheets.

[0046] S1012: Based on the model scoring result and the artificial scoring result, guide the large language model to reflect on itself to update the structured initial scoring knowledge base and form an enhanced scoring knowledge base.

[0047] Specifically, the model scoring result and the expert scoring result are input again into the large language model, and it is instructed to reflect on itself, analyze the reasons for the scoring bias, and update and refine the initial scoring knowledge base accordingly to form an enhanced scoring knowledge base.

[0048] ; Among them, is the instruction for updating the knowledge base, is the enhanced scoring knowledge base formed by the large language model after the first self-reflection.

[0049] It should be noted that the model self-reflection content includes identifying ambiguous points in the scoring rules, supplementing the un-covered boundary score explanation, and refining higher-level scoring principles.

[0050] S1013: Use the enhanced scoring knowledge base to perform a new round of scoring and reflection, and calculate the inter-rater reliability coefficient between the model score and the expert score after each iteration. When the inter-rater reliability coefficient is higher than the preset threshold value in multiple iterations, and the core rules of the scoring knowledge base do not change in continuous iterations, it is determined that the scoring knowledge base reaches a stable state, and the target scoring knowledge base is obtained.

[0051] Specifically, a new round of scoring and reflection is performed using the enhanced knowledge base, and the inter-rater reliability coefficient between the model score and the expert score is calculated after each iteration. When the inter-rater reliability coefficient is higher than the preset threshold value T in continuous S iterations, and the core rules of the knowledge base do not change in continuous iterations, it is determined that the knowledge base has reached a stable state, and the knowledge base at this time is determined as the final scoring knowledge base .

[0052] wherein the inter-rater reliability coefficient is the Cohen's kappa coefficient, and its calculation formula is: ; wherein, is the consistency rate of the model score and the expert score, i.e., the proportion of samples with consistent scores in the total number of samples; is the random consistency rate, which is the expected consistency rate under the assumption that the model score and the expert score occur randomly; for example, the preset threshold value T≥0.75.

[0053] wherein whether the scoring core rules of the knowledge base have substantial changes is determined by a large language model semantic judgment, which is performed in the following way: ; wherein, is an instruction for determining whether the scoring knowledge base has substantial changes, is a binary determination result: "yes" or "no". If is "yes" and the inter-rater reliability coefficient is greater than the preset threshold value T, the scoring knowledge base iteration stops, otherwise step 101 is continued.

[0054] For the illustrative example, the following operations are performed: Input: Compare the model score with the expert score. For example, for a certain answer, the model scores 15 points, and the expert only scores 12 points.

[0055] Large language model reflection task: "You gave answer A 15 points, but the expert score is 12 points. Please analyze the reasons for the scoring difference and reflect on how to modify the scoring knowledge base to avoid such bias." Large language model reflection output: "I found that answer A mentioned all the scoring points, but the discussion on 'cultural exchange' was shallow, only mentioning'spread of culture', while the expert valued specific and in-depth discussion. I need to update the knowledge base. Also..." Output enhanced knowledge base: (Add a supplementary note under the 'cultural exchange' scoring point) "...only mentioning'spread of culture' and other vague statements can only get half of the basic points for this point; must be combined with specific examples for full points." (Newly added 'discussion depth' dimension) "Add 'discussion depth' dimension (4 points) to evaluate whether the elaboration on each point is specific and logical." Repeat steps S1010-S1012 for a total of 10 rounds. The modification of the knowledge base tends to be subtle, the consistency between the model score and the expert score is stable above 95%, the Cohen kappa coefficient is about 0.81, and the content of the knowledge base is relatively stable and does not change substantially. The total time consumption is about 1 hour.

[0056] Step 102: Use the target scoring knowledge base to score the examinee's answer.

[0057] Specifically, all examinee answer sheets to be evaluated are batched into the large language model, and it is instructed to strictly follow the final scoring knowledge base for automatic scoring.

[0058]

[0059] Among them, is the examinee answer sample, is the large language model scoring result according to the final scoring knowledge base , containing two fields: "score" and "reason". "score" is the scoring value output by the large language model according to the final scoring knowledge base , and "reason" is the scoring reason, which needs to list specific rules as the basis for scoring.

[0060] It should be noted that the large language model is a parameter-frozen, pre-trained large language model in the target discipline field, and the improvement of its scoring ability is achieved through the iterative update of the external scoring knowledge base.

[0061] Exemplarily, 100,000 test takers' answer sheets are batch input into the large language model, and it is instructed to automatically score strictly according to the final scoring knowledge base. The following is an example of the large language model scoring intelligently: Test taker's answer sheet example: "The Silk Road is an ancient trade route connecting China and Europe. It promoted economic exchanges, such as silk, porcelain, and tea being transported to the West, while grapes, walnuts, and other products from the Western Regions were introduced to China. In terms of culture, papermaking was also spread from China to the West. Historically, the Silk Road helped the Han and Tang dynasties consolidate the border and promote cross-cultural exchanges. In the long run, the Silk Road has made significant contributions to world civilization." Large language model scoring result example: score: 20; reason: According to the stem requirements, the scoring knowledge base, the scoring of the test taker is as follows.

[0062] Economic exchanges (5 points): The answer clearly mentions "silk, porcelain, and tea being transported to the West" and "grapes, walnuts, and other products from the Western Regions being introduced", which fully meets the requirements of "mentioning the westward transmission of goods such as silk, porcelain, tea, and the eastward transmission of Western products" in the scoring rules, therefore 5 points are awarded.

[0063] Cultural exchanges (5 points): The answer mentions "the spread of papermaking to the West", covering the spread of technology (papermaking), which meets the requirement of "mentioning the spread of technology (such as papermaking), art, etc. along the Silk Road", therefore 5 points are awarded.

[0064] Historical exchanges (5 points): The answer points out "helping the Han and Tang dynasties consolidate the border" and "promoting cross-cultural exchanges", which directly corresponds to the rule "mentioning the role of consolidating the border of the Han and Tang dynasties and promoting cross-cultural exchanges", therefore 5 points are awarded.

[0065] Long-term impact (5 points): The answer emphasizes "significant contributions to world civilization", which meets the requirement of "mentioning contributions to the development of world civilization", therefore 5 points are awarded.

[0066] To verify the effectiveness of the present application, a comparative experiment was conducted.

[0067] Control group: Directly use the same large language model and score based on the original scoring standard.

[0068] Experimental group: Use the iterative knowledge base enhancement method described in the present application, after 10 rounds of "scoring-feedback-reflection" cycle, form a stable final scoring knowledge base, and then use the knowledge base to score.

[0069] Taking the expert artificial scoring as the benchmark, the consistency rate of the model scoring and the expert scoring within the 2-point difference threshold is calculated as the core evaluation index. The results are shown in Table 1: Table 1 Comparative experiments

[0070] The experimental results show that the score consistency is significantly improved by the method of the present application. This fully proves that the iterative knowledge base enhancement mechanism can effectively transfer the expert's scoring logic and internalization standard to the large language model, making its scoring performance close to the level of human experts.

[0071] On the basis of the above embodiment, it needs to be further pointed out that the system architecture of the present application has high flexibility and can be optimized for different application scenarios.

[0072] In the above embodiment, the model responsible for "self-reflection" to optimize the knowledge base and the model finally executing batch scoring are the same model, which is not a mandatory requirement of the present application. In actual deployment, a dual-model architecture of "reflection-scoring" separation can be adopted: a large model with more parameters and stronger ability can be used as a "reflection model", which focuses on high-precision and deep-level logic extraction from expert feedback to ensure the upper limit of the quality of the knowledge base; at the same time, a lightweight model with fewer parameters and faster reasoning speed can be selected as a "scoring model", which is responsible for efficient and low-cost batch scoring according to the final knowledge base. This role-separated architecture can significantly reduce the overall deployment and operation cost of the system while ensuring the highest scoring quality, improving the scoring efficiency, and is especially suitable for large-scale test scoring scenarios.

[0073] In the present embodiment, the model continuously absorbs expert experience through iteration, making the scoring standard more and more accurate and adapting to the changes of different tests and questions. Moreover, the scoring knowledge base is white-box, and the scoring rules and basis are clear and visible, avoiding the "black box" problem of neural network models and facilitating subsequent dispute handling. There is no need to fine-tune the large-scale model, saving huge computing cost. Only a small amount of expert feedback can drive the optimization of the entire system. Finally, all answer sheets are judged by the unified and stable knowledge base, effectively avoiding the inconsistency problem caused by fatigue or scale floating of human scorers.

[0074] In an embodiment, Figure 2 For the detailed flowchart of the large language model scoring method based on iterative knowledge enhancement provided by the embodiment of the present application, the present embodiment is further optimized and extended on the basis of the above embodiments.

[0075] S200: Based on the reference answer, scoring instructions and multiple expert scoring samples of the specified question, use the large language model to simulate the expert scoring logic to generate a structured initial scoring knowledge base.

[0076] S201: score the unseen examinee's answer sheet using the structured initial scoring knowledge base to obtain a model scoring result.

[0077] S202: obtain an artificial scoring result of the unseen examinee's answer sheet scored by an expert.

[0078] S203: based on the model scoring result and the artificial scoring result, guide the large language model to reflect on itself to update the structured initial scoring knowledge base to form an enhanced scoring knowledge base.

[0079] S204: use the enhanced scoring knowledge base to perform a new round of scoring and reflection, and calculate an inter-rater reliability coefficient between the model scoring and the expert scoring after each iteration.

[0080] S205: determine whether the inter-rater reliability coefficient is higher than a preset threshold in multiple iterations, and whether the core rules of the scoring knowledge base do not change in continuous iterations, if so, perform step S206, otherwise perform step S201.

[0081] S206: determine that the scoring knowledge base reaches a stable state to obtain a target scoring knowledge base.

[0082] S207: score the examinee's answer sheet to be scored using the target scoring knowledge base.

[0083] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] As Figure 3 shown below is an embodiment of the paper grading system of the large language model based on iterative knowledge enhancement provided by the embodiments of the present disclosure. The paper grading method based on iterative knowledge enhancement of the large language model described above belongs to the same inventive concept. The details not described in the embodiment of the paper grading system based on iterative knowledge enhancement of the large language model can be referred to the embodiment of the paper grading method based on iterative knowledge enhancement of the large language model.

[0085] An initial scoring knowledge base generation unit is configured to simulate expert grading logic using a large language model based on reference answers, scoring instructions and multiple expert scoring samples of a specified question to generate a structured initial scoring knowledge base. A scoring knowledge base iterative enhancement unit is configured to iteratively enhance the structured initial scoring knowledge base and repeat the following steps until the scoring knowledge base is stable: score the unseen examinee's answer sheet using the structured initial scoring knowledge base to obtain a model scoring result; obtaining an artificial scoring result of the unseen examinee's answer sheet by an expert; Based on the model scoring result and the artificial scoring result, guiding the large language model to reflect on itself to update the structured initial scoring knowledge base, forming an enhanced scoring knowledge base; Using the enhanced scoring knowledge base for a new round of scoring and reflection, and after each iteration, calculating the inter-rater reliability coefficient between the model scoring and the expert scoring, when the inter-rater reliability coefficient is higher than the preset threshold in multiple iterations, and the core rules of the scoring knowledge base do not change in continuous iterations, it is determined that the scoring knowledge base reaches a stable state, and a target scoring knowledge base is obtained; a scoring unit for scoring the examinee's answer sheet to be scored using the target scoring knowledge base.

[0086] Figure 4 is a hardware structure schematic diagram of an electronic device for implementing various embodiments of the present application.

[0087] The examination marking method based on iterative knowledge enhancement of the large language model provided by the embodiments of the present application can be applied to an electronic device. Those skilled in the art can understand that the electronic device structure involved in the embodiments of the present application does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the illustration, or combine certain components, or different component arrangements. In the embodiments of the present application, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described herein and / or claimed.

[0088] The electronic device can include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charge management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a key, a camera, a display screen, and a SIM card interface, etc.

[0089] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0090] The processor can include one or more processing units, such as: the processor can include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video code, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0091] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0092] The processor can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory can save instructions or data that the processor has just used or repeatedly uses. If the processor needs to use the instruction or data again, it can be directly called from the memory. Avoiding repeated access reduces the waiting time of the processor, thus improving the efficiency of the system.

[0093] The external memory interface can be used to connect an external storage card, such as a MicroSD card, to realize the expansion of the storage capacity of the electronic device. The external storage card communicates with the processor through the external memory interface to realize the data storage function. For example, files such as music and video are saved in the external storage card.

[0094] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory can include a program storage area and a data storage area. The internal memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0095] The wireless communication function of the electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor, etc.

[0096] The wireless communication module can provide solutions for wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), and the like.

[0097] The electronic device can implement audio functions and the like through an audio module, a speaker, a receiver, a microphone, a headset interface, an application processor, and the like.

[0098] The electronic device can implement a photographing function through an ISP, a camera, a video codec, a GPU, a display screen, an application processor, and the like.

[0099] The electronic device can implement a display function through a GPU, a display screen, an application processor, and the like.

[0100] The GPU is a microprocessor for image processing, connected to the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor can include one or more GPUs that execute program instructions to generate or change display information.

[0101] The display screen is used to display images, videos, and the like. The display screen includes a display panel.

[0102] In the storage medium provided in the present application, a program product capable of implementing the method for evaluating the large language model based on iterative knowledge enhancement is stored.

[0103] The method for marking an exam paper based on the large language model with iterative knowledge enhancement comprises: based on a reference answer, a scoring instruction and a plurality of expert scoring samples of a specified question, simulating expert marking logic using a large language model to generate a structured initial scoring knowledge base; iteratively enhancing the structured initial scoring knowledge base, and repeating the following steps until the scoring knowledge base is stable: scoring an unseen examinee's answer using the structured initial scoring knowledge base to obtain a model scoring result; obtaining an artificial scoring result of the unseen examinee's answer by an expert; based on the model scoring result and the artificial scoring result, guiding the large language model to reflect on itself to update the structured initial scoring knowledge base to form an enhanced scoring knowledge base; using the enhanced scoring knowledge base to perform a new round of scoring and reflection, and after each iteration, calculating an inter-rater reliability coefficient between the model scoring and the expert scoring, when the inter-rater reliability coefficient is higher than a preset threshold in multiple iterations, and the core rules of the scoring knowledge base do not change in continuous iterations, determining that the scoring knowledge base reaches a stable state, and obtaining a target scoring knowledge base; and scoring a to-be-scored examinee's answer using the target scoring knowledge base.

[0104] In some possible implementation manners, the method and system for marking an exam paper based on the large language model with iterative knowledge enhancement of the subject matter of the present disclosure can be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above "Exemplary Method" section of the present specification according to various exemplary embodiments of the present disclosure when the program product is run on the terminal device.

[0105] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0106] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A grading method based on an iterative knowledge-enhanced large language model, characterized in that, include: Based on the reference answers, scoring instructions and multiple expert scoring samples for the specified questions, a large language model is used to simulate the expert scoring logic and generate a structured initial scoring knowledge base. Iteratively enhance the structured initial scoring knowledge base, and repeat the following steps until the scoring knowledge base is stable: The model scores the unseen candidates' answer sheets using a structured initial scoring knowledge base. Obtain the manual scoring results of experts on the unseen candidates' answer sheets; Based on the model scoring results and the human scoring results, the large language model is guided to reflect on itself in order to update the structured initial scoring knowledge base and form an enhanced scoring knowledge base; A new round of scoring and reflection is conducted using the enhanced scoring knowledge base. After each iteration, the inter-rater reliability coefficient between the model score and the expert score is calculated. When the inter-rater reliability coefficient is higher than a preset threshold in multiple iterations and the core rules of the scoring knowledge base do not change in continuous iterations, the scoring knowledge base is determined to have reached a stable state, and the target scoring knowledge base is obtained. The target scoring knowledge base is used to score the candidates' answer sheets.

2. The grading method based on an iterative knowledge-enhanced large language model according to claim 1, characterized in that, The generated structured initial scoring knowledge base is represented as follows: ； In the formula, This represents a large language model pre-trained with professional knowledge in the target subject area. Instructions for generating a structured knowledge base, The question is titled "Exam Question". The reference answers are those corresponding to the questions in the test. For scoring purposes, A sample of expert ratings containing at least several with detailed reasoning. This serves as a structured initial scoring knowledge base.

3. The grading method based on an iterative knowledge-enhanced large language model according to claim 1, characterized in that, The model scores were obtained by using a structured initial scoring knowledge base to score the answer sheets of candidates who had not yet been seen. The results are as follows: ； In the formula, This is an instruction to score based on a scoring knowledge base. For questions for which no candidate's answer sheet has been found, It contains at least several sample answer sheets from candidates who have not been seen. For the first Each version of the rating knowledge base According to the first The scoring results of the large language model of the scoring knowledge base for each version.

4. The grading method based on an iterative knowledge-enhanced large language model according to claim 3, characterized in that, The resulting enhanced scoring knowledge base is as follows: ； In the formula, This is an instruction to update the scoring knowledge base. For questions for which no candidate's answer sheet has been found, It contains at least several sample answer sheets from candidates who have not been seen. For large language models after the first An enhanced scoring knowledge base was formed after repeated self-reflection.

5. The grading method based on an iterative knowledge-enhanced large language model according to claim 1, characterized in that, The formula for calculating the inter-rater reliability coefficient between model scores and expert scores is as follows: ； In the formula, The agreement rate between the model scoring results and the expert scoring results. The random consistency rate is the calculated value of the expected consistency rate under the assumption that the model and expert ratings occur randomly.

6. The grading method based on an iterative knowledge-enhanced large language model according to claim 4, characterized in that, Whether the core rules of the scoring knowledge base have changed during successive iterations is determined using the following method: ； In the formula, Instructions to determine whether there have been substantial changes to the scoring knowledge base. For questions for which no candidate's answer sheet has been found, The result is a binary judgment. For the first Each version of the rating knowledge base For large language models after the first An enhanced scoring knowledge base was formed after repeated self-reflection.

7. The grading method based on an iterative knowledge-enhanced large language model according to claim 1, characterized in that, The formula for scoring the candidates' answer sheets using the aforementioned target scoring knowledge base is as follows: ; In the formula, For the candidates' answer sheets, To base the target scoring knowledge base The large language model scoring results The questions for the candidates' answer sheets. The instruction is to score based on a scoring knowledge base, and the scoring result includes a score and a reason for the score.

8. A grading system based on an iterative knowledge-enhanced large language model, characterized in that, include: The initial scoring knowledge base generation unit is used to generate a structured initial scoring knowledge base based on the reference answers, scoring instructions and multiple expert scoring samples of the specified questions, using a large language model to simulate the expert scoring logic. The scoring knowledge base iterative enhancement unit is used to iteratively enhance the structured initial scoring knowledge base and repeat the following steps until the scoring knowledge base is stable: The model scores the unseen candidates' answer sheets using a structured initial scoring knowledge base. Obtain the manual scoring results of experts on the unseen candidates' answer sheets; Based on the model scoring results and the human scoring results, the large language model is guided to reflect on itself in order to update the structured initial scoring knowledge base and form an enhanced scoring knowledge base; A new round of scoring and reflection is conducted using the enhanced scoring knowledge base. After each iteration, the inter-rater reliability coefficient between the model score and the expert score is calculated. When the inter-rater reliability coefficient is higher than a preset threshold in multiple iterations and the core rules of the scoring knowledge base do not change in continuous iterations, the scoring knowledge base is determined to have reached a stable state, and the target scoring knowledge base is obtained. The marking unit is used to mark the candidates' answer sheets using the target marking knowledge base.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the grading method based on an iterative knowledge-enhanced large language model as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the grading method based on the large language model of iterative knowledge enhancement as described in any one of claims 1 to 7.

Images