Automatic evaluation method for subjective questions and answers in hydroelectric field of large language model

Abstract

The invention relates to the technical field of artificial intelligence and natural language processing, and discloses a large language model hydroelectric field subjective question and answer automatic evaluation method, which comprises the following steps: receiving a standard reference answer, and calling a large language model configured as an information extraction role to extract a discrete assessment point set; secondly, inputting the question into a to-be-tested large language model to generate a to-be-tested answer; then a large language model configured as a referee role is called to carry out two-dimensional evaluation, the content dimension judges the coverage condition of the assessment points based on semantic implication and supports a rigid or elastic strategy, and the language dimension executes score deduction according to a defect detection mechanism; and finally, performing weighting calculation according to the weight coefficient to obtain a final score. According to the method, the content and the language evaluation dimension are decoupled, semantic analysis is used for replacing mechanical character matching, the problem that traditional indexes are low in accuracy in the professional field is solved, increment pre-training or instruction fine adjustment of the model can be guided in a targeted mode according to the item scores, and the optimization efficiency of the vertical field model is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to an automated evaluation method for subjective question answering in the field of hydropower using a large language model. Background Technology

[0002] With the in-depth application of large language model technology in the field of hydropower engineering, subjective question answering based on vertical domain knowledge has become an important tool to assist technical personnel in decision support and experience transfer. For such generative tasks, how to efficiently and accurately evaluate the quality of model-generated answers has become a key factor restricting the practical application of the model.

[0003] In existing technologies, the evaluation of subjective question-and-answer questions mainly relies on human expert review or automated metrics based on statistics. While human review can ensure professionalism, it faces problems such as long review cycles, high labor costs, and score fluctuations due to inconsistent subjective standards among reviewers, making it difficult to meet the large-scale, high-frequency iteration requirements of model training. Regarding automated evaluation, traditional natural language processing metrics such as bilingual evaluation substitutes and recall-oriented summary evaluation essentially calculate the statistical features of N-gram co-occurrence between the test text and the reference text, focusing on the degree of overlap of literal characters.

[0004] However, in highly specialized fields such as hydropower, there are instances where standard answers and model responses differ in meaning but are actually the same, using different technical terms to describe the same technical principle, or where there is a trap of meaningless agreement, where high character overlap but incorrect logical connectors lead to completely opposite technical meanings. This makes it difficult to measure the semantic correctness and logical rigor of answers using traditional character matching metrics alone. Therefore, this invention proposes an automated evaluation method for subjective question answering in the hydropower field using a large language model to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an automated evaluation method for subjective question answering in the field of hydropower using a large language model. This method solves the problems of low efficiency and inconsistent standards caused by the reliance on manual evaluation in the field of hydropower, as well as the problems of general natural language evaluation indicators focusing too much on character overlap while ignoring semantic accuracy and professional logic.

[0006] To achieve the above objectives, the first aspect of this invention provides an automated evaluation method for subjective question answering in the field of hydropower using a large language model, which is implemented through the following technical solution: First, the assessment point extraction is performed. The standard reference answer for a specific question is received, and the large language model configured as the information extraction role is called to perform semantic analysis on the standard reference answer, extracting a set of assessment points consisting of multiple discrete assessment points, thereby realizing the transformation of unstructured text into structured evaluation indicators. Next, the test answer generation is performed by inputting the question into the large language model under test and obtaining the generated test answer. The two-dimensional automated scoring is executed again, and the large language model configured as the judge is invoked to evaluate the test answer from the content dimension and the language dimension respectively. The content dimension evaluation focuses on the semantic coverage of the set of assessment points in the test answer, while the language dimension evaluation focuses on detecting language defects according to preset standards. Finally, the final score is calculated by weighting the scores of the two dimensions according to the preset weighting coefficients and outputting an objective and quantitative final score.

[0007] Preferably, in the assessment point extraction stage, the system dynamically determines the number of points to be extracted based on the text length and information density of the standard reference answer. The large language model configured as the information extraction role identifies the core semantic units in the standard reference answer based on the extraction prompt words, removes modifying words, and generates a set of assessment points to ensure the objectivity and conciseness of the evaluation benchmark.

[0008] Preferably, a rigid assessment point matching strategy is adopted when calculating the content dimension score. This strategy hides the standard reference answer and only inputs the set of assessment points and the answer to be tested into the model. The model determines whether each assessment point is covered and calculates the coverage ratio. The content dimension score is determined according to the tiered mapping rules, that is, different coverage ratio ranges are mapped to corresponding tier scores, thereby achieving a hard constraint on the degree of mastery of core knowledge.

[0009] Preferably, when calculating the content dimension score, a reference-enhanced flexible evaluation strategy can be adopted. This strategy simultaneously inputs the set of assessment points, the test answer, and the complete standard reference answer into the model. The model then executes differentiated scoring logic: Rigid constraints are applied to assessment points involving definitions, formulas, or key data; information tolerance is applied to assessment points involving descriptive or explanatory information. The model directly outputs the content dimension score based on the semantic completeness score mapping table, making it suitable for subjective questions with strong open-endedness.

[0010] Preferably, when judging the coverage of assessment points, the judgment is based on semantic implications. If the expression of the answer to be tested contains the core meaning of the assessment point in semantic logic, the assessment point is judged to be covered. It is not required that the answer to be tested and the assessment point are completely matched at the character level, thereby avoiding the mechanical defects of traditional character matching algorithms.

[0011] Preferably, when calculating the language dimension score, a deduction system is used, setting a baseline maximum score for the language dimension, and the model detects three types of defects and deducts points accordingly: If logical flaws such as contradictory statements or reversed cause and effect are detected, the first preset score will be deducted; if expression flaws such as repetitive sentences, infinite loops, or grammatical errors are detected, the second preset score will be deducted; if redundancy flaws such as low effective information density due to the accumulation of general corpus are detected, the third preset score will be deducted. The final language dimension score is the baseline full score minus all deducted scores, and the minimum score is not less than zero.

[0012] Preferably, to improve the quality of the generated responses, the system appends system instructions containing role settings before generating the input question. The large language model under test receives the combined input and adjusts its attention mechanism and generation strategy according to the instructions, thereby generating responses that better conform to the business scenario specifications.

[0013] Preferably, this method utilizes decoupled content and language evaluation dimensions to accurately pinpoint the shortcomings of the large language model under test, thus assisting in subsequent optimization: when the content dimension score is lower than the preset content threshold, it indicates that the model lacks sufficient professional knowledge in the field of hydropower; when the language dimension score is lower than the preset language threshold, it indicates that the model has defects in instruction alignment, logic construction, and standardized response.

[0014] Preferably, the final score is calculated as follows: the content dimension score is multiplied by the content weight coefficient to obtain the content weight score, the language dimension score is multiplied by the language weight coefficient to obtain the language weight score, and the content weight score and the language weight score are added together to obtain the final score. By adjusting the weight coefficients, it can flexibly adapt to different evaluation scenarios that focus on the accuracy of knowledge or the comprehensive expression ability.

[0015] Preferably, a second aspect of the present invention provides an electronic device, including at least one processor and a memory communicatively connected to the at least one processor; the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the large language model-based automated evaluation method for subjective question answering in the hydropower field provided in the first aspect above.

[0016] This invention provides an automated evaluation method for subjective question answering in the field of hydropower using a large language model. It has the following beneficial effects: 1. This invention extracts discrete assessment points from standard reference answers by calling a large language model, and determines coverage based on semantic implication rather than simple character matching. Compared with traditional evaluation indicators based on N-gram overlap rate, it can avoid misjudgment caused by differences in synonymous expressions, realize the transformation of unstructured subjective question answers into structured evaluation standards, and improve the objectivity and accuracy of professional terminology evaluation in the hydropower field.

[0017] 2. This invention constructs a dual-dimensional evaluation mechanism that decouples the content dimension and the language dimension. It uses rigid or flexible strategies to evaluate knowledge coverage and a deduction system to detect logical fallacies and expression defects. This solves the problem that single-dimensional scoring cannot distinguish between factual illusions or correct knowledge but logical confusion, and ensures an independent and comprehensive measurement of the knowledge reserves and logical reasoning ability of the large language model in the vertical domain.

[0018] 3. This invention constructs a precise diagnostic mechanism based on dual-dimensional scoring, which can clearly distinguish the specific shortcomings of the model under test in terms of professional knowledge reserves and instruction alignment ability through the differentiated distribution of content scores and language scores. This changes the traditional general scoring mode that cannot locate the root cause of the problem, thereby providing an objective quantitative basis for the targeted optimization of vertical domain models, effectively reducing the trial and error cost in the model iteration process, and improving optimization efficiency. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a data diagram showing the consistency of content dimensions between the evaluation of various judge models and scoring strategies combined in this invention and the human evaluation. Figure 3 This is an error distribution diagram showing the evaluation of different task types and the human evaluation benchmark in this invention; Figure 4 This is a line graph showing the subjective question evaluation scores of the model at different training stages of this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Embodiments of the present invention can run on one or more computing devices. The computing device can be a server, personal computer (PC), tablet computer, or smartphone, etc. The hardware architecture of the computing device may include: The device includes at least one processor, a memory, and a bus for connecting the processor and the memory. The memory stores a computer program, and the processor is configured to execute the computer program to implement the automated evaluation method provided in embodiments of the present invention. The computing device may also include input or output interfaces for exchanging data with other devices.

[0022] This invention provides an automated evaluation system for subjective question-and-answer questions in the field of hydropower, based on a large language model. The system may include: an assessment point extraction module, an answer generation module, and an automated scoring module.

[0023] The assessment point extraction module is configured to receive standard reference answers. and from the standard reference answer Extract the core logic. Each assessment point should be considered, and the set of assessment points should be output. .

[0024] The answer generation module is configured to receive questions and call the large language model under test to generate answers based on the questions. .

[0025] The automated scoring module is configured to receive a set of assessment points. and the answer to be tested and answer the test Scoring is conducted on both content and language dimensions, and a comprehensive score is calculated.

[0026] Specifically, the automated scoring module includes a content scoring unit and a language scoring unit, with the content scoring unit configured to calculate scores for the content dimension. The language scoring unit is configured to calculate language dimension scores. .

[0027] The automated scoring module scores based on content dimensions. and language dimension score Calculate the final score The calculation method can be expressed by the following formula: ; in: This represents the final weighted total score; This indicates the score for the content dimension; Indicates the score for the language dimension; This represents the response text generated by the model under test; This represents the standard reference answer text used to extract assessment points; The weighting coefficients representing the content dimension scores; The weighting coefficients represent the scores for the language dimension.

[0028] See attached document Figure 1 The method provided in this embodiment decomposes the subjective evaluation task into a quantification process of two independent dimensions: content coverage and language clarity. By introducing a Large Language Model (LLM) to simulate the scoring logic of human experts, it achieves automated evaluation of unstructured text, including the following steps: During the assessment point extraction phase, the system receives responses to questions. Standard Reference Answer Standard Reference Answer The data is textual data from the water conservancy and hydropower sector that has been reviewed by experts. The system calls a large language model configured as an information extraction role and extracts data based on preset prompts. For the standard reference answer Perform semantic analysis and logical deconstruction to extract the standard reference answer for model recognition. The core semantic units are removed, modifying words, and unstructured text is transformed into text containing... A set of discrete assessment points , where the set Number of assessment points It is not a fixed value, but rather based on the standard reference answer. The text length and information density are dynamically determined, with a typical range of 3 to 5 values, for each assessment point. It represents independent knowledge elements, logical judgments, or data constraints.

[0029] During the model answer generation phase, the system will generate the question. The input is fed into the large language model to be evaluated. Based on its internal parameters and the knowledge gained during training, the large language model generates a response to the question. The answer to be tested Answer to be tested It is a piece of natural language text.

[0030] During the automated scoring phase, the system invokes a large language model configured as a judge to analyze the test responses. Conduct a two-dimensional assessment.

[0031] First, in terms of content, the large language model of the judge's role receives the set of assessment points. and the answers to be tested As input, the large language model, acting as the referee, performs a logical retrieval task, judging the set one by one. Each assessment point Is it in the answer to be tested? This is accurately reflected in the fact that the judgment process is based on semantic matching rather than complete character matching; that is, when the large language model of the judge determines the answer to be tested... The description contains assessment points. When the core semantics are determined, the system determines that the assessment point is covered. The system then calculates the content dimension score based on the coverage ratio of the assessment points. .

[0032] Meanwhile, in terms of language, the large language model of the judge receives the test responses. As input, and based on a preset language quality grading standard, the text's expressive quality is independently scored, and a large language model acting as a judge detects the test response. The system checks for logical inconsistencies, repetitive statements, or meaningless accumulation of generic language. If any of these defects are detected, the large language model, acting as the judge, will calculate a score based on a pre-defined baseline deduction model. Specifically, the system sets a maximum baseline score for the language dimension. (In this embodiment, it is set) (The score is consistent with the content dimension score). The large language model for the judge role is configured to identify the following three types of language defects and perform corresponding point deductions: Points deducted for logical flaws: If the answer to be tested... If there are self-contradictory statements, reversed causality, or serious factual illusions in the statements, the judge's model will deduct 2 points. Deduction for expression defects: If the response to be tested... If there are mechanical repetitions, infinite loops, or obvious grammatical errors in the statements, the judge model will deduct 1 point. Redundancy defect deduction: If the answer to be tested is negative. The system contains a large amount of meaningless general corpus (such as excessively long disclaimers used as artificial intelligence language models), resulting in a low effective information density. The judge model will deduct 1 point.

[0033] The large language model used by the referee calculates the score after deductions. If the result is less than 0, it is recorded as 0 points. This final score is output as the language dimension score. .

[0034] Finally, the system uses preset weighting coefficients. and Scoring based on content dimension and language dimension score Perform a weighted summation to calculate the result for the test response. Final score Final score This method is used to quantitatively characterize the comprehensive capabilities of the model under test in handling water conservancy and hydropower professional problems. By evaluating a batch of problems, an overall capability map of the model under test in the vertical field can be constructed.

[0035] See attached document Figure 1 The first step is to extract the assessment points. This step aims to extract the standard reference answers in unstructured text form. Transformed into a semi-structured set of assessment points containing key logical features. .

[0036] During the assessment point extraction step, the system obtains information related to the questions. Standard Reference Answer Standard Reference Answer This is a piece of natural language text that contains answers to questions. The complete set of knowledge points, logical relationships, and professional terminology required.

[0037] The system will provide the standard reference answer. and preset extraction prompts As input data, the data is fed into the assessment point extraction model, which is a pre-configured large language model configured to perform text understanding and information extraction tasks.

[0038] The assessment point extraction model extracts prompt words. The instructions are for the standard reference answer. The analysis process involves identifying core entities, key verbs, constraints, and logical connectors in the answer, and removing redundant descriptive phrases. The assessment point extraction model generates [a set of functions]. A collection of assessment points This process can be represented by the following formula: ; in, This represents the extracted set of assessment points, in the form of: ; Represents a set The first in There are 10 assessment points, each consisting of a short text string representing the standard reference answer. The core semantic unit in; This represents the mapping function executed by the assessment point extraction model.

[0039] Number of assessment points The value is determined by the system based on the standard reference answer. The text features are dynamically determined; specifically, the system determines them based on the standard reference answer. The character length and information density will Set to an integer between 3 and 5, when the standard reference answer is... When the system is short or has a simple logic, the system setting is smaller. Value; when the standard reference answer When the system is long or contains multiple layers of logic, a larger system setting is required. value.

[0040] As a specific example of this embodiment, if the standard reference answer... To define "ecological flow," the above extraction process generates a set of assessment points. It can include the following three assessment points: "Maintaining the health of river ecosystems"; "Minimum flow rate"; "Specific time period". Through the above steps, the system completes the transformation from unstructured text to structured knowledge features, providing an objective comparison benchmark for subsequent automated scoring of content dimensions.

[0041] See attached document Figure 1 After the assessment point extraction step, the test model response generation step is executed. This step aims to process specific domain problems through the large language model under test and obtain the generated natural language text as the evaluation object.

[0042] In the step of generating the model's answer to be tested, the system first determines the question to be evaluated. Issues to be evaluated The data is selected from a pre-built evaluation dataset for water conservancy and hydropower engineering, and is consistent with the standard reference answers used to extract assessment points in the assessment point extraction step. It has a unique corresponding mapping relationship.

[0043] The system will evaluate the issues to be tested. The input is converted into a vector sequence and fed into the large language model under test. The large language model under test is a generative model based on a Transformer architecture or a similar deep neural network architecture, internally storing parameter weights obtained through training. Based on its internal parameter distribution, the large language model under test evaluates the input question. Semantic encoding is performed, and output tags are generated one by one based on a probabilistic prediction mechanism. Finally, these tags are combined to form a complete natural language text, i.e., the test answer. .

[0044] During this process, the instruction text input by the system to the large language model under test can also include preset role-based prompts. For example, in the question to be evaluated, the system may provide pre-defined role-based prompts. Previously, the system command "You are a water conservancy and hydropower expert" was concatenated. The large language model under test received this combined input containing the role setting and adjusted its attention mechanism and generation strategy according to the setting, thereby generating the test response. It can invoke the potential knowledge features of a specific domain.

[0045] The large language model to be tested can be a model instance at different training stages, including but not limited to a general-purpose base model that has not been exposed to domain-specific corpora, a model that has undergone incremental pre-training with domain-specific corpora, or a model that has undergone supervised instruction fine-tuning (SFT). Regardless of the specific training state of the large language model to be tested, its function in this step is to receive text input and output the generated test answer. .

[0046] The system receives and stores the answers to the test. This data is labeled as unstructured text data to be evaluated, and a set of assessment points is established for it. The associated indexes are used by the subsequent automated scoring module.

[0047] See attached document Figure 1 After the test model's answer generation step, an automated scoring step is performed. Among these steps, content-level scoring is the core component of the automated scoring step, aiming to quantitatively evaluate the test answer. The system assesses the coverage and accuracy of core knowledge assessment points. In this embodiment, the system is configured with a large language model for the role of the referee. This large language model is pre-configured, and for different evaluation needs, the content dimension scoring provides two optional implementation strategies: a rigid assessment point matching strategy and a reference-enhanced flexible evaluation strategy.

[0048] In the first implementation, the system employs a rigid assessment point matching strategy (SCM). Under this strategy, the system sets the assessment points... and the answer to be tested The large language model is input to the referee role, but the standard reference answer is hidden. This prevents the large language model of the referee role from obtaining additional reference information besides the assessment points. The large language model of the referee role performs a strict logical retrieval task, traversing the set one by one. Each assessment point And determine whether the assessment point is in the answer to be tested. It was correctly mentioned or described.

[0049] In this implementation, the judgment logic of the large language model of the referee role is based on semantic implication rather than simple keyword matching; that is, if the answer to be tested... Existing and assessment points For semantically equivalent expressions, the large language model of the referee role outputs a judgment result of "covered"; otherwise, it outputs "not covered". Based on the judgment result of the large language model of the referee role, the system calculates the answer to be tested. Set of assessment points coverage ratio The calculation process is expressed by the following formula: ; in, This indicates the coverage ratio of assessment points, with a value range of [value range missing]. ; This is an indicator function.

[0050] When the large language model of the referee role is determined The semantics of are contained in China Times (referred to as) The function value is 1 when the condition is met (e.g., when the condition is not met, the function value is 0). The coverage ratio is then calculated. Then, the system determines the content dimension score according to the preset grading mapping rules. The specific mapping rules are as follows: Referring to Table 1, if If it is 100%, then It is recorded as 5 points; if Located in the interval ,but It is recorded as 4 points; if Located in the interval ,but It is recorded as 3 points; if Located in the interval ,but It is worth 2 points; if Located in the interval ,but It is recorded as 1 point; if Located in the interval ,but It is recorded as 0 points.

[0051] Table 1. Scoring Criteria for Rigid Assessment Point Matching Strategy

[0052] In the second implementation, the system employs a Reference Enhanced Resilience Evaluation (REE) strategy. Under this strategy, the system sets the evaluation points... Answer to be tested And complete standard reference answers Simultaneously, the large language model of the referee role is input, and the large language model of the referee role is configured to simulate the comprehensive judgment logic of human experts and perform flexible scoring.

[0053] In this implementation, the large language model of the referee role responds to the test according to the following logical rules. Rate it: First, enforce the core concept of rigid constraints for sets. The assessment points involve definitions, formulas, or key data; the large language model comparison of the referee role. and ,like If the statement contains factual errors or is missing key elements, the large language model of the judges will determine that this part will not receive any points.

[0054] Second, partial information fault tolerance is implemented for sets. If the assessment points involve descriptive, explanatory, or non-core aspects, The expression and While not entirely identical, if logically sound and not deviating from the core semantics, the large language model, acting as the judge, will assign points accordingly rather than directly awarding zero.

[0055] Ultimately, the large language model, acting as the referee, combines the judgment results of rigid constraints and flexible fault tolerance, quantifies the score based on a pre-set semantic completeness score mapping table, and directly outputs the content dimension score. .

[0056] The specific mapping logic is as follows: 5 points (complete coverage): Responses to be tested Accurately describes the set The core logic of all assessment points must be presented in a way that does not contain factual errors, and its expression and sentence structure are allowed to be similar to the standard reference answer. Different (i.e., satisfying semantic equivalence); 4 points (high coverage): Test response Covers the set It covers all the core assessment points, but omits minor modifying information or contains very slight ambiguity in its description; 3 points (moderate coverage): Test response Covers the set More than half of the assessment points are not mentioned, or although all assessment points are mentioned, the logical relationships are not expressed rigorously enough. 2 points (low coverage): Test response Only the set is covered Less than half of the assessment points are flawed, or contain obvious logical deviations. 1 point (very low coverage): Response to the test It only mentions keywords related to the problem, fails to reflect the specific assessment points, or contains serious factual errors; 0 points (not covered): Answer to be tested Common phrases for responding to questions as completely wrong, not answered, irrelevant, or a refusal to answer.

[0057] See attached document Figure 4 This application example aims to use the aforementioned automated evaluation method to quantitatively assess the capability evolution of a large language model during the adaptation training process in the vertical field of water conservancy and hydropower.

[0058] In this embodiment, the implementation background involves the technical process of transforming a general-purpose large language model into a model specifically for the field of water conservancy and hydropower. To analyze the specific impact of different training methods on model performance, this embodiment constructs and selects three model versions at different training stages as test objects. These three test objects share the same underlying network architecture (e.g., a Transformer architecture with 32 billion parameters), but exhibit significant differences in the training history of their model parameters, thus forming an experimental group sequence for comparative analysis.

[0059] The first test object is the base model, which is a general pre-trained large language model. Its training corpus mainly consists of general domain Internet text, encyclopedia data and books. Before being connected to this evaluation system, the base model has not been exposed to specially screened or organized water conservancy and hydropower domain corpus. The base model represents the initial capability state before domain adaptation and serves as the zero-point benchmark for evaluating the domain knowledge injection effect in this embodiment.

[0060] The second test object is the domain pre-trained model. This model is generated after incremental pre-training on the base model. During the construction process, the system collects and organizes unstructured text data in the field of water conservancy and hydropower, including but not limited to textbooks for higher education in water conservancy, academic literature on water conservancy and hydropower, national and industry technical specifications, as well as water resources bulletins and engineering reports from various administrative regions. The total amount of domain data is set to be more than 500GB. The system inputs the above unstructured text data into the base model for self-supervised learning. By predicting the next autoregressive language modeling task, the test object adjusts the internal parameter weights of the domain pre-trained model, thereby transforming professional terms, physical laws and engineering common sense in the field into implicit memory in the model parameters.

[0061] The third test object is the supervised fine-tuning model. The supervised fine-tuning model is a model version generated after performing supervised fine-tuning operations on the domain pre-trained model. In the specific construction process, the system automatically builds or manually annotates a dataset containing more than 100,000 questions and answers in the field of water conservancy and hydropower. The dataset covers subjective and objective question types and is uniformly formatted into a structured form of "instruction, input, and output". The system uses this structured dataset to perform full parameter fine-tuning or efficient parameter fine-tuning on the domain pre-trained model. By calculating the loss function between the predicted output and the standard output and backpropagating, the model learns to follow the human instruction format and activates the domain knowledge stored in the pre-training stage.

[0062] This embodiment uses the three test objects mentioned above as test models in the aforementioned automated evaluation system. Utilizing the same assessment point extraction mechanism and the same large language model configuration for the referee role, it obtains the content dimension scores for each stage of the model. and language dimension score The system uses these specific values ​​to... Figure 4 The distribution along the time axis or training phase axis is shown to generate evaluation conclusions about the model's training effectiveness.

[0063] The evaluation process in this application embodiment aims to obtain quantitative performance data of the aforementioned base model, domain pre-trained model, and supervised fine-tuning model in handling water conservancy and hydropower professional problems based on a unified evaluation benchmark.

[0064] During the evaluation process, the system first loads a pre-set subjective evaluation dataset for the water resources and hydropower profession. This dataset contains 409 pairs of "questions and standard reference answers," covering nine professional fields including hydraulics, engineering construction, and hydropower operation and maintenance. The questions are categorized into three types based on cognitive level: conceptual explanation, factual introduction, and comprehensive understanding. Following the assessment point extraction method, the system pre-calls the assessment point extraction model to evaluate each standard reference answer in the dataset. Process the data to generate a corresponding set of assessment points. And establish an index database.

[0065] The system configures a large language model with the role of a referee to perform automated scoring tasks. Based on the results of the verification examples, this process preferably uses a large language model with advantages in long text understanding (such as GLM-4-plus) as the large language model for the referee role, and configures a reference-enhanced resilient evaluation strategy (REE), that is, the large language model with advantages in long text understanding receives the set of assessment points when scoring. Answer to be tested and standard reference answers The complete context ensures the accuracy of assessments of comprehensive issues.

[0066] The evaluation process includes comparative experiments targeting the prompt word strategy. For each test subject (base model, PT model, SFT model), the system performs response generation under two different input conditions: The first condition is the no-character prompt mode, where the system directly presents the problem. Input the model to be tested; The second condition is a role-based prompting mode, where the system provides prompts for questions. The system-level instructions displayed before the main text are "You are a water conservancy and hydropower expert", which constructs an input sequence that includes the role settings.

[0067] The system performs the following operations in sequence: First, the model under test is invoked to generate test answers for all questions in the dataset under the different input conditions described above. ; Second, the large language model of the referee role utilizes a pre-stored set of assessment points. and standard reference answer For each answer to be tested A two-dimensional scoring method is used, with scores output for each content dimension. and language dimension score ; Third, the system statistically aggregates the scores of all individual questions for each model under a specific prompt word strategy and calculates the average score.

[0068] Ultimately, the system outputs six sets of data points, corresponding to the content and language scores of the pedestal model, PT model, and SFT model under both scenarios with and without role prompts. These data points form the numerical basis for analyzing the evolution of the model training effect.

[0069] This section details how the system analyzes the evolution of the large language model's capabilities during domain adaptation based on the acquired quantitative scoring data (i.e., content dimension scores and language dimension scores), and generates targeted training optimization strategies accordingly.

[0070] The system first analyzes the content dimension score. The system monitored the changes in the content dimension score across the three consecutive training stages—base model, domain pre-trained model (PTModel), and supervised fine-tuning model (SFTModel)—with role-based prompts. The score increased monotonically from 2.29 in the base model to 2.95 in the domain pre-trained model, and finally reached 3.19 in the supervised fine-tuning model stage. Based on this data trend, the system determined that the model exhibited a knowledge accumulation effect.

[0071] Specifically, the system's score improvement from the base model to the domain pre-trained model is attributed to the parameterized storage process of unstructured knowledge. During the domain pre-training phase, unstructured texts such as textbooks and standards input into the system are transformed into implicit memories in the model parameters through self-supervised learning, thereby improving the model's coverage of domain-specific terminology and physical laws. The system's score improvement from the domain pre-trained model to the supervised fine-tuning model is attributed to the enhanced knowledge retrieval capability. During the supervised fine-tuning phase, the question-answer pair data input into the system optimizes the model's retrieval path for implicit knowledge through an instruction alignment mechanism, enabling the model to respond to questions more accurately and output correct professional knowledge.

[0072] Secondly, the system analyzes the language dimension score. During the three training phases, the system detected a reversal pattern in the language dimension score: it dropped significantly from 3.71 points in the base model to 2.48 points in the domain pre-trained model, and then rebounded to 4.03 points in the supervised fine-tuning model phase.

[0073] Based on this data characteristic, the system determines that there is a temporary antagonistic relationship between domain knowledge injection and general language expression ability. During the domain pre-training stage, due to the intake of a large amount of high-density and obscure professional language data, the original general language generation probability distribution of the model is disturbed, resulting in a decrease in the fluency and logical clarity of the output text. During the supervised fine-tuning stage, the system realigns the language generation layer of the model by introducing instruction fine-tuning data that conforms to human interaction norms. This process not only repairs the damage to the expression ability caused in the pre-training stage, but also further improves the language score through standardized question-and-answer formats, achieving a dual optimization of knowledge density and expression clarity.

[0074] Furthermore, the system analyzes the impact of prompting strategies on model performance. Comparing data sequences with and without character prompts reveals that explicit character settings generate positive gains at all stages of model training, particularly in the base model stage. When the system instruction "You are a water conservancy and hydropower expert" is added, the content score increases from 1.81 to 2.29. Based on this data, the system determines that the general base model already contains some domain-specific common sense, but it is dormant. Explicit contextual guidance (i.e., character prompts) can act as an activation signal, adjusting the model's attention weights to prioritize the use of a subset of parameters relevant to a specific domain.

[0075] Finally, based on the above analysis results, the system outputs a training optimization strategy for large language models in the field of water conservancy and hydropower: First, in terms of training path planning, the system establishes a two-stage paradigm of incremental pre-training supervised fine-tuning. When a low content score is detected, the system generates an instruction to increase the size of the pre-training corpus; when a low language score is detected, the system generates an instruction to execute the fine-tuning instruction and improve expressive ability.

[0076] Second, in terms of inference deployment strategy, the system forces the loading of explicit role prompts in the input layer. Regardless of whether the model has been fine-tuned, the system regards role setting as a necessary preprocessing step to improve performance, ensuring that the model can output the best professional content and language expression in actual applications.

[0077] To verify the reliability of the automated evaluation method described in the above embodiments and to select the optimal model configuration for performing the refereeing task, this embodiment conducted a quantitative data analysis experiment. For example... Figure 2As shown in Tables 2 and 3, the verification process selected a subjective evaluation dataset for the water conservancy and hydropower profession containing 409 question-and-answer pairs. Several human experts were invited to score all the test answers based on two dimensions: content accuracy and language clarity. The standard human scores were then used as the baseline true values.

[0078] The system inputs the same batch of test responses into an automated evaluation module configured with different judging models (e.g., DeepSeek-R1, GLM-4-plus, Qwen-2.5) and different scoring strategies (Strict Assessment Point Matching Strategy, Reference Enhanced Flexible Evaluation Strategy, REE) to obtain automated scores. The system calculates the difference between the two. and through the sum of squared residuals To measure the degree of dispersion of automated scoring from human scoring, in the content dimension verification, different judge models showed obvious heterogeneity in their responses to scoring strategies.

[0079] Specifically, when using a rigid assessment point matching (SCM) strategy, the Qwen-2.5 model exhibits the highest consistency with human ratings, with a perfect consistency rate (i.e., The proportion of the total score reached 42.79%, and the sum of squared residuals was kept at a low level. This indicates that under the condition of shielding the complete reference answer and making logical judgments based only on discrete assessment points, the Qwen-2.5 model can perform the coverage calculation task and reduce the scoring bias caused by excessive association.

[0080] By comparing and analyzing the effects of introducing the Enhanced Resilience Evaluation (REE) strategy, the system monitored the sum of squared residuals of the GLM-4-plus and Qwen-2.5 models. Both remained in the lower range, and GLM-4-plus under both strategies The values ​​(824 and 767 respectively) were the lowest in the group. This data feature verifies that the two types of models have strong semantic alignment capabilities, can use complete standard reference answers to smoothly correct the matching results of the assessment points, and handle complex situations where the semantics are correct but the expression forms are diverse.

[0081] However, the system also detected that the sum of squared residuals of the DeepSeek-R1 model decreased after switching to the Reference Enhanced Resilience Evaluation Strategy (REE). A significant deterioration occurred, with the value increasing from 1235 under the rigid strategy to 1614, while the acceptable error rate (i.e. The percentage dropped from 88.51% to 79.95%, a phenomenon that reveals the limitations of some reasoning enhancement models in handling multi-source benchmarks (simultaneous input assessment points). With complete answer When dealing with fuzzy instructions (such as "handle at your discretion"), instruction compliance conflicts or over-reasoning occurred. This indicates that in the absence of strong domain prior knowledge constraints, introducing flexible strategies actually introduces scoring noise. Based on this verification result, when constructing the final automated evaluation process, the system configures rigid strategies for reasoning-sensitive models and flexible strategies for semantic understanding models to maximize evaluation accuracy.

[0082] Table 2. Consistency of Language Dimension Between Automated and Manual Evaluations of Various Referee Models

[0083] Table 3. Sum of Squared Residuals Between Question Evaluation and Human Evaluation Benchmarks for Different Task Types

[0084] Furthermore, in the validation of the language dimension, the system analyzed the consistency data between automated and human scoring. Statistical results showed that the sum of squared residuals of all participating judge models in the language dimension... The error rate is distributed within the range of 500 to 700, which is significantly lower than the error level of the content dimension (range of 700 to 1700). Meanwhile, the acceptable error rate for each model in the language dimension exceeds 90%. For example, the acceptable error rate of GLM-4-plus reaches 95.35%. This validation result indicates that current large language models have reached a high level of maturity in judging general language capabilities such as text fluency, logical coherence, and the presence of repetitive generation, and can serve as a reliable evaluation tool independent of specialized content. Based on the above validation data, this embodiment confirms the effectiveness of decoupling content evaluation from language evaluation and configuring weight coefficients separately. and The effectiveness of the technical approach.

[0085] See attached document Figure 3 The adaptive verification process in this embodiment aims to evaluate the robustness of the aforementioned automated evaluation method in handling water conservancy and hydropower professional problems with different cognitive levels and logical complexities. Based on Bloom's Taxonomy of Educational Objectives and domain knowledge characteristics, the system divides the 409 questions and standard reference answer pairs in the verification dataset into three task subsets: concept explanation, fact presentation, and comprehensive understanding.

[0086] Conceptual explanation questions focus on the standardized explanation of proper nouns, physical definitions, or engineering terms, and their standard answers have clear semantic boundaries; factual description questions focus on the enumeration of specific engineering parameters, geographical distributions, or historical data, and their standard answers contain high-density entity information; comprehensive understanding questions focus on the analysis of complex water conservancy mechanisms, scheme comparison, or causal reasoning, and their standard answers have complex logical structures and divergent expressions.

[0087] The system selects the referee model configuration that performed best in the previous embodiment (e.g., the GLM-4-plus model combined with a reference-enhanced resilient evaluation strategy), performs automated scoring on the three task subsets mentioned above, and calculates the automated content dimension score for each subset. Compared with human rating Sum of squared residuals between Since the sample size of each subset is different, the system further calculates the average sum of squared residuals to eliminate the influence of sample size differences, thereby achieving horizontal comparison.

[0088] Figure 3 As shown in Table 3, the statistical data revealed a positive correlation between evaluation error and the cognitive complexity of the questions. Specifically, for concept explanation and fact description questions, the system calculated the sum of squared residuals. It is in the low range. Data shows that in both types of tasks, the standard reference answer... Assessment points It directly corresponds to specific entity nouns or definition phrases, with low semantic ambiguity. When performing rigid matching or flexible evaluation, the judging model can accurately match the test answer. The system identifies the relevant knowledge features, thus the automated scoring and the human expert scoring show a high degree of consistency.

[0089] In contrast, for comprehensive comprehension questions, the system detected the sum of squared residuals. The error rate shows an upward trend, with values ​​significantly higher than the first two types of tasks. Technical analysis indicates that this increase in error primarily stems from the semantic diffusion of answers to comprehensive questions. In comprehensive comprehension tasks, correct answers are not limited to hitting specific keywords but rely on the completeness of multi-step logical reasoning. Although the assessment point extraction module can extract key logical nodes, the tested models use sentence structures or argumentation paths that are drastically different from the standard answers when expressing these logics. This diversity of expression increases the computational difficulty for the judging model to perform semantic alignment, leading to scoring discrepancies in some marginal cases.

[0090] Despite the aforementioned error fluctuations, system verification data shows that the acceptable error rate for comprehensive comprehension questions (i.e., the absolute value of the difference between automated and human scoring) is within acceptable limits. The sample share remained above 85%, which proves the effectiveness of the reference-enhanced resilience evaluation strategy. That is, by introducing complete context as the basis for judgment, the judge model can, to some extent, overcome the limitations of literal matching and capture the consistency of deep logic when dealing with divergent problems.

[0091] In summary, this embodiment verifies the generalization ability of the automated evaluation method on different types of tasks in the field of water conservancy and hydropower. The system confirms that the method is not only suitable for objective concept and fact-based assessments, but also provides statistically significant and reliable scoring references when dealing with high-level comprehensive analysis problems, meeting the application needs of all scenarios from basic knowledge examination to complex ability assessment.

[0092] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0093] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. An automated evaluation method for subjective question answering in the field of hydropower using a large language model, characterized in that, Includes the following steps: S1. Receive the standard reference answer to the question, call the large language model configured as the information extraction role, perform semantic analysis on the standard reference answer based on the hydropower professional knowledge system, and extract a set of assessment points consisting of multiple discrete assessment points; S2. Input the question into the large language model to be tested, and receive the test answer generated by the large language model to be tested based on the question; S3. Call the large language model configured as the referee to score the content dimension and language dimension of the answer to be tested. The content dimension scoring process includes: inputting the set of assessment points into the large language model configured as the judge role, determining whether the professional knowledge element assessment points in the set of assessment points are covered in the answer to be tested, and calculating the content dimension score based on the coverage. The language dimension scoring process includes: detecting whether the test answer has language defects according to the preset language quality grading standard. The language defects include illogical sentences, incomplete semantic expression, and repetitive and redundant expressions. The language dimension score is calculated based on the detection results. S4. Based on the preset content weight coefficient and language weight coefficient, the content dimension score and the language dimension score are weighted and summed to obtain the final score for the answer to be tested.

2. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S1, the number of assessment points in the assessment point set is dynamically determined based on the text length and information density of the standard reference answer, and the number ranges from 3 to 5. The large language model configured as the information extraction role is configured to identify core professional terms, physical constraints, and logical connectors in the standard reference answer based on preset extraction prompts, and remove modifying words to generate the assessment point set.

3. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S3, a rigid assessment point matching strategy is used when calculating the content dimension score. The rigid assessment point matching strategy specifically includes: The set of assessment points and the answer to be tested are input into the large language model configured as a referee, and the standard reference answer is hidden, so that the large language model configured as a referee makes judgments only based on the set of assessment points; The large language model configured as the referee determines one by one whether each assessment point in the assessment point set is covered in the answer to be tested, and the ratio of the number of covered assessment points to the total number of assessment points is calculated as the coverage ratio. According to the preset grading mapping rules, the coverage ratio is mapped to the content dimension score. The grading mapping rules include: when the coverage ratio is 100%, it is recorded as a full score; when the coverage ratio falls into different preset percentage ranges, it is recorded as the score of the corresponding grading level.

4. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S3, a reference-enhanced resilience evaluation strategy is used when calculating the content dimension score. This reference-enhanced resilience evaluation strategy specifically includes: The set of assessment points, the answer to be tested, and the complete standard reference answer are simultaneously input into the large language model configured as the referee role; Elastic scoring is performed through the large language model configured as the referee role. Rigid constraints are applied to assessment points in the assessment point set that involve hydropower professional definitions, calculation formulas or key engineering data. Information fault tolerance is applied to assessment points in the assessment point set that involve engineering phenomenon descriptions or qualitative explanations. The large language model configured as the referee outputs the content dimension score based on a pre-set semantic completeness score mapping table.

5. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 3, is characterized in that... The step of determining whether each assessment point in the assessment point set is covered in the answer to be tested by using the large language model configured as a referee includes: When the large language model configured as the referee determines that the expression in the answer to be tested contains the core meaning of the assessment point or an equivalent professional expression, it determines that the assessment point is covered, without requiring the answer to be tested to perform a complete character match with the assessment point.

6. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S3, calculating the language dimension score based on the detection results includes: A baseline maximum score is set for the language dimension, and the following three types of language defects are identified and deducted points using the large language model configured as the judge: If the answer to be tested contains internal logical conflicts such as contradictions between statements or reversals of cause and effect, it is judged as a logical defect and the first preset score is deducted. If the answer to be tested contains repeated sentences, infinite loops, or grammatical errors, it is judged as an expression defect and a second preset score is deducted. If the answer to be tested contains a general corpus of language that results in the effective professional information density being lower than a preset threshold, it is judged as a redundancy defect and a third preset score is deducted. Calculate the baseline full score minus all deducted points. If the result is less than zero, it is recorded as zero points. The result is used as the language dimension score.

7. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S2, before inputting the question into the large language model to be tested, the following steps are also included: Prepend system instructions containing preset role settings before the aforementioned issue; The large language model to be tested is configured to receive a combination of input including system instructions set by the preset role and the question, and to adjust the attention mechanism according to the system instructions set by the preset role to generate the answer to be tested.

8. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S4, the final score is calculated as follows: the content dimension score is multiplied by the content weight coefficient to obtain the content weighted score, the language dimension score is multiplied by the language weight coefficient to obtain the language weighted score, and the content weighted score and the language weighted score are added together to obtain the final score.

9. The automated evaluation method for subjective question answering in the hydropower field using a large language model, as described in claim 1, is characterized in that... In step S4, the final score is calculated as follows: The content dimension score is multiplied by the content weight coefficient to obtain the content-weighted score, the language dimension score is multiplied by the language weight coefficient to obtain the language-weighted score, and the content-weighted score and the language-weighted score are added together to obtain the final score.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

Images