Model evaluation method and apparatus

Abstract

A model evaluation method and device are disclosed. The method comprises: for each knowledge explanation video, inputting an ordered explanation step of the knowledge explanation video into a generation model to obtain a plurality of questions, a plurality of assertions, a target answer and an explanation step set corresponding to each question, a target label and an explanation step set corresponding to each assertion for the knowledge explanation video; merging the explanation step sets corresponding to all questions and all assertions respectively into a preliminary evidence set; for each explanation step set in the preliminary evidence set, obtaining support information of the explanation step set through at least two different verification models; in response to the indication results of the support information being all yes, retaining the explanation step set; merging the retained explanation step sets into a standard evidence set; and evaluating a to-be-evaluated model based on the standard evidence set.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence and multimodal model evaluation technology, and more specifically, to model evaluation methods and apparatus. Background Technology

[0002] Multimodal large models are increasingly being applied to educational support scenarios. Educational lecture videos typically include teacher explanations, slides, diagrams, blackboard writing, and an explanation process that unfolds over time. Unlike problems that rely on only a single local fact, problems in educational scenarios often require reconstructing the introduction of concepts, the setting of conditions, intermediate derivations, and the explanation of conclusions by following multiple ordered explanatory steps.

[0003] Existing evaluations typically place more emphasis on whether the final answer is correct. However, the model being evaluated may rely on parameter memory, general knowledge, or local visual cues to generate fluent and semantically correct answers. The explanation steps provided are not the actual explanation steps that support the answer in the lecture. In other words, such answers are correct on the surface, but do not reflect the specific teaching path in the lecture.

[0004] To address the aforementioned issues, common retrieval evaluation methods typically treat evidence location (i.e., the chain of explanatory steps that actually supports the answer) as a problem of ranking relevant segments. For procedural knowledge in educational videos, simply finding segments with similar themes is insufficient; it's also necessary to retain essential bridging steps, exclude visually recurring steps that no longer correspond to the current content, and avoid referencing subsequent extended content. Therefore, it's necessary to use the minimum, sufficient, and process-dependent chain of explanatory steps as the supervised object, evaluating the model's evidence recognition, evidence-conditional response, and evidence interpretation and verification capabilities. However, currently, there is a lack of methods for obtaining the minimum, sufficient, and process-dependent chain of explanatory steps. Summary of the Invention

[0005] The embodiments of this disclosure provide a model evaluation method and apparatus that can effectively solve the problem that the prior art lacks a method for obtaining a minimal, sufficient, and process-dependent chain of explanatory steps.

[0006] In one general aspect, a model evaluation method is provided, comprising: for each knowledge explanation video, inputting the ordered explanation steps of the knowledge explanation video into a generative model to obtain multiple questions, multiple assertions, a set of target answers and explanation steps corresponding to each question, and a set of target labels and explanation steps corresponding to each assertion. Each explanation step in the ordered explanation steps includes at least the corresponding subtitle text and key image frames. Assertions are knowledge points inferred from the knowledge explanation video. The set of explanation steps corresponding to each question is a set of explanation steps supporting the target answer of the question. Target labels include support labels, rebuttal labels, and empty labels. When the target label corresponding to an assertion is an empty label, the set of explanation steps corresponding to the assertion is empty; when the target label corresponding to an assertion is a support label or a rebuttal label, the set of explanation steps corresponding to the assertion is empty. The set of explanation steps for supporting or refuting assertions; merging the sets of explanation steps corresponding to all questions and all assertions into a preliminary evidence set; for each set of explanation steps in the preliminary evidence set, obtaining supporting information for the set of explanation steps through at least two different verification models, wherein the supporting information indicates whether the label information of the corresponding object in the set of explanation steps is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted; if the object is a question or an assertion, the label information is the target answer when the object is a question, and the label information is the target label when the object is an assertion; in response to the indication result of the supporting information being yes, retain the set of explanation steps; merge the retained set of explanation steps into a standard evidence set; evaluate the model to be evaluated based on the standard evidence set.

[0007] Optionally, supporting information for the set of explanation steps is obtained through at least two different verification models, including: inputting the set of explanation steps, the object, and the object's label information into a first verification model to obtain first supporting information, which indicates whether the object's label information is correct, whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; inputting the set of explanation steps, the object, and the object's label information into a second verification model to obtain second supporting information, which indicates whether the object's label information is correct, whether the set of explanation steps can obtain the object's label information, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; and determining the first supporting information and the second supporting information as supporting information.

[0008] Optionally, before obtaining supporting information for the set of explanation steps through at least two different verification models, the method further includes: inputting the set of explanation steps, the object, and the object's label information into a third verification model to obtain third supporting information. The third supporting information indicates whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted. In response to the indication result of the third supporting information not being all yes, the set of explanation steps is corrected to obtain a corrected set of explanation steps.

[0009] Optionally, before merging the retained set of explanation steps into the standard evidence set, the method further includes: in response to the fact that the indication results of the supporting information are not all yes, inputting the set of explanation steps, the object, and the object's label information into the fourth verification model to obtain the fourth supporting information, which indicates whether the object's label information is correct, whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; in response to the fact that the indication results of the fourth supporting information are all yes, retaining the set of explanation steps.

[0010] Optionally, the evaluation of the model to be evaluated is performed based on a standard evidence set, including: for each question in the set of explanation steps in the standard evidence set, the following processing is performed: inputting the question and the explanation step library into the model to be evaluated to obtain a first estimated set of explanation steps for the question, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; based on the first estimated set of explanation steps and the set of explanation steps corresponding to the question in the standard evidence set, obtaining a first number of steps and a second number of steps, wherein the first number of steps is the number of explanation steps in the first estimated set of explanation steps that exceeds the set of explanation steps, and the second number of steps is the number of explanation steps in the set of explanation steps that exceeds the first estimated set of explanation steps; based on the first number of steps and the second number of steps, obtaining the first double-penalty evidence information of the model to be evaluated, wherein the first double-penalty evidence information indicates the ability of the model to be evaluated to obtain the explanation steps it relies on in the process of obtaining the answer to the question.

[0011] Optionally, the model to be evaluated is evaluated based on a standard evidence set, including: for each question in the set of explanation steps in the standard evidence set, the following processing is performed: inputting the question and the candidate set of explanation steps into the model to be evaluated to obtain the predicted answer and the reference step identifier for the question, wherein the candidate set of explanation steps is either the first set of predicted explanation steps or the set of explanation steps in the standard evidence set corresponding to the question, and the reference step identifier is the identifier of the explanation step in the candidate set of explanation steps referenced by the model to be evaluated when reasoning about the predicted answer; based on the explanation steps corresponding to the reference step identifier and the set of explanation steps in the standard evidence set corresponding to the question, the number of third steps and the number of fourth steps are obtained, wherein the number of third steps is the number of explanation steps that the explanation steps corresponding to the reference step identifier exceed the number of explanation steps in the set of explanation steps, and the number of fourth steps is the number of explanation steps that the explanation steps in the set exceed the number of explanation steps corresponding to the reference step identifier; based on the number of third steps and the number of fourth steps, the second double-penalty evidence information of the model to be evaluated is obtained, wherein the second double-penalty evidence information indicates the ability of the model to be evaluated to obtain the answer to the question.

[0012] Optionally, after obtaining the second double-penalty evidence information of the model to be evaluated based on the number of the first step and the number of the second step, the method further includes: evaluating the answer quality of the predicted answer based on a predetermined evaluation model; and determining, in response to the answer quality being greater than a first preset value and the second double-penalty evidence information being less than a second preset value, that the model to be evaluated exhibits a false knowledge phenomenon where the predicted answer is correct but the explanation steps on which it is based are incorrect.

[0013] Optionally, the model to be evaluated is evaluated based on a standard evidence set, including: for each assertion in the set of explanatory steps corresponding to the standard evidence set, the following processing is performed: inputting the assertions and the explanatory step library into the model to be evaluated to obtain the predicted label for the assertion and the second predicted explanatory step set; based on the second predicted explanatory step set and the set of explanatory steps corresponding to the assertion in the standard evidence set, obtaining the number of fifth steps and the number of sixth steps, wherein the number of fifth steps is the number of explanatory steps in the second predicted explanatory step set that is greater than the explanatory step set, and the number of sixth steps is the number of explanatory steps in the explanatory step set that is greater than the second predicted explanatory step set; based on the number of fifth steps and the number of sixth steps, obtaining the third double-penalty evidence information of the model to be evaluated, wherein the third double-penalty evidence information indicates the ability of the model to be evaluated to verify the truth of the assertions.

[0014] Optionally, the above method further includes: for each set of explanation steps in the standard evidence set, performing the following processing: inputting the object corresponding to the set of explanation steps and the explanation step library into the initial process-aware step retrieval model to obtain a third estimated set of explanation steps for the object, wherein the explanation step library includes the ordered explanation steps of each knowledge explanation video in multiple knowledge explanation videos; determining the target loss based on the third estimated set of explanation steps and the set of explanation steps; adjusting the parameters of the initial process-aware step retrieval model based on the target loss to obtain the process-aware step retrieval model.

[0015] In another general aspect, a model evaluation device is provided, comprising: a generation unit configured to, for each knowledge explanation video, input the ordered explanation steps of the knowledge explanation video into a generation model to obtain multiple questions, multiple assertions, a set of target answers and explanation steps corresponding to each question, and a set of target labels and explanation steps corresponding to each assertion for the knowledge explanation video, wherein each explanation step in the ordered explanation steps includes at least subtitle text and key image frames corresponding to the explanation step, the assertion is a knowledge point inferred based on the knowledge explanation video, the set of explanation steps corresponding to each question is a set of explanation steps supporting the target answer corresponding to the question, and the target labels include support labels, rebuttal labels, and empty labels; when the target label corresponding to an assertion is an empty label, the set of explanation steps corresponding to the assertion is empty; when the target label corresponding to an assertion is a support label or a rebuttal label, the set of explanation steps corresponding to the assertion is a set of explanation steps supporting or rebutting the assertion; and a first merging unit configured to... The system merges the sets of explanation steps corresponding to all questions and all assertions into a preliminary evidence set. A verification unit is configured to, for each set of explanation steps in the preliminary evidence set, obtain supporting information for the set of explanation steps using at least two different verification models. This supporting information indicates whether the label information of the corresponding object in the set of explanation steps is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted. If the object is a question or an assertion, the label information is the target answer when the object is a question, and the label information is the target label when the object is an assertion. A retention unit is configured to retain the set of explanation steps in response to the indication of the supporting information being "yes". A second merging unit is configured to merge the retained set of explanation steps into a standard evidence set. An evaluation unit is configured to evaluate the performance of the model to be evaluated based on the standard evidence set.

[0016] Optionally, the verification unit is further configured to input the set of explanation steps, the object, and the object's label information into a first verification model to obtain first supporting information. The first supporting information indicates whether the object's label information is correct, whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; input the set of explanation steps, the object, and the object's label information into a second verification model to obtain second supporting information. The second supporting information indicates whether the object's label information is correct, whether the set of explanation steps can obtain the object's label information, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; and determine the first supporting information and the second supporting information as supporting information.

[0017] Optionally, the verification unit is further configured to input the set of explanation steps, the object, and the object's label information into a third verification model before obtaining supporting information for the set of explanation steps through at least two different verification models, thereby obtaining third supporting information. The third supporting information indicates whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted. In response to the indication result of the third supporting information not being all yes, the set of explanation steps is corrected to obtain a corrected set of explanation steps.

[0018] Optionally, the verification unit is also configured to, in response to the indication result of the supporting information not being all yes, input the set of explanation steps, the object, and the label information of the object into the fourth verification model to obtain the fourth supporting information. The fourth supporting information indicates whether the label information of the object is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted; in response to the indication result of the fourth supporting information being all yes, retain the set of explanation steps.

[0019] Optionally, the evaluation unit is further configured to perform the following processing for each question in the set of explanation steps corresponding to the standard evidence set: inputting the question and the explanation step library into the model to be evaluated to obtain a first estimated set of explanation steps for the question, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; based on the first estimated set of explanation steps and the set of explanation steps corresponding to the question in the standard evidence set, obtaining a first number of steps and a second number of steps, wherein the first number of steps is the number of explanation steps in the first estimated set of explanation steps that exceeds the set of explanation steps, and the second number of steps is the number of explanation steps in the set of explanation steps that exceeds the first estimated set of explanation steps; based on the first number of steps and the second number of steps, obtaining the first double-penalty evidence information of the model to be evaluated, wherein the first double-penalty evidence information indicates the ability of the model to be evaluated to obtain the explanation steps it relies on in the process of obtaining the answer corresponding to the question.

[0020] Optionally, the evaluation unit is further configured to perform the following processing for each question in the set of explanation steps corresponding to the standard evidence set: inputting the question and the candidate set of explanation steps into the model to be evaluated to obtain the predicted answer and the reference step identifier for the question, wherein the candidate set of explanation steps is either the first set of predicted explanation steps or the set of explanation steps corresponding to the question in the standard evidence set, and the reference step identifier is the identifier of the explanation step in the candidate set of explanation steps referenced by the model to be evaluated when reasoning about the predicted answer; based on the explanation steps corresponding to the reference step identifier and the set of explanation steps corresponding to the question in the standard evidence set, obtaining the number of third steps and the number of fourth steps, wherein the number of third steps is the number of explanation steps that the explanation steps corresponding to the reference step identifier exceed the number of explanation steps in the set of explanation steps, and the number of fourth steps is the number of explanation steps that the explanation steps in the set exceed the number of explanation steps corresponding to the reference step identifier; based on the number of third steps and the number of fourth steps, obtaining the second double-penalty evidence information of the model to be evaluated, wherein the second double-penalty evidence information indicates the ability of the model to be evaluated to obtain the answer to the question.

[0021] Optionally, the evaluation unit is further configured to, after obtaining the second double-penalty evidence information of the model to be evaluated based on the first step number and the second step number, evaluate the answer quality of the predicted answer based on a predetermined evaluation model; in response to the answer quality being greater than a first preset value and the second double-penalty evidence information being less than a second preset value, determine that the model to be evaluated exhibits a false knowledge phenomenon where the predicted answer is correct but the explanation steps on which it is based are incorrect.

[0022] Optionally, the evaluation unit is further configured to perform the following processing for each assertion in the set of explanatory steps corresponding to the standard evidence set: inputting the assertion and the explanatory step library into the model to be evaluated to obtain the predicted label for the assertion and the second predicted explanatory step set; based on the second predicted explanatory step set and the set of explanatory steps corresponding to the assertion in the standard evidence set, obtaining the number of fifth steps and the number of sixth steps, wherein the number of fifth steps is the number of explanatory steps in the second predicted explanatory step set that is greater than the explanatory step set, and the number of sixth steps is the number of explanatory steps in the explanatory step set that is greater than the second predicted explanatory step set; based on the number of fifth steps and the number of sixth steps, obtaining the third double-penalty evidence information of the model to be evaluated, wherein the third double-penalty evidence information indicates the ability of the model to be evaluated to verify the truth of the assertion.

[0023] Optionally, the above apparatus further includes: a training unit configured to perform the following processing for each set of explanation steps in the standard evidence set: inputting the object corresponding to the set of explanation steps and the explanation step library into the initial process-aware step retrieval model to obtain a third estimated set of explanation steps for the object, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; determining the target loss based on the third estimated set of explanation steps and the set of explanation steps; and adjusting the parameters of the initial process-aware step retrieval model based on the target loss to obtain the process-aware step retrieval model.

[0024] In another general aspect, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform any of the model evaluation methods described above.

[0025] In another general aspect, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform any of the model evaluation methods described above.

[0026] In another general aspect, a computer program product is provided, including computer instructions that, when executed by a processor, implement any of the model evaluation methods described above.

[0027] According to the model evaluation method and apparatus of the embodiments of this disclosure, the acquired preliminary evidence set is verified by at least two different verification models. Only the set of explanation steps that can support obtaining the correct answer or label (i.e., the label information of the object is correct and can be obtained according to the set of explanation steps) and whose answer or label cannot be obtained after any explanation step is deleted, is retained. This ensures that the retained set of explanation steps is a minimal sufficient chain of steps rather than a broadly related fragment, thereby distinguishing necessary explanation steps from contexts that are similar to the topic but not essential, improving the reliability of the evaluation samples. Then, the model to be evaluated is evaluated using the retained set of explanation steps, resulting in relatively accurate evaluation results. Therefore, this disclosure effectively solves the problem of the lack of a method in the prior art for obtaining a minimal, sufficient chain of explanation steps that conforms to the process dependency order.

[0028] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0029] The above and other objects and features of the embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings illustrating the embodiments, wherein: Figure 1 This is a flowchart illustrating a model evaluation method according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating a standard chain of evidence according to an embodiment of this disclosure; Figure 3 This is a flowchart illustrating a multi-stage construction process of an evaluation dataset according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating a pseudo-knowledge phenomenon according to an embodiment of the present disclosure; Figure 5 This is a comparison diagram showing the occurrence of false knowledge in multiple evaluation models of embodiments of this disclosure; Figure 6 This is a diagram illustrating a multimodal model evaluation framework based on the minimum sufficient chain of evidence, according to an embodiment of this disclosure. Figure 7 This is a schematic diagram illustrating the evaluation results of various models representing embodiments of the present disclosure; Figure 8 This is a schematic diagram illustrating the training process of a process-aware step retrieval model according to an embodiment of the present disclosure; Figure 9 This is a schematic diagram illustrating the training results of a process-aware step retrieval model according to an embodiment of the present disclosure; Figure 10 This is a block diagram illustrating a model evaluation apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0030] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0031] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0032] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0033] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0034] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0035] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0037] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0038] The model evaluation method and apparatus of this disclosure will be described in detail below with reference to the accompanying drawings.

[0039] This disclosure proposes a model evaluation method. Figure 1 This is a flowchart illustrating a model evaluation method according to an embodiment of the present disclosure. (Refer to...) Figure 1 The model evaluation method includes the following steps: In step S101, for each knowledge explanation video, the ordered explanation steps of the knowledge explanation video are input into the generation model to obtain multiple questions, multiple assertions, a set of target answers and explanation steps for each question, and a set of target labels and explanation steps for each assertion. Each explanation step in the ordered explanation steps includes at least the corresponding subtitle text and key image frames. Assertions are knowledge points inferred from the knowledge explanation video. The set of explanation steps for each question is a set of explanation steps that support the target answer for the question. Target labels include support labels, rebuttal labels, and empty labels. When the target label corresponding to an assertion is an empty label, the set of explanation steps corresponding to the assertion is empty. When the target label corresponding to an assertion is a support label or a rebuttal label, the set of explanation steps corresponding to the assertion is a set of explanation steps that support or refute the assertion.

[0040] In step S102, the sets of explanatory steps corresponding to all questions and all assertions are merged into a preliminary evidence set; In step S103, for each set of explanation steps in the preliminary evidence set, support information for the set of explanation steps is obtained through at least two different verification models. The support information indicates whether the label information of the corresponding object in the set of explanation steps is correct. Based on whether the label information of the object can be obtained from the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted, the object is a question or an assertion. When the object is a question, the label information is the target answer; when the object is an assertion, the label information is the target label. In step S104, if the indication result of the supporting information is "yes", the set of explanatory steps is retained; In step S105, the retained set of explanatory steps is merged into a standard set of evidence; In step S106, the model to be evaluated is evaluated based on the standard evidence set. As an example, the above statement regarding whether the object's label information can be obtained based on the set of explanation steps also implies that all explanation steps in the set of explanation steps collectively support the acquisition of the corresponding label information, without any redundant explanation steps.

[0041] The explanatory steps in the above standard evidence set are arranged in the order of reasoning dependencies required by the question or assertion.

[0042] As an example, the aforementioned knowledge explanation videos can be educational lecture videos or other explanatory videos, and this disclosure does not limit them.

[0043] For example, taking educational lecture videos as an example, the process begins by acquiring the educational lecture video, its subtitles, sampled keyframes, and Optical Character Recognition (OCR) text. Then, the annotator divides each educational lecture video into sequentially arranged teaching steps (i.e., the aforementioned ordered explanation steps) based on the boundaries of the teaching content, and can merge these steps into a teaching step library (i.e., the explanation step library) for subsequent model evaluation. It should be noted that each teaching step is a coherent teaching unit (such as a video segment), which can correspond to units such as concept introduction, formula derivation for the concept, or causal explanation; the boundaries of the teaching content can be determined by the topic transitions observed directly by humans, including but not limited to concept switching, changes in the derivation stage (such as a change in the derivation stage when two concepts are derived), and changes in the program state (such as a chemical process, where each step can be regarded as a program state); each teaching step can include step identifiers, subtitle text, sampled keyframes, and OCR text. Among them, step identifiers are used for subsequent evidence chain annotation, model output reference, and evaluation, subtitle text anchors the content actually explained by the teacher in the current teaching step, and sampled keyframes and OCR text are used to supplement visual information aligned with the subtitles.

[0044] This embodiment no longer uses automatically segmented video segments as evaluation samples, but determines the evaluation samples through pre-formed ordered teaching steps. This allows for the separation of upstream video boundary detection errors from subsequent process evidence tracking capabilities, thereby enabling a focused evaluation of the evidence selection and evidence-supported reasoning capabilities of the multimodal model.

[0045] As an example, after obtaining the teaching steps library, for each teaching lecture video, a multimodal large model (i.e., the generative model mentioned above) can be used to generate process-oriented questions, process-oriented assertions, target answers for each question, and a set of teaching steps (i.e., a set of explanation steps), as well as target labels for each assertion and a set of teaching steps (i.e., a set of explanation steps). It should be noted that this generation stage can be performed on five types of processes in the teaching lecture video: causality, deduction, sequence, state change, and counterfactual (e.g., deleting a necessary step). This disclosure does not limit the scope of these processes.

[0046] For example, regarding a problem or assertion, suppose To express a question or assertion, Indicates the target answer or target label. Indicates that it is able to Support under conditions The evidence set (i.e., the set of explanation steps). This ultimately leads to the standard chain of evidence. (That is, the minimum sufficient chain of evidence, also known as the standard set of evidence mentioned above) needs to satisfy the following sufficiency and minimum conditions: belong (Right now Correct and based on able to obtain ),for Each teaching step ,delete The set after Not belonging to (any) The item cannot be obtained after being deleted. ).

[0047] like Figure 2 As shown in the example illustrating homogeneous transformation matrices, the question is why the end effector's pose uses a 4×4 homogeneous matrix instead of a 3×3 rotation matrix. The standard chain of evidence includes: step_01, defining the rigid body pose as consisting of rotation R and translation t; step_02, explaining that a 3×3 rotation matrix can only encode rotation, not translation; and step_04, introducing homogeneous coordinates [x,y,z,1]. T Step 5: Merge R and t into a single 4×4 matrix T. All steps in the standard chain of evidence support the answer, and removing any step leaves the remaining steps unable to support the answer. Step 3 only explains the orthogonality property of rotation matrices, which is a property of rotation matrices, but it is not necessary to answer why the matrix dimension changes. Step 6 further explains that the inverse transformation is an extension of the inverse transformation after the definition of a matrix, and it is not necessary to answer why a homogeneous matrix is ​​needed. Therefore, these two steps are not necessary steps to answer the question.

[0048] It should be noted that for samples with multiple feasible supporting paths or incorrect process order, revisions or deletions can be made during the verification stage. Through this process, the standard chain of evidence is no longer an open-ended interpretation preference, but a controlled and supervised object established based on ordered teaching steps and target questions.

[0049] As an example, the target labels mentioned above include support, refute, and NEI. When the target label corresponding to an assertion is NEI, the set of explanatory steps corresponding to the assertion is empty, indicating that the assertion is related to the topic, but no evidence can be found to support or refute it from the knowledge explanation video. When the target label corresponding to an assertion is support, the set of explanatory steps corresponding to the assertion is the set of explanatory steps that support the assertion. When the target label corresponding to an assertion is refute, the set of explanatory steps corresponding to the assertion is the set of explanatory steps that refute the assertion.

[0050] According to embodiments of this disclosure, obtaining supporting information for the set of explanation steps in step S103 using at least two different verification models may include: inputting the set of explanation steps, the object, and the object's tag information into a first verification model to obtain first supporting information, the first supporting information indicating whether the object's tag information is correct, whether the object's tag information can be obtained based on the set of explanation steps, and whether the object's tag information cannot be obtained after any explanation step in the set of explanation steps is deleted; inputting the set of explanation steps, the object, and the object's tag information into a second verification model to obtain second supporting information, the second supporting information indicating whether the object's tag information is correct, whether the object's tag information can be obtained based on the set of explanation steps, and whether the object's tag information cannot be obtained after any explanation step in the set of explanation steps is deleted; and determining the first supporting information and the second supporting information as supporting information.

[0051] In this embodiment, the first and second supporting information obtained from two independent verification models are used as supporting information, which can realize cross-model verification and avoid the inaccuracy caused by verification of the same model.

[0052] As an example, the first and second verification models mentioned above are different and independent models. Each set of explanation steps is verified for sufficiency (whether the target answer or target label is correct, and whether the set of explanation steps can obtain the target answer or target label) and minimumity (i.e., whether deleting any explanation step in the set of explanation steps will prevent obtaining the target answer or target label). Only when all three checks (whether the target answer or target label is correct, whether the set of explanation steps can obtain the target answer or target label, and whether deleting any explanation step in the set of explanation steps will prevent obtaining the target answer or target label) are completely consistent, i.e., all indications are yes, is the set of explanation steps directly retained.

[0053] According to the embodiments of this disclosure, before obtaining the supporting information of the explanation step set through at least two different verification models in the above step S103, the explanation step set, the object, and the object's tag information can be input into a third verification model to obtain third supporting information. The third supporting information indicates whether the object's tag information can be obtained based on the explanation step set, and whether the object's tag information cannot be obtained after any explanation step in the explanation step set is deleted. In response to the indication result of the third supporting information not being all yes, the explanation step set is corrected to obtain a corrected explanation step set.

[0054] This embodiment allows for a revision of the set of explanation steps, thereby increasing the number of qualified explanation steps and preventing an insufficient number of explanation steps.

[0055] As an example, before performing sufficiency and minimum verification using two independent verification models, a third verification model can be used to perform sufficiency and minimum verification on each set of explanatory steps in the initial evidence set. If the supporting information output by the third verification model does not all indicate "yes," then it is possible to check whether there are any redundant steps that can be deleted in the set of explanatory steps, and whether any bridging steps have been omitted, and to delete redundant steps or supplement any missing steps accordingly; if the supporting information output by the third verification model all indicate "yes," then proceed to the sufficiency and minimum verification using the two independent verification models.

[0056] It should be noted that checking whether there are redundant steps that can be deleted in the set of explanatory steps and whether bridging steps are omitted can also be output by the third verification model, and this disclosure does not limit this.

[0057] According to the embodiments of this disclosure, before merging the retained set of explanation steps into a standard evidence set in step S105 above, in response to the fact that the indication results of the supporting information are not all yes, the set of explanation steps, the object, and the object's label information are input into the fourth verification model to obtain the fourth supporting information. The fourth supporting information indicates whether the object's label information is correct, whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; in response to the fact that the indication results of the fourth supporting information are all yes, the set of explanation steps is retained.

[0058] This embodiment allows for further verification using an enhanced fourth verification model, preventing the omission of any correct steps in the explanation.

[0059] As an example, the fourth verification model mentioned above can be the first verification model, the second verification model, or other enhanced models, and this disclosure does not limit it in this regard.

[0060] If the supporting information output by the two independent verification models above is not both indicated as yes, the corresponding set of explanation steps can be input into an additional arbitration model (i.e., the fourth verification model above). If the arbitration model indicates yes in all three checks (whether the target answer or target label is correct, whether the set of explanation steps can obtain the target answer or target label, and whether the target answer or target label cannot be obtained after any explanation step in the set of explanation steps is deleted), then the set of explanation steps is retained; otherwise, the set of explanation steps is deleted.

[0061] As an example, for the set of explanatory steps that have passed model validation, further manual review can be performed. This manual review can adopt a sampling review approach, and the sampling dimensions may include, but are not limited to, task type, process type, evidence chain length, fidelity verification labels, validation model divergence samples, and high-risk samples where visual content of adjacent steps continues to appear. This manual review evaluates sufficiency, minimality, and problem clarity. For minimality, a leave-one-step-out check can be used, that is, removing steps from the standard evidence chain one by one. After removing any step, the remaining evidence should be insufficient to support the objective. Samples that fail any of the conditions can be revised and re-validated or directly deleted; this disclosure does not limit this.

[0062] It should be noted that in one implementation, after final manual review, the sufficiency, minimality, and clarity of all explanation steps were 94.5%, 91.0%, and 96.0%, respectively, which was effectively improved.

[0063] Figure 3 This demonstrates a multi-stage process for constructing assessment samples based on educational lecture videos, such as... Figure 3 As shown, firstly, 1750 real teaching videos (educational lecture videos + MOOC videos + multilingual videos, etc.) and their subtitle text, sampled keyframes, and Optical Character Recognition (OCR) text are acquired; secondly, the annotator divides each educational lecture video into ordered teaching steps (i.e., the ordered explanation steps mentioned above) according to the boundaries of the teaching content, and can merge them into a teaching step library (i.e., an explanation step library, such as containing 9047 steps); thirdly, for each teaching lecture video, a multimodal large model (i.e., the generative model mentioned above) can be used to generate process-oriented questions, process-oriented assertions, the target answer and teaching step set (i.e., the explanation step set) corresponding to each question, and the target label and teaching step set (i.e., the explanation step set) corresponding to each assertion, such as... Approximately 20,000 question-answer and assertion samples were generated. Then, preliminary validation and correction were performed using a first validation model (such as a multimodal large language model), resulting in approximately 1.5,000 question-answer and assertion samples. Next, cross-model validation was performed using two independent validation models, followed by a second validation using an arbitration model, yielding 11,621 question-answer and assertion samples. These samples generally fall into five categories: causal, inference, sequence, state change, and counterfactual. Finally, further manual review was conducted, for example, by setting up a manual annotation team and selecting 500 samples for cross-validation, thus obtaining the final evaluation dataset.

[0064] According to an embodiment of this disclosure, the evaluation of the model to be evaluated based on a standard evidence set in step S106 may include: for each question in the question corresponding to the set of explanation steps in the standard evidence set, performing the following processing: inputting the question and the explanation step library into the model to be evaluated to obtain a first estimated set of explanation steps for the question, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; based on the first estimated set of explanation steps and the set of explanation steps corresponding to the question in the standard evidence set, obtaining a first number of steps and a second number of steps, wherein the first number of steps is the number of explanation steps in the first estimated set of explanation steps that exceeds the set of explanation steps, and the second number of steps is the number of explanation steps in the set of explanation steps that exceeds the first estimated set of explanation steps; based on the first number of steps and the second number of steps, obtaining first double-penalty evidence information of the model to be evaluated, wherein the first double-penalty evidence information indicates the ability of the model to be evaluated to obtain the explanation steps it relies on in the process of obtaining the answer corresponding to the question.

[0065] In this embodiment, the standard evidence chain obtained through the above verification can be used to evaluate the model's ability to identify the explanatory steps on which the answer is based. At the same time, the first double-penalty evidence information takes into account additional and omitted citations, thereby enabling effective evaluation of the model's performance.

[0066] As an example, suppose the first estimated set of explanation steps includes steps 1, 2, 3, and 4, while the standard set of evidence includes steps 2, 3, 5, and 6. Then the first estimated set of explanation steps has two more steps than the corresponding set of explanation steps, so the first set of steps has two more steps. The corresponding set of explanation steps has two more steps than the first estimated set of explanation steps, so the second set of steps has two more steps.

[0067] The aforementioned model to be evaluated can be any multimodal large model, and this disclosure does not impose any limitations on it.

[0068] As an example, the aforementioned standard evidence set can be used to construct a dataset for evaluating a model. For instance, the dataset for evaluating a model can be constructed using the questions, assertions, and explanation steps corresponding to the standard evidence set. Assume a dataset for evaluating a model contains 1750 knowledge explanation videos, 9047 explanation steps, 2549 question-and-answer samples (including questions, target answers, and the set of explanation steps supporting those target answers), and 1174 assertion samples (including assertions, target labels, and the set of explanation steps supporting those target labels). These question-and-answer samples can cover five types of processes: causality, deduction, sequence, state change, and counterfactual. The explanation step sets in most question-and-answer samples are 2 to 4 explanation steps selected from the ordered explanation steps of the corresponding knowledge explanation videos.

[0069] For example, for the evidence retrieval (ER) task, under a unified input setting, the model to be evaluated receives the question q and the corresponding explanation steps library. Each explanation step can include at least subtitle text and sampled key image frames. The model to be evaluated outputs a set of predicted explanation steps. (That is, the set of explanation steps for the first estimation mentioned above). The set of explanation steps corresponding to this issue in the standard evidence set is as follows: It should be noted that, unlike the sorting of related segments, ER is evaluated based on the set of predicted explanatory steps. The goal is to assess whether the model can identify the accurate set of explanatory steps required to complete the reasoning process.

[0070] Then, obtain the number of additional reference steps. (i.e., the number of steps in the first step mentioned above), the number of omitted reference steps is... (i.e., the number of steps in the second step), and then obtain the first pair of punishment evidence information. For example, the first pair of punishment evidence information can be calculated using the following formula:

[0071] in, Controlling additional reference penalties To control the penalty for missed references, τ is the minimum recall threshold. For the recall constraint, F1(P,G) is and The F1 score of the set dimension is a comprehensive metric in the evidence retrieval task (ER) specifically used to evaluate the degree of matching between the predicted explanatory steps set P and the standard evidence set G. The higher the value, the stronger the ability to explain the steps involved in obtaining the answer to a question.

[0072] It should be noted that when If it is not empty, empty or Compared to The recall rate is lower than τ ,but r =0; when Non-empty and Compared to The recall rate is no less than τ hour, r The value is 1. One implementation uses ( , , τ =(2.0,1.8,2 / 3). This embodiment also uses recall constraints to suppress outputs that only cover a small number of evidence steps, further ensuring effective evaluation of the model's performance.

[0073] As an example, for ER tasks, in addition to reports, In addition, it can also report the complete match rate of the evidence set (i.e. and Match rate), full coverage (i.e.) cover The coverage and F1 value are not limited in this disclosure.

[0074] According to an embodiment of this disclosure, the evaluation of the model to be evaluated based on a standard evidence set in step S106 may include: for each question in the question corresponding to the explanation step set in the standard evidence set, performing the following processing: inputting the question and the candidate explanation step set into the model to be evaluated to obtain the predicted answer and the reference step identifier for the question, wherein the candidate explanation step set is either the first predicted explanation step set or the explanation step set corresponding to the question in the standard evidence set, and the reference step identifier is the identifier of the explanation step in the candidate explanation step set referenced by the model to be evaluated when reasoning about the predicted answer; based on the explanation steps corresponding to the reference step identifier and the explanation step set corresponding to the question in the standard evidence set, obtaining the number of third steps and the number of fourth steps, wherein the number of third steps is the number of explanation steps that the explanation steps corresponding to the reference step identifier exceed the number of explanation steps in the explanation step set, and the number of fourth steps is the number of explanation steps that the explanation steps in the explanation step set exceed the number of explanation steps corresponding to the reference step identifier; based on the number of third steps and the number of fourth steps, obtaining the second double-penalty evidence information of the model to be evaluated, wherein the second double-penalty evidence information indicates the ability of the model to be evaluated to obtain the answer to the question.

[0075] In this embodiment, the standard evidence chain obtained by the above verification can be used to evaluate the model's ability to obtain the corresponding answer to the question. At the same time, the second double-penalty evidence information takes into account additional and omitted references, thereby enabling effective evaluation of the model's performance.

[0076] As an example, the dataset constructed in the above embodiments can still be used for the Evidence-Supported Question Answering (EGQA) task. Under a uniform input setting, the model to be evaluated receives a given question q and a set of candidate explanation steps. The model to be evaluated outputs the answer to the question q. a and the step identifiers of the steps referenced in the candidate explanation steps. C The EGQA task includes two settings: In the Gold-EGQA setting, the candidate explanatory steps set is the set of explanatory steps corresponding to the question in the standard evidence set, used to evaluate reasoning ability under controlled evidence conditions; In the Retrieved-EGQA setting, the candidate explanatory steps set is the first predicted explanatory steps set output by the ER task, used to evaluate end-to-end processes.

[0077] Then, obtain the number of additional reference steps. (i.e., the number of steps in the third step above), the number of omitted reference steps is (i.e., the number of steps in step four), and then obtain the second pair of penalty evidence information. For example, the second pair of penalty evidence information can be calculated using the following formula:

[0078] in, Controlling additional reference penalties To control the penalty for missed references, τ is the minimum recall threshold. For the recall constraint, F1(P,G) is and The F1 score for the set dimension between the two is a comprehensive metric specifically used in the task to evaluate the degree of matching between the set of predicted explanatory steps P and the set of standard evidence G. The higher the value, the stronger the ability to obtain the corresponding answers to questions.

[0079] It should be noted that when If it is not empty, empty or Compared to The recall rate is lower than τ ,but r =0; when Non-empty and Compared to The recall rate is no less than τ hour, r The value is 1. One implementation uses ( , , τ =(2.0,1.8,2 / 3). This embodiment also uses recall constraints to suppress outputs that only cover a small number of evidence steps, further ensuring effective evaluation of the model's performance.

[0080] As an example, the EGQA task can also report answer quality, citation quality, and can be compared with evidence quality scores. A unified overall score for the EGQA task is calculated, which can be calculated as follows: Score=α × Semantic+β × +γ × Citation in, α, β, γ All are not less than 0 and their sum is 1. Semantic Indicates the quality of the semantic answer. Citation Indicates the quality of the citation.

[0081] It should be noted that the quality of the above answers and citations can be obtained through a predetermined evaluation model, and this disclosure does not limit this.

[0082] According to embodiments of this disclosure, after obtaining the second double-penalty evidence information of the model to be evaluated based on the first step quantity and the second step quantity, the answer quality of the predicted answer can also be evaluated based on a predetermined evaluation model; in response to the answer quality being greater than a first preset value and the second double-penalty evidence information being less than a second preset value, it is determined that the model to be evaluated exhibits a false knowledge phenomenon where the predicted answer is correct but the explanation steps on which it is based are incorrect.

[0083] This embodiment evaluates the correctness of the final answer separately from the fidelity of the explanation steps, which can diagnose pseudo-knowledge phenomena where the answer is correct but the evidence is insufficient, the evidence is redundant, or the order of steps deviates from the teaching path.

[0084] As an example, the corresponding question, predicted answer, and cited step identifiers, along with the set of candidate cited steps, can be input into a predefined evaluation model to obtain the answer quality and cite quality. If the answer quality is high but the second double-penalty evidence is low, the model may have entered the correct topic area but missed necessary bridging steps, crossed adjacent step boundaries, or cited incorrect teaching steps when visual persistence bias occurs. This type of output constitutes pseudo-knowing, where the answer is fluent but the process evidence is insufficient.

[0085] Figure 4 This demonstrates the phenomenon of pseudo-knowledge, such as Figure 4 As shown, when a model only provides fluent answers but fails to reconstruct the minimum chain of teaching steps required to guide students to understand knowledge and engage in self-directed learning, the phenomenon of "pseudo-knowing" occurs. Figure 4 While the answers in the case on the left were fluent, they lacked supporting evidence; the case on the right, however, had concrete evidence regarding the process.

[0086] Figure 5 A comparison chart showing the occurrence of false knowledge in multiple evaluation models. Figure 5 In Figure (a), the faithfulness diagram of Retrieved-EGQA is shown. Models above the contour lines are mainly characterized by "false knowledge" phenomena, while models below the contour lines are mainly characterized by "faithfulness" phenomena (i.e., being able to arrive at the correct answer and the corresponding correct reasoning chain). Figure 5 (b) describes the performance degradation caused by increasing the length of the evidence chain. As the evidence chain length increases from 2 steps to 4 or more steps, the performance of all models deteriorates. The scores all showed a monotonically decreasing trend, indicating that the phenomenon of "false knowledge" is becoming increasingly serious.

[0087] Therefore, evaluations of multiple multimodal models show that even if a model can generate high-quality answers, generating the minimum sufficient chain of evidence (standard chain of evidence) remains significantly more difficult. As the chain of evidence length increases, the double-penalty evidence score continuously decreases; visually recurring formulas or diagrams can also cause confusion between adjacent instructional steps. Therefore, relying solely on the correctness of answers may overestimate the model's ability to comprehend the video explanation process.

[0088] According to an embodiment of this disclosure, the evaluation of the model to be evaluated based on a standard evidence set in step S106 may include: for each assertion in the set of explanatory steps corresponding to the assertions in the standard evidence set, performing the following processing: inputting the assertions and the explanatory step library into the model to be evaluated to obtain a predicted label for the assertion and a second predicted explanatory step set; based on the second predicted explanatory step set and the set of explanatory steps corresponding to the assertions in the standard evidence set, obtaining the number of fifth steps and the number of sixth steps, wherein the number of fifth steps is the number of explanatory steps in the second predicted explanatory step set that is greater than the explanatory step set, and the number of sixth steps is the number of explanatory steps in the explanatory step set that is greater than the second predicted explanatory step set; based on the number of fifth steps and the number of sixth steps, obtaining the third double-penalty evidence information of the model to be evaluated, wherein the third double-penalty evidence information indicates the ability of the model to be evaluated to verify the truth of the assertions.

[0089] In this embodiment, the standard evidence chain obtained by the above verification can be used to evaluate the model's ability to obtain the corresponding answer to the question. At the same time, the third double-penalty evidence information takes into account additional and omitted references, thereby enabling effective evaluation of the model's performance.

[0090] As an example, the dataset constructed in the above embodiments can still be used for the fidelity verification task (FCV). Under the uniform input settings, the model to be evaluated receives a given assertion. c The model to be evaluated outputs a predicted label and a set of explanatory steps, S, which includes the predicted label and the explanatory steps supporting that predicted label. The predicted label may include SUPPORTED, REFUTED, and NEI.

[0091] Then, obtain the number of additional reference steps. (i.e., the number of steps in step 5 above), the number of omitted reference steps is (i.e., the number in step six), and then obtain the third pair of penalty evidence information. For example, the third pair of penalty evidence information can be calculated using the following formula:

[0092] in, Controlling additional reference penalties To control the penalty for missed references, τ is the minimum recall threshold. For the recall constraint, F1(P,G) is and The F1 score for the set dimension between the two is a comprehensive metric specifically used in the task to evaluate the degree of matching between the set of predicted explanatory steps P and the set of standard evidence G. The higher the value, the stronger the ability to verify the truth of the assertion.

[0093] It should be noted that when If it is not empty, empty or Compared to The recall rate is lower than τ ,but r =0; when Non-empty and Compared to The recall rate is no less than τ hour, r The value is 1. One implementation uses ( , , τ =(2.0,1.8,2 / 3). This embodiment also uses recall constraints to suppress outputs that only cover a small number of evidence steps, further ensuring effective evaluation of the model's performance.

[0094] As an example, the FCV task can also report label accuracy and can be compared with the evidence quality score. The joint score for the FCV task is calculated uniformly as follows: Joint=Label Acc.× in, Label Acc. This indicates the label accuracy rate.

[0095] As an example, in an overview evaluation, one can also use... Summarize the performance of the three types of tasks, where ER represents the performance of the ER task. , express Overall score of the task Score , express Joint score of the task Joint .

[0096] Figure 6 This diagram illustrates a multimodal model evaluation framework based on the minimum sufficient evidence chain, such as... Figure 6As shown, firstly, the evaluation dataset needs to be obtained. Specifically, this involves acquiring the teaching videos and their subtitles, sampled keyframes, and Optical Character Recognition (OCR) text. Then, the annotator divides each educational lecture video into ordered teaching steps (i.e., the ordered explanation steps mentioned above) based on the boundaries of the teaching content, and these steps can be merged into a teaching step library (i.e., an explanation step library or step pool). Next, a multimodal large model (i.e., the generative model mentioned above) is used to generate procedural questions, procedural assertions, the target answer for each question, and a set of teaching steps (i.e., the explanation step set), as well as the target label for each assertion and a set of teaching steps (i.e., the explanation step set).

[0097] Secondly, the model was evaluated using the ER, EGQA, and FCV tasks. The specific evaluation process has been discussed in detail above and will not be repeated here. It should be noted that... Figure 6 This paper only demonstrates the evaluation process for one question and one assertion. In reality, multiple questions and multiple assertions can be evaluated, and the scores of the evaluation results can be averaged. This disclosure does not limit this process. Moreover, after the generative model generates a question or assertion, its corresponding set of teaching steps can be specifically verified to obtain the minimum sufficient evidence chain (i.e., the minimum sufficient step chain).

[0098] Figure 7 The evaluation results of various models are presented, such as Figure 7 As shown, each task reports two metrics, and further reports are provided for overall comparison. The second line shows the human performance, i.e., the performance resulting from human intervention in place of the model being evaluated. The remaining lines show the performance of the model being evaluated. The two lines below the human performance represent the closed-source and open-source models, respectively, with bold numbers indicating the best and worst results. It should be noted that the last line shows the comparison between the subsequently trained PASR and its base model; bold numbers represent the PASR results.

[0099] Evaluation results after testing on three tasks indicate that the current multimodal models under evaluation generally outperform the minimum sufficient chain of evidence generation in terms of answer or label generation, even when stronger models achieve higher task scores. However, its performance on the ER task is still lower than that of human performance, which shows that the evaluation method disclosed in this paper has the limitation of making it difficult to obtain accurate process evidence when the correctness of the answer is revealed.

[0100] According to embodiments of this disclosure, for each set of explanation steps in the standard evidence set, the following processing can also be performed: inputting the object corresponding to the set of explanation steps and the explanation step library into the initial process-aware step retrieval model to obtain a third estimated set of explanation steps for the object; determining the target loss based on the third estimated set of explanation steps and the set of explanation steps; adjusting the parameters of the initial process-aware step retrieval model based on the target loss to obtain the process-aware step retrieval model.

[0101] In this embodiment, the above-mentioned standard evidence set can be used to directly train a performance-enhanced process-aware step retrieval model, thereby improving the recovery capability of the minimum sufficient evidence set.

[0102] As an example, such as Figure 8 As shown, the questions and their corresponding explanatory step sets for ER and the assertions and their corresponding explanatory step sets for FCV are unified into the same format. Using this data, an Initial Process Aware Step Retrieval (PASR) model is trained. PASR predicts the minimum sufficient step chain corresponding to the question or assertion from the complete explanatory step library S. It should be noted that candidate step pre-pruning is not performed during this training phase; that is, explanatory steps are not divided from the video, but rather a pre-defined explanatory step library is used. Furthermore, the selector in PASR needs to distinguish the steps that truly support the question answer or assertion from lexically or visually similar distracting steps.

[0103] For example, PASR uses a multimodal model as its backbone and employs LoRA-based supervised fine-tuning. The training input consists of objects (questions or assertions) and a library of explanatory steps, S. For ER samples derived from a question (AQ) (i.e., the question and its corresponding explanatory step set), PASR retrieves the minimum sufficient step chain supporting the corresponding answer to the question (i.e., the set of explanatory steps corresponding to the question in the standard evidence set), meaning PASR outputs the minimum sufficient step chain as its prediction. For FCV samples derived from an assertion (i.e., the assertion and its corresponding explanatory step set), PASR retrieves the minimum sufficient step chain that can support or refute the assertion (i.e., the set of explanatory steps corresponding to the assertion in the standard evidence set), but does not need to predict labels; that is, PASR outputs the minimum sufficient step chain as its prediction, and when the label is NEI, this minimum sufficient step chain is empty.

[0104] The PASR training process can use the following objective loss function: .

[0105] in, The token-level cross-entropy loss is used for the complete structured output. To predict the number of evidence steps (i.e., the number of steps in the third estimated explanation step set). This refers to the number of standard evidence steps (i.e., the number of steps in the set of explanatory steps corresponding to the object within the standard evidence set). The cross-entropy loss described above is used to learn the complete output format, and the auxiliary counting loss is used to constrain the length of the evidence list, and only applies to the evidence list portion.

[0106] As an example, when PASR uses Qwen3-VL-8B as the backbone, it can be supervised and fine-tuned for two training epochs using LoRA rank 8 and α=32, with a learning rate of The input consists of the explanation steps library S and the object (problem or assertion). For example... Figure 9 As shown, compared to the untrained backbone model, the trained PASR performs better on the ER task. The perfect match rate under the ER task increased from 0.357 to 0.445, and the joint score under the FCV task increased from 0.510 to 0.547.

[0107] It should be noted that, Figure 9 Direct ER: This step directly evaluates the evidence found by retrieval tools such as BM25 and Dense. The retrieval results do not need to be input into other models for filtering. This corresponds to the Evidence Retrieval (ER) task, which does not undergo additional filtering.

[0108] Figure 9 The ER selector (also known as Final ER) adds a trained selector (such as PARS) to the initial retrieval results, removes interference steps, and outputs the final minimum sufficient evidence set, still corresponding to the ER task.

[0109] Figure 9 The +FCV verifier in the context of search results adds a verifier (such as PARS) to the search results, predicting the SUPPORTED / REFUTED / NEI tags and their supporting steps, corresponding to the FCV task. PARS is essentially a selector, filtering the explanatory steps in the search results, but the terminology differs between ER and FCV.

[0110] Figure 9 Exact, or Exact-Set Accuracy, means that only when the predicted set of steps is completely consistent with the gold evidenceset, without any omissions or over-selections, is it considered correct.

[0111] Figure 9The 1-stage PASR does not require the two-stage process of "first searching (using search engines such as BM25, Dense, etc.) and then filtering," but instead directly selects the final evidence set from the complete lecture step pool in one go. Therefore, its final result is reported in the Final ER.

[0112] According to embodiments of this disclosure, this disclosure also provides an evaluation system for a multimodal model of educational videos based on a minimum sufficient chain of evidence. The system includes: a step-by-step library construction module, used to organize knowledge explanation videos into a sequentially arranged library of explanation steps according to the boundaries of the explanation content, and record step identifiers, subtitle text, sampled keyframes, and text recognition text; a sample generation module, used to generate procedural questions and / or procedural assertions based on the explanation step library; an evidence chain verification module, used to check the sufficiency and minimumity of the evidence chain corresponding to the question or assertion through a first verification module; a consistency verification and arbitration module, used to filter samples that cannot be confirmed through at least two independent verification models and an arbitration model; a task construction module, used to construct at least one of an evidence retrieval task, an evidence-supported question-and-answer task, and a fidelity verification task; and a scoring module, used to calculate a double-penalty evidence score based on the additional and omitted reference steps between the predicted explanation step set output by the model to be evaluated and the standard evidence chain, and to give an evaluation result of the process evidence tracking ability based on the score, with higher scores indicating stronger ability.

[0113] In summary, the purpose of this disclosure is to provide a multimodal model evaluation method and system based on the minimum sufficient evidence chain, so that the evaluation results not only reflect whether the model under evaluation generates the correct answer or the correct label, but also whether it recovers the necessary teaching step chain supporting the answer or label. Specifically: First, a library of explanation steps is established for knowledge explanation videos. Each explanation step corresponds to a coherent teaching unit, and step identifiers, subtitle text, sampled keyframes, and OCR text are recorded. The boundaries of explanation steps are determined based on human-observable changes in teaching content, such as concept switching, changes in the derivation stage, or changes in program state.

[0114] Secondly, process-oriented questions and assertions are generated based on the set of explanation steps for each knowledge explanation video. The types of processes can include causal, deductive, sequential, state-change, and counterfactual. For each question or assertion, a preliminary standard chain of evidence (any set of explanation steps in the preliminary evidence set) is established. The standard chain of evidence must simultaneously satisfy sufficiency (all steps in the chain jointly support the objective, with no redundant steps) and minimality (after removing any step from the chain, the remaining steps are no longer sufficient to fully support the objective). For process-dependent questions, the corresponding set of explanation steps is also arranged according to the dependency order required to complete the reasoning.

[0115] Secondly, multi-stage quality control is performed on the generated preliminary standard chain of evidence. First, a multimodal large model is used to initially check the sufficiency and minimum validity of the evidence; then, two independent verification models are used for cross-model consistency verification. When the two verification models are inconsistent, an additional arbitration model is used for confirmation; samples that cannot obtain clear confirmation from the arbitration model are deleted. Furthermore, layered manual review can be performed according to abstract dimensions such as task type, process type, evidence chain length, fidelity verification labels, and high-risk samples. Samples that do not meet the requirements of sufficiency, minimum validity, or clarity of issue are revised and re-verified, or directly deleted.

[0116] Finally, the evidence retrieval task (ER), evidence-supported question answering task (EGQA), and fidelity verification task (FCV) are constructed using the validated set of explanatory steps, and double-penalty evidence scores are used to penalize both extra citations and missing citations.

[0117] In addition, the explanation steps after verification in this disclosure can also be used to train a process-aware step retrieval model, and the process-aware step retrieval model trained in this way has relatively stronger capabilities.

[0118] Through the above embodiments of this disclosure, at least the following technical effects are achieved: Evaluating the correctness of the final answer separately from the fidelity of the evidence in the explanation process can diagnose pseudo-knowledge phenomena, such as correct answers but insufficient evidence, redundant evidence, or steps deviating from the teaching path.

[0119] Using the minimum sufficient step chain rather than broad related fragments as the evaluation target can distinguish between necessary teaching steps and contexts that are related to the topic but not necessary.

[0120] By employing sufficiency checks, progressively eliminating minimum checks, cross-model consistency verification, arbitration, and tiered manual review to form a failure-closed filtering process, the reliability of the evaluation samples is improved.

[0121] The double-penalty evidence score takes into account both additional and omitted citations, and suppresses the output of steps that only cover a small number of evidence steps through recall constraints.

[0122] The validated set of explanatory steps disclosed herein can be directly used to train a process-aware step retrieval model, improving the recovery capability of the minimum sufficient evidence set.

[0123] Figure 10 This is a block diagram illustrating a model evaluation apparatus according to an embodiment of the present disclosure, such as... Figure 10 As shown, the device includes: The generation unit 1000 is configured to input the ordered explanation steps of each knowledge explanation video into the generation model for each knowledge explanation video, thereby obtaining multiple questions, multiple assertions, a set of target answers and explanation steps for each question, and a set of target labels and explanation steps for each assertion. Each explanation step in the ordered explanation steps includes at least the corresponding subtitle text and key image frames. Assertions are knowledge points inferred from the knowledge explanation video. The set of explanation steps for each question is a set of explanation steps supporting the target answer for that question. Target labels include support labels, rebuttal labels, and empty labels. When the target label corresponding to an assertion is an empty label, the set of explanation steps corresponding to the assertion is empty; when the target label corresponding to an assertion is a support label or a rebuttal label, the set of explanation steps corresponding to the assertion is a set of explanation steps supporting or rebutting the assertion. The first merging unit 1001 is configured to merge all questions and all assertions. The solution steps are merged into a preliminary evidence set; the verification unit 1002 is configured to obtain supporting information for each set of explanation steps in the preliminary evidence set through at least two different verification models. The supporting information indicates whether the label information of the corresponding object in the set of explanation steps is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted. The object is a question or an assertion. When the object is a question, the label information is the target answer. When the object is an assertion, the label information is the target label. The retention unit 1003 is configured to retain the set of explanation steps in response to the indication result of the supporting information being yes. The second merging unit 1004 is configured to merge the retained set of explanation steps into a standard evidence set. The evaluation unit 1005 is configured to evaluate the performance of the model to be evaluated based on the standard evidence set.

[0124] According to an embodiment of this disclosure, the verification unit 1002 is further configured to input the set of explanation steps, the object, and the object's label information into a first verification model to obtain first support information. The first support information indicates whether the object's label information is correct, whether the object's label information can be obtained based on the set of explanation steps, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; input the set of explanation steps, the object, and the object's label information into a second verification model to obtain second support information. The second support information indicates whether the object's label information is correct, whether the set of explanation steps can obtain the object's label information, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted; and determine the first support information and the second support information as support information.

[0125] According to an embodiment of this disclosure, the verification unit 1002 is further configured to input the set of explanation steps, the object, and the object's tag information into a third verification model before obtaining supporting information for the set of explanation steps through at least two different verification models, thereby obtaining third supporting information. The third supporting information indicates whether the object's tag information can be obtained based on the set of explanation steps, and whether the object's tag information cannot be obtained after any explanation step in the set of explanation steps is deleted. In response to the indication result of the third supporting information not being all yes, the set of explanation steps is corrected to obtain a corrected set of explanation steps.

[0126] According to an embodiment of this disclosure, the verification unit 1002 is further configured to, in response to the indication result of the supporting information not being all yes, input the set of explanation steps, the object, and the label information of the object into the fourth verification model to obtain the fourth supporting information. The fourth supporting information indicates whether the label information of the object is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted; in response to the indication result of the fourth supporting information being all yes, retain the set of explanation steps.

[0127] According to an embodiment of this disclosure, the evaluation unit 1005 is further configured to perform the following processing for each question in the set of explanation steps corresponding to the question in the standard evidence set: inputting the question and the explanation step library into the model to be evaluated to obtain a first estimated set of explanation steps for the question, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; based on the first estimated set of explanation steps and the set of explanation steps corresponding to the question in the standard evidence set, obtaining a first number of steps and a second number of steps, wherein the first number of steps is the number of explanation steps in the first estimated set of explanation steps that are more than the set of explanation steps, and the second number of steps is the number of explanation steps in the set of explanation steps that are more than the first estimated set of explanation steps; based on the first number of steps and the second number of steps, obtaining first double-penalty evidence information of the model to be evaluated, wherein the first double-penalty evidence information indicates the ability of the model to be evaluated to obtain the explanation steps it relies on in the process of obtaining the answer corresponding to the question.

[0128] According to an embodiment of this disclosure, the evaluation unit 1005 is further configured to perform the following processing for each question in the set of explanation steps corresponding to the standard evidence set: inputting the question and the candidate set of explanation steps into the model to be evaluated to obtain the predicted answer and the reference step identifier for the question, wherein the candidate set of explanation steps is either the first set of predicted explanation steps or the set of explanation steps corresponding to the question in the standard evidence set, and the reference step identifier is the identifier of the explanation step in the candidate set of explanation steps referenced by the model to be evaluated when reasoning about the predicted answer; based on the explanation steps corresponding to the reference step identifier and the set of explanation steps corresponding to the question in the standard evidence set, obtaining the number of third steps and the number of fourth steps, wherein the number of third steps is the number of explanation steps that the explanation steps corresponding to the reference step identifier exceed the number of explanation steps in the set of explanation steps, and the number of fourth steps is the number of explanation steps that the explanation steps in the set of explanation steps exceed the number of explanation steps corresponding to the reference step identifier; based on the number of third steps and the number of fourth steps, obtaining the second double-penalty evidence information of the model to be evaluated, wherein the second double-penalty evidence information indicates the ability of the model to be evaluated to obtain the answer to the question.

[0129] According to an embodiment of this disclosure, the evaluation unit 1005 is further configured to, after obtaining the second double-penalty evidence information of the model to be evaluated based on the first number of steps and the second number of steps, evaluate the answer quality of the predicted answer based on a predetermined evaluation model; in response to the answer quality being greater than a first preset value and the second double-penalty evidence information being less than a second preset value, determine that the model to be evaluated exhibits a false knowledge phenomenon where the predicted answer is correct but the explanation steps on which it is based are incorrect.

[0130] According to an embodiment of this disclosure, the evaluation unit 1005 is further configured to perform the following processing for each assertion in the set of explanation steps corresponding to the standard evidence set: inputting the assertion and the explanation step library into the model to be evaluated to obtain the predicted label for the assertion and obtain the second predicted explanation step set; based on the second predicted explanation step set and the set of explanation steps corresponding to the assertion in the standard evidence set, obtaining the number of fifth steps and the number of sixth steps, wherein the number of fifth steps is the number of explanation steps in the second predicted explanation step set that are more than the set of explanation steps, and the number of sixth steps is the number of explanation steps in the set of explanation steps that are more than the second predicted explanation step set; based on the number of fifth steps and the number of sixth steps, obtaining the third double-penalty evidence information of the model to be evaluated, wherein the third double-penalty evidence information indicates the ability of the model to be evaluated to verify the truth of the assertion.

[0131] According to embodiments of this disclosure, the apparatus further includes: a training unit configured to perform the following processing for each set of explanation steps in a standard evidence set: inputting the object corresponding to the set of explanation steps and the explanation step library into an initial process-aware step retrieval model to obtain a third estimated set of explanation steps for the object, wherein the explanation step library includes ordered explanation steps for each knowledge explanation video in multiple knowledge explanation videos; determining a target loss based on the third estimated set of explanation steps and the set of explanation steps; and adjusting the parameters of the initial process-aware step retrieval model based on the target loss to obtain a process-aware step retrieval model.

[0132] According to embodiments of the present disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a model evaluation method as described in any of the above embodiments.

[0133] According to embodiments of the present disclosure, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform a model evaluation method as described in any of the above embodiments.

[0134] According to embodiments of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement any of the model evaluation methods described above.

[0135] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A model evaluation method, characterized in that, include: For each knowledge explanation video, the ordered explanation steps of the knowledge explanation video are input into the generation model to obtain multiple questions, multiple assertions, a set of target answers and explanation steps for each question, and a set of target tags and explanation steps for each assertion. Each explanation step in the ordered explanation steps includes at least the corresponding subtitle text and key image frames. The assertions are knowledge points inferred from the knowledge explanation video. The set of explanation steps for each question is a set of explanation steps that support the target answer for the question. The target tags include support tags, refutation tags, and empty tags. When the target tag corresponding to an assertion is an empty tag, the set of explanation steps corresponding to the assertion is empty. When the target tag corresponding to an assertion is a support tag or a refutation tag, the set of explanation steps corresponding to the assertion is a set of explanation steps that support or refute the assertion. The explanatory steps for each question and each assertion are combined into a preliminary evidence set. For each set of explanation steps in the preliminary evidence set, supporting information for the set of explanation steps is obtained through at least two different verification models. The supporting information indicates whether the label information of the corresponding object in the set of explanation steps is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted. The object is a question or an assertion. When the object is a question, the label information is the target answer. When the object is an assertion, the label information is the target label. If the indication result of the supporting information is "yes", the set of explanation steps is retained; The retained set of explanation steps will be merged into a standard set of evidence. The model to be evaluated is evaluated based on the aforementioned standard evidence set.

2. The method as described in claim 1, characterized in that, The step of obtaining supporting information for the set of explanatory steps through at least two different verification models includes: The set of explanation steps, the object, and the object's tag information are input into the first verification model to obtain the first support information. The first support information indicates whether the object's tag information is correct, whether the object's tag information can be obtained based on the set of explanation steps, and whether the object's tag information cannot be obtained after any explanation step in the set of explanation steps is deleted. The set of explanation steps, the object, and the object's label information are input into the second verification model to obtain second support information. The second support information indicates whether the object's label information is correct, whether the set of explanation steps can obtain the object's label information, and whether the object's label information cannot be obtained after any explanation step in the set of explanation steps is deleted. The first support information and the second support information are determined as the support information.

3. The method as described in claim 2, characterized in that, Before obtaining supporting information for the set of explanatory steps through at least two different verification models, the following is also included: The set of explanation steps, the object, and the object's tag information are input into the third verification model to obtain the third support information. The third support information indicates whether the object's tag information can be obtained based on the set of explanation steps, and whether the object's tag information cannot be obtained after any explanation step in the set of explanation steps is deleted. In response to the fact that the indication results of the third supporting information are not all yes, the set of explanation steps is modified to obtain the modified set of explanation steps.

4. The method as described in claim 1, characterized in that, Before merging the retained set of explanatory steps into the standard set of evidence, it also includes: In response to the fact that the indication results of the supporting information are not all yes, the set of explanation steps, the object, and the label information of the object are input into the fourth verification model to obtain the fourth supporting information. The fourth supporting information indicates whether the label information of the object is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted. If the indication result of the fourth supporting information is "yes", the set of explanation steps is retained.

5. The method as described in claim 1, characterized in that, The evaluation of the model to be evaluated based on the standard evidence set includes: For each question in the set of explanation steps within the standard evidence set, the following processing is performed: The problem and the explanation step library are input into the model to be evaluated to obtain a first estimated set of explanation steps for the problem, wherein the explanation step library includes the ordered explanation steps of each knowledge explanation video in multiple knowledge explanation videos; Based on the first estimated set of explanation steps and the set of explanation steps corresponding to the question in the standard evidence set, the first number of steps and the second number of steps are obtained, wherein the first number of steps is the number of explanation steps in the first estimated set of explanation steps that are more than the set of explanation steps, and the second number of steps is the number of explanation steps in the set of explanation steps that are more than the first estimated set of explanation steps. Based on the number of the first step and the number of the second step, the first double-penalty evidence information of the model to be evaluated is obtained, wherein the first double-penalty evidence information indicates the ability of the model to be evaluated to obtain the explanation steps in the process of obtaining the answer to the question.

6. The method as described in claim 5, characterized in that, The evaluation of the model to be evaluated based on the standard evidence set also includes: For each question in the set of explanation steps within the standard evidence set, the following processing is performed: The question and the set of candidate explanation steps are input into the model to be evaluated to obtain the predicted answer and the reference step identifier for the question. The set of candidate explanation steps is the set of explanation steps corresponding to the question in the first set of predicted explanation steps or the set of standard evidence. The reference step identifier is the identifier of the explanation step in the set of candidate explanation steps referenced by the model to be evaluated when reasoning the predicted answer. Based on the explanation steps corresponding to the reference step identifier and the set of explanation steps corresponding to the question in the standard evidence set, the number of third steps and the number of fourth steps are obtained. The number of third steps is the number of explanation steps that the explanation steps corresponding to the reference step identifier exceed the number of explanation steps in the set of explanation steps, and the number of fourth steps is the number of explanation steps that the number of explanation steps in the set of explanation steps exceeds the number of explanation steps corresponding to the reference step identifier. Based on the number of steps in the third step and the number of steps in the fourth step, the second double-penalty evidence information of the model to be evaluated is obtained, wherein the second double-penalty evidence information indicates the ability of the model to be evaluated to obtain the corresponding answer to the question.

7. The method as described in claim 6, characterized in that, After obtaining the second double-penalty evidence information of the model to be evaluated based on the number of steps in the first step and the number of steps in the second step, the method further includes: The quality of the predicted answer is evaluated based on a predetermined evaluation model. In response to the answer quality being greater than a first preset value and the second double-penalty evidence information being less than a second preset value, it is determined that the model to be evaluated exhibits a false knowledge phenomenon where the predicted answer is correct but the explanation steps on which it is based are incorrect.

8. The method as described in claim 5, characterized in that, The evaluation of the model to be evaluated based on the standard evidence set also includes: For each assertion in the set of assertions corresponding to the explanation steps in the standard evidence set, the following processing is performed: Input the assertion and the explanation step library into the model to be evaluated to obtain the predicted label and the second predicted explanation step set for the assertion; Based on the second estimated set of explanation steps and the set of explanation steps corresponding to the assertion in the standard evidence set, the number of fifth steps and the number of sixth steps are obtained, wherein the number of fifth steps is the number of explanation steps in the second estimated set of explanation steps that is more than the set of explanation steps, and the number of sixth steps is the number of explanation steps in the set of explanation steps that is more than the second estimated set of explanation steps. Based on the number of steps in the fifth step and the number of steps in the sixth step, the third double-penalty evidence information of the model to be evaluated is obtained, wherein the third double-penalty evidence information indicates the ability of the model to be evaluated to verify the truth of assertions.

9. The method as described in claim 1, characterized in that, The method further includes: For each set of explanatory steps in the standard evidence set, the following processing is performed: The object corresponding to the set of explanation steps and the explanation step library are input into the initial process perception step retrieval model to obtain the third estimated set of explanation steps for the object. The explanation step library includes the ordered explanation steps of each knowledge explanation video in multiple knowledge explanation videos. Based on the third set of estimated explanation steps and the set of explanation steps, the target loss is determined; The parameters of the initial process-aware step retrieval model are adjusted based on the target loss to obtain the process-aware step retrieval model.

10. A model evaluation device, characterized in that, include: The generation unit is configured to input the ordered explanation steps of each knowledge explanation video into the generation model for each knowledge explanation video, thereby obtaining multiple questions, multiple assertions, a set of target answers and explanation steps for each question, and a set of target tags and explanation steps for each assertion. Each explanation step in the ordered explanation steps includes at least the corresponding subtitle text and key image frames. The assertions are knowledge points inferred from the knowledge explanation video. The set of explanation steps for each question is a set of explanation steps that support the target answer for the question. The target tags include support tags, refutation tags, and empty tags. When the target tag corresponding to an assertion is an empty tag, the set of explanation steps corresponding to the assertion is empty. When the target tag corresponding to an assertion is a support tag or a refutation tag, the set of explanation steps corresponding to the assertion is a set of explanation steps that support or refute the assertion. The first merging unit is configured to merge the sets of explanatory steps corresponding to all questions and all assertions into a preliminary set of evidence. The verification unit is configured to, for each set of explanation steps in the preliminary evidence set, obtain supporting information for the set of explanation steps through at least two different verification models. The supporting information indicates whether the label information of the corresponding object in the set of explanation steps is correct, whether the label information of the object can be obtained based on the set of explanation steps, and whether the label information of the object cannot be obtained after any explanation step in the set of explanation steps is deleted. The object is a question or an assertion. When the object is a question, the label information is the target answer. When the object is an assertion, the label information is the target label. The retention unit is configured to retain the set of explanation steps in response to the indication result of the supporting information being "yes". The second merging unit is configured to merge the retained set of explanatory steps into a standard set of evidence. The evaluation unit is configured to evaluate the model to be evaluated based on the standard evidence set.

11. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the model evaluation method as described in any one of claims 1 to 9.

12. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the model evaluation method as described in any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the model evaluation method as described in any one of claims 1 to 9.

Images