Text generation model evaluation method and apparatus

Abstract

This invention provides a method and apparatus for evaluating a text generation model, applied in the field of natural language processing technology. The method includes: acquiring test data, the test data including entity attributes and entity categories; generating multiple test tasks based on the test data, each test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into a text generation model respectively, obtaining corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and apparatus for evaluating text generation models. Background Technology

[0002] In recent years, large language models (LLMs) based on massive pre-trained data have demonstrated outstanding performance in tasks such as natural language understanding, text generation, and question answering systems. Their core advantage lies in encoding rich world knowledge through self-supervised learning mechanisms, enabling flexible application in scenarios such as dialogue interaction, information retrieval, and knowledge reasoning. However, existing research largely focuses on the model's ability to store and retrieve discrete factual knowledge (such as entities, dates, and events), neglecting the more complex logical connections and hierarchical structures within knowledge systems. This isolated knowledge representation differs significantly from the concept network-based knowledge organization model in human cognition, leading to problems such as logical breaks and semantic biases in tasks requiring deep contextual reasoning or cross-domain knowledge connections.

[0003] From a cognitive science perspective, the efficient use of knowledge relies on the hierarchical system of concepts (classes) and the structured expression of their attribute relationships. Ontology, as a formal knowledge representation framework, provides a standardized semantic model for knowledge organization by defining domain-specific concepts, attributes, and logical relationships. Research shows that integrating ontology knowledge into language model training can significantly enhance its accuracy in understanding domain-specific terminology and the coherence of contextual reasoning. However, current technologies lack a systematic evaluation framework to quantify the completeness and structure of ontology knowledge in language models, resulting in a lack of targeted guidance for model optimization. Summary of the Invention

[0004] This invention provides a text generation model evaluation method and apparatus to address the problem that existing technologies have not yet established a systematic evaluation system to quantify the completeness and structure of ontology knowledge in language models, resulting in a lack of targeted guidance for model optimization.

[0005] This invention provides a method for evaluating a text generation model, comprising: acquiring test data, the test data including entity attributes and entity categories; generating multiple test tasks based on the test data, each test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into a text generation model respectively to obtain corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0006] According to a text generation model evaluation method provided by the present invention, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchy structures, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in real-world scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in real-world scenarios.

[0007] According to the text generation model evaluation method provided by the present invention, when the test task is the entity category-hierarchical memory test, the plurality of test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment.

[0008] According to the text generation model evaluation method provided by the present invention, when the test task is the entity category-hierarchy application test, the multiple test questions include: different class judgment, different level judgment, and category hierarchy arrangement.

[0009] According to the text generation model evaluation method provided by the present invention, when the test task is the entity attribute memory test, the plurality of test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment.

[0010] According to the text generation model evaluation method provided by the present invention, when the test task is the entity attribute application test, the plurality of test questions include: negation object judgment and negation subject judgment.

[0011] This invention also provides a text generation model evaluation device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire test data, the test data including entity attributes and entity categories; the processing module is used to generate multiple test tasks based on the test data, each test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein, the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0012] According to a text generation model evaluation device provided by the present invention, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchy structures, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in real-world scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in real-world scenarios.

[0013] According to the text generation model evaluation device provided by the present invention, when the test task is the entity category-hierarchical memory test, the plurality of test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment.

[0014] According to the text generation model evaluation device provided by the present invention, when the test task is the entity category-hierarchy application test, the plurality of test questions include: different class judgment, different level judgment, and category hierarchy arrangement.

[0015] According to the text generation model evaluation device provided by the present invention, when the test task is the entity attribute memory test, the plurality of test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment.

[0016] According to a text generation model evaluation device provided by the present invention, when the test task is an application test of the entity attributes, the plurality of test questions include: negation object judgment and negation subject judgment.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the text generation model evaluation method as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text generation model evaluation method as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the text generation model evaluation method as described above.

[0020] The text generation model evaluation method and apparatus provided by this invention can test the degree of mastery of ontology knowledge of the text generation model through multiple test tasks such as entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test. It can not only reveal the defects in the internal knowledge structure of the text generation model, but also provide directional guidance for the optimization of the text generation model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the text generation model evaluation method provided by the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of the text generation model evaluation device provided by the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0028] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0029] This application describes some exemplary embodiments for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.

[0030] like Figure 1 As shown, this application provides a text generation model evaluation method, which can be applied to a text generation model evaluation device. The text generation model evaluation method may include steps S101-S103:

[0031] S101, The text generation model evaluation device acquires test data.

[0032] The test data mentioned above includes entity attributes and entity categories.

[0033] Optionally, the ontology knowledge framework of the test data may specifically include entities of multiple categories organized in a hierarchical relationship. These entities of multiple categories may be distributed in three levels, namely top category, middle category and bottom category. Each category of entities may correspond to multiple instances, and each category may correspond to at least one attribute.

[0034] For example, in a hierarchical relationship, the top class can be "person", the middle class can be "athlete", and the bottom class can be "football player". An instance under the bottom class "football player" can be athlete a. The attribute of the top class "person" can be "possessing certain cognitive and behavioral abilities", the attribute of the middle class "athlete" can be "participating in sports competitions", and the attribute of the bottom class "football player" can be "mainly engaged in football".

[0035] S102, The text generation model evaluation device generates multiple test tasks based on the test data.

[0036] One test task corresponds to multiple test questions, including entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0037] Optionally, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in real-world scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in real-world scenarios.

[0038] Optionally, when the test task is the entity category-hierarchy memory test, the plurality of test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment.

[0039] Specifically, instance-base class determination refers to, given an instance, causing the text generation model to select the most specific category (base class) to which the entity in that instance belongs.

[0040] For example, the problem of input text generation models can be:

[0041] To which category does the entity “Athlete A” belong?

[0042] A. Football player; B. Basketball player; C. Swimmer; D. Boxer.

[0043] Instance-top-class determination refers to, given an instance, causing the text generation model to select the most abstract category (top class) to which the entity in that instance belongs.

[0044] For example, the problem of input text generation models can be:

[0045] To which category does the entity “Athlete A” belong?

[0046] A. People; B. Events; C. Equipment; D. Organizations.

[0047] Base class-instance judgment refers to the process of a text generation model selecting an instance belonging to a given specific category (base class).

[0048] For example, the problem for input text generation models could be: Which of the following is an instance of "football player"?

[0049] A. Athlete a; B. Athlete b;

[0050] C. Athlete c; D. Athlete d.

[0051] Top-class-instance judgment refers to the process of a text generation model selecting an instance belonging to a given abstract category (top class).

[0052] For example, the problem for input text generation models could be: Which of the following is an instance of "human"?

[0053] A. Athlete a; B. Football match; C. Smartphone; D. Refrigerator.

[0054] Base class-intermediate class determination refers to, given a base class, causing the text generation model to select the intermediate class to which that class belongs.

[0055] For example, the problem for input text generation models could be: What is the middle class to which "football player" belongs?

[0056] A. Athlete; B. Entertainment star; C. Scientist; D. Artist.

[0057] Base-to-top class determination refers to the process of determining the top class to which a text generation model should belong, given a base class.

[0058] For example, the problem for input text generation models could be: What is the top class of "football player"?

[0059] A. Person; B. Event; C. Object; D. Location.

[0060] The middle class-base class decision refers to the process of determining which base class a given middle class belongs to the text generation model.

[0061] For example, the problem for input text generation models could be: Which of the following base classes does "athlete" belong to?

[0062] A. Football player; B. Writer; C. Painter; D. Engineer.

[0063] Top-to-bottom class determination refers to, given a top class, causing the text generation model to select the bottom class that the class belongs to.

[0064] For example, the problem for input text generation models could be: Which of the following base classes does "human" belong to?

[0065] A. Football player; B. Sports meet; C. Car; D. City.

[0066] Optionally, when the test task is the entity category-hierarchy application test, the multiple test questions include: different class judgment, different level judgment, and category hierarchy arrangement.

[0067] Specifically, different class determination refers to selecting one entity from a group of entities that is different from the other entities in the same class.

[0068] For example, the problem for an input text generation model could be: Please identify entities that differ from other entities of different types.

[0069] A. Athlete c; B. Athlete a; C. Stadium; D. Athlete b.

[0070] Different level judgment refers to selecting one entity from a group of entities that is at a different level from the other entities.

[0071] For example, the problem for an input text generation model could be: determining the options that are different from other options at the category level.

[0072] A. Football player; B. Player a; C. Player b; D. Player c.

[0073] Category hierarchy arrangement refers to sorting a group of categories according to a hierarchical structure from the most abstract to the most concrete.

[0074] For example, the problem for an input text generation model could be: Please sort the following categories in order of their hierarchical structure from most abstract to most concrete:

[0075] A. Person; B. Athlete; C. Football player.

[0076] Optionally, when the test task is the entity attribute memory test, the plurality of test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment.

[0077] Class-definition judgment refers to the process of selecting the most appropriate definition for a text generation model given a category.

[0078] For example, the problem for input text generation models could be: Which of the following is the most appropriate definition for the category "football player"?

[0079] A. Individuals who participate in football competitions, acquire football skills through professional training, and compete in football as a profession or hobby.

[0080] B. Individuals who engage in ball sports, with any ball sport as their primary activity.

[0081] C. Individuals who are skilled in track and field events such as running and jumping, and who can also participate in football matches.

[0082] D. People who mainly engage in indoor sports activities, and whose activities are related to football.

[0083] Definition-class judgment refers to the process of a text generation model selecting the corresponding category given a definition.

[0084] For example, the problem for the input text generation model could be: "Personnel who have received professional training, possess skills in specific sports, and participate in sports competitions." Which of the following categories does this definition correspond to?

[0085] A. Athlete; B. Actor; C. Teacher; D. Engineer.

[0086] Subject-relationship-object judgment refers to the process by which a text generation model selects a suitable object given a subject and a relationship.

[0087] For example, the problem of input text generation model can be: the subject is "football player" and the relation is "effective to". Which of the following is the appropriate object?

[0088] A. Football club; B. Music album; C. Mathematical formula; D. Painting.

[0089] Optionally, when the test task is to test the entity attribute, the plurality of test questions include: negation object judgment and negation subject judgment.

[0090] Negation object judgment refers to, given a subject and a relation, causing the text generation model to select an object that is impossible to appear.

[0091] For example, the problem of input text generation model can be: the subject is "football player" and the relation is "participate in the game". Which of the following objects is impossible?

[0092] A. Football match; B. Basketball match; C. Track and field event; D. Cooking competition.

[0093] Negative subject judgment refers to, given a relation and an object, causing the text generation model to select a subject that is impossible to appear.

[0094] For example, the problem of input text generation model can be: given the relation "creation" and the object "song", which of the following is the impossible subject?

[0095] A. Football player; B. Songwriter; C. Music producer; D. Singer.

[0096] It should be noted that this application can design the input as a multiple-choice question, requiring the text generation model to output the number of the correct option. This ensures that the text generation model fully understands the intent of the natural language input and avoids ambiguity during the text generation process.

[0097] S103. The text generation model evaluation device inputs the test questions of the multiple test tasks into the text generation model respectively, obtains the corresponding test results, and determines the degree of mastery of the ontology knowledge by the text generation model based on the test results.

[0098] Specifically, the accuracy rate of the text generation model's answers to all questions in each task category is the model's score for that task. The text generation model evaluation device can analyze the model's mastery of ontology knowledge based on its scores across different tasks, thereby determining the extent to which the text generation model has grasped ontology knowledge.

[0099] In this embodiment, the text generation model's mastery of ontology knowledge can be tested through multiple test tasks such as entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test. This not only reveals the defects in the text generation model's internal knowledge structure but also provides directional guidance for the optimization of the text generation model.

[0100] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] It should be noted that the apparatus in the embodiments of this application includes a virtual apparatus and a physical apparatus. The virtual apparatus can be a text generation model evaluation apparatus, and the physical apparatus can include electronic devices, computer storage media, and computer program products.

[0102] The text generation model evaluation method provided in this application can be executed by a text generation model evaluation device or a control module within that device for text generation model evaluation. This application uses the text generation model evaluation device executing the text generation model evaluation method as an example to illustrate the text generation model evaluation device provided in this application.

[0103] It should be noted that the embodiments of this application can divide the text generation model evaluation device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. Optionally, the module division in the embodiments of this application is illustrative and is only a logical functional division; other division methods may be used in actual implementation.

[0104] like Figure 2 As shown in the figure, this application embodiment provides a text generation model evaluation device 200. The text generation model evaluation device 200 includes an acquisition module 201 and a processing module 202. The acquisition module 201 is used to acquire test data, which includes entity attributes and entity categories. The processing module 202 is used to generate multiple test tasks based on the test data, with each test task corresponding to multiple test questions; input the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determine the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein, the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0105] Optionally, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in real-world scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in real-world scenarios.

[0106] Optionally, when the test task is the entity category-hierarchy memory test, the plurality of test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment.

[0107] Optionally, when the test task is the entity category-hierarchy application test, the multiple test questions include: different class judgment, different level judgment, and category hierarchy arrangement.

[0108] Optionally, when the test task is the entity attribute memory test, the plurality of test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment.

[0109] Optionally, when the test task is to test the entity attribute, the plurality of test questions include: negation object judgment and negation subject judgment.

[0110] In this embodiment, the text generation model's mastery of ontology knowledge can be tested through multiple test tasks such as entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test. This not only reveals the defects in the text generation model's internal knowledge structure but also provides directional guidance for the optimization of the text generation model.

[0111] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a text generation model evaluation method. This method includes: acquiring test data, the test data including entity attributes and entity categories; generating multiple test tasks based on the test data, each test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0112] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text generation model evaluation method provided by the above methods. The method includes: acquiring test data, the test data including entity attributes and entity categories; generating multiple test tasks based on the test data, one test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determining the degree of mastery of the text generation model of ontology knowledge based on the test results; wherein the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0114] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the text generation model evaluation method provided by the above methods. The method includes: acquiring test data, the test data including entity attributes and entity categories; generating multiple test tasks based on the test data, each test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into a text generation model respectively to obtain corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model based on the test results; wherein the multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating a text generation model, characterized in that, include: Acquire test data, which is constructed based on an ontology knowledge framework. The ontology knowledge framework includes entity categories organized according to a top-middle-bottom class hierarchy, and entity attributes corresponding to each entity category. Multiple test tasks are generated based on the test data, and one test task corresponds to multiple test questions. The test questions of the multiple test tasks are input into the text generation model to obtain the corresponding test results, and the degree of mastery of the text generation model of ontology knowledge is determined based on the test results. The multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test; The step of determining the degree of mastery of ontology knowledge by the text generation model based on the test results includes: evaluating the text generation model's ability to memorize ontology knowledge through the entity category-hierarchy memory test and the entity attribute memory test, and evaluating the text generation model's ability to apply ontology knowledge through the entity category-hierarchy application test and the entity attribute application test. Based on the scores of memory ability and application ability, the degree of mastery of ontology knowledge by the text generation model is determined. When the test task is the entity category-hierarchy memory test, the multiple test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment; When the test task is the entity category-hierarchy application test, the multiple test questions include: different category judgment, different level judgment, and category hierarchy arrangement; When the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment; When the test task is to test the entity attribute, the multiple test questions include: negation object judgment and negation subject judgment.

2. The text generation model evaluation method according to claim 1, characterized in that, The entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in real-world scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in real-world scenarios.

3. A text generation model evaluation device, characterized in that, include: Acquisition module and processing module; The acquisition module is used to acquire test data, which is constructed based on an ontology knowledge framework. The ontology knowledge framework includes entity categories organized according to a top-middle-bottom class hierarchical relationship, and entity attributes corresponding to each entity category. The processing module is used to generate multiple test tasks based on the test data, with each test task corresponding to multiple test questions; input the test questions of the multiple test tasks into the text generation model respectively to obtain the corresponding test results, and determine the degree of mastery of the text generation model of ontology knowledge based on the test results; The multiple test tasks include entity category-hierarchy memory test, entity category-hierarchy application test, entity attribute memory test, and entity attribute application test; The processing module is used to evaluate the text generation model's ability to memorize ontology knowledge through the entity category-hierarchy memory test and the entity attribute memory test, and to evaluate the text generation model's ability to apply ontology knowledge through the entity category-hierarchy application test and the entity attribute application test. Based on the scores of memory ability and application ability, the degree of mastery of ontology knowledge by the text generation model is determined. When the test task is the entity category-hierarchy memory test, the multiple test questions include: instance-base class judgment, instance-top class judgment, base class-instance judgment, top class-instance judgment, base class-middle class judgment, base class-top class judgment, middle class-base class judgment, and top class-base class judgment; When the test task is the entity category-hierarchy application test, the multiple test questions include: different category judgment, different level judgment, and category hierarchy arrangement; When the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment, and subject-relationship-object judgment; When the test task is to test the entity attribute, the multiple test questions include: negation object judgment and negation subject judgment.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the text generation model evaluation method as described in any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text generation model evaluation method as described in any one of claims 1 to 2.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text generation model evaluation method as described in any one of claims 1 to 2.

Images