Text generation model evaluation method and device

Abstract

The invention provides a text generation model evaluation method and device, and is applied to the technical field of natural language processing. The method comprises the following steps: acquiring test data, wherein the test data comprises entity attributes and entity categories; generating a plurality of test tasks according to the test data, wherein one test task corresponds to a plurality of test problems; respectively inputting the test problems of the plurality of test tasks into a text generation model to obtain corresponding test results, and determining the mastering degree of the text generation model for ontology knowledge according to the test results; wherein the plurality of test tasks comprise an entity category-hierarchy memory test, an entity category-hierarchy application test, an entity attribute memory test and an entity attribute application test.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text generation model evaluation method and device. Background Art

[0002] In recent years, large language models (LLMs) based on massive pre-training data have demonstrated excellent performance in tasks such as natural language understanding, text generation, and question-answering systems. Its core advantage is that it encodes rich world knowledge through a self-supervised learning mechanism, and can be flexibly applied to scenarios such as dialogue interaction, information retrieval, and knowledge reasoning. However, existing research has mostly focused on the model's ability to store and extract discrete factual knowledge (such as entities, dates, and events), while ignoring the more complex logical associations and hierarchical structures in the knowledge system. This isolated way of representing knowledge is significantly different from the knowledge organization model based on concept networks in human cognition, which makes the model prone to problems such as logical breaks and semantic deviations in tasks that require deep context reasoning or cross-domain knowledge associations.

[0003] From the perspective of cognitive science, the efficient use of knowledge depends on the hierarchical system of concepts (classes) and the structured expression of their attribute relationships. Ontology, as a formal knowledge representation framework, can provide a standardized semantic model for knowledge organization by defining concepts, attributes and logical relationships within a domain. Studies have shown that incorporating ontology knowledge into language model training can significantly enhance the accuracy of its understanding of professional domain terms and the coherence of contextual reasoning. However, the existing technology has not yet established a systematic evaluation system to quantify the completeness and structured degree of ontology knowledge in language models, resulting in a lack of targeted guidance for model optimization. Summary of the invention

[0004] The present invention provides a text generation model evaluation method and device to solve the problem that the prior art has not yet established a systematic evaluation system to quantify the completeness and structured degree of ontological knowledge in a language model, resulting in a lack of targeted guidance for model optimization.

[0005] The present invention provides a text generation model evaluation method, comprising: acquiring test data, the test data comprising entity attributes and entity categories; generating multiple test tasks according to the test data, one test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into a text generation model respectively to obtain corresponding test results, and determining the degree of mastery of ontology knowledge by the text generation model according to the test results; wherein the multiple test tasks comprise an entity category-level memory test, an entity category-level application test, an entity attribute memory test, and an entity attribute application test.

[0006] According to a text generation model evaluation method provided by the present invention, the entity category-level memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-level application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in actual scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in actual scenarios.

[0007] According to a text generation model evaluation method provided by the present invention, when the test task is the entity category-level memory test, the multiple test questions include: instance-bottom class judgment, instance-top class judgment, bottom class-instance judgment, top class-instance judgment, bottom class-middle class judgment, bottom class-top class judgment, middle class-bottom class judgment and top class-bottom class judgment.

[0008] According to a text generation model evaluation method provided by the present invention, when the test task is the entity category-level application test, the multiple test questions include: different class judgments, different level judgments and category level arrangement.

[0009] According to a text generation model evaluation method provided by the present invention, when the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment and subject-relationship-object judgment.

[0010] According to a text generation model evaluation method provided by the present invention, when the test task is the entity attribute application test, the multiple test questions include: negative object judgment and negative subject judgment.

[0011] The present invention also provides a text generation model evaluation device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire test data, the test data including entity attributes and entity categories; the processing module is used to generate multiple test tasks based on the test data, and one test task corresponds to multiple test questions; the test questions of the multiple test tasks are respectively input into the text generation model to obtain corresponding test results, and the degree of mastery of the ontology knowledge by the text generation model is determined based on the test results; wherein the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

[0012] According to a text generation model evaluation device provided by the present invention, the entity category-level memory test is used to evaluate the text generation model's ability to remember the category hierarchy, the entity category-level application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in actual scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in actual scenarios.

[0013] According to a text generation model evaluation device provided by the present invention, when the test task is the entity category-level memory test, the multiple test questions include: instance-bottom class judgment, instance-top class judgment, bottom class-instance judgment, top class-instance judgment, bottom class-middle class judgment, bottom class-top class judgment, middle class-bottom class judgment and top class-bottom class judgment.

[0014] According to a text generation model evaluation device provided by the present invention, when the test task is the entity category-level application test, the multiple test questions include: different class judgments, different level judgments and category level arrangement.

[0015] According to a text generation model evaluation device provided by the present invention, when the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment and subject-relationship-object judgment.

[0016] According to a text generation model evaluation device provided by the present invention, when the test task is the entity attribute application test, the multiple test questions include: negative object judgment and negative subject judgment.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text generation model evaluation method described in any one of the above methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the text generation model evaluation method as described in any one of the above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the text generation model evaluation method as described in any one of the above is implemented.

[0020] The text generation model evaluation method and device provided by the present invention can test the degree of mastery of the text generation model of ontology knowledge through multiple test tasks such as entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test. It can not only reveal the defects of the inherent knowledge structure of the text generation model, but also provide directional guidance for the optimization of the text generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of the text generation model evaluation method provided by the present invention; Figure 2 It is a structural schematic diagram of a text generation model evaluation device provided by the present invention; Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0026] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and order of execution.

[0027] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.

[0028] like Figure 1 As shown, the embodiment of the present application provides a text generation model evaluation method, which can be applied to a text generation model evaluation device. The text generation model evaluation method may include S101-S103: S101. The text generation model evaluation device obtains test data.

[0029] The above test data includes entity attributes and entity categories.

[0030] Optionally, the ontology knowledge framework of the test data may specifically include entities of multiple categories organized in a hierarchical relationship. The entities of the multiple categories may be distributed in three levels, namely, top category, middle category and bottom category. Each category of entities may correspond to multiple instances, and each category may correspond to at least one attribute.

[0031] For example, the top class in the hierarchical relationship can be "person", the middle class can be "athlete", the bottom class can be "football player", and the instance under the bottom class of football player can be athlete a. The attributes of the top class "person" can be "possessing certain cognitive and behavioral abilities", the attributes of the middle class "athlete" can be "participating in sports competitive activities", and the attributes of the bottom class "football player" can be "mainly engaged in football sports."

[0032] S102: The text generation model evaluation device generates a plurality of test tasks according to the test data.

[0033] Among them, one test task corresponds to multiple test questions, and the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

[0034] Optionally, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in actual scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in actual scenarios.

[0035] Optionally, when the test task is the entity category-hierarchy memory test, the multiple test questions include: instance-bottom class judgment, instance-top class judgment, bottom class-instance judgment, top class-instance judgment, bottom class-middle class judgment, bottom class-top class judgment, middle class-bottom class judgment and top class-bottom class judgment.

[0036] Specifically, instance-base class judgment means that given an instance, the text generation model selects the most specific category (base class) to which the entity in the instance belongs.

[0037] For example, the input question for a text generation model could be: To what category does the entity "athlete a" belong? A. Football player; B. Basketball player; C. Swimmer; D. Boxer.

[0038] Instance-top category judgment means that given an instance, the text generation model selects the most abstract category (top category) to which the entity in the instance belongs.

[0039] For example, the input question for a text generation model could be: To what category does the entity "athlete a" belong? A. People; B. Events; C. Equipment; D. Organization.

[0040] Base class-instance judgment means that given a specific category (base class), the text generation model selects an instance belonging to this category.

[0041] For example, the question input to a text generation model could be: Which of the following is an instance of “football player”? A. Athlete a; B. Athlete b; C. athlete c; D. athlete d.

[0042] Top-class-instance judgment means that given an abstract category (top class), the text generation model selects an instance belonging to this category.

[0043] For example, the question input to a text generation model could be: Which of the following is an instance of “person”? A. Athlete a; B. Football match; C. Smart phone; D. Refrigerator.

[0044] The base-middle class judgment means that given the base class, the text generation model selects the middle class to which the class belongs.

[0045] For example, the question input to a text generation model could be: "What class does 'football player' belong to?" A. Athlete; B. Entertainment star; C. Scientist; D. Artist.

[0046] Bottom-top category judgment means that given a bottom category, the text generation model selects the top category to which the category belongs.

[0047] For example, the question input to a text generation model could be: "What is the top category to which 'football player' belongs?" A. People; B. Events; C. Objects; D. Places.

[0048] Middle-bottom class judgment means that given a middle class, the text generation model selects the bottom class of this category.

[0049] For example, the question input to a text generation model could be: "Which of the following underlying categories does the word 'athlete' have?" A. Football player; B. Writer; C. Painter; D. Engineer.

[0050] Top-class-bottom-class judgment means that given a top class, the text generation model selects the bottom class that the class has.

[0051] For example, the question input to a text generation model could be: Which of the following underlying categories does the word "person" have? A. Football player; B. Sports meeting; C. Car; D. City.

[0052] Optionally, in the case where the test task is the entity category-level application test, the multiple test questions include: different class judgments, different level judgments and category level arrangement.

[0053] Specifically, different-class judgment refers to selecting one entity from a group of entities that is different from the other entity categories.

[0054] For example, the question for the text generation model could be: Please identify entities of different types from: A. Athlete c; B. Athlete a; C. Stadium; D. Athlete b.

[0055] Different-level judgment refers to selecting one entity from a group of entities that is at a different level from the other entities.

[0056] For example, the input problem for a text generation model could be: identify the option that is categorically distinct from the other options: A. Football player; B. Athlete a; C. Athlete b; D. Athlete c.

[0057] Category hierarchy refers to the arrangement of a set of categories into a hierarchical structure from the most abstract to the most specific.

[0058] For example, the input question to a text generation model could be: Please rank the following categories in a hierarchy from most abstract to most specific: A. People; B. Athletes; C. Football players.

[0059] Optionally, when the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment and subject-relationship-object judgment.

[0060] Class-definition judgment means that given a category, the text generation model selects the most appropriate definition.

[0061] For example, the question that could be input to a text generation model could be: "Which of the following is the most appropriate definition for the category 'football player'?" A. People who participate in football games, acquire football skills through professional training, and compete in football as a profession or hobby.

[0062] B. People who engage in ball sports, with free kick events as their main activity.

[0063] C. People who are good at track and field events such as running and jumping, and can also participate in football games.

[0064] D. People who mainly engage in sports activities indoors and the content of the activities is related to football.

[0065] Definition-class judgment means that given a definition, the text generation model selects the corresponding category.

[0066] For example, the question that can be input into the text generation model can be: "People who have undergone professional training, acquired specific sports skills, and participate in sports competitions" To which of the following categories does this definition correspond? A. Athlete; B. Actor; C. Teacher; D. Engineer.

[0067] Subject-relation-object judgment means that given a subject and a relationship, the text generation model selects a suitable object.

[0068] For example, the question input to the text generation model could be: the subject is "football player", the relation is "plays for", which of the following is the appropriate object? A. Football club; B. Music album; C. Mathematical formula; D. Painting.

[0069] Optionally, in the case where the test task is a test of the entity attribute application, the multiple test questions include: negative object judgment and negative subject judgment.

[0070] Negative object judgment means that given a subject and a relationship, the text generation model selects an object that is impossible to appear.

[0071] For example, the question input to the text generation model can be: the subject is "football player", the relationship is "participating in the game", which of the following objects cannot appear? A. Football match; B. Basketball match; C. Track and field event; D. Cooking competition.

[0072] Negative subject judgment means that given a relationship and an object, the text generation model chooses a subject that is impossible to appear.

[0073] For example, the question input to the text generation model can be: The relation is "creation", the object is "song", which of the following subjects cannot appear? A. Football player; B. Songwriter; C. Music producer; D. Singer.

[0074] It should be noted that the application can design the input in the form of multiple-choice questions, requiring the text generation model to output the serial number of the correct option. In this way, it can ensure that the text generation model fully understands the intention of the natural language input and avoids ambiguity in the text generation model generation process.

[0075] S103, the text generation model evaluation device inputs the test questions of the multiple test tasks into the text generation model respectively, obtains corresponding test results, and determines the degree of mastery of the text generation model on the ontology knowledge according to the test results.

[0076] Specifically, the correct rate of answers given by the text generation model in all questions in each type of task is the score of the model on this type of task. The text generation model evaluation device can analyze the model's mastery of ontological knowledge based on the scores of the text generation model on different tasks to determine the degree of mastery of the text generation model on ontological knowledge.

[0077] In an embodiment of the present application, the degree of mastery of the text generation model for ontological knowledge can be tested through multiple test tasks such as entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test. This can not only reveal the defects of the inherent knowledge structure of the text generation model, but also provide directional guidance for the optimization of the text generation model.

[0078] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0079] It should be noted that the device in the embodiment of the present application includes a virtual device and a physical device. The virtual device can be a text generation model evaluation device, and the physical device can include an electronic device, a computer storage medium, and a computer program product.

[0080] The text generation model evaluation method provided in the embodiment of the present application can be executed by a text generation model evaluation device or a control module for text generation model evaluation in the text generation model evaluation device. In the embodiment of the present application, the text generation model evaluation method is executed by a text generation model evaluation device as an example to illustrate the text generation model evaluation device provided in the embodiment of the present application.

[0081] It should be noted that the embodiment of the present application can divide the text generation model evaluation device into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0082] like Figure 2 As shown, an embodiment of the present application provides a text generation model evaluation device 200. The text generation model evaluation device 200 includes: an acquisition module 201 and a processing module 202. The acquisition module 201 is used to acquire test data, and the test data includes entity attributes and entity categories; the processing module 202 is used to generate multiple test tasks according to the test data, and one test task corresponds to multiple test questions; the test questions of the multiple test tasks are respectively input into the text generation model to obtain corresponding test results, and the degree of mastery of the ontology knowledge by the text generation model is determined according to the test results; wherein the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

[0083] Optionally, the entity category-hierarchy memory test is used to evaluate the text generation model's ability to remember category hierarchies, the entity category-hierarchy application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in actual scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in actual scenarios.

[0084] Optionally, when the test task is the entity category-hierarchy memory test, the multiple test questions include: instance-bottom class judgment, instance-top class judgment, bottom class-instance judgment, top class-instance judgment, bottom class-middle class judgment, bottom class-top class judgment, middle class-bottom class judgment and top class-bottom class judgment.

[0085] Optionally, in the case where the test task is the entity category-level application test, the multiple test questions include: different class judgments, different level judgments and category level arrangement.

[0086] Optionally, when the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment and subject-relationship-object judgment.

[0087] Optionally, in the case where the test task is an application test of the entity attribute, the multiple test questions include: negative object judgment and negative subject judgment.

[0088] In an embodiment of the present application, the degree of mastery of the text generation model for ontological knowledge can be tested through multiple test tasks such as entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test. This can not only reveal the defects of the inherent knowledge structure of the text generation model, but also provide directional guidance for the optimization of the text generation model.

[0089] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the text generation model evaluation method, which includes: obtaining test data, wherein the test data includes entity attributes and entity categories; generating multiple test tasks according to the test data, wherein one test task corresponds to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively, obtaining corresponding test results, and determining the degree of mastery of the ontology knowledge by the text generation model according to the test results; wherein the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

[0090] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text generation model evaluation method provided by the above-mentioned methods, and the method includes: obtaining test data, and the test data includes entity attributes and entity categories; generating multiple test tasks according to the test data, and one test task corresponds to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determining the degree of mastery of the ontology knowledge by the text generation model according to the test results; wherein the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

[0092] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text generation model evaluation method provided by the above-mentioned methods, the method comprising: obtaining test data, the test data comprising entity attributes and entity categories; generating multiple test tasks according to the test data, one test task corresponding to multiple test questions; inputting the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determining the degree of mastery of the ontology knowledge by the text generation model according to the test results; wherein the multiple test tasks comprise an entity category-level memory test, an entity category-level application test, an entity attribute memory test, and an entity attribute application test.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text generation model evaluation method, characterized in that: include: Acquire test data, wherein the test data includes entity attributes and entity categories; Generate multiple test tasks according to the test data, one test task corresponds to multiple test questions; Input the test questions of the multiple test tasks into the text generation model respectively to obtain corresponding test results, and determine the degree of mastery of the ontology knowledge by the text generation model according to the test results; The multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

2. The text generation model evaluation method according to claim 1, characterized in that: The entity category-level memory test is used to evaluate the text generation model's ability to remember the category hierarchy, the entity category-level application test is used to evaluate the text generation model's ability to apply hierarchical knowledge in actual scenarios, the entity attribute memory test is used to evaluate the text generation model's ability to remember category definitions and attributes, and the entity attribute application test is used to evaluate the text generation model's ability to apply attribute knowledge in actual scenarios.

3. The text generation model evaluation method according to claim 1 or 2, characterized in that: When the test task is the entity category-level memory test, the multiple test questions include: instance-bottom class judgment, instance-top class judgment, bottom class-instance judgment, top class-instance judgment, bottom class-middle class judgment, bottom class-top class judgment, middle class-bottom class judgment and top class-bottom class judgment.

4. The text generation model evaluation method according to claim 1 or 2, characterized in that: In the case where the test task is the entity category-level application test, the plurality of test questions include: different-category judgment, different-level judgment, and category-level arrangement.

5. The text generation model evaluation method according to claim 1 or 2, characterized in that: In the case where the test task is the entity attribute memory test, the multiple test questions include: class-definition judgment, definition-class judgment and subject-relationship-object judgment.

6. The text generation model evaluation method according to claim 1 or 2, characterized in that: In the case where the test task is an application test of the entity attribute, the plurality of test questions include: negative object judgment and negative subject judgment.

7. A text generation model evaluation device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire test data, wherein the test data includes entity attributes and entity categories; The processing module is used to generate multiple test tasks based on the test data, and one test task corresponds to multiple test questions; the test questions of the multiple test tasks are respectively input into the text generation model to obtain corresponding test results, and the degree of mastery of the ontology knowledge by the text generation model is determined according to the test results; wherein the multiple test tasks include entity category-level memory test, entity category-level application test, entity attribute memory test and entity attribute application test.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the text generation model evaluation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text generation model evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the text generation model evaluation method according to any one of claims 1 to 6 is implemented.

Images