Learning material evaluation system, method, and program

Abstract

To provide a learning material evaluation system, a method, and a program capable of automatically evaluating learning materials.SOLUTION: A learning material evaluation system according to the present invention comprises: acquisition means for acquiring text information indicating learning materials; extraction means for extracting keyword information indicating words and related information relating to the words based on the text information acquired by the acquisition means; and output means for inputting the keyword information and related information extracted by the extraction means into a base model and outputting evaluation information indicating an evaluation of the learning material.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a learning material evaluation system, method, and program for evaluating educational learning materials. [Background technology]

[0002] Currently, there is much research being done on automatic text generation using AI (artificial intelligence). In particular, by using natural language processing models trained on large amounts of text data, it is possible to, for example, classify input text, analyze emotions from input text, summarize input text, and generate responses to input questions.

[0003] Patent Document 1 discloses a system that uses a natural language processing model to generate questions in natural language and answers (Q&A) to those questions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-2475 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the generation system disclosed in Patent Document 1, a natural language processing model is used to generate a natural language question using a first phrase that expresses a cause or effect, and candidate second phrases are extracted from candidate text, and the candidate second phrase with the highest probability is selected as having a causal relationship with the first phrase, thereby generating a combination of question and answer.

[0006] However, the generation system disclosed in Patent Document 1 does not disclose, for example, automatic evaluation of the generated learning materials. Therefore, the generation system disclosed in Patent Document 1 cannot automatically proofread the generated learning materials if there are errors in the generated learning materials.

[0007] The present invention was devised in consideration of the above-mentioned problems, and its purpose is to provide a learning material evaluation system, method, and program that can automatically evaluate learning materials. [Means for solving the problem]

[0008] The learning material evaluation system of the first invention is characterized by comprising an acquisition means for acquiring text information indicating learning materials, and an output means for inputting the text information acquired by the acquisition means into a base model and outputting evaluation information indicating an evaluation of the learning materials.

[0009] The learning material evaluation system of the second invention is characterized in that, in the first invention, it further comprises an extraction means for extracting keyword information indicating a word and related information related to the word based on the text information acquired by the acquisition means, and the output means inputs the keyword information and related information extracted by the extraction means into a base model and outputs the evaluation information.

[0010] The learning material evaluation system according to the third invention is characterized in that, in the second invention, the output means inputs keyword information and related information extracted by the extraction means into the basic model trained using learning data that takes keyword information and related information as input and outputs evaluation information, and outputs the evaluation information.

[0011] The learning material evaluation system of the fourth invention is characterized in that, in the third invention, it further comprises a search means that refers to a database that stores keyword information and related information in association with each other, and searches for related information stored in the database based on the keyword information extracted by the extraction means, and the output means inputs the keyword information and related information extracted by the extraction means into the basic model, and outputs the evaluation information based on the related information searched by the search means.

[0012] The learning material evaluation system of the fifth invention is characterized in that, in the first invention, the output means inputs the text information acquired by the acquisition means into a base model and outputs the evaluation information including grounds information indicating the grounds for the evaluation.

[0013] The learning material evaluation system of the sixth invention is characterized in that, in the first invention, the output means inputs the text information acquired by the acquisition means into a large language model and outputs the evaluation information.

[0014] The learning material evaluation system of the seventh invention is characterized in that, in the first invention, it further comprises a correction means for generating corrected text information that corrects the learning material based on the text information acquired by the acquisition means and the evaluation information output by the output means.

[0015] The learning material evaluation system of the eighth invention is characterized in that, in the seventh invention, it further comprises a generation means for generating text information based on a generation theme of the learning material by referring to a generation model trained using learning data including corrected text information corrected by the correction means and evaluation information output by the output means.

[0016] The learning material evaluation method of the ninth invention is characterized by having an acquisition step of acquiring text information indicating learning material, and an output step of inputting the text information acquired by the acquisition step into a base model and outputting evaluation information indicating an evaluation of the learning material.

[0017] The learning material evaluation program of the 10th invention is characterized in that it causes a computer to execute an acquisition step of acquiring text information indicating learning material, and an output step of inputting the text information acquired by the acquisition step into a base model and outputting evaluation information indicating an evaluation of the learning material. [Effects of the Invention]

[0018] According to the first to tenth aspects of the present invention, the learning material evaluation system, method, and program of the present invention inputs text information into a base model and outputs evaluation information, thereby enabling automatic evaluation of learning materials.

[0019] In particular, according to the second aspect of the present invention, the learning material evaluation system of the present invention inputs keyword information and related information into a base model and outputs evaluation information. This allows for consideration of the consistency between the words contained in the learning material and the related information. This makes it possible to evaluate the learning material with higher accuracy.

[0020] In particular, according to the third aspect of the present invention, the learning material evaluation system of the present invention inputs keyword information and related information to a base model trained using training data that inputs keyword information and related information and outputs evaluation information, and outputs evaluation information. This makes it possible to consider the correlation between the keyword information and related information and the evaluation information. This makes it possible to evaluate learning materials with higher accuracy.

[0021] In particular, according to the fourth aspect of the present invention, the learning material evaluation system inputs keyword information and related information into a base model and outputs evaluation information based on the related information retrieved by the search means. This makes it possible to evaluate learning materials with higher accuracy by using a base model in which the probability of occurrence between words contained in the related information retrieved by the search means is set high, for example.

[0022] In particular, according to the fifth aspect of the present invention, the learning material evaluation system inputs text information into a base model and outputs evaluation information including evidence information, thereby making it possible to output the evidence for the evaluation of the learning material.

[0023] In particular, according to the sixth aspect of the present invention, the learning material evaluation system of the present invention uses a large-scale language model. This makes it possible to train the model from a huge amount of data. This allows for more flexible answers to questions.

[0024] In particular, according to the seventh aspect of the present invention, the learning material evaluation system generates corrected text information based on text information and evaluation information, thereby making it possible to automatically correct learning materials in consideration of the evaluation information.

[0025] In particular, according to the eighth aspect of the present invention, the learning material evaluation system of the present invention generates text information based on the learning material generation theme by referring to a generative model trained using learning data including corrected text information and evaluation information, thereby making it possible to generate new learning materials that reflect the corrections to the learning materials. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a schematic diagram showing an example of a learning material evaluation system according to this embodiment. [Figure 2] 2(a) and 2(b) are schematic diagrams showing an example of the configuration of a learning material evaluation system according to this embodiment. [Figure 3] FIG. 3 is a schematic diagram showing an example of a detailed configuration of the learning material evaluation system according to this embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the operation of generating learning materials in the learning material evaluation system according to this embodiment. [Figure 5] Figure 5(a) is a schematic diagram showing an example of the operation of generating learning materials in the learning material evaluation system in this embodiment, and Figure 5(b) is a schematic diagram showing a modified example of the operation of generating learning materials in the learning material evaluation system in this embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of the operation of generating generation theme information in the learning material evaluation system according to this embodiment. [Figure 7] FIG. 7 is a schematic diagram showing an example of the operation of generating related generation theme information in the learning material evaluation system according to this embodiment. [Figure 8] FIG. 8 is a schematic diagram showing an example of a database used in the operation of generating learning materials in the learning material evaluation system according to this embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the operation of evaluating a learning material by the learning material evaluation system in this embodiment. [Figure 10] Figure 10(a) is a schematic diagram showing an example of the operation of evaluating learning materials by the learning material evaluation system in this embodiment, and Figure 10(b) is a schematic diagram showing a modified example of the operation of evaluating learning materials by the learning material evaluation system in this embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example of the correlation of the evaluation model. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, examples of a learning material evaluation system 100, a learning material evaluation method, and a learning material evaluation program as embodiments of the present invention will be described in detail with reference to the drawings. Note that the configurations in each drawing are depicted schematically for the purpose of explanation, and for example, the size of each component, the size comparison between components, etc. may differ from those shown in the drawings.

[0028] An example of the configuration of a learning material evaluation system 100 according to this embodiment will be described with reference to FIGS.

[0029] As shown in Fig. 1, the learning material evaluation system 100 includes a user terminal 1, a server 2, and a communication network 9. The learning material evaluation system 100 receives input from a user U via the user terminal 1, and acquires a query to be executed on a database that stores a pre-stored model such as a large-scale language model. The learning material evaluation system 100 then executes the acquired query, thereby automatically generating and evaluating learning materials based on the model such as a large-scale language model.

[0030] The user U refers to a user who presents learning materials generated and evaluated using the learning material evaluation system 100 to answering users who are the subjects of education. By providing the answering users with automatically generated learning materials, the user U can improve the efficiency of educational activities for the answering users. The user U may also be, for example, the answering user. In other words, the user U can improve the efficiency of self-study by answering the learning materials generated using the learning material evaluation system 100 by himself.

[0031] Learning materials refer to tests that test the knowledge of the respondent user, including either questions or answers, and include question formats such as essay questions, multiple-choice questions, and matching questions. Learning materials may also be texts that provide explanations, such as textbooks. Learning materials may also be texts that include keywords indicating words such as names of people, places, and events, and explanations of the keywords. Learning materials may also include any teaching materials that are output to the respondent user as visual or auditory information. Examples of learning materials include learning material data generated by the learning material evaluation system 100, images and videos displayed on a display based on the learning material data, 3D images and 3D videos displayed via a holographic display based on the learning material data, printed materials printed based on the learning material data, and audio played via an audio device based on the learning material data.

[0032] In this embodiment, a database that stores models such as a large-scale language model is stored in the server 2, and the learning material evaluation system 100 generates and evaluates learning materials in the server 2. However, the configuration is not limited to this. The learning material evaluation system 100 may store a database that stores a large-scale language model in the user terminal 1, and generate learning materials in the user terminal 1, in which case the server 2 need not be provided.

[0033] <User device 1> The user terminal 1 is a terminal that controls the automatic generation of learning materials. The user terminal 1 is a terminal that is operated by a user U. The user terminal 1 is connected to a server 2 via a communication network 9 for communication.

[0034] As shown in FIG. 2(a), the user terminal 1 includes a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The components 101 to 107 are connected via an internal bus 110. The user terminal 1 may be a well-known electronic device such as a laptop PC, a desktop PC, a tablet terminal, or a smartphone.

[0035] The CPU 101 controls the entire user terminal 1. The ROM 102 stores operation code for the CPU 101. The RAM 103 is a working area used when the CPU 101 is operating. The storage unit 104 stores various information such as backups of data stored in the ROM, databases, and learning target data. The storage unit 104 may be, for example, a data storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). For example, the user terminal 1 may have a GPU (Graphics Processing Unit) (not shown).

[0036] The I / F 105 is an interface for transmitting and receiving various information to and from the server 2 as needed via the communication network 9. The I / F 106 is an interface for transmitting and receiving information to and from the input unit 108. The input unit 108 may be, for example, a keyboard or a mouse, and the user U inputs various information via the input unit 108. The I / F 107 is an interface for transmitting and receiving various information to and from the display unit 109. The display unit 109 displays various information stored in the storage unit 104, evaluation results, etc. A display is used as the display unit 109, and in the case of a touch panel type, it is provided integrally with the input unit 108. The display unit 109 displays learning materials generated by each component of the learning material evaluation system 100. The display unit 109 displays, for example, learning materials generated by a generation unit 12 (described below) included in the server 2.

[0037] The display unit 109 may display the learning material on a monitor, for example. The display unit 109 may display the learning material in a format that allows the user U to understand the characteristics of the learning material, for example, by using a display format that is pre-stored in the storage unit 104, for example.

[0038] 3, the user terminal 1 includes an acquisition unit 11, a generation unit 12 and an evaluation unit 13 connected to the acquisition unit 11, a correction unit 14 connected to the evaluation unit 13, and a storage unit 15 connected to the acquisition unit 11, the generation unit 12, and the evaluation unit 13. Each component of the user terminal 1 is realized by the CPU 101 using the RAM 103 as a working area to execute programs stored in the ROM 102, the storage unit 104, etc.

[0039] <Acquisition part 11> The acquisition unit 11 acquires various types of information. The acquisition unit 11 transmits the information to, for example, the server 2. The acquisition unit 11 transmits, for example, information acquired, generated, evaluated, or stored by each component of the user terminal 1 to the server 2 via the communication network 9. The acquisition unit 11 receives, for example, information acquired, generated, or stored by the server 2 via the communication network 9.

[0040] The acquisition unit 11 also acquires a generation request, which is a query (query data) requesting a model such as a large-scale language model to generate learning materials, or an evaluation request, which is a query requesting evaluation. The acquisition unit 11 acquires a query requesting a model such as a large-scale language model to generate learning materials or an evaluation, for example, by a method of accepting input from the user U via the input unit 108.

[0041] <Generation part 12> The generating unit 12 executes the generation request acquired or generated by each component of the learning material evaluation system 100. The generating unit 12 executes the generation request acquired by the acquiring unit 11 to a database 7 in which a model such as a large-scale language model for generating learning materials is pre-stored. The generating unit 12 may output text information D22 indicating the generated learning material to the evaluating unit 13 or the like.

[0042] <Evaluation Section 13> The evaluation unit 13 executes an evaluation request acquired or generated by each component of the learning material evaluation system 100. The evaluation unit 13 executes the evaluation request acquired by the acquisition unit 11 against a database 8 in which a model such as a large-scale language model for evaluating learning materials is pre-stored. The evaluation unit 13 evaluates, for example, text information D22 indicating the learning material generated by the generation unit 12. The evaluation unit 13 outputs evaluation information indicating the evaluation of the learning material to the correction unit 14.

[0043] <Revised part 14> The correction unit 14 generates corrected text information by correcting the text information D22, for example, based on the evaluation information evaluated by the evaluation unit 13 and the text information D22 generated by the generation unit 12. The correction unit 14 outputs the generated corrected text information to the storage unit 15.

[0044] <Storage section 15> The storage unit 15 stores, as needed, for example, information acquired or generated by each component of the user terminal 1 in a database stored in the storage unit 104. The storage unit 15 retrieves, as needed, various types of information stored in the database stored in the storage unit 104, for example.

[0045] <Server 2> The server 2 stores a database in which models such as large-scale language models are stored in advance. The server 2 is communicatively connected to the user terminal 1 via, for example, a communication network 9. The server 2 generates, stores, and transmits / receives information in response to a processing request received from, for example, the user terminal 1 via the communication network 9.

[0046] As shown in FIG. 2(b), the server 2 includes a housing 20, a CPU 201, a ROM 202, a RAM 203, a storage unit 204, and an I / F 205. The components 201 to 205 are connected via an internal bus 210. The server 2 may be a well-known electronic device such as a laptop PC or a desktop PC. Alternatively, the server 2 may be a cloud server such as Amazon Web Services (registered trademark) (AWS).

[0047] The CPU 201 controls the entire server 2. The ROM 202 stores the operation code of the CPU 201. The RAM 203 is a working area used when the CPU 201 is operating. The storage unit 204 stores various information such as backups of data stored in the ROM, databases, and learning target data. As the storage unit 204, for example, a data storage device such as an HDD or an SSD is used. Note that the server 2 may also have a GPU (not shown), for example. The I / F 205 is an interface for transmitting and receiving various information to and from the user terminal 1 as needed via the communication network 9.

[0048] <Communication Network 9> The communication network 9 is, for example, the Internet network to which the user terminal 1 and the server 2 are connected via a communication circuit. The communication network 9 may be configured as a so-called optical fiber communication network. Furthermore, the communication network 9 may be realized by known communication technologies such as a wired communication network or a wireless communication network.

[0049] Next, as a learning material evaluation method in this embodiment, an example of an operation of generating learning materials in the learning material evaluation system 100 will be described with reference to Figures 4 and 5. The learning material evaluation system 100 is executed via a learning material evaluation program installed in the user terminal 1 and the server 2, for example.

[0050] The operation of generating learning materials in the learning material evaluation system 100 includes, for example, a query acquisition step S11, a learning material generation step S12, and a learning material output step S13, as shown in Fig. 4. Note that the operation of generating learning materials in the learning material evaluation system 100 may involve performing the query acquisition step S11 multiple times, and may involve performing the learning material generation step S12 multiple times.

[0051] First, various types of information involved in the operation of generating learning materials in the learning material evaluation system 100 in this embodiment will be described. The various types of information handled by the learning material evaluation system 100 include, for example, a query D1 and generated data D71, as shown in FIG. 5(a). The learning material evaluation system 100 stores a database 7 in advance.

[0052] <Query D1> The query D1 is information including a processing request for the database 7 described below, which stores a generation model 71 such as a large-scale language model described below. The query D1 includes, for example, learning material generation request information D11 and generation theme information D12. The learning material generation request information D11 and the generation theme information D12 are, for example, linked to each other.

[0053] The query D1 is, for example, a prompt including a processing instruction for the generative model 71, and is composed of a tag specifying a role and a natural language specifying the instruction content. <system>You are an excellent Japanese history teacher. <user>Please create one difficult question for the History Aptitude Test in the following format. This is a string containing the combination "[Question][Choices][Answer][Explanation]." By executing query D1 containing this combination, it is possible to generate study materials containing questions, choice answers, answers, and explanations related to Japanese history, for example.

[0054] The instructions may also include a request to generate a problem set consisting of multiple problems, and may also include the title of the problem set, the structure of each chapter in the problem set, the goal or objective of each chapter in the problem set, keywords for each chapter in the problem set, the number of problems in each chapter, and the output format (such as HTML format).

[0055] <Learning material generation request information D11> The learning material generation request information D11 is information that identifies a request to generate learning material among processing requests. The learning material generation request information D11, for example, constitutes part of the character string of the query D1. For example, the learning material generation request information D11 corresponds to "Please create a question" in the above example prompt sentence.

[0056] <Generated theme information D12> The generation theme information D12 is information that identifies a theme related to a request to generate learning materials. The generation theme information D12 is information that identifies a theme of learning materials requested to be generated by, for example, the learning material generation request information D11. The generation theme information D12, for example, constitutes part of the character string of the query D1. The generation theme information D12 is composed of one or more words that identify, for example, the content of the learning materials to be generated, such as the subject, the name of the qualification or examination, the field, the era, the person, the question format, the answer format (presence or absence of multiple choice, presence or absence of explanation), the number of questions, the difficulty level, and the educational classification (e.g., primary education, secondary education, higher education, etc.) in the school system corresponding to the curriculum guidelines. The generation theme information D12 corresponds, for example, to the above-mentioned prompt example sentences: "History Proficiency Test," "Difficult," "The following format," "One," "[Question], "[Choices], "[Answer], [Explanation]."

[0057] The generation theme information D12 may also be, for example, one or more words that specify the final goal of learning. In this case, the generation theme information D12 may be a string of characters such as "acquire basic knowledge for taking a qualification exam," "cover common questions in qualification exams," or "deepen understanding of subjects that are difficult to study through repetitive study."

[0058] The generation theme information D12 may use, for example, one or more words input by the user U via the input unit 108. Furthermore, the learning material generation request information D11 may use, for example, a syntax pre-stored in the user terminal 1. That is, the learning material evaluation system 100 can acquire the query D1 and generate learning materials simply by the user U inputting one or more words. In this case, even a user U with little specialized knowledge about the query D1 can automatically generate learning materials. This can improve the workability of creating learning materials.

[0059] <Generated data D71> The generated data D71 indicates data generated based on the query D1. The generated data D71 includes one or more pieces of data indicating the content of the learning material, and includes, for example, learning material data D711.

[0060] The learning material data D711 indicates the content of learning materials related to the generation theme included in the generation theme information D12, and consists of a combination of questions and answers. The number of characters in the questions and answers in the learning material data D711 may be determined arbitrarily by the generative model 71. The learning material data D711 may also include an explanation for the answer. In this case, the explanation may be generated by the generative model 71 together with the answer.

[0061] An example of the learning material data D711 corresponding to the creation theme information D12 is as shown in Table 1.

[0062] [Table 1] As in Category 1, when the generated theme includes the word "history," the learning material data D711 may be a question-and-answer style learning material consisting of a combination of a question related to the generated theme "history," such as "Who carried out the Taiko land survey?", and the answer to that question, "Toyotomi Hideyoshi."

[0063] As in Category 2, when the generated theme includes the word "Azuchi-Momoyama period," i.e., is a sub-concept of "history," the learning material data D711 may be a question-and-answer style learning material consisting of a combination of a question similar to the question related to the generated theme "history," "Who carried out the Taiko land survey?", and the answer to that question, "Toyotomi Hideyoshi."

[0064] As in Category 3, when the generated theme includes the word "Toyotomi Hideyoshi," the learning material data D711 may be a question-and-answer type learning material consisting of a combination of the question "Who carried out the Taiko land survey?" and the answer "Toyotomi Hideyoshi," which is generated with the generated theme as the answer.

[0065] As in Category 4, when the generated theme includes the word "Junior High School Social Studies," the learning material data D711 may be a question-and-answer style learning material consisting of a combination of the question "Who carried out the Taiko land survey?" and the answer to that question "Toyotomi Hideyoshi," which is generated by referring to an external database such as a curriculum guideline.

[0066] As in Category 5, when the generated theme includes the words "Azuchi-Momoyama period" and "question and answer type question," the learning material data D711 may be a question and answer type learning material consisting of a combination of a question about the generated theme "Azuchi-Momoyama period," "Who carried out the Taiko land survey?", and the answer to that question, "Toyotomi Hideyoshi," which is generated based on the premise that the question format is a question and answer type question.

[0067] As in Category 6, when the generated theme includes the words "history" and "basic," the learning material data D711 may be a question-and-answer style learning material consisting of a combination of the question "Who carried out the Taiko land survey?" and the answer to that question, "Toyotomi Hideyoshi," which is generated by referring to an external database such as a curriculum guideline, as well as a database in which the generated theme is linked to difficulty levels such as basic / advanced, easy / difficult, etc.

[0068] When the generated theme includes the words "history" and "application," as in Category 7, the learning material data D711 may be a question-and-answer style learning material consisting of a combination of the question "What is the principle of peasant control established by the Taiko land survey called?" and the answer to that question "The principle of one land, one farmer," which is generated by referring to a database similar to that described in Category 6.

[0069] As per category 8, when the generated theme includes the words "Azuchi-Momoyama period" and "multiple choice question," the learning material data D711 may be a multiple choice learning material consisting of a combination of a question about the generated theme "Azuchi-Momoyama period" - "Who carried out the Taiko land survey? 1. Oda Nobunaga 2. Toyotomi Hideyoshi 3. Tokugawa Ieyasu" - and the answer to that question "2" - generated with the premise that the question format is a multiple choice question.

[0070] As per category 9, when the generated theme includes the words "Azuchi-Momoyama period" and "combination question," the learning material data D711 may be a combination-type learning material consisting of a question about the generated theme "Azuchi-Momoyama period," "Which did Toyotomi Hideyoshi carry out? 1. Taiko land survey 2. Sword hunt 3. Rakuichi Rakuza," and the answer to that question, "1 and 2," which is generated based on the premise that the question format is a combination question.

[0071] The learning material data D711 may also be learning material showing explanations from a textbook or the like. The learning material data D711 may be learning material including, for example, keyword information D23 indicating a word and related information D24 relating to the word. The learning material data D711 may also be learning material including, for example, keyword information D23 of "Azuchi-Momoyama period" and related information D24 of "Toyotomi Hideyoshi," "Taiko land survey," and the like relating to the "Azuchi-Momoyama period."

[0072] <Database 7> The database 7 is a collection of data stored in advance in the learning material evaluation system 100. The database 7 is stored in advance in, for example, the storage unit 204 of the server 2. The database 7 stores in advance a generation model 71 for generating learning materials.

[0073] <Generative Model 71> The generative model 71 is a type of natural language processing model used for automatically generating learning materials by the learning material evaluation system 100. The generative model 71 may be a known large-scale language model (LLM) that has been machine-learned in advance using a large amount of text data.

[0074] As the generative model 71, specifically, GPT-3, GPT-3.5, or GPT-4 related to "GPT (Generative Pre-trained Transformer) (registered trademark)", "BERT (Bidirectional Encoder Representations from Transformers)", "LaMDA (Language Model for Dialogue Applications)", "PaLM (Scaling Language Modeling with Pathways)", "LLaMA (Large Language Model Meta AI)", etc. may be used.

[0075] Next, the operation of generating learning materials in the learning material evaluation system 100 according to this embodiment will be described in detail.

[0076] <Advance preparation> Before causing the learning material evaluation system 100 to generate a learning material, the user U confirms that the database 7 in which the generative model 71 is stored is saved in the server 2.

[0077] <Query acquisition step S11> In a query acquisition step S11, the acquisition unit 11 accepts input from the user U via the input unit 108 and acquires a query D1 including learning material generation request information D11 and generation theme information D12.

[0078] <Learning material generation step S12> In the learning material generation step S12, the generation unit 12 executes the query D1 (generation request based on the learning material generation request information D11) acquired in the query acquisition step S11 against the database 7. When the acquisition unit 11 acquires the query D1, the generation unit 12 may automatically execute the acquired query D1, or may execute the query D1 according to input from the user U. The generation unit 12 may be executed separately before the learning material generation step S12 as a query execution step distinct from the learning material generation step S12.

[0079] When the query D1 is executed on the database 7, the generation unit 12 refers to the database 7 and generates generation data D71 including learning material data D711 based on the generation theme included in the generation theme information D12. In this case, the generation model 71 can automatically generate learning materials (learning material data D711) related to the generation theme (generation theme information D12). This can improve the efficiency of creating learning materials. Furthermore, as long as the query D1 can be acquired and executed, the user U can have the learning material evaluation system 100 generate the generation data D71 without having to prepare existing learning materials. This can improve the user U's workability in creating learning materials.

[0080] <Learning material output step S13> In the learning material output step S13, when the user terminal 1 receives the generated data D71 generated by the generation unit 12 via the communication network 9, the display unit 109 outputs the data to the user U via the display unit 109 or the like.

[0081] In addition, in the learning material output step S13, the generating unit 12 may output the generated learning material to the evaluating unit 13 for evaluation of the learning material.

[0082] After the above-described steps are performed, the operation of generating learning materials in the learning material evaluation system 100 in this embodiment is completed. Note that the learning material evaluation system 100 may perform the above-described steps repeatedly, for example.

[0083] (First embodiment: Modified example of operation of generating learning materials in the learning material evaluation system 100) <Query D1> The query D1 may further include table of contents information D13, for example, as shown in Fig. 5(b) The query D1 includes one or more pieces of table of contents information D13.

[0084] The table of contents information D13 is information that specifies the table of contents of the questions included in the learning material to be generated. Here, the table of contents specifies the number of questions, order, chapters (including chapters, sections, paragraphs, etc.), number of questions per chapter, title of each chapter, etc. of each learning material generated by the learning material evaluation system 100.

[0085] For example, one piece of table of contents information D13 is linked to one piece of generation theme information D12 to form one data set D2. At this time, each piece of generation data D71 generated based on a generation theme included in each piece of generation theme information D12 is assigned a table of contents according to the table of contents information D13 linked to the corresponding piece of generation theme information D12.

[0086] Specifically, when the table of contents information D13 includes information specifying the number of questions in the study material as "2," two pieces of generated data D71 are generated based on the generation theme included in the generation theme information D12, in accordance with the number of questions "2" specified by the table of contents information D13. Here, even if the query D1 includes only one data set D2, multiple pieces of generated data D71 may be generated in accordance with the number of questions specified by the table of contents information D13.

[0087] When the table of contents information D13 includes information specifying the orders "1" and "2" of the learning materials, two pieces of generation data D71 are generated based on the generation themes included in the generation theme information D12, arranged in the order specified by the table of contents information D13. For example, when the query D1 includes a dataset D2a in which table of contents information D13a specifying the order "1" is linked to generation theme information D12a specifying the generation theme "Theme A," and a dataset D2b in which table of contents information D13a specifying the order "2" is linked to generation theme information D12a specifying the generation theme "Theme B," generation data D71 is generated, including learning material data D711 consisting of a combination of questions and answers for "Question 1" related to "Theme A," and learning material data D712 consisting of a combination of questions and answers for "Question 2" related to "Theme B."

[0088] When the table of contents information D13 includes information specifying the chapter divisions of the learning material, such as "Chapter 1" and "Chapter 2," the generated data D71 is arranged according to the chapter divisions specified by the table of contents information D13 and generated based on the generation theme included in the generation theme information D12, similar to the case where the table of contents information D13 includes information specifying the order described above. When the table of contents information D13 includes information specifying the combinations of the chapter divisions and the number of questions of the learning material, such as "Chapter 1, 2 Questions" and "Chapter 2, 3 Questions," the generated data D71 is arranged according to the combinations of the chapter divisions and the number of questions specified by the table of contents information D13 and generated based on the generation theme included in the generation theme information D12. When the table of contents information D13 includes information specifying the titles of the chapter divisions of the learning material, the titles may be included as part of each learning material data D712 and output together with the questions.

[0089] Next, the operation of generating learning materials in the learning material evaluation system 100 according to this embodiment will be described in detail.

[0090] <Query acquisition step S11> In the query acquisition step S11, the acquisition unit 11 accepts input from a user U, for example, via the input unit 108, and acquires a query D1 including learning material generation request information D11 and multiple data sets D2 each consisting of a set of generation theme information D12 and table of contents information D13.

[0091] Instead of or in addition to the method of accepting input from the user U, the acquisition unit 11 may acquire a query D1 including a dataset D2 in which at least one of the generated theme information D12 and the table of contents information D13 is extracted from existing learning material data.

[0092] The acquisition unit 11 may acquire information extracted from existing learning material data as at least one of the generated theme information D12 and the table of contents information D13 using a known program or software capable of phrase extraction using, for example, TF-IDF (Term Frequency-Inverse Document Frequency), RAKE (Rapid Automatic Keyword Extractor), or YAKE (Yet Another Keyword Extractor). Specifically, all or part of the existing learning material data may be provided as context, and nouns and unique expressions (such as expressions with low frequency of occurrence) may be extracted from the context and acquired as the generated theme information D12. Furthermore, example combinations of chapters and number of questions in the existing learning material data, such as "Chapter 1, 2 Questions" and "Chapter 2, 3 Questions," may be extracted and acquired as the table of contents information D13. In this case, the user U's work of creating the query D1 can be omitted. This further improves the efficiency of creating learning materials. Furthermore, the generated learning materials are generated based on existing learning materials that have higher quality as learning materials than texts unrelated to the learning materials. This improves the quality of the learning materials. The number of characters in the questions and answers of the learning material data D711 generated based on these data sets D2 may be determined by the generative model 71 based on the number of characters in the context of the existing learning material data given as the extraction range.

[0093] <Learning material generation step S12> In the learning material generation step S12, when the query D1 acquired in the query acquisition step S11 is executed on the database 7, the generation unit 12 refers to the database 7 and generates, based on the generation theme information D12 and table of contents information D13 contained in the multiple data sets D2, generation data D71 including multiple learning material data D711, D712 corresponding to the table of contents information D13. In this case, the generation model 71 can automatically generate multiple learning materials (multiple learning material data D711, D712) related to the generation theme (generation theme information D12) according to the table of contents (table of contents information D13). This can improve the efficiency of creating a problem set including multiple learning materials.

[0094] According to this embodiment, the generation unit 12 generates learning materials based on a generation theme included in a query D1 acquired by referring to a database 7 in which a generative model 71 is pre-stored. This allows the generative model 71 to automatically generate learning materials related to the generation theme. This improves the efficiency of creating learning materials.

[0095] According to this embodiment, the acquisition unit 11 acquires a query D1 including a plurality of data sets D2, each of which includes a request for generating study materials, a generation theme, and a table of contents, and the generation unit 12 generates a plurality of study materials based on the acquired plurality of data sets D2. Therefore, the generation model 71 can automatically generate study materials related to the generation theme according to the table of contents. This improves the efficiency of creating a workbook including a plurality of study materials.

[0096] According to this embodiment, the acquisition unit 11 acquires a query D1 including a dataset D2 in which at least one of a generation theme and a table of contents is extracted from existing learning material data. This eliminates the need to create the query D1. This further improves the efficiency of creating learning materials. Furthermore, it is possible to generate learning materials based on existing learning materials. This improves the quality of the learning materials.

[0097] According to this embodiment, the generative model 71 has a learning material generation step S12 in which the generative model 71 generates learning materials based on a generation theme included in the query D1 acquired by referring to the database 7 stored in advance. This allows the generative model 71 to automatically generate learning materials related to the generation theme. This improves the efficiency of creating learning materials.

[0098] According to this embodiment, the computer executes a learning material generation step S12 in which the generative model 71 generates learning materials based on a generation theme included in the query D1 acquired by referring to the database 7 in which the generative model 71 is pre-stored. This allows the generative model 71 to automatically generate learning materials related to the generation theme. This improves the efficiency of creating learning materials.

[0099] Next, with reference to FIG. 6, an example of the operation of generating generation theme information of the learning material evaluation system 100 in this embodiment will be described.

[0100] First, we will explain various types of information involved in the operation of generating learning materials in the learning material evaluation system 100 in this embodiment. The various types of information handled by the learning material evaluation system 100 further include, for example, generation theme generation request information D14, classification theme information D15, and intermediate acquisition data D8, as shown in Fig. 6.

[0101] <Query D1> The query D1 may further include, for example, creation theme generation request information D14 and classification theme information D15. The creation theme generation request information D14 and the classification theme information D15 are, for example, linked to each other.

[0102] The query D1 may further include intermediately acquired data D8 that is newly acquired based on the creation theme generation request information D14 and the classification theme information D15.

[0103] <Generation theme generation request information D14> The creation theme generation request information D14 is information that specifies a request to create a creation theme for a learning material among the processing requests. Like the learning material generation request information D11, the creation theme generation request information D14 constitutes, for example, a part of the character string of the query D1.

[0104] <Classification Theme Information D15> The classification theme information D15 is information that identifies a theme related to a request for generating a generation theme for a learning material. The classification theme information D15 includes information similar to the generation theme information D12, and is information that identifies the theme of the learning material requested to be generated by, for example, the learning material generation request information D11. Like the generation theme information D12, the classification theme information D15 is, for example, part of the character string of the query D1 and is composed of one or more words.

[0105] The classification theme included in the classification theme information D15 may be, for example, a content corresponding to a superordinate concept of the generated theme, a content corresponding to a subordinate concept, or a content corresponding to an identical concept.

[0106] The classification theme information D15 may use, for example, one or more words input by the user U via the input unit 108. The generation theme generation request information D14 may use, for example, a syntax pre-stored in the user terminal 1. In other words, the learning material evaluation system 100 can acquire a query D1 simply by the user U inputting one or more words, and newly acquire intermediate acquisition data D8 as a generation theme for the learning material.

[0107] <Interim data D8> The intermediately acquired data D8 indicates data generated before the generated data D71, among data newly acquired based on the query D1. The intermediately acquired data D8 may also acquire existing web article texts and article URLs newly acquired based on the query D1 together with the data to be generated, or may acquire them instead of the data to be generated. At least a portion of the acquired article texts, article URLs, etc. may be used as part or all of the explanations included in the learning material data D711. The intermediately acquired data D8 includes generation theme information D81.

[0108] <Generated theme information D81> The generation theme information D81 is information that identifies a theme related to a request to generate a learning material, and is similar to the generation theme information D12. The generation theme information D81 is, for example, information that identifies a theme of the learning material requested to be generated by the learning material generation request information D11. The generation theme information D81, for example, constitutes part of the character string of the query D1 and is composed of one or more words.

[0109] The generation theme information D81 indicates the generation theme of the learning material related to the classification theme included in the classification theme information D15, and is composed of, for example, one or more words. An example of the generation theme information D81 corresponding to the classification theme information D15 is as shown in Table 2.

[0110] [Table 2] When the classification theme includes the word "Azuchi-Momoyama period" as in Class 10, the generation theme information D81 may include the generation theme "history" which corresponds to the superordinate concept of the classification theme.

[0111] When the classification theme includes the word "Azuchi-Momoyama period," as in Class 11, the generation theme information D81 may include a generation theme "Toyotomi Hideyoshi," which is a subordinate concept of the classification theme.

[0112] When the classification theme includes the word "Azuchi-Momoyama period" as in Class 12, the generation theme information D81 may include the generation theme "Azuchi-Momoyama period" which is equivalent to the same concept as the classification theme.

[0113] When the classification theme including the difficulty level includes the words "Azuchi-Momoyama period basics" as in Class 13, the generation theme information D81 may include a generation theme "Toyotomi Hideyoshi basics" that does not change the difficulty level part.

[0114] As in category 14, when the classification theme including the question format includes the words "Azuchi-Momoyama period multiple-choice questions," the generated theme information D81 may include a generated theme "Toyotomi Hideyoshi multiple-choice questions" that does not change the difficulty level portion.

[0115] <Database 7a> The database 7a is stored in advance in the server 2. The database 7a may be the same as the database 7, or may be a similar database different from the database 7. The database 7a stores a generative model 71a in advance.

[0116] <Generative Model 71a> The generative model 71a may be the same large-scale language model as the generative model 71, or may be a similar large-scale language model different from the generative model 71.

[0117] Next, the operation of generating learning materials in the learning material evaluation system 100 according to this embodiment will be described in detail.

[0118] <Query acquisition step S11> In a query acquisition step S11, the acquisition unit 11 acquires a query D1 including learning material generation request information D11, generation theme generation request information D14, and classification theme information D15.

[0119] <Learning material generation step S12> In the learning material generation step S12, the generation unit 12 executes the query D1 (generation request based on the generation theme generation request information D14) acquired in the query acquisition step S11 to the database 7a in which the generation model 71a is stored in advance.

[0120] When the query D1 is executed on the database 7a, the generation unit 12 refers to the database 7a and then newly acquires intermediately acquired data D8 including generation theme information D81 based on the classification theme included in the classification theme information D15. Here, the intermediately acquired data D8 may be information generated by the generation model 71a based on the classification theme, or may be information (including the above-mentioned existing article text on the web and article URLs) acquired as a result of the learning material evaluation system 100 performing a web search or an external database search using the information generated by the generation model 71a based on the classification theme as a theme.

[0121] Thereafter, the generation unit 12 refers to the database 7 in response to the query D1 (generation request based on the learning material generation request information D11) acquired in the query acquisition step S11, and generates generation data D71 including learning material data D711 based on the generation theme included in the newly acquired generation theme information D81. In this case, the generation model 71a can automatically generate a generation theme (generation theme information D81) related to the classification theme (classification theme information D15). This can further improve the efficiency of creating learning materials. In addition, since there is no need to select a generation theme for each learning material, even a user U with little specialized knowledge of learning materials can automatically generate learning materials. This can improve the workability of creating learning materials.

[0122] According to this embodiment, the generation unit 12 generates learning materials based on a newly acquired generation theme based on the classification theme after referring to the database 7a in which the generation model 71a is pre-stored. Therefore, the generation model 71a can automatically generate a generation theme related to the classification theme. This further improves the efficiency of creating learning materials. In addition, since there is no need to select a generation theme for each learning material, even users with little specialized knowledge of learning materials can automatically create learning materials. This improves the workability of creating learning materials.

[0123] Next, an example of the operation of generating related generation theme information of the learning material evaluation system 100 will be described with reference to FIG. 7 as a learning material evaluation method in this embodiment.

[0124] The operation of generating related generation theme information of the learning material evaluation system 100 further includes a query conversion step S14, as shown in Fig. 4, for example. The query conversion step S14 is executed after the query acquisition step S11. Note that the operation of generating related generation theme information of the learning material evaluation system 100 may perform the query conversion step S14 multiple times.

[0125] First, a description will be given of various types of information associated with the operation of generating related generation theme information in the learning material evaluation system 100 in this embodiment. The various types of information handled by the learning material evaluation system 100 further include related generation theme information D82, as shown in FIG.

[0126] <Interim data D8> The intermediately acquired data D8 includes related generated theme information D82.

[0127] <Related generated theme information D82> The related generation theme information D82 is information that specifies a theme related to a request to generate a learning material, and is the same information as the generation theme information D81. The related generation theme information D82 is composed of a character string, for example, like the generation theme information D12, and is composed of one or more words.

[0128] The generated theme information D81 differs from the generated theme information D81 in that the generated theme information D81 is newly generated based on the classified theme information D15 by referring to the generated model 71a, whereas the related generated theme information D82 is obtained from the related word database 72 by referring to the related word database 72 described below.

[0129] <Database 7b> The database 7b is stored in advance in the server 2. The database 7b may be the same as the database 7, or may be a similar database different from the database 7. The database 7b stores a related word database 72 in advance.

[0130] <Related Word Database 72> The related word database 72 is stored in advance in the database 7b. For example, as shown in FIG. 8, the related word database 72 stores reference generation theme information D721 and reference related generation theme information D722 that are linked to each other in advance. The related word database may be a database created in advance by the user U, or may use publicly known concept dictionary data such as "WordNet."

[0131] The reference generating theme information D721 is made up of one or more pieces of generating theme information. The reference generating theme information D721 may include information corresponding to the generating theme information D12. The reference related generating theme information D722 is made up of one or more pieces of related generating theme information. The reference related generating theme information D722 may include information corresponding to the related generating theme information D82.

[0132] The reference generation theme information D721 is linked to reference related generation theme information D722 that includes, for example, one or more related generation themes. The generation theme "X" included in the reference generation theme information D721 may be linked to related generation theme A "X1" and related generation theme B "X2" included in the reference related generation theme information D722. Specifically, the generation theme "Azuchi-Momoyama period" may be linked to related generation theme A "history" and related generation theme B "Toyotomi Hideyoshi."

[0133] Next, the operation of generating related generation theme information in the learning material evaluation system 100 according to this embodiment will be described in detail.

[0134] <Query conversion step S14> In the query conversion step S14, the generation unit 12 converts the query D1 by replacing the generation theme information D12 included in the query D1 acquired in the query acquisition step S11 with a piece of related generation theme information D82 acquired from the related word database 72 (database 7b) after referring to the related word database 72. That is, the query D1 executed on the database 7 includes the learning material generation request information D11 and the related generation theme information D82, but does not include the generation theme information D12 replaced with the related generation theme information D82.

[0135] Note that the generating unit 12 may arbitrarily select one related generation theme to replace the generation theme information D12 from one or more related generation themes included in the reference related generation theme information D722 that constitutes the related word database 72. In the example of Fig. 8, the generating unit 12 may acquire, as related generation theme information D82, the generation theme information D12 including the generation theme "X" by arbitrarily selecting one of the generation themes "X1" and "X2" included in the reference related generation theme information D722 linked to the reference generation theme information D721 corresponding to the generation theme "X," and replace the generation theme information D12 with the generation theme information D82.

[0136] Furthermore, the generation unit 12 may select one related generation theme based on the track record, such as the number of times each related generation theme has been retrieved from the related word database 72. For example, one related generation theme "X1" that has been retrieved many times from the related word database 72 may be selected as a related generation theme with an easy level of difficulty, and one related generation theme "X2" that has been retrieved few times from the related word database 72 may be selected as a related generation theme with a difficult level of difficulty.

[0137] <Learning material generation step S12> In the learning material generation step S12, the generation unit 12 refers to the database 7 in response to the query D1 (generation request based on the learning material generation request information D11) acquired in the query acquisition step S11, and generates generation data D71 including learning material data D711 based on the related generation theme included in the query D1 converted in the query conversion step S14. In this case, the generation model 71 can automatically generate learning materials with different contents from the same generation theme (generation theme information D12). This improves the convenience of the created learning materials.

[0138] According to this embodiment, the generation theme (generation theme information D12) included in the acquired query D1 is converted into one related generation theme (related generation theme information D82). Therefore, the generation model 71 can automatically generate learning materials with different contents from the same generation theme. This improves the convenience of the created learning materials.

[0139] 9 to 11, an example of the operation of evaluating a learning material by the learning material evaluation system 100 will be described as a learning material evaluation method in this embodiment. The learning material evaluation system 100 is executed via a learning material evaluation program installed in the user terminal 1 and the server 2, for example.

[0140] The operation of evaluating learning materials by the learning material evaluation system 100 includes, for example, a learning material acquisition step S21, a learning material extraction step S22, a related information search step S23, a learning material evaluation step S24, a text information correction step S25, and an evaluation information storage step S26, as shown in Figure 9.

[0141] First, various types of information associated with the operation of evaluating learning materials by the learning material evaluation system 100 in this embodiment will be described. The various types of information handled by the learning material evaluation system 100 include, for example, a query D2 and generated data D72, as shown in Fig. 10(a). The learning material evaluation system 100 stores a database 8 in advance.

[0142] <Query D2> The query D2 is information including a processing request for a database 8 described later in which an evaluation model 81, such as a large-scale language model described later, is stored. The query D2 includes, for example, evaluation instruction information D21 and text information D22. The evaluation instruction information D21 and the text information D22 are, for example, linked to each other.

[0143] Query D2 is, for example, a prompt including a processing request for evaluation model 81. Query D2 is composed of, for example, natural language specifying the instruction content. By executing query D2, it is possible to evaluate learning materials including questions, options, answers, and explanations related to Japanese history, for example.

[0144] <Evaluation Instruction Information D21> The evaluation instruction information D21 is information that identifies a request for evaluation of learning materials among processing requests. The evaluation instruction information D21 constitutes, for example, a part of the character string of the query D2. The evaluation instruction information D21 includes, for example, "Please evaluate the learning materials."

[0145] <Text information D22> The text information D22 is information indicating the study material. The text information D22 refers to a test that tests the knowledge of the respondent user, including either questions or answers, and includes question formats such as essay questions, multiple-choice questions, and matching questions. The text information D22 may also include one or more words that identify the content of the study material to be evaluated, such as the name of a subject, qualification or examination, field, era, person, question format, answer format (presence or absence of multiple choice, presence or absence of explanation, etc.), number of questions, difficulty level, and educational classification (e.g., primary education, secondary education, higher education, etc.) in the school system corresponding to the curriculum guidelines. The text information D22 may also include keyword information D23 indicating words included in the study material and related information D24 related to the keyword. The text information D22 includes, for example, keyword information D23 such as "Toyotomi Hideyoshi" and related information D24 such as "the Taiko conducted a land survey."

[0146] The text information D22 may use, for example, one or more learning materials input by the user U via the input unit 108. Alternatively, the text information D22 may use, for example, learning materials stored in advance in the user terminal 1. Alternatively, the text information D22 may use learning materials output from the generation unit 12 in the learning material output step S13.

[0147] The keyword information D23 is, for example, information indicating a word, but is not limited to a word and may be information indicating any term such as a person, a place name, or an event. The related information D24 is information related to the term indicated by the keyword information D23. The related information D24 may be, for example, an explanation and description of the term indicated by the keyword information D23, or a question and answer regarding the term indicated by the keyword information D23.

[0148] <Generated data D72> The generated data D72 indicates data generated based on the query D2. The generated data D72 includes one or more pieces of evaluation information indicating the evaluation of the learning material, such as learning material evaluation information D731.

[0149] <Learning Materials Evaluation Information D731> The learning material evaluation information D731 includes one or more of evaluation information indicating the evaluation of the learning material, grounds information indicating the grounds for the evaluation of the learning material, and corrected text information obtained by correcting the text information D22. Note that the number of characters in the learning material evaluation information D731 may be determined arbitrarily by the evaluation model 81.

[0150] The evaluation information is information indicating an evaluation of the learning material. The evaluation information may be, for example, information indicating the validity of the text written in the learning material. The evaluation information may be, for example, information indicating errors or corrections in the text written in the learning material.

[0151] The basis information is information that indicates the basis for the evaluation of the learning material. The basis information may be information such as information contained in publications, specifications, guidelines, papers, patent documents, etc.

[0152] <Database 8> The database 8 is a collection of data stored in advance in the learning material evaluation system 100. The database 8 is stored in advance in, for example, the storage unit 204 of the server 2. The database 8 stores an evaluation model 81 for evaluating learning materials in advance.

[0153] <Evaluation Model 81> The evaluation model 81 is a type of natural language processing model used for automatically evaluating learning materials by the learning material evaluation system 100. The evaluation model 81 may be a known large-scale language model that has been machine-learned in advance using a large amount of text data.

[0154] As the evaluation model 81, specifically, GPT-3, GPT-3.5 or GPT-4 related to "GPT (Generative Pre-trained Transformer) (registered trademark)", "BERT (Bidirectional Encoder Representations from Transformers)", "LaMDA (Language Model for Dialogue Applications)", "PaLM (Scaling Language Modeling with Pathways)", "LLaMA (Large Language Model Meta AI)", etc. may be used.

[0155] The evaluation model 81 may also be a base model. The base model may be a natural language model. The base model may be a model generated by unsupervised learning. The base model may also be a generative AI. The natural language model may be an interactive, so-called chat-type or conversation-type model that alternates between receiving instruction sentences and generating response sentences.

[0156] A large-scale language model is a deep learning model that pre-trains a language model, which models human spoken language based on its occurrence probability, from a huge amount of data. In other words, a large-scale language model is a natural language processing model trained using a large amount of text data, which inputs sentences as questions or instructions and outputs sentences as answers. When a large-scale language model is applied to a question-and-answer system, when a question is input into the large-scale language model, the LLM outputs an answer.

[0157] When the large-scale language model receives text data (prompt), it uses the large-scale language model to statistically estimate the probability of generating the next word from the sentence included in the received prompt and outputs an answer based on the estimation result. As the large-scale language model, for example, known technologies described on internet sites such as "https: / / chatgpt-lab.com / n / n418d3aa56f0b" and "https: / / agirobots.com / chatgpt-mechanism-and-problem / " can be adopted. Furthermore, for example, GPT-4 provided by OpenAI, Inc. of the United States may also be used as the large-scale language model.

[0158] The evaluation model 81 may be a model different from the generative model 71. For example, the evaluation model 81 may be a large-scale language model trained using training data different from that of the generative model 71.

[0159] The evaluation model 81 may be generated using machine learning based on a neural network, for example. The evaluation model 81 is, for example, an AI neural network. The evaluation model 81 may be trained using machine learning based on a neural network such as a convolution neural network (CNN), or any other model may be used. The evaluation model 81 may be generated using, for example, retrieval-augmented generation (RAG), sequence-to-sequence (seq2seq) linear discriminant analysis, support vector machines, k-nearest neighbors, random forests, deep learning, or the like.

[0160] In this case, the evaluation model 81 stores associations having degrees of association between the keyword information D23 and related information D24, which are input data, and the evaluation information, which is output data, as shown in FIG. 11 . Alternatively, text information D22 may be used as input data. In this case, morphemes, words, or the like included in the text information D22 may be used as input data. The degree of association indicates the degree of association between the input data and the output data. For example, the higher the degree of association, the stronger the connection between the data. The degree of association may be expressed, for example, as a percentage, or in three or more levels, or may be expressed as two values ​​or two levels. The keyword information D23, related information D24, and evaluation information used in learning the degree of association are, for example, the keyword information D23, related information D24, and evaluation information used in advance for learning data. However, the present invention is not limited to this, and information acquired at any timing may be used.

[0161] For example, the association is constructed based on the degree of connection between multiple input data pairs and multiple output data. The association is appropriately updated during the machine learning process and indicates a classifier using a function optimized, for example, based on multiple input data and multiple output data. Note that the association may have multiple association degrees indicating the degree of connection between each piece of data. For example, when the database is constructed using a neural network, the association degree may correspond to a weight variable. For example, as shown in FIG. 11, the association may indicate the degree of connection between multiple input data and multiple output data. In this case, by using the association, the degree of relationship between each piece of input data, "Keyword Information A and Related Information A" to "Keyword Information C and Related Information C" in FIG. 11, and multiple output data, "Evaluation Information A" to "Evaluation Information C," can be linked and stored. Therefore, for example, multiple input data can be linked to one output data via the association. This enables multifaceted selection of output data for the input data. Furthermore, the input data and output data are not limited to these, and any type of information may be used. The input data may also be, for example, image information included in the text information D22.

[0162] The correlation has, for example, multiple correlations that link each input data with each output data. The correlation is expressed in three or more levels, such as a percentage, a 10-point scale, or a 5-point scale, and is expressed, for example, by line characteristics (such as thickness). For example, "keyword information A and related information A" included in the input data shows a correlation AA of "73%" with "rating information A" included in the output data, and a correlation AB of "12%" with "rating information B" included in the output data. In other words, the "correlation" indicates the degree of connection between each piece of data; for example, the higher the correlation, the stronger the connection between each piece of data.

[0163] Three or more levels of correlation as shown in Fig. 11 are acquired in advance. In other words, past data sets are accumulated to determine which of the input data and output data was adopted and evaluated when determining the actual solution, and these are analyzed to create the correlation shown in Fig. 11.

[0164] For example, suppose that in the past, "rating information B" was judged to be the most relevant and evaluated for input data of "keyword information B and related information B." By collecting and analyzing such data sets, the correlation between input data and output data becomes stronger.

[0165] This analysis may be performed by artificial intelligence. In such a case, if there are many cases where "rating information B" is estimated for input data such as "keyword information B and related information B," the degree of association between this "keyword information B and related information B" and "rating information B" is set higher.

[0166] This correlation may be formed by a node of a neural network in artificial intelligence. That is, the weighting coefficient for the output of the node of this neural network corresponds to the correlation. Furthermore, it is not limited to a neural network, and may be formed by any decision-making factor that constitutes artificial intelligence.

[0167] Furthermore, the evaluation model 81 may be configured to perform machine learning by providing at least one or more hidden layers between the input data and the output data. The above-mentioned correlation is set in either or both of the input data and the hidden layer data, and this serves as a weighting for each data, and output selection is based on this. Then, when this correlation exceeds a certain threshold, that output may be selected.

[0168] Such correlations are what is referred to in artificial intelligence as training data. Such training data is trained in advance, and new keyword information D23 and related information D24 are actually input into the evaluation model 81 to output evaluation information. When outputting, the correlations shown in FIG. 11 , for example, are referenced. For example, if the newly acquired keyword information D23 and related information D24 are identical to or similar to "keyword information A and related information A," they are associated with "evaluation information A" at a correlation level of AA of 73% and with "evaluation information B" at a correlation level of AB of 12%. In this case, "keyword information A and related information A," which has the highest correlation, is selected as the optimal solution. However, selecting the one with the highest correlation as the optimal solution is not essential; "evaluation information B," which has a low correlation but is recognized as having correlation, may be selected as the optimal solution. Of course, other output solutions without connected arrows may also be selected, and any other priority may be selected based on the correlation.

[0169] By referring to such degrees of association, it is possible to quantitatively select output data suitable for the input data, not only when the keyword information D23 and related information D24 are identical to or similar to the input data, but also when they are dissimilar.

[0170] Next, the operation of evaluating a learning material by the learning material evaluation system 100 in this embodiment will be described in detail.

[0171] <Learning material acquisition step S21> In the learning material acquisition step S21, the acquisition unit 11 accepts input from the user U, for example, via the input unit 108, and acquires a query D2 including multiple data sets D3, each of which is a set of evaluation instruction information D21 and text information D22.

[0172] Instead of or in addition to the method of receiving an input from the user U, the acquisition unit 11 may acquire a query D2 including a data set D3 in which text information D22 is extracted from existing learning material data.

[0173] The acquiring unit 11 may also acquire text information D22 including the learning material generated in the learning material generating step S12 and output in the learning material outputting step S13. In this case, the acquiring unit 11 may further acquire evaluation instruction information D21 that is automatically generated when the learning material is output in the learning material outputting step S13. This makes it possible to automatically evaluate the generated learning material.

[0174] <Learning material extraction step S22> In the learning material extraction step S22, when the query D2 acquired in the learning material acquisition step S21 is executed, the evaluation unit 13 may extract keyword information D23 and related information D24 from the text information D22. In this case, for example, the text information D22 is subjected to morphological analysis to extract words contained in the text information D22. In this case, the extracted words are, for example, nouns indicating people, place names, events, etc., but are not limited thereto. For example, important or frequently occurring words within the text information D22 may be extracted. The evaluation unit 13 extracts keyword information D23 containing the extracted words and related information D24 related to the words. The related information D24 is, for example, text-format information indicating an explanation of the word. The related information D24 is, for example, information on sentences highly relevant to the word.

[0175] <Related information search step S23> In the related information search step S23, the evaluation unit 13 refers to a database that stores keyword information D23 and related information D24 in association with each other, and searches for related information D24 stored in the database based on the keyword information D23 extracted in the learning material extraction step S22. In this case, the evaluation unit 13 may search a database stored in the server 2, for example, via the communication network 9. In this case, the evaluation unit 13 searches, for example, a database stored in the server 2 based on the keyword information D23 extracted in the learning material extraction step S22, to search for keyword information D23 stored in the database that is similar to the keyword information D23 extracted in the learning material extraction step S22, and extracts related information D24 stored in association with the searched keyword information D23. The evaluation unit 13 may also search, for example, a database based on the keyword information D23 extracted in the learning material extraction step S22, to retrieve two or more pieces of keyword information D23 stored in the database whose similarity to the keyword information D23 extracted in the learning material extraction step S22 is higher than a reference value, and acquire two or more pieces of related information D24 stored in association with the retrieved two or more pieces of keyword information D23. In this case, the evaluation unit 13 may also acquire similarity information indicating the similarity between the keyword information D23 extracted in the learning material extraction step S22 and the keyword information D23 stored in the database. The evaluation unit 13 may also search, for example, a database stored in the server 2 based on the keyword information D23 extracted in the learning material extraction step S22, to retrieve the top K pieces of keyword information D23 with the highest similarity to the keyword information D23 extracted in the learning material extraction step S22, and extract K pieces of related information D24 stored in association with the retrieved K pieces of keyword information D23. Furthermore, the database searched by the evaluation unit 13 only needs to contain the keyword information D23 and the related information D24, and for example, the related information D24 may be extracted by performing morphological analysis on text containing the keyword information D23.

[0176] In addition, in the related information search step S23, the evaluation unit 13 may use, for example, a maximum inner-product search (MIPS) to search for keyword information D23 stored in the database that is similar to the keyword information D23 extracted in the learning material extraction step S22, and extract related information D24 that is stored in association with the searched keyword information D23.

[0177] Furthermore, in the related information search step S23, the evaluation unit 13 may perform the search by referring to a database stored in a communicable memory or the like, without using a public communication network, and is not limited to a database stored in the server 2. Furthermore, in the related information search step S23, if basis information is linked to the searched related information D24, the evaluation unit 13 may acquire the linked basis information.

[0178] Furthermore, the evaluation unit 13 may proceed to the next learning material evaluation step S24 without performing the learning material extraction step S22 and the related information search step S23.

[0179] <Learning material evaluation step S24> In the learning material evaluation step S24, the evaluation unit 13 inputs, for example, the text information D22 generated in the learning material generation step S12 into an evaluation model, outputs evaluation information, and evaluates the learning material. The evaluation unit 13 executes, for example, a query D2 on the database 8. When the acquisition unit 11 acquires the query D2, the evaluation unit 13 may automatically execute the acquired query D2, or may execute the query D2 in accordance with an input from the user U.

[0180] When the query D2 is executed on the database 8, the evaluation unit 13 refers to the database 8 and generates generated data D72 including learning material evaluation information D731 based on the text information D22. In this case, the learning material can be automatically evaluated by the evaluation model 81. This can improve the efficiency of creating learning materials.

[0181] Furthermore, when the query D2 includes multiple data sets D3 each including keyword information D23 and related information D24, as shown in Figure 10(b), when the query D2 is executed against the database 8, the evaluation unit 13 generates generated data D72 including learning material evaluation information D731 based on the keyword information D23 and related information D24 after referring to the database 8.

[0182] Furthermore, in the learning material evaluation step S24, the evaluation unit 13 may use a simulator stored in the database 8. In this case, for example, a natural science simulator stored in the database 8 may be used.

[0183] A natural science simulator is a tool that performs simulations related to the above-mentioned natural sciences based on input parameters. Any existing tool may be used as this natural science simulator. Examples of natural science simulators include FEM simulators that analyze stress fields and strains based on the finite element method, chemical reaction simulators, chemical engineering simulators including plants, electrical circuit simulators, fluid dynamics simulators, quantum mechanics simulators, and other tools that can simulate physical and chemical phenomena, as well as any natural science.

[0184] The natural science simulator performs a natural science simulation based on parameters corresponding to the input text information D22 and a known simulation method. The natural science simulator may then determine whether the content of the text information D22 is valid. For example, if the content violates the law of gravitation, which is a law of nature, the simulator may add a note to the evaluation information.

[0185] In addition to outputting the simulation results themselves from the natural science simulator as evaluation information, the large-scale language model may perform a new solution search based on the simulation results and output evaluation information. In such a case, the evaluation information is output by inputting a prompt that reflects the simulation results from the natural science simulator into the large-scale language model.

[0186] Furthermore, in the learning material evaluation step S24, the evaluation unit 13 may input, for example, the text information D22 generated in the learning material generation step S12 to an evaluation model that has been trained on associations as shown in Fig. 11, and output evaluation information. In this case, the evaluation model may be one that has been trained using training data that takes the text information D22 as input and outputs evaluation information including grounds information that indicates the grounds for the evaluation.

[0187] In addition, in the learning material evaluation step S24, the evaluation unit 13 may input, for example, the keyword information D23 and related information D24 extracted in the learning material extraction step S22 into an evaluation model that has learned the associations shown in Figure 11, and output evaluation information.

[0188] Furthermore, in the learning material evaluation step S24, if the evaluation unit 13 is unable to obtain evaluation information with sufficient accuracy using the above-mentioned method, it may output the evaluation information by further using the related information D24 retrieved in the related information search step S23. In such a case, the evaluation unit 13 inputs the keyword information D23 extracted in the learning material extraction step S22 and the related information D24 into the evaluation model, and outputs the evaluation information based on the related information D24 retrieved in the related information search step S23.

[0189] For example, in the learning material evaluation step S24, the evaluation unit 13 may input the keyword information D23 extracted in the learning material extraction step S22, the related information D24, and the related information D24 retrieved in the related information search step S23 into an evaluation model and output evaluation information. In such a case, the evaluation model may be trained using training data that receives the keyword information D23 and two pieces of related information D24 or the difference between the two pieces of related information D24 as input and outputs evaluation information. Also, in the learning material evaluation step S24, if the evaluation unit 13 obtains evidence information in the related information search step S23, it may output evaluation information including the evidence information. This makes it possible to evaluate the learning material based on the difference between the related information D24 extracted in the learning material extraction step S22 and the related information D24 retrieved in the related information search step S23, thereby enabling more accurate evaluation.

[0190] Furthermore, when a large-scale language model is used as the evaluation model, a generation probability for generating the next word from the sentence included in the received prompt may be set based on the related information D24 retrieved in the related information retrieval step S23. In such a case, in the learning material evaluation step S24, the evaluation unit 13 may set a high generation probability for the evaluation model of the word or a similar word included in the related information D24 retrieved in the related information retrieval step S23.

[0191] <Text information correction step S25> In the text information correction step S25, the correction unit 14 corrects the text information D22 based on the text information D22 acquired in the learning material acquisition step S21 and the evaluation information output in the learning material evaluation step S24, and outputs the corrected text information. In this case, the correction unit 14 may, for example, refer to a correction model that inputs the text information D22 and the evaluation information and outputs the corrected text information, and output the corrected text information based on the text information D22 acquired in the learning material acquisition step S21 and the evaluation information output in the learning material evaluation step S24. Alternatively, the correction unit 14 may generate the corrected text information by reflecting the correction indicated by the evaluation information in the text information D22. This enables the generated learning material to be automatically evaluated and corrected. The correction unit 14 stores the generated corrected text information in the storage unit 15.

[0192] <Evaluation information storage step S26> In evaluation information storage step S26, the storage unit 15 stores the evaluation information output in the learning material evaluation step S24 and the corrected text information D22 generated in the text information correction step S25 in the database 7. This allows the generation unit 12 to refer to the database 7 in the learning material generation step S12, thereby generating learning materials that reflect the evaluation information.

[0193] For example, in the learning material generation step S12, the generation unit 12 generates text information D22 based on the generation theme of the learning material by referring to a generation model trained using learning data including the evaluation information output in the learning material evaluation step S24 and the corrected text information generated in the text information correction step S25. By having the generation model learn the evaluation information output in the learning material evaluation step S24 and the corrected text information generated in the text information correction step S25, the probability of generating matches between words included in the evaluation information and the corrected text information, or similar words, increases, making it possible to generate new learning material that reflects the corrections to the learning material.

[0194] After performing the above steps, the operation of evaluating the learning material of the learning material evaluation system 100 in this embodiment is completed. The learning material evaluation system 100 in this embodiment inputs the text information D22 into the base model and outputs evaluation information. This makes it possible to automatically evaluate the learning material.

[0195] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0196] 100 Learning Material Evaluation System 1. User terminal 10. Cabinet 101 CPU 102 ROM 103 RAM 104 Preservation Department 105~107 I / F 108 Input section 109 Display section 110 Internal Bus 11 Acquisition Department 12 Generation part 13 Evaluation Section 14 Correction section 15 Storage section 2 Server 20 Case 201 CPU 202 ROM 203 RAM 204 Preservation Department 205 Interface 210 Internal Bus 7 Database 71 Generative Model 72 Related Words Database 8 Database 81 Evaluation Model 9. Communication Network U User S11 Query acquisition step S12 Learning material generation step S13 Learning material output step S14 Query transformation step S21 Learning material acquisition step S22 Learning material extraction step S23 Related information search step S24 Learning Material Evaluation Steps S25 Text information correction step S26: Evaluation information storage step D1 Query D11 Learning material generation request information D12 Generating Theme Information D13 Table of Contents Information D14 Theme Generation Request Information D15 Classification Theme Information D2 queries D21 Evaluation Instruction Information D22 Text Information D23 Keyword Information D24 Related Information D71 Generated Data D711, D712 learning material data D721 Reference generated theme information D722 Related generated theme information for reference D72 Generated Data D731 Learning Materials Evaluation Information D8 Interim data D81 Generated Theme Information D82 Related generated theme information< / user> < / system>

Claims

1. An acquisition means for acquiring text information indicating learning materials; and an output means for inputting the text information acquired by the acquisition means into a base model and outputting evaluation information indicating an evaluation of the learning material. A learning material evaluation system that features:

2. The system further includes an extraction unit that extracts keyword information indicating a word and related information relating to the word based on the text information acquired by the acquisition unit, The output means inputs the keyword information and related information extracted by the extraction means into a base model and outputs the evaluation information.

2. The learning material evaluation system according to claim 1,

3. The output means inputs the keyword information and related information extracted by the extraction means into the base model trained using training data that receives keyword information and related information as input and outputs evaluation information, and outputs the evaluation information.

3. The learning material evaluation system according to claim 2, wherein:

4. The system further comprises a search unit that refers to a database that stores keyword information and related information in association with each other, and searches for related information stored in the database based on the keyword information extracted by the extraction unit; The output means inputs the keyword information and related information extracted by the extraction means into the base model, and outputs the evaluation information based on the related information searched by the search means.

4. The learning material evaluation system according to claim 3, wherein:

5. The output means inputs the text information acquired by the acquisition means into a base model, and outputs the evaluation information including grounds information indicating the grounds for the evaluation.

2. The learning material evaluation system according to claim 1,

6. The output means inputs the text information acquired by the acquisition means into a large language model and outputs the evaluation information.

2. The learning material evaluation system according to claim 1,

7. The learning material may further include a correction means for generating corrected text information by correcting the learning material based on the text information acquired by the acquisition means and the evaluation information output by the output means.

2. The learning material evaluation system according to claim 1,

8. The system further comprises a generation unit that generates text information based on a generation theme of the learning material by referring to a generation model trained using learning data including the corrected text information corrected by the correction unit and the evaluation information output by the output unit.

8. The learning material evaluation system according to claim 7,

9. an acquisition step of acquiring text information indicating learning materials; and an output step of inputting the text information acquired by the acquisition step into a base model and outputting evaluation information indicating an evaluation of the learning material. A learning material evaluation method characterized by:

10. an acquisition step of acquiring text information indicating learning materials; and an output step of inputting the text information acquired by the acquisition step into a base model and outputting evaluation information indicating an evaluation of the learning material. A learning materials evaluation program that features:

Images