Programs, information processing devices, methods, and systems

Abstract

This approach separates learners' writing ability from their language proficiency, reduces unfairly high scores based on a string of difficult expressions, and objectively assesses language proficiency. [Solution] A program that causes a computer processor to execute the following steps: acquire a first text input by a learner; analyze the content of the first text and generate a second text in which grammatical errors contained in the first text have been corrected, and a third text in which the naturalness of expression in the first text has been improved; calculate a first index, which is the coverage rate of items belonging to the item group that appear in the third text, for a group of language items defined for each predetermined language proficiency level; normalize the first index for each level according to the text length of the third text and calculate a second index; and determine an upper limit level indicating the upper limit of the learner's language proficiency based on the second index for each level.

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing apparatus, a method, and a system.

Background Art

[0002] General grammar checkers and proofreading tools present correct data for the original text. Further, Patent Document 1 discloses a technique for associating sentences between an English composition input by a user and a proofread text, determining and calculating the vocabulary level of words included in the English composition, and feeding back an evaluation analysis result together with the Levenshtein distance or the like between the English composition and the proofread text.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional techniques as described above, the writing ability such as the logical structure and expressiveness of a composition and the pure language proficiency are often confused, and it has been difficult to evaluate the appropriate proficiency. Further, it has been difficult to prevent evaluation hacking in which a high evaluation is obtained simply by unnaturally listing difficult words and grammar.

[0005] An object of the present disclosure is to separate the writing ability and language proficiency of a learner, reduce inappropriate high evaluation due to listing of difficult expressions, and objectively determine the language proficiency.

Means for Solving the Problems

[0006] To solve the aforementioned problems, a program according to one aspect of the present disclosure is a program for operating a computer equipped with a processor, which causes the processor to perform the following steps: acquire a first text which is a sentence input by a learner; analyze the content of the acquired first text and generate a second text which corrects grammatical errors contained in the first text and a third text which improves the naturalness of the expression of the first text; calculate a first index which is the coverage rate of appearing items belonging to the item group that appear in the third text for a group of items which is a set of language items defined for each predetermined language proficiency level; normalize the calculated first index for each level according to the text length of the third text for each level to calculate a second index; and determine an upper limit level which indicates the upper limit of the learner's language proficiency level based on the calculated second index for each level. [Effects of the Invention]

[0007] According to this disclosure, it is possible to separate learners' writing ability from their language proficiency, reduce unfairly high scores due to a string of difficult expressions, and objectively determine language proficiency. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the overall system configuration. [Figure 2] This is a block diagram showing the functional configuration of terminal devices. [Figure 3] This is a block diagram showing the functional section realized by the control unit of the terminal device. [Figure 4] This is a block diagram showing the functional configuration of the server. [Figure 5] This is a block diagram showing the functional parts implemented by the server's control unit. [Figure 6] This figure shows an example of the data structure of a learner management table. [Figure 7] This figure shows an example of the data structure of the evaluation history table. [Figure 8]This figure shows an example of the data structure of a text analysis table. [Figure 9] This figure shows an example of the data structure of a classification master table. [Figure 10] This figure shows an example of the data structure of the evaluation settings master table. [Figure 11] This flowchart shows an example of server operation. [Figure 12] This is a schematic diagram showing an example of a results display screen. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. A processor may be expressed, for example, as processing circuitry. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be other types of processors such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core. Also, at least one processor may be a general-purpose processor or a purpose-specific processor.

[0011] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit)).

[0012] In the following description, the expression such as "xxx table" may be used to describe information from which an output can be obtained for an input. However, this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx table" can be referred to as "xxx information".

[0013] In the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0014] The program may be pre-installed in the information processing device shown below. For example, it may be on a recording medium (e.g., non-temporary) readable by the information processing device, and this program may be installed in the information processing device. Also, the program may be transmitted from a program distribution server to the information processing device and installed. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0015] In the following description, identification information for various objects is used. However, the identification information may be any information indicating a predetermined object, and the specific data is not limited to the embodiments. The identification information may be an identification number or an identifier including letters and symbols.

[0016] 〔Overview〕 The system according to this embodiment objectively and quantitatively evaluates the grammar accuracy and expression fluency of compositions of language learners including Japanese learners, distinguishing between them. Specifically, the system according to this embodiment generates two pieces of corrected data, namely, a second text which is the minimum necessary corrected text with only grammar errors corrected for the learner's first text, and a third text which is a fluent corrected text sublimated into a more natural expression than the second text.

[0017] The evaluation is performed in two stages. As the first stage, the system according to this embodiment determines the upper limit stage that the learner can reach from the coverage rate and density of the vocabulary and grammar appearing in the third text. As the second stage, the system according to this embodiment classifies the evaluation elements into unnecessary for correction (correct use), grammatical correction (misuse), and improvement of fluency (insufficiency) from the differences among the first text, the second text, and the third text, and determines the attainment level of each language level.

[0018] Furthermore, according to the system according to this embodiment, by combining a hacking prevention mechanism by listing difficult words and a proficiency exemption mechanism for preventing underestimation due to insufficient sample size peculiar to advanced learners, a refined and objective determination of language proficiency can be realized.

[0019] 〔Embodiment〕 <Overall Configuration of the System> FIG. 1 is a diagram showing an example of the overall configuration of a system 1 according to this embodiment. The system 1 is an information processing system for objectively and precisely determining the writing ability of a learner, particularly the language proficiency. As shown in FIG. 1, the system 1 includes a terminal device 10 used by the learner and a server 20 that executes various processes related to writing ability evaluation. The terminal device 10 and the server 20 are communicably connected to each other via a network 80. The system 1 provides a function of analyzing the text created by the learner from various angles and evaluating the language ability step by step.

[0020] The terminal device 10 is an information processing device for the learner to input a composition to be evaluated and confirm the evaluation result transmitted from the server 20. The terminal device 10 is, for example, a personal computer, a smartphone, a tablet terminal, or other portable information terminals. The terminal device 10 uses the functions of the system 1 via a dedicated application program or a web browser.

[0021] Server 20 is an information processing device that performs a series of processes to evaluate language proficiency based on the learner's composition transmitted from the terminal device 10. Server 20 is, for example, a server computer composed of one or more computers. Server 20 may be physically implemented in a single enclosure. Server 20 may be distributed and implemented in multiple enclosures. Furthermore, the functions of Server 20 may be implemented as a virtual machine in a cloud computing environment.

[0022] Network 80 is a communication network for sending and receiving data between terminal device 10 and server 20. Network 80 is composed of, for example, the Internet, Wide Area Ethernet (registered trademark), dedicated line, virtual private network, or any combination thereof. A communication protocol such as TCP / IP is used.

[0023] <Terminal device hardware configuration> Next, the hardware configuration of the terminal device 10 will be described. The terminal device 10 has the hardware configuration of a typical computer. Specifically, the terminal device 10 includes a processor, main memory, auxiliary storage, a communication interface, input devices, and output devices. The processor is, for example, a central processing unit (CPU), a microprocessing unit (MPU), or a system-on-a-chip (SoC). The main memory is, for example, random access memory (RAM). The auxiliary storage is, for example, a solid-state drive (SSD), a hard disk drive (HDD), or flash memory. The communication interface is a network card or wireless communication module for connecting to the network 80. The input devices are, for example, a keyboard, mouse, touch panel, or microphone. The output devices are, for example, a display, speaker, or printer. The various functions of the terminal device 10 described later are realized through the cooperation of these hardware resources.

[0024] <Functional Configuration of Terminal Devices> Figure 2 is a block diagram showing the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 comprises a control unit 101, a storage unit 102, a communication unit 103, an input unit 104, and an output unit 105 as functional components.

[0025] The control unit 101 has the function of comprehensively controlling each part of the terminal device 10. The control unit 101 is realized, for example, by the processor executing a predetermined program stored in the memory unit 102. The control unit 101 receives input from the input unit 104, controls data transmission and reception via the communication unit 103, and controls the display to the output unit 105.

[0026] The memory unit 102 has the function of storing programs executed by the control unit 101, an operating system, and various data. The memory unit 102 is implemented, for example, by the aforementioned main memory or auxiliary memory. The memory unit 102 temporarily or permanently stores text data of compositions entered by learners, evaluation result data received from the server 20, and learners' account information.

[0027] The communication unit 103 has the function of performing data communication with the server 20 via the network 80. The communication unit 103 is implemented, for example, by the aforementioned communication interface and a program that controls the communication. The communication unit 103 sends the text data of the composition entered by the learner to the server 20. The communication unit 103 receives evaluation result data from the server 20.

[0028] The input unit 104 has the function of receiving various inputs from the learner. The input unit 104 is implemented, for example, by the input device described above. The learner inputs the text of the essay to be evaluated via the input unit 104. The learner also performs operations such as giving instructions to start the evaluation, requesting the display of evaluation results, or other various operations via the input unit 104.

[0029] The output unit 105 has the function of presenting information such as evaluation results received from the server 20 to the learner. The output unit 105 is implemented, for example, by the output device described above. The output unit 105 displays a results display screen 1200 on a display, as shown in Figure 12. The results display screen 1200 includes, for example, a text display area 1201, a score display area 1202, and a feedback comment area 1203.

[0030] Figure 3 is a block diagram showing in more detail the functional configuration of the control unit 101 of the terminal device 10. As shown in Figure 3, the control unit 101 includes an operation reception unit 131, a transmission / reception unit 132, and a presentation control unit 133.

[0031] The operation reception unit 131 has the function of receiving operation input from the learner via the input unit 104. The operation reception unit 131 receives, for example, input of composition text, instructions to start evaluation, or instructions to switch display content.

[0032] The transmitting / receiving unit 132 controls the communication unit 103 to send and receive data with the server 20. The transmitting / receiving unit 132 transmits the composition text received by the operation reception unit 131 to the server 20 in a predetermined format. The transmitting / receiving unit 132 also receives evaluation result data transmitted from the server 20.

[0033] The display control unit 133 has the function of generating screen data to be displayed on the output unit 105 based on the evaluation result data received by the transmitting / receiving unit 132, and controlling the display on the output unit 105. For example, the display control unit 133 generates a result display screen 1200 (see Figure 12) and displays it on the output unit 105.

[0034] <Server Hardware Configuration> Next, the hardware configuration of server 20 will be described. Server 20 has the hardware configuration of a typical server computer. Specifically, server 20 includes a processor, main memory, auxiliary storage, a communication interface, and an input / output interface. The processor is, for example, one or more CPUs. The main memory is, for example, a large-capacity RAM. The auxiliary storage is, for example, an SSD or HDD in a RAID configuration. The communication interface is a device for connecting to network 80. The input / output interface is responsible for connecting to a console device for managing server 20. The various functions of server 20 described later are realized through the cooperation of these hardware resources.

[0035] <Server Functional Configuration> Figure 4 is a block diagram showing the functional configuration of server 20. As shown in Figure 4, server 20 comprises a control unit 201, a storage unit 202, a communication unit 203, and an input / output IF 204 as functional components.

[0036] The control unit 201 has the function of comprehensively controlling each part of the server 20 and executing the main processing for evaluating writing ability in this embodiment. The control unit 201 is realized, for example, by the processor executing a predetermined program stored in the memory unit 202. The specific functions of the control unit 201 will be described later.

[0037] The storage unit 202 has the function of storing the program executed by the control unit 201, the operating system, and various data used for evaluating writing ability. The storage unit 202 is implemented, for example, by the aforementioned main memory or auxiliary memory. As shown in Figure 4, the storage unit 202 holds a database 220 for storing various data. The specific configuration of the database 220 will be described later.

[0038] The communication unit 203 has the function of performing data communication with the terminal device 10 via the network 80. The communication unit 203 is implemented, for example, by the aforementioned communication interface and a program that controls the communication. The communication unit 203 receives the text data of the composition from the terminal device 10. The communication unit 203 transmits the evaluation result data generated by the control unit 201 to the terminal device 10.

[0039] The input / output interface 204 has the function of inputting and outputting data to and from a console device used by the administrator of server 20. The input / output interface 204 is implemented, for example, by the input / output interface described above. The administrator can perform tasks such as updating the master table in the database 220, which will be described later, via the input / output interface 204.

[0040] Figure 5 is a block diagram showing in more detail the functional configuration of the control unit 201 of the server 20. As shown in Figure 5, the control unit 201 comprises an information acquisition unit 211, a text generation unit 212, an upper limit stage determination unit 213, an evaluation element classification unit 214, an achievement level determination unit 215, and a presentation control unit 216. These functional units are functional modules realized by the processor executing a predetermined program.

[0041] The execution order of the processing by the upper limit stage determination unit 213 and the processing by the evaluation element classification unit 214 is not limited. For example, either the processing of the upper limit stage determination unit 213 or the evaluation element classification unit 214 may be executed before the processing of the other, or both the processing of the upper limit stage determination unit 213 and the evaluation element classification unit 214 may be executed in parallel.

[0042] The information acquisition unit 211 has the function of acquiring a first text, which is a document entered by the learner and transmitted from the terminal device 10 via the communication unit 203. The first text is the original text created by the learner. Along with the first text, the information acquisition unit 211 also acquires information such as a learner ID to identify the learner.

[0043] The text generation unit 212 has the function of analyzing the content of the first text acquired by the information acquisition unit 211. The content analysis of the first text is performed using natural language processing techniques such as morphological analysis, syntactic analysis, or dependency parsing. Through the analysis process, the text generation unit 212 decomposes the grammatical structure, vocabulary, or semantic content of the first text and converts it into structured data.

[0044] The text generation unit 212 has the function of generating a second text in which grammatical errors contained in the first text have been corrected based on the analysis results. The second text is a sentence with the minimum necessary corrections, such as correcting only grammatical, word order, or conjugation errors, or only typos. When generating the second text, the text generation unit 212 processes it under the constraint that the semantic content of the first text is not changed.

[0045] The text generation unit 212 has the function of generating a third text that improves the naturalness of expression of the first text. Naturalness of expression refers to qualities related to linguistic fluency, such as the appropriateness of vocabulary selection, the naturalness of word combinations, or the consistency of writing style, in addition to grammatical correctness. The third text is a sentence that has been modified to be more natural in expression while retaining the semantic content of the second text. The generation of the third text includes, for example, vocabulary selection, improvement of collocations, or changes to more natural phrasing.

[0046] The text generation unit 212 uses, for example, a large-scale language model, a rule-based algorithm, or a statistical machine translation model to generate the second and third texts. When using a large-scale language model, the text generation unit 212 generates the second and third texts by using different instructions for different purposes. For example, it generates the second text using the instruction "correct only grammatical errors without changing the meaning," and generates the third text using the instruction "rewrite into a natural-sounding sentence like a native speaker."

[0047] The text generation unit 212 has a function to impose predetermined constraints when generating the third text. These predetermined constraints include a constraint not to add information not included in the second text. These predetermined constraints also include a constraint not to replace language items included in the second text with language items of a higher level of language proficiency than the original language item. This prevents the evaluation results from becoming overly dependent on the characteristics of the model used for text generation, thereby achieving an objective evaluation.

[0048] The upper limit stage determination unit 213 has the function of determining an upper limit stage that indicates the upper limit of the learner's language proficiency, based on the third text generated by the text generation unit 212. Language proficiency refers to the level of attainment of knowledge and ability to use a second language. Language proficiency is defined in multiple stages, such as the stages from N1 to N5 of the Japanese Language Proficiency Test (JLPT), or the stages from A1 to C2 of the Common European Framework of Reference for Languages ​​(CEFR). The upper limit stage indicates the highest level of language proficiency that the learner is estimated to be able to attain with their current ability.

[0049] The upper limit stage determination unit 213 has a function to calculate a first indicator, a function to calculate a second indicator, and a function to determine the upper limit stage.

[0050] The function for calculating the first indicator calculates the first indicator for a set of language items, which are defined for each predetermined level of language proficiency. A language item refers to vocabulary, grammatical rules, or expression patterns that are expected to be acquired at a particular level of language proficiency. For example, the vocabulary word "economics" or the grammatical expression "must be" are examples of language items. The set of items is defined, for example, in the level classification master table 224 shown in Figure 9.

[0051] The first indicator is the coverage rate of items belonging to the group of items that appeared in the third text. An item refers to a given language item that actually appears in the third text. Coverage rate is the proportion of the total set of language items belonging to a certain language proficiency level that are unique items that appear in the third text. For example, if there are 1000 language items in a certain level and 50 of them appear in the third text, the coverage rate is 5%.

[0052] The function for calculating the second indicator normalizes the first indicator calculated for each stage according to the text length of the third text, and then calculates the second indicator. The second indicator is an indicator of coverage normalized by text length, and it shows the frequency of occurrence of language items. Normalization according to text length is, for example, a process of dividing the calculated coverage rate by the number of words, characters, or morphemes in the third text. This normalization eliminates the influence of sentence length on evaluation and prevents unfairly high ratings due to the creation of artificially long sentences.

[0053] The function for determining the upper limit stage determines the upper limit stage based on the second indicator calculated for each stage. There are several variations in the method for determining the upper limit stage. For example, in the gate judgment method, whether or not the second indicator for each stage meets a predetermined threshold is determined sequentially from stages with relatively low language proficiency to stages with high language proficiency. Then, the stage immediately preceding the stage that did not meet the threshold is determined as the upper limit stage.

[0054] Another example of a method for determining the upper limit stage is the profile matching method. In the profile matching method, the distribution of the second indicator for each calculated stage is treated as a feature vector. The similarity to a pre-prepared density profile for each proficiency stage is then calculated, and the stage with the most similarity is determined as the upper limit stage. For example, cosine similarity can be used to calculate the similarity.

[0055] Another example of a method for determining the upper limit stage is the machine learning model approach. In the machine learning model approach, the second metric of each stage is input as a feature into a pre-trained classifier. The classifier outputs the upper limit stage based on the input features. Examples of classifiers used include support vector machines and random forests.

[0056] The upper limit stage determination unit 213 may have the function of determining the upper limit stage using not only a single first text, but also multiple texts obtained from the learner's past input history. In that case, the upper limit stage determination unit 213 determines the upper limit stage based on a statistic of a second indicator calculated using the multiple texts. The statistic may be, for example, the mean, median, or mode. This suppresses fluctuations in evaluation that depend on specific tasks, enabling more stable evaluation.

[0057] The evaluation element classification unit 214 has the function of classifying each element in the first text based on the differences between the first text and the second text, and the differences between the second text and the third text. Specifically, the evaluation element classification unit 214 classifies elements into first evaluation elements, second evaluation elements, and third evaluation elements. The first evaluation elements are elements that did not require correction in the first text and correspond to correctly used vocabulary or grammar (correct usage). The second evaluation elements are elements that require correction from the first text to the second text and correspond to grammatical errors (misuse). The third evaluation elements are elements that require correction from the second text to the third text and correspond to elements that were lacking to improve the naturalness of expression (deficiencies).

[0058] The evaluation element classification unit 214 identifies differences by, for example, alignment analysis comparing the morphological analysis results of the first text, the second text, and the third text, and classifies each element. Alternatively, the evaluation element classification unit 214 may perform classification based on tag information assigned by the text generation unit 212 when generating the second text and the third text. The tag information may be, for example, a "grammar error correction tag" or a "fluency improvement tag."

[0059] The evaluation element classification unit 214 may have the function of further classifying the classified first to third evaluation elements into multiple categories. These multiple categories may include grammar and vocabulary. This classification makes it possible to analyze the learner's abilities from a more multifaceted perspective.

[0060] The evaluation element classification unit 214 may have the function of identifying a predetermined error among the second evaluation elements. The predetermined error is a grammatical error that makes it difficult to interpret the basic meaning of the sentence. The predetermined error includes at least one selected from the group consisting of errors in case relations, errors in negative expressions, errors in tense, and errors in subject-verb agreement.

[0061] The achievement level determination unit 215 has the function of determining the learner's level of language proficiency based on the classification results by the evaluation element classification unit 214. For this purpose, the achievement level determination unit 215 has an evaluation index calculation function and an achievement level determination function.

[0062] The evaluation index calculation function calculates an evaluation index based on the occurrence of the classified first to third evaluation elements. The evaluation index indicates the learner's level of language proficiency. The evaluation index is calculated, for example, as the ratio of the first evaluation element to the total of all evaluation elements. The evaluation index calculation function may also have a function to calculate the evaluation index by setting a predetermined upper limit on the number of occurrences of the same evaluation element. This prevents manipulation of evaluations by repeating specific words.

[0063] The achievement level determination function performs predetermined processing on the calculated evaluation indicators to ultimately determine the learner's level of language proficiency. The achievement level determination is performed within the range of the upper limit levels determined by the upper limit level determination unit 213. The predetermined processing includes correction processing based on the occurrence amounts of the classified first to third evaluation elements. The predetermined processing also includes limitations on the correction processing based on evaluation indicators at levels where language proficiency is relatively lower than the level being judged.

[0064] The achievement level determination function may have the function of determining the achievement level by performing a stepwise judgment process from a relatively low level of language proficiency to a higher level. Furthermore, the achievement level determination function may have the function of restricting the transition to a level of language proficiency that is relatively higher than the level being judged if the number of occurrences of predetermined errors identified by the evaluation element classification unit 214 exceeds a predetermined threshold.

[0065] The presentation control unit 216 has the function of generating data that includes evaluation results such as the upper limit stage determined by the upper limit stage determination unit 213 and the achievement level determined by the achievement level determination unit 215. The presentation control unit 216 has the function of transmitting the generated data to the terminal device 10 via the communication unit 203. The presentation control unit 216 can also transmit the second text and third text generated by the text generation unit 212, as well as information related to the classification results by the evaluation element classification unit 214.

[0066] <Data structure> Figures 6 to 10 show the data structure of the database 220 stored by the server 20. The following explanation and Figures 6 to 10 illustrate an example in which each table is included in the database 220. Note that Figures 6 to 10 are just examples and do not exclude any data not shown. Also, even data listed in the same table may be stored in separate memory areas in the storage unit 202.

[0067] The databases shown in Figures 6 to 10 refer to relational databases, which are used to manage data sets called tables, which are structurally defined by rows and columns, by associating them with each other. In a database, tables are called tables, the columns of a table are called columns, and the rows of a table are called records. In a relational database, relationships can be established and linked between tables.

[0068] Typically, each table has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit 201 can add, delete, or update records in specific tables stored in the storage unit 202 according to various programs.

[0069] Figure 6 shows an example of the data structure of the learner management table 221. The learner management table 221 is a table for identifying learners who use the service. The learner management table 221 includes, for example, columns for learner ID and attribute information. The "learner ID" column stores the learner ID, which is an identifier that uniquely identifies each learner on the system. The "attribute information" column stores information such as the learner's age, native language, learning history, and external test scores. External test scores include, for example, JLPT, TOEIC, etc. The server 20 can estimate the learner's level based on the attribute information and refer to it when fine-tuning the evaluation logic and switching the feedback content.

[0070] Figure 7 shows an example of the data structure of the evaluation history table 222. The evaluation history table 222 is a table that stores past evaluation results. The evaluation history table 222 includes columns such as evaluation ID, learner ID, first text, grade level, score for each language level, overall score, and evaluation date and time. The "Evaluation ID" column stores the evaluation ID, which is an identifier that uniquely identifies a single evaluation process. The "Learner ID" column is a key linked to the learner management table 221. The "First Text" column stores the text itself that the learner entered. The "Grade Level" column stores the final determined level of achievement. The "Score for Each Language Level" and "Overall Score" columns store quantitative score values ​​calculated by the evaluation logic. The "Evaluation Date and Time" column stores the date and time when the evaluation was performed.

[0071] Figure 8 shows an example of the data structure of the text analysis table 223. The text analysis table 223 is a table that manages the text and difference information generated in a single evaluation process. The text analysis table 223 includes columns such as evaluation ID, second text, third text, alignment information, classification flag, and unclassified element percentage. The "evaluation ID" column is a key linked to the evaluation history table 222. The "second text" and "third text" columns store the text data generated by the text generation unit 212. The "alignment information / classification flag" column stores the classification results of the first evaluation element, second evaluation element, and third evaluation element. Specifically, the "alignment information / classification flag" column stores the classification results of correct use, misuse, and omission at the morphological level identified by the evaluation element classification unit 214. The "unclassified element percentage" column stores the percentage of vocabulary, etc., that do not exist in the level classification master table 224.

[0072] Figure 9 shows an example of the data structure of the Level Classification Master Table 224. The Level Classification Master Table 224 is a table that defines a set of language items that serve as the basis for each level of language proficiency. The Level Classification Master Table 224 includes, for example, columns for Item ID, Item Name, and Language Proficiency Level. The "Item ID" column stores the Item ID, which is an identifier that uniquely identifies each language item. The "Item Name" column stores the specific vocabulary and grammatical rules. Specific vocabulary is, for example, "milk," and grammatical rules are, for example, "~I want." The "Language Proficiency Level" column stores the level to which each language item belongs. Levels are, for example, JLPT levels 1 to 5, or levels 1 to 4.

[0073] Figure 10 shows an example of the data structure of the evaluation setting master table 225. The evaluation setting master table 225 is a table that manages various thresholds and parameters used in the evaluation logic. The evaluation setting master table 225 includes columns such as language proficiency level, judgment threshold, penalty coefficient setting value, minimum number of elements, and lower-level proficiency exemption condition value. The "language proficiency level" column is a key linked to the grade classification master table 224. The "judgment threshold" column stores the score threshold required to advance to the next level in gate judgment for each grade. The "minimum number of elements" column stores the sample size required to avoid applying a penalty. The "lower-level proficiency exemption condition value" column stores the range of positive scores at lower levels required to exempt a penalty at a higher level. The range of positive scores is, for example, +0.20. By managing these values ​​as a master table, the evaluation logic can be easily fine-tuned.

[0074] <Example of operation> Figure 11 is a flowchart illustrating an example of the operation of the server 20 according to this embodiment. First, in step S11, the information acquisition unit 211, which constitutes the control unit 201 of the server 20, acquires the first text. The first text is a document entered by the learner via the terminal device 10. Specifically, the operation reception unit 131, which constitutes the control unit 101 of the terminal device 10, accepts the input of the composition by the learner. After inputting, the learner performs a transmission operation. The control unit 101 of the terminal device 10 generates the input composition data as the first text. The transmission / reception unit 132, which constitutes the control unit 101 of the terminal device 10, transmits the generated first text to the server 20 via the communication unit 103 and the network 80. The communication unit 203 of the server 20 receives the transmitted first text. The information acquisition unit 211 acquires the received first text. The information acquisition unit 211 stores the acquired first text, along with related information such as the learner ID, in the evaluation history table 222.

[0075] Next, in step S12, the text generation unit 212, which constitutes the control unit 201 of the server 20, analyzes the content of the acquired first text. Based on the analysis results, the text generation unit 212 generates the second text and the third text. The text generation unit 212 can perform the generation process of the second text and the third text using, for example, a generative AI model such as a large-scale language model.

[0076] The text generation unit 212 can also generate the second and third texts using methods other than a large-scale language model. For example, the text generation unit 212 can generate text using a rule-based algorithm. In a rule-based algorithm, the text generation unit 212 performs morphological analysis on the first text. The text generation unit 212 refers to a pre-stored grammar rule dictionary. The text generation unit 212 corrects grammatical errors based on the analysis results and the grammar rule dictionary. Alternatively, the text generation unit 212 can generate text using a statistical machine translation model. A statistical machine translation model is a model trained from a large amount of source-and-corrected text pair data. The text generation unit 212 outputs the second or third text by inputting the first text into the trained model.

[0077] The text generation unit 212 inputs a predetermined prompt and the first text to, for example, a large-scale language model in order to generate the second text. The prompt is an instruction to the effect of, "Do not change the meaning of the input text, and only correct grammatical errors." The text generation unit 212 controls the process to prevent changes such as adding semantic content or substituting with more advanced vocabulary. Changes that go beyond simple grammatical correction are not performed in the generation of the second text.

[0078] The text generation unit 212 takes a predetermined prompt and the first or second text as input to, for example, a large-scale language model, in order to generate a third text. The prompt is an instruction to the effect of, "Rewrite the input text into a more natural and fluent native-level expression while retaining its semantic content." The text generation unit 212 generates the third text based on the input. The third text generation process includes vocabulary selection, collocation optimization, and changes to more natural phrasing. The text generation unit 212 stores the generated second and third texts in a text analysis table 223 or the like, in association with the original first text.

[0079] Next, in step S13, the upper limit stage determination unit 213, which constitutes the control unit 201 of the server 20, calculates a first index. The first index is the coverage rate of items belonging to the item group that appeared in the third text. The item group is a set of language items defined for each predetermined language proficiency stage. The language proficiency stages refer to levels that hierarchically indicate language ability, such as N1 to N5 of the Japanese Language Proficiency Test. The item group is a list of vocabulary and grammatical items that are predefined to belong to each language proficiency stage. The item groups are stored, for example, in the grade classification master table 224.

[0080] The upper level determination unit 213 first decomposes the third text into language items using methods such as morphological analysis. Language items include words and grammatical elements. Next, the upper level determination unit 213 refers to the level classification master table 224. The upper level determination unit 213 identifies which language proficiency level each decomposed language item belongs to. Next, the upper level determination unit 213 calculates the percentage of the total number of items belonging to each language proficiency level. The percentage is the proportion of unique items that appeared in the third text. The calculated percentage becomes the first index, which is the coverage rate. For example, suppose there are 1000 vocabulary words defined in the item group of a certain language proficiency level. Suppose 50 unique vocabulary words appear in the third text. In this case, the first index for that language proficiency level will be 5%.

[0081] Next, in step S14, the upper limit stage determination unit 213 calculates the second index. The upper limit stage determination unit 213 normalizes the first index for each stage of language proficiency calculated according to the text length of the third text, and calculates the second index. The upper limit stage determination unit 213 can use, for example, the total number of words, total number of characters, or total number of tokens contained in the third text as the text length. The upper limit stage determination unit 213 calculates the second index as, for example, the density obtained by dividing the first index by the text length. The upper limit stage determination unit 213 performs a normalization process. The normalization process eliminates bias in evaluation that depends on the length of the text. It becomes possible to evaluate the quality of the text more objectively.

[0082] Finally, in step S15, the upper limit stage determination unit 213 determines the upper limit stage. The upper limit stage determination unit 213 determines the upper limit stage based on the second indicator for each stage of language proficiency that has been calculated. The upper limit stage determination process is a process that estimates the ceiling of ability that the learner can reach. The ceiling of ability indicates up to which stage of language proficiency the composition being evaluated reflects. The upper limit stage determination unit 213 uses the determined upper limit stage to define the evaluation range when determining the level of achievement, which will be described later. The upper limit stage determination unit 213 records the determined upper limit stage in the evaluation history table 222, etc.

[0083] <Summary> According to this embodiment, the server 20 generates a second text indicating accuracy and a third text indicating fluency as separate data. The server 20 determines the upper limit stage based on the third text. This allows the server 20 to separate and handle "grammatical errors" and "unnatural expressions," which were previously confused, within the system. Even if a learner lists unnaturally difficult words, the text generation unit 212 simplifies them into natural expressions in the third text. The upper limit stage determination unit 213 calculates the density based on the simplified third text. Therefore, the server 20 can prevent hacking activities that would result in unfairly high ratings beyond the learner's actual ability. As a result, the server 20 performs objective and precise language proficiency assessments.

[0084] [Variation] <First variation: Achievement evaluation process> In this modified example, the control unit 201 of the server 20 performs a process to specifically evaluate the learner's level of language proficiency. The evaluation element classification unit 214 and the achievement level determination unit 215 mainly perform the evaluation process.

[0085] First, the evaluation element classification unit 214 classifies the evaluation elements based on the differences between the first text and the second text, and between the second text and the third text. The differences between the first text and the second text indicate grammatical errors in the learner's original text. Specific examples include misuse of particles, incorrect verb conjugation, or incorrect word order. The differences between the second text and the third text indicate areas for improvement that are grammatically correct but would enhance the naturalness or fluency of expression. Specific examples include the removal of redundant expressions, substitution with more natural vocabulary, or the completion of subjects omitted in context.

[0086] The evaluation element classification unit 214 classifies the evaluation elements based on these differences. An evaluation element is the smallest unit of evaluation that constitutes a text, and corresponds, for example, to each word or phrase after morphological analysis. The evaluation elements are classified into first evaluation elements, second evaluation elements, and third evaluation elements. The first evaluation elements are elements that did not require correction in the first text and are also called correct usage elements. The second evaluation elements are elements that require correction from the first text to the second text and are also called misuse elements. The third evaluation elements are elements that require correction from the second text to the third text and are also called deficiency elements. The evaluation element classification unit 214 performs a classification process. The classification process compares and analyzes three texts. The three texts are the first text, the second text, and the third text. The purpose of the classification process is to distinguish between the accuracy and fluency of the learner's language use. Accuracy indicates the presence or absence of errors, and fluency indicates a lack of expression.

[0087] The evaluation element classification unit 214 can also use a dependency structure / syntax tree comparison method as an alternative classification method. The evaluation element classification unit 214 generates syntax trees by performing dependency parsing or syntactic parsing on each of the first text, second text, and third text. Next, the evaluation element classification unit 214 compares the syntax trees of each text to detect structural changes. For example, the evaluation element classification unit 214 classifies modifications to the basic structure of a sentence, such as the relationship between subject and predicate, as a second evaluation element. The evaluation element classification unit 214 classifies structural changes, such as the natural addition of modifying components, as a third evaluation element.

[0088] Next, the achievement determination unit 215 calculates an evaluation index. The evaluation index indicates the learner's level of language proficiency. The achievement determination unit 215 calculates the evaluation index based on the occurrence of the classified first to third evaluation elements. Calculating the evaluation index based on the occurrence of the first to third evaluation elements refers to a process, for example, of aggregating the number of occurrences of the classified evaluation elements in each language proficiency stage and each category, and applying a predetermined calculation formula to calculate a score. The achievement determination unit 215 calculates the evaluation index for each language proficiency stage, for example.

[0089] As another example of how to calculate the evaluation index, the achievement determination unit 215 may calculate an error rate, which is the ratio of the second evaluation element to the sum of all evaluation elements (first to third evaluation elements), and a deficiency rate, which is the ratio of the third evaluation element to the sum of all evaluation elements. The achievement determination unit 215 may then determine the evaluation index based on the balance between the calculated error rate and deficiency rate.

[0090] The achievement determination unit 215 can calculate the correct usage rate as an example of an evaluation index. The achievement determination unit 215 calculates the correct usage rate, for example, using a predetermined formula. The formula is "Correct Usage Rate = (Number of First Evaluation Elements) / (Number of First Evaluation Elements + Number of Second Evaluation Elements + Number of Third Evaluation Elements)". The denominator of the formula includes the third evaluation element. By including the third evaluation element in the denominator, the achievement determination unit 215 can evaluate more practical language ability. Evaluating practical language ability takes into account not only the absence of grammatical errors but also the naturalness of expression. The achievement determination unit 215 records the calculated evaluation index in the evaluation history table 222, etc.

[0091] The achievement determination unit 215 can also use an additive / deductive scoring method as a method for calculating the evaluation index. The achievement determination unit 215 classifies the first evaluation element as an additive element. The achievement determination unit 215 classifies the second and third evaluation elements as deductive elements. The achievement determination unit 215 calculates the final evaluation index by subtracting the number or percentage of deductive elements from a predetermined initial score.

[0092] <Second variation: Correction process and stepwise judgment> In this modified version, the achievement level determination unit 215 performs further refined correction and judgment processing on the calculated evaluation index. The achievement level determination unit 215 determines the final achievement level through the correction and judgment processing.

[0093] First, the achievement determination unit 215 calculates the raw score, which is the evaluation index before correction. Next, the achievement determination unit 215 performs a correction process to discount the calculated evaluation index if the total number of classified first to third evaluation elements is small. For example, the achievement determination unit 215 calculates a penalty coefficient as the ratio of the actual total number of evaluation elements to the minimum required sample size defined for each language proficiency level. Then, the achievement determination unit 215 multiplies the calculated raw score by the penalty coefficient to calculate the corrected evaluation index.

[0094] Next, if the achievement determination unit 215 shows an extremely high level of proficiency at a stage where language proficiency is relatively lower than the stage being judged, it disables or mitigates the correction process due to insufficient sample size at the stage being judged. For example, if the evaluation index of the immediately preceding stage significantly exceeds a predetermined threshold, the achievement determination unit 215 skips multiplication by the penalty coefficient at the higher stage. The achievement determination unit 215 allows the determination based solely on the raw score. Furthermore, if the raw score of the stage passed through the exemption is sufficiently high, the achievement determination unit 215 can also perform a cascading process that chains exemptions to even higher stages.

[0095] Finally, the achievement determination unit 215 performs a stepwise determination process. The achievement determination unit 215 performs a stepwise determination process using evaluation indicators that have undergone correction processing and the limitations imposed by the correction processing. The stepwise determination process is performed within the range of the upper limit stages determined in the basic flow. The stepwise determination process is performed sequentially from stages with relatively low language proficiency to stages with high language proficiency. The stepwise determination process is a gate determination method. The achievement level of the learner's language proficiency is determined by the stepwise determination process.

[0096] Specifically, the achievement level determination unit 215 first determines the evaluation index for the lowest level of language proficiency. For example, the achievement level determination unit 215 determines whether the evaluation index for level 4 is equal to or greater than the judgment threshold defined in the evaluation setting master table 225. The judgment threshold is, for example, 0.60. Only if the condition that the evaluation index is equal to or greater than the judgment threshold is met, the achievement level determination unit 215 proceeds to the determination of the next stage, level 3. Similarly, if the evaluation index for level 3 is equal to or greater than the judgment threshold, the achievement level determination unit 215 proceeds to the determination of level 2. The judgment threshold is, for example, 0.65. The achievement level determination unit 215 repeats the determination process sequentially up to the upper limit stage determined by the upper limit stage determination unit 213. Let's assume that at any stage the evaluation index falls below the judgment threshold. The achievement level determination unit 215 terminates the determination process. The achievement level determination unit 215 determines the stage immediately preceding the stage in which the index fell below the judgment threshold as the learner's final level of language proficiency.

[0097] Furthermore, if the occurrence of both the second and third evaluation elements in the calculated evaluation index meets the predetermined criteria, a correction process may be performed to determine the final level of achievement as a level of language proficiency that is relatively higher than the determined upper limit level. Examples of predetermined criteria include cases where both the error rate and deficiency rate are zero, cases where both the error rate and deficiency rate are below a predetermined small threshold (e.g., 0.1%), or cases where the sum of the error rate and deficiency rate is below a predetermined threshold.

[0098] <Third variation: Constraints on text generation processing> This modified example relates to the processing performed when the text generation unit 212 generates a third text. When generating the third text, the text generation unit 212 performs processing under certain constraints. Specifically, the text generation unit 212 generates the third text under at least one of the following constraints: not to add information not included in the second text, and not to replace language items included in the second text with language items of a higher level of language proficiency. The text generation unit 212 can also perform the generation process under the constraint of setting an upper limit on the amount of editing of the third text. The upper limit on the amount of editing is defined, for example, as the increase in the number of characters or tokens of the third text relative to the first text being less than or equal to a predetermined percentage.

[0099] <Fourth variation: Evaluation by category> In this modified version, the evaluation element classification unit 214 and the achievement level determination unit 215 perform more detailed category-based evaluations. The evaluation element classification unit 214 further classifies the classified first to third evaluation elements into multiple categories such as "grammar" and "vocabulary." The classification is performed, for example, based on the linguistic properties of the modified sections. The evaluation element classification unit 214 scans the morphological analysis results of the first text, second text, and third text, for example, and generates an array of evaluation elements corresponding to each word or expression. The evaluation element classification unit 214 assigns a category to each evaluation element by referring to tag information, etc., which will be described later. The achievement level determination unit 215 then independently performs the calculation of the evaluation index, correction processing, and step-by-step judgment processing for each category.

[0100] <Fifth Variation: Restrictions Based on Material Errors> In this modified example, the achievement determination unit 215 imposes restrictions based on specific critical errors in the step-by-step judgment process. First, the evaluation element classification unit 214 or the achievement determination unit 215 identifies predetermined errors from the second evaluation elements that make it difficult to interpret the basic meaning of a sentence. The basic meaning of a sentence refers to the propositional meaning conveyed by the sentence. The propositional meaning is the core part that affects the truth value of the sentence, such as who did what and how. The predetermined errors are also called critical errors. The predetermined errors include at least one selected from the group consisting of errors in case relations, errors in negative expressions, errors in tense, and errors in subject-verb agreement. Next, the achievement determination unit 215 determines whether the number of occurrences of the identified predetermined errors exceeds a predetermined threshold. If the number of occurrences exceeds the threshold, the achievement determination unit 215 forcibly restricts the transition to a stage with a relatively higher level of language proficiency than the stage being judged, even if the learner's evaluation index value is high.

[0101] <Sixth variation: Use of multiple texts> This modified example relates to the process of determining the upper limit stage. When determining the upper limit stage, the upper limit stage determination unit 213 considers not only the single first text input in the current instance, but also multiple texts obtained from the learner's past input history. Multiple texts refer to a series of composition data previously created by the same learner and stored in the system. Specifically, the server 20 stores past composition data, generated third texts, calculated second indicators, etc., associated with a specific learner ID in the evaluation history table 222, etc. The upper limit stage determination unit 213 then integrates the second indicator calculated from the current composition with the second indicator calculated from multiple past compositions. The upper limit stage determination unit 213 determines the upper limit stage based on the resulting statistics. As statistics, for example, the mean, median, or mode of multiple second indicators can be used.

[0102] <Seventh Modification: Gate-based determination of upper limit stage> This modified example illustrates a specific logic for determining the upper limit stage. The upper limit stage determination unit 213 determines whether the second indicator for each stage of language proficiency meets a predetermined threshold defined in the evaluation setting master table 225, etc. The determination is performed sequentially from the stages with relatively low language proficiency to the stages with high proficiency. For example, the upper limit stage determination unit 213 first determines whether the second indicator for the lowest stage, level 4, is above the threshold. If the second indicator meets the threshold, the upper limit stage determination unit 213 proceeds to the determination for level 3. The upper limit stage determination unit 213 repeats the determination sequentially, and if the second indicator fails to meet the threshold for the first time at a certain stage, it determines the stage immediately preceding that as the final upper limit stage.

[0103] <Variation 8: Hacking Prevention> This modified example relates to hacking prevention measures when calculating evaluation metrics. The achievement determination unit 215 sets a predetermined upper limit on the number of occurrences of the same evaluation element when calculating evaluation metrics. For example, consider a case where a learner intentionally manipulates the evaluation metrics by unnaturally repeating a specific word that they already use correctly. In counting each evaluation element used to calculate the evaluation metrics, the achievement determination unit 215 sets a predetermined upper limit on the number of occurrences of evaluation elements corresponding to an index indicating the same position in the first text. The upper limit is, for example, two times. This prevents the evaluation metrics from increasing simply by repeating the same expression, allowing the server 20 to prevent artificial score manipulation and achieve a fairer evaluation.

[0104] <9th variation: Use of tag information> This modified example relates to a specific method for classifying evaluation elements. When classifying the first, second, and third evaluation elements, the evaluation element classification unit 214 refers to the tag information assigned by the text generation unit 212 when it generates the second and third texts. Tag information is metadata assigned by the text generation unit 212 to the parts that have been modified, in order to indicate the intention, type, or basis of the modification. Specific examples include "grammar error correction tags," "fluency improvement tags," or more detailed "particle error tags."

[0105] The classification process of evaluation elements based on tag information is specifically performed as follows: First, the text generation unit 212 analyzes the first text and adds tag information to the modified parts when generating the second and third texts. Next, the evaluation element classification unit 214 checks the morphological analysis results of the first text, second text, and third text. For each morpheme of the first text, the evaluation element classification unit 214 adds a reference to the tag information added by the text generation unit 212 at the locations where there have been changes between the second and third texts. For example, locations with tags correcting grammatical errors are classified as second evaluation elements. Locations with tags improving fluency are classified as third evaluation elements. Locations without tag information are classified as first evaluation elements. By directly referencing the tag information, the evaluation element classification unit 214 can perform the classification of evaluation elements quickly and accurately.

[0106] <Other Embodiments> In this embodiment, modifications regarding the processing of language items not present in the grading master table 224, i.e., unclassified elements, are also possible. When the third text is analyzed, the upper limit stage determination unit 213 or the achievement level determination unit 215 identifies language items not registered in the grading master table 224 as unclassified elements. The upper limit stage determination unit 213 calculates the unclassification rate, which is the proportion of unclassified elements in the entire third text. If the unclassification rate exceeds a predetermined threshold, the upper limit stage determination unit 213 can treat the determination of the upper limit stage as provisional. The achievement level determination unit 215 also estimates the language proficiency stage to which the unclassified element would belong, based on the frequency of occurrence of the unclassified element, the character type, or its generality in other text corpora. The achievement level determination unit 215 can also reflect the estimated stage in the calculation of the evaluation index.

[0107] System 1 can also provide functionality via an API. When Server 20 receives the first text from Terminal Device 10, it performs a series of evaluation processes described herein. The presentation control unit 216 of Server 20 then returns the final evaluation result to Terminal Device 10 as an API response. The final evaluation result includes, for example, the language proficiency level determined as the level of achievement, the score for each category, the generated second text, and the third text. Based on the API response, Terminal Device 10 displays the evaluation result on its screen.

[0108] Furthermore, the presentation control unit 216 can also change the information it presents depending on the type of user. For example, if the user is a learner belonging to an academic institution, the presentation control unit 216 will only present the second text and evaluation indicators, and will restrict the display of the third text. On the other hand, if the user is a general learner, the presentation control unit 216 will present all information, including the second text, the third text, and the evaluation indicators. This allows the server 20 to differentiate its functions according to the form of service delivery.

[0109] <Screen example> Figure 12 shows an example of a results display screen 1200 displayed on the output unit 105 of the terminal device 10. The results display screen 1200 is a user interface for clearly presenting evaluation results and feedback information on language proficiency for the first text input by the learner. The results display screen 1200 includes a text display area 1201, a score display area 1202, and a feedback comment area 1203.

[0110] The text display area 1201 displays the first text entered by the learner, the second text with grammatical errors corrected, and the third text improved to be more natural. The presentation control unit 133 of the terminal device 10 displays each text in a comparable format. The comparable format is either parallel display or tab-switching display. In the text display area 1201, the presentation control unit 133 highlights the corrected grammatical errors in the first text in red. The presentation control unit 133 highlights the parts that have been added or changed to make the expression more natural in blue. When the learner taps on a highlighted part, the presentation control unit 133 may display a detailed explanation of the error in a pop-up window. This allows the learner to visually distinguish and understand between "mere grammatical errors" and "points for improvement to make the expression better."

[0111] The score display area 1202 displays a graph of scores indicating the level of achievement for each language level. The language levels are, for example, JLPT N5 to N1. A radar chart or a bar graph is used as the graph. The radar chart shows the achievement status for each level from multiple perspectives. The bar graph shows the correct answer rate for each level. The presentation control unit 133 also displays the final judgment level determined by the server 20 in large emphasis in the score display area 1202. The final judgment level is an overall evaluation based on the upper limit level and the level of achievement.

[0112] The feedback comment area 1203 displays detailed evaluation comments for each category, such as grammar and vocabulary. For example, the presentation control unit 133 provides specific learning advice tailored to the learner's level, such as, "You have few grammatical errors, but your expression lacks variety. Let's learn more natural ways of speaking."

[0113] 〔supplement〕 In the embodiments described above, the cases in which the units and means are implemented by a processor were explained, but the invention is not limited to this. The units and means may be any hardware known to perform the operation.

[0114] Furthermore, although the above-described embodiment explained an example in which the server 20 provides each function, the configuration is not limited to this. Some or all of the functions provided by the server 20 in this embodiment may be provided by the terminal device 10 instead of the server 20.

[0115] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. For example, a configuration or process in one embodiment may be combined with a configuration or process in another embodiment, or a modification of one embodiment may be applied to another embodiment. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0116] [Note] The details described in each of the above embodiments are noted below.

[0117] <Note 1> A program for operating a computer equipped with a processor, the program causing the processor to perform the following steps: acquire a first text which is a sentence input by a learner; analyze the content of the acquired first text and generate a second text which corrects grammatical errors contained in the first text and a third text which improves the naturalness of the expression of the first text; calculate a first index which is the coverage rate of items belonging to the item group that appear in the third text for a group of items which is a set of language items defined for each predetermined language proficiency level; normalize the calculated first index for each stage according to the text length of the third text for each and calculate a second index; and determine an upper limit stage which indicates the upper limit of the learner's language proficiency level based on the calculated second index for each stage.

[0118] <Note 2> The program described in Appendix 1, which further causes the processor to perform the following steps: classify the first text into first evaluation elements that did not require modification in the first text, second evaluation elements corresponding to modifications from the first text to the second text, and third evaluation elements corresponding to modifications from the second text to the third text, based on the difference between the first text and the second text, and the difference between the second text and the third text; and calculate an evaluation index indicating the degree of the learner's language proficiency based on the occurrence of the classified first to third evaluation elements.

[0119] <Note 3> The program described in Appendix 2, which further causes the processor to perform a correction process on the calculated evaluation index based on the occurrence amounts of the classified first to third evaluation elements, and to limit the correction process based on the evaluation index in the stage where language proficiency is relatively lower than the stage to be judged, and then to perform a stepwise judgment process from the stage where language proficiency is relatively lower to the stage where it is relatively higher, within the range of the upper limit stage, or within the range up to the stage obtained by correcting the upper limit stage if the occurrence status of the second evaluation element and the occurrence status of the third evaluation element meet predetermined criteria, thereby determining the degree of language proficiency achieved by the learner.

[0120] <Note 4> The program described in any of Appendix 1 to Appendix 3, wherein in the generation step, the third text is generated under the constraint that no information not included in the second text is added, and that language items included in the second text are not replaced with language items of a higher level of language proficiency than the language item in question.

[0121] <Note 5> The program according to Appendix 2, which causes the processor to further perform the step of classifying the classified first to third evaluation elements into a plurality of categories, including grammar and vocabulary, and independently calculating the evaluation index for each category.

[0122] <Note 6> The processor includes the steps of: identifying a predetermined error among the classified second evaluation elements, which is a grammatical error that makes it difficult to interpret the basic meaning of a sentence, and which includes at least one selected from the group consisting of case relationship errors, negative expression errors, tense errors, and subject-verb agreement errors; and, if the number of occurrences of the identified predetermined error exceeds a predetermined threshold, restricting the transition in the stepwise determination process to a stage in which language proficiency is relatively higher than the stage being determined. The program described in Appendix 3 further executes the above.

[0123] <Note 7> In the step of determining the above, the upper limit stage is determined based on a statistic of the second indicator calculated using multiple texts obtained from the learner's past input history in addition to the first text, as described in any of the programs described in Appendix 1 to Appendix 6.

[0124] <Note 8> The program described in any of Appendix 1 to Appendix 7, wherein in the step of determining, it is determined whether the second indicator for each stage satisfies a predetermined threshold, in order from the stage with relatively low language proficiency to the stage with high relative proficiency, and the stage immediately preceding the stage that did not satisfy the threshold is determined to be the upper limit stage.

[0125] <Note 9> The program described in Appendix 2, wherein in the step of calculating the evaluation index, a predetermined upper limit is set on the number of occurrences of the same evaluation element before calculating the evaluation index.

[0126] <Note 10> The program described in Appendix 2 classifies the first evaluation element, the second evaluation element, and the third evaluation element based on tag information assigned when generating the second text and the third text, in the classification step.

[0127] <Note 11> An information processing device comprising a processor, wherein the processor executes all steps in any of the programs described in Appendix 1 to Appendix 10.

[0128] <Note 12> A method to be performed on a computer equipped with a processor, wherein the processor performs all steps in any of the programs described in Appendix 1 to Appendix 10.

[0129] <Note 13> A system comprising one or more of the aforementioned processors that execute all the steps in any of the programs described in Appendix 1 to Appendix 10. [Explanation of Symbols]

[0130] 1... System 10…Terminal device 20... Server 80…Network 101, 201… Control Units 102, 202...Storage section 103, 203... Communications Department 104...Input section 105...Output section 204… Input / Output Interface 131...Operation reception unit 132... Transmitter / Receiver 133…Display Control Unit 211…Information acquisition department 212...Text generation unit 213... Upper limit stage determination unit 214…Evaluation Element Classification Section 215…Achievement level determination section 216…Display Control Unit 220... Database 221... Learner Management Table 222... Evaluation History Table 223...Text analysis table 224...Classification Master Table 225…Evaluation setting master table 1200…Result display screen 1201...Text display area 1202...Score display area 1203...Feedback comment area

Claims

1. A program that operates a computer equipped with a processor, The aforementioned processor, The first step is to obtain the first text, which is a sentence entered by the learner, The steps include: analyzing the content of the acquired first text, generating a second text in which grammatical errors contained in the first text have been corrected, and generating a third text in which the semantic content of the second text has been preserved while improving the naturalness of expression compared to the second text; A step of calculating a first index, which is the coverage rate of the items belonging to the item group that appear in the third text, for a group of items that is a set of language items defined for each predetermined level of language proficiency, The first indicator calculated for each of the aforementioned stages is normalized according to the text length of the third text to calculate a second indicator, A step of determining an upper limit stage that indicates the upper limit of the learner's language proficiency, based on the second indicator calculated for each stage, A program that executes something.

2. The aforementioned processor, A step of classifying the first evaluation elements into a first evaluation element that did not require modification in the first text, a second evaluation element that corresponds to modification from the first text to the second text, and a third evaluation element that corresponds to modification from the second text to the third text, based on the difference between the first text and the second text, and the difference between the second text and the third text. A step of calculating an evaluation index indicating the degree of language proficiency achieved by the learner based on the occurrence of the classified first to third evaluation elements, The program according to claim 1, which further executes the following.

3. The program according to claim 2, further comprising the step of causing the processor to perform a correction process on the calculated evaluation index based on the occurrence amounts of the classified first to third evaluation elements, and to limit the correction process based on the evaluation index in the stage where language proficiency is relatively lower than the stage to be judged, and then, within the range of the upper limit stage, or within the range up to the stage obtained by correcting the upper limit stage if the occurrence status of the second evaluation element and the occurrence status of the third evaluation element meet a predetermined standard, to perform a stepwise judgment process from the stage where language proficiency is relatively lower to the stage where it is higher, thereby determining the degree of language proficiency achieved by the learner.

4. The program according to claim 1, wherein in the generation step, the third text is generated under the constraint that no information not included in the second text is added, and the constraint that language items included in the second text are not replaced with language items of a higher level of language proficiency than the language items.

5. The program according to claim 2, further comprising the step of causing the processor to classify the classified first to third evaluation elements into a plurality of categories, including grammar and vocabulary, and to independently calculate the evaluation index for each category.

6. The aforementioned processor, The steps include identifying a predetermined error among the classified second evaluation elements that makes it difficult to interpret the basic meaning of the sentence, and which includes at least one selected from the group consisting of errors in case relations, errors in negative expressions, errors in tense, and errors in subject-verb agreement, If the number of occurrences of the identified predetermined errors exceeds a predetermined threshold, the step-by-step judgment process includes a step of restricting the transition to a stage with a relatively higher level of language proficiency than the stage being judged, The program according to claim 3, which further executes the following.

7. The program according to claim 1, wherein in the step of determining, the upper limit is determined based on a statistic of the second indicator calculated using a plurality of texts obtained from the learner's past input history in addition to the first text.

8. The program according to claim 1, in the step of determining, determines whether the second indicator for each stage satisfies a predetermined threshold, in order from the stage with relatively low language proficiency to the stage with high relative proficiency, and determines the stage immediately preceding the stage that did not satisfy the threshold as the upper limit stage.

9. The program according to claim 2, wherein in the step of calculating the evaluation index, a predetermined upper limit is set on the number of occurrences of the same evaluation element when calculating the evaluation index.

10. The program according to claim 2, wherein the classification step classifies the first evaluation element, the second evaluation element, and the third evaluation element based on tag information assigned when generating the second text and the third text.

11. An information processing device comprising a processor, wherein the processor executes all steps in any one of claims 1 to 10.

12. A method to be performed on a computer having a processor, wherein the processor performs all steps of a program according to any one of claims 1 to 10.

13. A system comprising one or more of the processors that perform all steps in the program according to any one of claims 1 to 10.

Images