User management system and user management method

Abstract

To provide a mechanism for predicting continuous usability of a user using education service.SOLUTION: A user management system includes: an acquisition unit for acquiring learning history information on a user; an evaluation unit for evaluating whether or not the user can obtain a learning effect by education service, on the basis of the acquired learning history information; and an output unit for outputting an evaluation result indicating whether or not the user can obtain the obtained learning effect by the education service.SELECTED DRAWING: Figure 35

Description

[Technical field]

[0001] The present invention relates to a user management system and a user management method. [Background technology]

[0002] As a background technology of this technical field, there is JP 2021-111007 A (Patent Document 1). This publication states that "GRU11, which is a trained recurrent neural network, is input with a visit interval sequence and its difference sequence in addition to the customer's gender, age, etc. The visit interval sequence is composed of a sequence of multiple numbers indicating the visit interval corresponding to the most recent visit. The visit interval difference sequence is composed of a sequence of multiple numbers indicating the difference between the visit intervals corresponding to the most recent visit. GRU11 performs calculations on these input values ​​and outputs multiple output values ​​to the fully connected layer 12. The fully connected layer 12 adds up all the results of multiplying these output values ​​by a predetermined weight. The output layer 20 further performs calculations on the calculation results of the fully connected layer 12 using a sigmoid function to derive a predicted value indicating the likelihood of the customer dropping out." (See summary) [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-111007 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is described in Patent Document 1 that the technology can appropriately evaluate the likelihood of a customer dropping out. However, even if the technology in Patent Document 1 is applied to predicting the learning compatibility of a user who uses an educational service, there is a problem in that it is not possible to appropriately predict the learning compatibility.

[0005] Therefore, the present invention provides a mechanism for predicting the learning suitability of a user who uses an educational service. [Means for solving the problem]

[0006] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above problems, but one example is a user management system comprising an acquisition unit that acquires a user's learning history information, an evaluation unit that evaluates whether the user has been able to enjoy the learning benefits of the educational service based on the acquired learning history information, and an output unit that outputs the evaluation result of whether the user has been able to enjoy the learning benefits of the educational service. Effect of the Invention

[0007] According to the present invention, a mechanism for predicting the learning suitability of a user who uses an educational service can be provided. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an example of an overall configuration diagram of a user management system 100. [Diagram 2] FIG. 2 shows an example of the hardware configuration of the management server 101. [Diagram 3] FIG. 3 shows an example of the hardware configuration of the user terminal 102. [Figure 4] FIG. 4 shows an example of the hardware configuration of the educational facility terminal 103. [Diagram 5] FIG. 5 is an example of user information 500. [Figure 6] FIG. 6 is an example of learning unit information 600. [Figure 7] FIG. 7 is an example of study content information 700. [Figure 8] FIG. 8 is an example of learning history information 800. [Figure 9] FIG. 9 is an example of educational facility information 900. [Figure 10]FIG. 10 is an example of a learning fitness estimation flow 1000. [Figure 11] FIG. 11 is an example of a machine learning model generation flow 1100. [Figure 12] FIG. 12 shows another example of a learning adaptability estimation flow 1200. [Figure 13] FIG. 13 is an example of a screen 1300 showing the contribution of each feature amount in the learning history information to the learning adaptability. [Figure 14] FIG. 14 is an example of a screen 1400 showing the contribution of each feature amount in the learning history information of a high-achieving user to the learning suitability. [Figure 15] FIG. 15 is an example of a screen 1500 showing the contribution of each feature amount in the learning history information of a low-ability user to the learning suitability. [Figure 16] FIG. 16 is an example of a screen 1600 showing the results of an AB test regarding whether the feature amount "number of problem-solving alerts per time" contributes to the learning adaptability. [Figure 17] FIG. 17 is an example of a screen 1700 showing the results of an AB test regarding whether or not the feature amount "number of stumble alerts per unit time" contributes to the learning adaptability. [Figure 18] FIG. 18 is an example of a screen 1800 showing the results of an AB test regarding whether the feature "number of units passed on the first try per unit time@non-target units" contributes to the learning adaptability. [Figure 19] FIG. 19 is an example of a screen 1900 showing the results of an AB test regarding whether the feature amount "number of short-term goals achieved per unit time" contributes to the learning adaptability. [Figure 20] FIG. 20 is an example of a screen 2000 showing the results of an AB test regarding whether the feature amount "cumulative top recommendation selection rate" contributes to the learning adaptability. [Figure 21] FIG. 21 is an example of a screen 2100 showing the results of an AB test regarding whether the feature amount "number of educational material list execution alerts per hour" contributes to the learning suitability. [Figure 22]FIG. 22 is an example of a screen 2200 showing the results of an AB test regarding whether or not the feature "learning time per active days" contributes to learning adaptability. [Diagram 23] FIG. 23 is an example of a screen 2300 showing the results of an AB test regarding whether the feature "study time per number of non-target units passed in the most recent week" contributes to the learning fitness. [Figure 24] FIG. 24 is an example of a screen 2400 showing an ALE plot of the feature "number of problem-solving alerts per unit time" versus the learning adaptability. [Diagram 25] FIG. 25 is an example of a screen 2500 showing an ALE plot of the feature "number of unit stumble alerts per unit time" versus learning adaptability. [Figure 26] FIG. 26 is an example of a screen 2600 showing an ALE plot of the feature "Number of units passed first time per unit time@non-target units" versus learning fitness. [Figure 27] FIG. 27 is an example of a screen 2700 showing an ALE plot of the feature "number of short-term goals achieved per unit time" versus learning fitness. [Figure 28] FIG. 28 is an example of a screen 2800 showing an ALE plot of the feature "cumulative top recommendation selection rate" versus the learning relevance. [Figure 29] FIG. 29 is an example of a screen 2900 showing an ALE plot of the feature "number of times learning material list is performed and number of alerts per unit time" against learning suitability. [Diagram 30] FIG. 30 is an example of a screen 3000 showing an ALE plot of the feature "learning time per active day" versus learning fitness. [Diagram 31] FIG. 31 is an example of a screen 3100 showing an ALE plot of the feature "total hint viewing rate" versus learning fitness. [Diagram 32] FIG. 32 shows an example of a feature setting screen 3200. [Diagram 33] FIG. 33 is an example of a display screen 3300 for the model analysis results. [Diagram 34] FIG. 34 is an example of a unit relation diagram 3400. [Diagram 35]FIG. 35 is an example of a learning compatibility display screen 3500. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a user management system, a user management method, and a program according to the present technology will be described with reference to the drawings. The user management system according to the present technology predicts the learning suitability of a user who uses an educational service for the educational service. The predicted learning suitability of the user for the educational service can be provided to, for example, a service provider (e.g., an educational facility terminal) that provides the educational service to the user.

[0010] [User Management System] FIG. 1 is an example of an overall configuration diagram of an overall user management system 100 according to an embodiment. The user management system 100 comprises one or more management servers 101, one or more user terminals 102, and one or more educational facility terminals 103. The one or more user terminals 102 and the one or more educational facility terminals 103 are each configured to be connectable to one or more management servers 101 via a network. The network may be wired or wireless, and each terminal can transmit and receive information to and from each other via the network. The educational facility terminal 103 may be provided separately from the management server 101, or may be provided integrally with the management server 101.

[0011] The management server 101, the user terminal 102, and the educational facility terminal 103 in the user management system 100 (hereinafter, the management server 101, the user terminal 102, and the educational facility terminal 103 may be simply referred to as "terminals"), for example, a mobile terminal such as a smartphone, a tablet, a mobile phone, or a personal digital assistant (PDA), or a wearable terminal such as a glasses-type (including goggles-type), a wristwatch-type, or a clothing-type. Each terminal may be a stationary or mobile computer, or a server located on a cloud or a network. From the viewpoint of functionality, each terminal may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, each terminal may be a combination of a plurality of these terminals. For example, a combination of one smartphone and one wearable terminal may function logically as one terminal. Each terminal may also be an information processing terminal other than these.

[0012] Each terminal of the user management system 100 may optionally include a processor that executes an operating system, applications, programs, etc., a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), a flash memory, etc., a communication control unit such as a network card, a wireless communication module, a mobile communication module, etc., an input device that detects motion by capturing images of a touch panel, a keyboard, a mouse, a voice input device, a motion controller, a camera unit, etc., and an output device such as a monitor, a display, a printer, a voice output device, an oscillator, etc. The input device may include a sensor such as a GPS, a gyro sensor, an acceleration sensor, etc. The output device may be a device or a terminal that transmits information to be output to an external monitor, a display, a printer, a device, etc.

[0013] The main memory stores various programs and applications (software modules), and the processor executes these programs and applications to realize each functional element of the overall system. Each module may be implemented by one or more programs or applications. Each module may be implemented by an independent program or application, or may be implemented as a subprogram or function within an integrated program or application. Each module may be implemented as hardware by integrating circuits or employing a microcomputer (hardware module).

[0014] Furthermore, each module may be realized by a single processor or by multiple processors. Also, each module may be provided in a single terminal (including a management server) or may be provided separately in two or more terminals (including a management server) connected to each other via a network. Also, each module may be provided in each or any one or more of two or more terminals (including a management server) connected to each other via a network. Cases in which some modules are realized in a different country from the other modules are also considered.

[0015] In this specification, each module is described as an entity (subject) that performs processing, but in reality, processing is performed by a processor executing various programs, applications, and the like.

[0016] The auxiliary storage device stores various databases (DB). A "database" is a set of data that is organized and collected so as to be compatible with any data operation (e.g., extraction, addition, deletion, overwriting, etc.) from the processor or an external computer. The auxiliary storage device is a functional element (storage unit) that stores one or more sets of data. There are no limitations on the method of implementing the database, and it may be, for example, a database management system, spreadsheet software, or text files such as XML and JSON. Some or all of this information may be stored in a relational database or a non-relational database. The database may be provided independently in a state that is connectable to the processor, etc.

[0017] The user management system 100 evaluates an index (hereinafter, sometimes simply referred to as "learning suitability") indicating whether a user is fully enjoying an educational service that supports the user's learning. The learning suitability to be evaluated may be, for example, the degree to which the user uses the educational service for a period longer than a predetermined period from the time of evaluation, or the degree to which the user stops using the educational service within a predetermined period from the time of evaluation. Alternatively, the learning suitability may be an index that quantitatively evaluates the results of learning, such as the degree of improvement in the user's grades, the number of passes, or the frequency of study. Furthermore, the learning suitability may be an index such as the user's satisfaction with the learning service, obtained by a questionnaire survey of the users, etc.

[0018] The learning contents supported by the educational service are not particularly limited, and may be, for example, learning contents such as Japanese language (including classics, modern Japanese, etc.), arithmetic (including mathematics, etc.), science (including chemistry, physics, biology, etc.), social studies (including world history, Japanese history, geography, civics, etc.), foreign language (including English, German, etc.), music, calligraphy, etc., learned at various schools, etc. Furthermore, learning contents supported by the educational service may be learning specialized academic subjects such as information engineering, fashion business, physical / occupational therapy, learning of exercises such as gymnastics and e-sports, learning of academic hobbies, learning for various qualification examinations, and various other types of learning.

[0019] Furthermore, the embodiment of the educational service is not particularly limited, and may be, for example, a service provided at an educational support facility such as a school or a cram school, or may be a service provided online or offline at any location not limited to these educational support facilities. Furthermore, the teaching materials used in these educational services are not limited to those provided online, and may be paper textbooks, DVD textbooks, tablet teaching materials equipped with learning applications, and the like, provided offline.

[0020] In the following, the present technology will be described taking as an example a case where the user management system 100 evaluates the user's learning suitability for the online learning service provided by the management server 101.

[0021] [Management Server] FIG. 2 illustrates an example of the hardware configuration of the management server 101. The management server 101 is an element that manages this user management system 100. The management server 101 is typically a terminal used by an administrator who manages and operates this user management system 100. The management server 101 is configured, for example, by a computer server or the like that is placed on a cloud.

[0022] The management server 101 includes a main storage device 201 and an auxiliary storage device 202. The management server 101 also includes a processor 203, an input device 204, an output device 205, and a communication control unit 206 as described above.

[0023] The main memory device 201 stores programs and applications such as a user terminal management module 211, an educational facility terminal management module 212, a learning execution module 213, an evaluation module 214, an output module 215, a machine learning model generation module 216, and an advice generation module 217. Each functional element of the management server 101 is realized by the processor 203 executing these programs and applications stored in the main memory device 201.

[0024] The advice generation module 217 is not an essential component of the present technology, but may be an additional element. The implementation method of each module in the main storage device 201 is not limited to the embodiment, and each module may be realized by a plurality of servers or cloud applications cooperating with each other.

[0025] The auxiliary storage device 202 stores information necessary for the operation of the user management system 100. The auxiliary storage device 202 stores, for example, user information 500, learning unit information 600, learning content information 700, learning history information 800, educational facility information 900, and machine learning model 220. Details of this information will be described later. Note that the implementation method of each piece of information stored in the auxiliary storage device 202 is not limited to this embodiment, and each piece of information may be realized in a form in which it is distributed and arranged in multiple database servers or cloud applications.

[0026] Each functional element of the management server 101 will now be briefly described. The user terminal management module 211 manages the operation of the user terminal 102. The user terminal management module 211 cooperates with a user learning management module 311 of the user terminal 102 to control basic operations for executing the learning support service in the user terminal 102.

[0027] For example, the user terminal management module 211 cooperates with the user learning management module 311 to output (display) a login screen, a user information management screen, and the like used for learning management executed by the management server 101 on an output device 305 such as a display of the user terminal 102. The user terminal management module 211 also cooperates with the user learning management module 311 of the user terminal 102 to acquire input information input from the user terminal 102 via these display screens. The user terminal management module 211 operates based on the acquired information (various information, instructions, and the like) and outputs (records) the acquired information to the auxiliary storage device 202 as user information 500, and the like.

[0028] The education facility terminal management module 212 manages the operation of the education facility terminal 103. The education facility terminal management module 212 cooperates with the education facility terminal management module 411 of the education facility terminal 103 to control basic operations for executing a learning support service in the education facility terminal 103. The education facility terminal management module 212 also cooperates with the education facility terminal management module 411 of the education facility terminal 103 to control basic operations for evaluating a user's learning suitability in the education facility terminal 103 using the user management system 100.

[0029] For example, the education facility terminal management module 212 cooperates with the education facility terminal management module 411 to output (display) a login screen for the education facility service provided by the management server 101, a lecturer information management page, various user learning management pages, etc., to the education facility terminal 103. The education facility terminal management module 212 also cooperates with the education facility terminal management module 411 of the education facility terminal 103 to acquire input information entered from the education facility terminal 103. The education facility terminal management module 212 operates based on the acquired information (various information, instructions, etc.), and outputs (records) the acquired information to the auxiliary storage device 202 as education facility information 900, etc.

[0030] The learning execution module 213 controls the operation for executing learning by the learning support service in the user terminal 102. For example, the learning execution module 213 outputs learning content to the user terminal 102. For example, the learning execution module 213 manages the output of learning content, connects to learning unit information 600, learning content information 700, learning history information 800, etc. according to the user's learning content, and outputs the learning content to the user terminal 102. The learning execution module 213 cooperates with the user learning execution module 312 of the user terminal 102, and outputs learning content to the output device 305 such as a display of the user terminal 102, for example, based on an input (e.g., an instruction) from the user terminal 102. The learning execution module 213 cooperates with the user learning execution module 312 of the user terminal 102, and obtains information related to the user's learning history based on an input (e.g., an answer) from the user terminal 102, and outputs (records) the obtained information to the auxiliary storage device 202 as learning history information 800, etc. The user learning execution module 312 is an example of an acquisition unit in the present technology.

[0031] The evaluation module 214 evaluates whether the user is fully enjoying the learning effect of the educational service based on the learning history information. For example, in response to a request from the educational facility terminal 103, the evaluation module 214 evaluates the learning suitability using an index indicating whether the user is fully enjoying the learning effect of the educational service. As a specific example, the evaluation module 214 evaluates whether the user will use the educational service for a period longer than a predetermined period. The method of evaluating the learning suitability will be described later.

[0032] The output module 215 outputs the obtained degree of continuous use of the educational service. The output module 215, for example, cooperates with the educational facility terminal management module 411 of the educational facility terminal 103 to display (output) the learning suitability of the educational service for the user on the output device 405 of the educational facility terminal 103. The output module 215 outputs, for example, an evaluation result indicating whether or not the user will use the educational service for longer than a predetermined period of time. The output module 215 also outputs advice generated by the advice generation module 217 (described later) to the output device 405 of the educational facility terminal 103, for example.

[0033] The machine learning model generation module 216 generates a machine learning model 220 that evaluates the degree to which the user will continuously use this educational service. The machine learning model generation module 216 generates the machine learning model 220 as a trained model by having the learning model learn the user's learning history information 800 as input and the user's learning suitability for the educational service as output.

[0034] The advice generation module 217 generates advice for managing the user. For example, the advice generation module 217 generates advice to be provided to the user when the instructor intervenes in the user's learning. For example, specifically, the advice generation module 217 generates recommended advice in natural language that the instructor should give to the user based on the learning suitability evaluated by the evaluation module 214. The advice generation module 217 also generates information on the user's learning tendency in natural language based on the learning suitability evaluated by the evaluation module 214.

[0035] [User device] FIG. 3 shows an example of the hardware configuration of the user terminal 102. The user terminal 102 is a terminal used by a user (a learner, for example, a student, etc.) who studies using the educational service provided by the management server 101. The user terminal 102 is composed of a terminal such as a mobile information terminal such as a tablet, a notebook PC, or a desktop PC, for example.

[0036] The user terminal 102 includes a main memory device 301 and an auxiliary memory device 302. The user terminal 102 also includes a processor 303, an input device 304, an output device 305, a camera 306, and a communication control unit 307 as described above. The main memory device 301 stores programs and applications such as a user learning management module 311 and a user learning execution module 312, and each functional element of the user terminal 102 is realized by the processor 303 executing these programs and applications.

[0037] The user learning management module 311 controls the basic operations of the user terminal 102. The user learning management module 311 cooperates with, for example, the user terminal management module 211 of the management server 101 to control the basic operations for utilizing the educational services provided by the management server 101.

[0038] The user learning execution module 312 controls the operation for executing learning by the educational service provided by the management server 101 in the user terminal 102. For example, the user learning execution module 312 cooperates with the learning execution module 213 of the management server 101 to output (display) various learning contents to the user terminal 102 and transmit user input to the management server 101. For example, the user learning execution module 312 cooperates with the learning execution module 213 of the management server 101 to output (transmit) answers and instructions to the user terminal 102 and transmit user learning history information to the management server 101.

[0039] The auxiliary storage device 302 stores information necessary for the operation of the user terminal 102. For example, the auxiliary storage device 302 stores user information 500, learning unit information 600, learning content information 700, learning history information 800, etc. This information may be some or all of the information related to the user's learning among the user information 500, learning unit information 600, learning content information 700, learning history information 800, etc. stored in the management server 101.

[0040] [Educational facility terminal] FIG. 4 shows an example of the hardware configuration of the educational facility terminal 103. The educational facility terminal 103 is an example of a terminal used by a service provider that introduces the educational service provided by the management server 101 and provides the educational service to users. The service provider may be the same as or different from the administrator of the management server 101. The educational facility terminal 103 may be provided separately from the management server 101, or may be provided integrally with the management server 101. In this embodiment, for example, a lecturer of a cram school (service provider) that has introduced this educational service uses the educational facility terminal 103. The educational facility terminal 103 is composed of a terminal such as a smartphone, a tablet, a notebook PC, or a desktop PC. The educational facility terminal 103 may typically be installed in a cram school, but does not necessarily have to be installed in a physical educational facility.

[0041] The main memory device 401 stores programs and applications such as an educational facility terminal management module 411 and an educational facility terminal execution module 412, and each functional element of the educational facility terminal 103 is realized by the processor 403 executing these programs and applications.

[0042] The education facility terminal management module 411 controls the basic operations of the education facility terminal 103. For example, in cooperation with the education facility terminal management module 212 of the management server 101, the education facility terminal management module 411 controls the basic operations for managing the learning performed by the user using the education service provided by the management server 101. For example, in cooperation with the education facility terminal management module 212, the education facility terminal management module 411 outputs various user learning management pages, etc. to the education facility terminal 103 and transmits input from the instructor to the management server 101.

[0043] The education facility terminal execution module 412 controls operations related to the provision of education services in the education facility terminal 103. The education facility terminal execution module 412 cooperates with, for example, the learning execution module 213 of the management server 101 to support user learning. For example, the education facility terminal execution module 412 cooperates with the learning execution module 213 and the output module 215, etc., to obtain each user's learning suitability for the education service and display it on the output device 405.

[0044] The auxiliary storage device 402 stores information necessary for the operation of the educational facility terminal 103. The auxiliary storage device 402 stores, for example, user information 500, learning unit information 600, learning content information 700, learning history information 800, educational facility information 900, etc. This information may be some or all of the information related to the learning support of the instructor among the user information 500, learning unit information 600, learning content information 700, learning history information 800, educational facility information 900, etc. stored in the management server 101.

[0045] 5 to 9 show various types of information stored in the management server 101. FIG. FIG. 5 is an example of user information 500. User information 500 is information about a user (e.g., a student) who studies using the educational services provided by management server 101. User information 500 includes, for example, user ID, user display ID, name, affiliated school, grade, desired school, address, course ID, course taken, and other information, and values ​​such as those shown as sample values ​​520 are input for each field name (item name) 510. The contents indicated by sample values ​​520 are written in a summary 530 column.

[0046] The user ID is a symbol given to each user to identify the user, and information about each user is basically linked by this user ID. The user ID is a primary key referenced from other information. The user display ID is a display ID for the user displayed on the screen of the management server 101, the user terminal 102, etc., and can be set arbitrarily. The user display ID can be set arbitrarily by the user using a predetermined symbol within a range of predetermined conditions (for example, 10 characters or less). The name is the user's name information. The school and grade are information indicating the name of the school (educational institution) to which the user currently belongs and the grade to which the user belongs. For example, "JH2" indicates the second year of junior high school. The school of choice is the name of the school to which the user wishes to advance, for example, the name of a junior high school, high school, university, etc. for which the user plans to take an entrance exam. The school of choice may include information such as faculty and department.

[0047] The address is information indicating the address or residence of the user, or the address or residence of the user's guardian. The course ID indicates an identification ID of a course the user is taking among the learning courses set in this educational service. The course ID stores the course ID of the learning unit information 600, and is linked to the content of the learning unit information 600 by this course ID. The course to be taken indicates a course the user is taking among the learning courses set in this educational service. The content of the user information 500 is not limited to this example. The user information 500 may also include other information such as information on the user's guardian, various procedural information, the user registration date (contract date), the start date of use of the educational service, and the cancellation date of the educational service.

[0048] FIG. 6 is an example of learning unit information 600. The learning unit information 600 is information about a learning course set in this educational service, and the learning course is prepared for each subject, grade, and textbook used in school, for example. The learning subjects in this embodiment include studies in multiple fields, and each field includes multiple learning units. A "learning unit" is a unit of learning set for each group of learning elements that is a coherent whole. A "learning unit" may or may not match a unit specified in a textbook or the like (i.e., a unit according to a curriculum guideline). In this educational service, for example, one or more sets of learning contents are prepared for the study of each learning unit. A set of learning contents may include, for example, one or more lecture contents, exercise contents, homework contents, etc. Note that the learning unit information 600 may be further divided in consideration of, for example, the learning progress, etc., or may be set from other perspectives.

[0049] The learning unit information 600 includes information such as course name, course ID, field, field ID, unit, unit ID, content, content ID, backward unit ID, difficulty level, estimated time to master, etc., and values ​​such as those shown as sample values ​​620 are entered for each field name 610. The course name and course ID are a name and an ID for identifying a course to be studied. In this embodiment, the course name may be the same as the name of a course to be studied, for example.

[0050] The field and field ID are the name of a category and its ID when each course (which may be a subject) is classified according to its content. The unit and unit ID are the name of a category and its ID when each field is further classified according to its content. The content and content ID are the type of learning content prepared for each unit and its ID. In this embodiment, for example, lecture content including a lecture video of the learning unit, exercise content including exercise problems of the learning unit, homework content including homework problems of the learning unit, etc. are prepared as learning content.

[0051] The backward unit ID is an ID for identifying data indicating the relationship between a certain learning unit and its backward unit, which is provided for each learning unit. A backward unit is a learning unit in which prerequisite knowledge for understanding a certain learning unit is provided. FIG. 34 is an example of a unit relation diagram showing the connection between a certain learning unit and its backward unit. Here, the unit relation diagram of "unit 1" is shown. In the unit relation diagram, the arrows connecting the learning units are oriented so that the learning unit to be understood is the end point and the backward unit for the learning unit to be understood is the starting point. Therefore, if a certain learning unit cannot be understood, the understanding of the learning unit can be improved by going back to the learning unit (backward unit) that is the starting point of the arrow according to the unit relation diagram. Although there are three backward units for unit 1 in FIG. 34, there may be one, two, or more backward units for one learning unit. In addition, a learning unit may be a learning unit that is a different learning unit from the learning unit. In addition, a learning unit may be a learning unit that is a different learning unit. The learning units may be learning units in different learning fields (e.g., different subjects, learning fields in different grades, etc.).

[0052] The difficulty level is information indicating the difficulty level when learning each unit, and can be determined based on, for example, the percentage of correct answers in past and / or current tests for the unit. The standard learning estimate time is information that is an estimate of the time required to learn the learning content of each unit, and can be determined based on, for example, the time required by any student in the past and / or present to learn the learning content of the unit. As an example, the learning estimate time can be calculated based on a standard learning time (for example, average learning time) calculated based on big data based on past learning history. Whether or not the learning content has been mastered can be determined, for example, by obtaining a passing score once or multiple times in a confirmation test for the unit. The content of the learning unit information 600 is not limited to this example.

[0053] FIG. 7 is an example of study content information 700. Learning content information 700 is information related to learning content prepared by this educational service. Learning content information 700 includes, for example, content ID, data URL, question ID, correct answer ID, point allocation, standard answer time, question frequency, etc., and values ​​such as those shown as sample values ​​720 are entered for each field name 710. The contents indicated by sample values ​​720 are also written in the summary 730 column.

[0054] The data URL is information that indicates the storage location of the data of the learning content used in the study of each unit. The data URL is, for example, the URL of a storage folder that stores various data that constitutes the learning content for each unit.

[0055] The question ID and the correct answer ID are the IDs of the questions and the correct answers given in the learning content of the unit, respectively. The points allocated indicate the points allocated to each question. The standard answer time indicates the time necessary and sufficient for a user who has the ability to solve each question to solve the question.

[0056] The question frequency is information determined based on the question tendency of past exam questions, etc., and may be information expressed by a numerical value (e.g., -1 to +1 or 0 to 5) that increases as the question frequency increases. Also, the question frequency may be input as 100% for learning content that is always tested, or 40% for learning content that has a 40% probability of being tested.

[0057] The frequency of questions may vary according to the current learning goal, such as a quiz for each unit, a midterm test, a final test, an entrance exam, etc. The frequency of questions may be determined in advance, for example, by analyzing statistical data on similar past tests, machine learning, etc. The learning content information 700 is linked to the learning unit information 600, for example, via a field ID, etc. The content of the learning content information 700 is not limited to this example.

[0058] FIG. 8 is an example of learning history information 800. Learning history information 800 is information on the learning records of users who have used educational services, and is provided, for example, for each user and each unit. Learning history information 800 includes, for example, user ID, content ID, start date and time, end date and time, completion date and time, number of times the unit has been studied, score, score rate, retention rate (which may be the degree of mastery), number of hint views, and other information, and values ​​such as those shown as sample values ​​820 are input for each field name 810. The contents indicated by sample values ​​820 are described in the summary 830 column.

[0059] The start date and time, end date and time, completion date and time, and number of times of unit study indicate information on the start date and time of study using the learning content of the unit, the end date and time, the completion date and time of study, and the number of times the study was completed, respectively. Note that completion means, for example, that the study using the learning content has been completed, and completion means, for example, that the study using the learning content has been completed (finished) by satisfying a predetermined completion criterion. Therefore, in order to "complete" the study using one learning content, the study using the learning content may be completed (conducted) multiple times. The completion criterion may be determined, for example, based on the correct answer rate, score rate, retention rate, etc. described later (for example, correct answer rate of 80% or more, score rate of 90% or more, retention rate of 95% or more, etc.). The number of times of unit study completion indicates the number of times the study of the unit has been completed for a unit that can be studied multiple times, and may be information such as 0 times (not completed), 1 time, 2 times or more, etc.

[0060] The score rate is the percentage of the user's score out of the total score allocated for the content. The retention rate is an index showing the degree of mastery of the learning content of the unit, and can be determined, for example, based on the correct answer rate, score rate, number of times of implementation, etc., when learning is performed using the learning content of the unit. The retention rate can be determined, for example, so that 0% is set for no learning, and the higher the correct answer rate, score rate, and number of times of implementation, the higher the retention rate (approaching 100%).

[0061] The number of times a hint has been viewed indicates the number of times a hint for solving a question, which is provided for that question, has been viewed. The number of times a hint has been viewed is, for example, a value that serves as an index showing the user's level of understanding of the question. The learning history information 800 may also include records of the user's learning and learning behavior. Other learning history information 800 will be described later. The learning history information 800 is linked to the user information 500, for example, via a user ID. The learning history information 800 is also linked to the learning unit information 600, for example, via a content ID.

[0062] FIG. 9 is an example of educational facility information 900. The educational facility information 900 is information about instructors who manage and support users' learning using the user management system 100. The educational facility information 900 includes, for example, information such as an educational facility ID, an educational facility display ID, an affiliation ID, an offered course ID, and a responsible user ID, and values ​​such as those shown as sample values ​​920 are entered for each field name 910. The contents indicated by the sample values ​​720 are also written in the summary 930 column.

[0063] The educational facility ID is an ID given to each educational facility to identify the educational facility that has introduced this educational service. The educational facility display ID is an ID for displaying the educational facility, which is displayed on the screen of the management server 101, the educational facility terminal 103, etc. The affiliation ID indicates, for example, information regarding the group or school building to which the educational facility belongs. The course ID indicates a course offered at the educational facility among the learning courses set in the educational service. The user ID in charge is set, for example, for each course offered at the educational facility, and indicates the ID of the user (student) who takes the course at the educational facility. The educational facility information 900 is linked to the learning history information 800, for example, via the user ID and the offered course ID.

[0064] [User management method] A user management method according to the present technology will be described based on an operation example of the user management system 100. First, the recording of learning history information will be described, and then the user management method will be described.

[0065] <Recording of learning history information> When using this educational service, for example, the user can determine in advance the learning unit to be studied in the coming week as a learning plan. The user then specifies the learning unit and learning content to be studied sequentially in accordance with this learning plan on the user terminal 102. The user terminal management module 211 cooperates with the user learning management module 311 of the user terminal 102, and receives information specifying, for example, the learning unit and learning content to be studied by the user.

[0066] The learning execution module 213 cooperates with the user learning execution module 312 of the user terminal 102, and outputs (displays) the specified learning content on the display or the like of the user terminal 102 based on the received information. Here, the user answers, for example, questions included in the learning content. The learning execution module 213 receives the answers from the user terminal 102, judges whether the answers are correct, and outputs the result to the user terminal 102.

[0067] The learning execution module 213 also measures and records the time taken by the user to answer the questions posed. The learning execution module 213 calculates and stores information, for example, on the number of correct answers and the correct answer rate (for example, the ratio of the number of correct answers to the total number of questions) from the result of judging whether the user's answers are correct or not. The learning execution module 213 also calculates the score and score rate (for example, the ratio of the total score to the total score) from the user's answers to the questions posed, based on the question ID, correct answer ID, score allocation, etc.

[0068] Furthermore, in addition to the problem, the learning execution module 213 displays a hint button for displaying a hint for solving the problem on the display of the user terminal 102. When the user wishes to view a hint for solving the problem, the user, for example, selects the hint button displayed on the display. The learning execution module 213, for example, accepts the selection of the hint button from the user terminal 102, and outputs a hint for solving the problem to the user terminal 102. Furthermore, when an event occurs in which the user selects the hint button, the learning execution module 213 records that an event of viewing a hint for the problem has occurred.

[0069] In this way, the user performs learning using the predetermined learning content. Learning execution module 213 records various events that occur during the user's learning as learning history information 800 by accepting information associated with the user's operation of user terminal 102.

[0070] In addition to the above, the learning history information 800 recorded by the learning execution module 213 may include, for example, the number of stumble alerts per unit time, the number of stumble alerts per unit time, the number of alerts of the number of times the learning material list has been executed per unit time, the cumulative top recommendation selection rate, the study time per active day, the number of non-target units passed per unit time, the number of short-term goals achieved per unit time, and the study time per number of non-target units passed in the past week (see, for example, FIG. 32). Details of these pieces of information will be described later.

[0071] <Estimation of the learning compatibility of educational services 1> The user management system 100 can estimate the learning suitability of the educational service. For example, the instructor can request the management server 101 via the educational facility terminal 103 to provide an estimation result of the learning suitability of the educational service for the user who is the target of learning support.

[0072] FIG. 10 is an example of a learning fitness estimation flow 1000. The user management method according to the present technology typically includes acquiring learning history information of a user (S1010), evaluating whether the user has sufficiently enjoyed the learning effect of the educational service based on the acquired learning history information (S1020), and outputting the obtained evaluation result (S1030). Whether the user has sufficiently enjoyed the learning effect of the educational service can be evaluated, for example, by utilizing an index indicating whether the user has sufficiently enjoyed the learning effect of the educational service and whether this index exceeds a predetermined threshold.

[0073] In this case, if a user is able to fully enjoy the learning benefits of an educational service, it can be said that the user will continue to use the educational service. Therefore, here, the degree to which a user continuously uses an educational service (degree of continuous use) is estimated as an example of learning suitability, which is an index showing whether a user is able to fully enjoy the learning benefits of an educational service.

[0074] Specifically, for example, an instructor requests the user's learning suitability for the educational service from the educational facility terminal 103. The management server 101 then receives information requesting the user's learning suitability for the educational service from the educational facility terminal 103, and the evaluation module 214 acquires the user's learning history information 800 from, for example, the auxiliary storage device 202 (S1010). Next, the evaluation module 214 estimates the learning fitness based on the acquired learning history information (S1020).

[0075] In the following, we will describe in detail an example of estimating the degree to which a user continuously uses an educational service (degree of continuous use) as an example of learning suitability, which is an indicator of whether a user is fully enjoying the learning benefits of an educational service. Fig. 13 is an example of a screen 1300 showing the contribution of each feature in the learning history information to the learning suitability (degree of continued use). As shown in Fig. 13, various information stored in the auxiliary storage device 202 includes, for example, information showing a high correlation with users with low learning suitability, in other words, with the degree of continued use, which is an index in this embodiment, and users who cancel the educational service early. The evaluation module 214 uses such information (for example, the user information 500 and the learning history information 800) to estimate the learning suitability (degree of continued use) of the educational service for each user.

[0076] Regarding the estimation of the degree of continuous use, which is an example of learning suitability, the typical period of use of an educational service may be at most 3 to 6 years depending on the changes in the user's educational environment, and the degree to which the use continues beyond this period is inevitably low. Therefore, the degree of continuous use of an educational service may be understood, for example, as the degree to which the educational service is continued for a period longer than a predetermined period, or, for example, as the degree to which the educational service is terminated within a predetermined period. Examples of the predetermined period include, but are not limited to, 3 months, 4 months, 6 months, 1 year, 2 years, 3 years, etc.

[0077] For example, for user information 500 and learning history information 800 that are highly correlated with the degree of continued use, the management server 101 stores in advance the correspondence between the contents of the information and the degree of contribution to the degree of continued use as a correlation function, a correspondence map, or the like based on a method such as regression analysis. The evaluation module 214 estimates the degree of continued use of a given user by applying the user information 500 and learning history information 800 of the given user to the prepared correlation function, correspondence map, or the like.

[0078] Here, the evaluation module 214 uses at least the learning history information 800 to estimate the continued use degree. The evaluation module 214 preferably uses a combination of the user information 500 and the learning history information 800 to estimate the continued use degree. The evaluation module 214 stores the estimated continued use degree of the user in the user information 500, for example.

[0079] The output module 215 cooperates with, for example, the education facility terminal execution module 412 of the education facility terminal 103 to display the obtained degree of continuous use of the education service for each user on the output device 405 of the education facility terminal 103 or the like. With this configuration, the instructor can support users with low learning suitability for the educational service by increasing the level of assistance intervention through interviews, etc., so that the users can appropriately enjoy the learning effect.

[0080] <Estimation of the learning compatibility of educational services 2> The learning relevance of an educational service may be estimated using, for example, a machine learning model 220 . Fig. 11 is an example of a machine learning model generation flow 1100. Fig. 12 is another example of a learning fitness estimation flow 1200.

[0081] In this embodiment, the degree to which a user continuously uses an educational service (degree of continuous use) is estimated as an example of the learning suitability, which is an index showing whether the user is sufficiently enjoying the learning effect of the educational service. The user management method of the present technology typically includes acquiring learning history information of the user (S1210), evaluating the degree to which the user continuously uses educational services (degree of continuous use) using machine learning model 220 based on the acquired learning history information (S1220), and outputting the obtained evaluation result (S1230).

[0082] <Machine learning model generation> The machine learning model can be prepared based on, for example, a machine learning model generation method according to the present technology, but is not limited thereto. The machine learning model generation method typically includes acquiring learning history information (S1110), and performing machine learning based on the acquired learning history information to generate a machine learning model (S1120).

[0083] Specifically, for example, the machine learning model generation module 216 acquires learning history information stored in the auxiliary storage device 202 (S1110). Then, the machine learning model generation module 216 uses the acquired learning history information as input and the learning compatibility as output to train a learning device, thereby generating a machine learning model (S1120). As the learning compatibility, for example, information regarding the contract period from the contract date to the cancellation date of the educational service (for example, information indicating that the contract period is less than 120 days) can be used.

[0084] Although not limited thereto, the machine learning model generation module 216 may generate a machine learning model by training a learner based on a decision tree algorithm such as XGBoost, Catboost, or LightGBM.

[0085] Furthermore, the machine learning model generation module 216 uses at least the learning history information 800 as input data for the teacher data of machine learning. The machine learning model generation module 216 preferably uses a combination of the user information 500 and the learning history information 800 as input data for the teacher data. The machine learning model generation module 216 may use, for example, the following learning history information that has a high contribution to the learning suitability as input data for the teacher data.

[0086] In the above embodiment, the degree to which the user continuously uses the educational service (degree of continuous use) is used as the objective variable of the learning suitability. However, the present invention is not limited to this. As the objective variable of the learning suitability, for example, the learning frequency indicating that the number of times of learning is a certain number or more per week, the total learning time indicating that the learning time is a certain number or more per week, the learning progress indicating that the number of passed units is a certain number or more per week, the user satisfaction indicating the user's satisfaction with the educational service, etc. may be used as the objective variable of the learning suitability.

[0087] FIG. 32 is another example of the feature amount setting screen 3200. For example, the machine learning model generation module 216 may be configured to display the feature amount setting screen 3200 on the display of the educational facility terminal 103. The machine learning model generation module 216 displays a feature amount type column 3210, a feature amount column 3220, and a selection check column 3230 on the feature amount setting screen 3200. The machine learning model generation module 216 displays the feature amounts by type, for example. The instructor selects the corresponding feature amount by checking the selection check column 3230 by selecting with the cursor 3250 on the feature amount setting screen 3200. The machine learning model generation module 216 may be configured to receive information on the feature amount selected by the instructor on the feature amount setting screen 3200, and generate the machine learning model 220 or perform additional learning using the received feature amount as input data. The machine learning model generation module 216 may display the features on the feature setting screen 3200 in a state in which the contribution of the features can be seen, for example, as in a model analysis result display screen 3300 shown in FIG. 33, and accept a selection.

[0088] (1) Learning difficulties The learning stumbling block information is information that indicates some kind of stumbling block in learning. Examples of the learning stumbling block information include an index related to browsing hints for answering a question, an index related to the time required to answer a question, and the like. For example, the learning stumbling block information may be the number of problem stumbling block alerts per unit time, the number of unit stumbling block alerts per unit time, and the like. Details will be described later.

[0089] (2) Non-target unit learning information The non-target unit learning information is information on learning of a unit different from the original learning unit, such as a retroactive unit. Examples of the non-target unit learning information include an index that focuses on the progress of the non-target unit learning and indicates how smoothly the non-target unit learning was completed, and an index that indicates how difficult it was to complete the non-target unit learning. For example, one example may be the number of units passed on the first try per unit time @ non-target units. Details will be described later.

[0090] (3) Learning efficiency information The learning efficiency information is information that indicates how efficiently the learning has progressed. Examples of the learning efficiency information include an index that indicates the degree of goal achievement (preferably, the degree of short-term goal achievement), a correct answer rate, and an index that indicates the number of learning units passed per unit time. For example, the information may be the cumulative top recommendation selection rate, the number of times the learning material list is executed per unit time, the number of alerts, the study time per active day, the number of short-term goal achievements per unit time, the study time per number of non-target units passed in the last week, etc. Details will be described later.

[0091] The evaluation module 214 stores the estimated learning fitness of the user in the user information 500, for example. The output module 215 cooperates with, for example, the education facility terminal execution module 412 of the education facility terminal 103 to display the obtained learning suitability of the education service for each user on the output device 405 of the education facility terminal 103 or the like. With this configuration, the instructor can provide appropriate learning support to each user based on the learning suitability of the educational service for each user. For example, the instructor can appropriately intervene to increase the motivation of a user with a low learning suitability for the educational service to continue using the educational service.

[0092] [Example] <Creating a machine learning model> A machine learning model was generated using the following objective variables, explanatory variables, and machine learning algorithms. Objective variable: A binary flag indicating whether the contract was terminated within 120 days (early termination) or continued for 120 days or more Explanatory variables: Based on 320 different features of the learning history information, the following features were calculated and used as explanatory variables. 1. Cumulative feature value over the entire period 2. Cumulative feature amount from the first week of learning 3. Cumulative feature value for the past week since the last training Algorithm: LightGBM

[0093] In addition, the distribution of durations for those who cancelled their subscriptions to the educational service was as follows: 25%ile = 29 days, 50%ile = 121 days, and 75%ile = 281 days. In addition, among the features, those that are clearly correlated with the duration of use (e.g., total study time, total number of responses, etc.) were excluded because they would lead to leaks. To avoid bias, we excluded features that are difficult to implement policy interventions on and for which only a portion of users provided information, such as motivation surveys. Features related to post-pass activities (warm-up, confirmation tests, etc.) were excluded because they represent a long period of use of the educational service.

[0094] <Verification of machine learning models> The predictive accuracy of the obtained machine learning model was evaluated using a confusion matrix, and a ROC curve was created, with an AUC (Area Under Curve) of 0.892. A PR curve (Precision Recall Curve) was created, with a recall rate of 39.7% and a precision rate of 86.0%. This confirmed that it was possible to generate a machine learning model with extremely high predictive accuracy.

[0095] Next, we investigated the influence of explanatory variables on the objective variable of the machine learning model (explanation of predictions). The machine learning model generation module 216 is configured to output information (for example, a graph) that indicates the degree of contribution of the input data (i.e., explanatory variables, also called feature amounts) used to generate the machine learning model 220 to the prediction result of the machine learning model 200. Although not limited thereto, as an index that indicates the degree of contribution of each feature amount to the prediction result of the machine learning model 200, for example, an index called FI (Feature Importance), PDP (Partial Dependence Plot), SHAP (SHapley Additive exPlanation), etc. can be applied. This allows visual understanding of which explanatory variables the machine learning model considers important (is learning) and how each explanatory variable affects the predicted value.

[0096] (SHAP) The machine learning model generation module 216 of this embodiment is configured to use SHAP to display, in a bar graph, the feature importance of a predetermined number of features that significantly influenced the output among 500 or more types of input data as features, as shown in FIG. 13. The machine learning model generation module 216 calculates the sum of the absolute values ​​of the SHAP values ​​of each data, as shown in FIG. 13, to display the feature importance at the model level as a global index (similar to FIG. 14 and FIG. 15). The machine learning model generation module 216 may also display the feature importance at the model level as a display screen 3300 of the model analysis result, as shown in FIG. 33, for example.

[0097] SHAP is originally an index (local index) that evaluates how much each explanatory variable in a model contributes to a prediction at a sample level, and the degree of contribution (also called prediction contribution degree, SHAP value here) can take positive or negative values. Therefore, the machine learning model generation module 216 may plot the SHAP value for each data as a local index and display it in the form of a distribution diagram or the like. When interpreting the background of the prediction result of the machine learning model for each data, the machine learning model generation module 216 may apply indices such as LIME (Local Interpretable Modelagnostic Explanations) and ICE (Individual Conditional Expectation) in addition to SHAP.

[0098] Furthermore, the machine learning model generation module 216 may be configured to output information indicating the degree of contribution to a prediction result for a machine learning model trained using data satisfying a predetermined condition from the learning history information 800 as input data. The machine learning model generation module 216 may be configured to output information indicating the degree of contribution of a feature to a prediction result for a user with relatively high academic ability or a user with relatively low academic ability, as shown in Figs. 14 and 15, for example.

[0099] In addition, in Fig. 13, the machine learning model generation module 216 displays the feature amount that cannot be intervened to improve academic ability and its bar surrounded by a dotted line. In Fig. 14 and Fig. 15, the machine learning model generation module 216 extracts and displays the contribution degree of the feature amount other than the feature amount that cannot be intervened to improve academic ability. For example, for each information of the learning history information 800, the feature amount that can be intervened to improve academic ability and the feature amount that cannot be intervened may be distinguished in advance by tagging, and the machine learning model generation module 216 may be configured to be able to switch between a case where the feature amount that cannot be intervened is displayed (for example, Fig. 13) and a case where the feature amount is not displayed (for example, Fig. 14 and Fig. 15).

[0100] (AB testing) For features that were evaluated as having high feature importance by SHAP, we used AB testing to evaluate whether there was a correlation between the objective variable and the features. As shown in Figures 16 to 23, the machine learning model generation module 216 is configured to display the frequency of a response variable obtained based on certain feature data in a bar graph, as well as to display the difference between those frequencies and whether or not there is a significant difference between those frequencies.

[0101] (Sensitivity analysis) For features that were evaluated as having high feature importance by SHAP, the impact of changes in the features on the objective variable (the predicted value of the model) was evaluated by calculating the accumulated local effects (ALE) on the objective variable. The machine learning model generation module 216 is configured to be able to display ALE plots, which show the change in the degree of early cancellation when a certain explanatory variable is varied, as shown in FIGS.

[0102] (Model output results) The machine learning model generation module 216 may display the output result of the machine learning model (model output value) as a model output result in the form of a cumulative line graph from the feature value with the large SHAP value, with the output value (predicted value) as the start and end points, as shown in Fig. 33. Such a model output result is preferable because it allows visual understanding of what feature value affects the output value and how it affects it.

[0103] Below, we will explain the learning history information (features) that contribute highly to the learning adaptability of the generated machine learning model, with reference to the AB test results and / or ALE plots. In the following learning history information, "per unit time" means the sum of values ​​related to learning characteristics divided by a given learning time [hr]. Unless otherwise specified, such values ​​related to learning characteristics per unit time can be calculated based on any one of the following: the learning time required for each learning content, the learning time required for each type of learning content (e.g., lecture, exercise, homework, etc.), the learning time required for each learning unit, and the learning time required for the entire study.

[0104] Number of problem alerts per unit time The number of stumbling block alerts per unit time is an index that can evaluate the user's learning stumbling block per problem. For example, when the user's answer time for a problem exceeds a time obtained by adding a predetermined time (for example, a time of standard deviation x 2) to the standard answer time for the problem, the learning execution module 213 is configured to notify the educational facility terminal 103 of information indicating that it takes some time to solve the problem (in other words, the user is having trouble answering). The standard answer time may be anything that can be considered as a standard answer time for solving the problem, and may be, for example, an average or median answer time of users who have answered the same problem in the past, or an average answer time of users with a deviation score of around 50. For example, the learning execution module 213 counts the number of such notifications and stores the value divided by the time [hr] of using the educational service as the number of stumbling block alerts per unit time.

[0105] Fig. 16 is an example of a screen 1600 showing the results of an AB test on whether the feature "number of problem alerts per unit time" contributes to early cancellation (cancellation within 120 days; the same applies below). As shown in Fig. 16, the results of the AB test show that there is a significant difference. 24 is an example of a screen 2400 showing an ALE plot of the feature "number of problematic alerts per unit time" against early churn. When the number of problematic alerts per unit time becomes larger than a certain threshold, the degree of early churn tends to increase rapidly. Therefore, the machine learning model generation module 216 may include the number of problematic alerts per unit time in the input data.

[0106] Number of unit failure alerts per unit time The number of unit stumbling alerts per unit time is an index that can evaluate the user's stumbling in learning for each learning unit. The learning unit information 600 includes, for example, information on the standard unit learning completion time, which is the standard time required to complete learning for each learning unit. The learning execution module 213 can calculate the standard unit learning completion time for each learning unit based on the time required for a user who has completed the learning to complete the learning of the learning unit, and store it as the learning unit information 600.

[0107] The learning execution module 213 is configured to store information indicating that the user is having difficulty learning a learning unit, for example, when the user's learning time for a learning unit exceeds a standard unit learning completion time for that learning unit plus a predetermined time (for example, a time of twice the standard deviation). The learning execution module 213 counts the number of pieces of information indicating that the user is having difficulty learning this learning unit, and stores the value divided by the usage time [hr] of the educational service as the number of problem-solving alerts per unit time.

[0108] Fig. 17 is an example of a screen 1700 showing the results of an AB test regarding whether the feature amount "number of unit stumbling alerts per unit time" contributes to early cancellation. As shown in Fig. 17, the results of the AB test show that there is a significant difference. 25 is an example of a screen 2500 showing an ALE plot of the feature "number of unit stumbling alerts per unit time" for early cancellation. The number of unit stumbling alerts per unit time generally tends to increase the degree of early cancellation as the value increases. Therefore, the machine learning model generation module 216 may include the number of unit stumbling alerts per unit time in the input data.

[0109] Number of non-target units passed per unit time (Number of units passed on the first try per unit time @ non-target units) The number of non-target units passed per unit time can be an index that can evaluate the degree to which the user was not able to study as intended in this educational service. Here, the learning unit that the user is originally studying / trying to study is called the target unit, and the learning unit that the educational service recommends to study before studying the target unit is called the non-target unit. In this educational service, for example, a learning unit presented as a retroactive unit is an example of a non-target unit. For example, the learning execution module 213 stores the value obtained by dividing the number of non-target units that the user has completed studying (the number of passes) by the total time of use of this educational service as the number of non-target units passed per unit time.

[0110] In this educational service, for example, if the user has a low level of understanding of a certain learning unit, the user is required to go back and study the previous learning unit of that learning unit. The user is not allowed to return to the learning of the previous learning unit unless he or she completes the learning of the previous learning unit. Therefore, users with relatively low academic ability or users who make many careless mistakes are more likely to be required to study the previous learning unit, and are unable to study the original learning unit, which tends to reduce their satisfaction with learning. Therefore, the machine learning model generation module 216 may include the learning time per active days in the input data.

[0111] Fig. 18 is an example of a screen 1800 showing the results of an AB test on whether the feature "number of units passed on the first try per unit time @ non-target units (number of units passed on non-target units per unit time)" contributes to early cancellation. As shown in Fig. 18, the results of the AB test show that there is a significant difference. 26 is an example of a screen 2600 showing an ALE plot of the feature "number of units passed on the first try per unit time @ non-target units (number of units passed in non-target units per unit time)" for early cancellation. The number of units passed in non-target units per unit time tends to increase with increasing value. Therefore, the machine learning model generation module 216 may include the number of units passed in non-target units per unit time in the input data.

[0112] -Number of short-term goals achieved per unit time The number of short-term goal achievements per unit time can be an index that can evaluate the degree to which the user has been able to execute learning according to his / her own goal in this educational service. For example, in this educational service, the user can select one or more learning units that he / she intends to study in the nearest predetermined period (for example, a short period of about one week to three months) as a short-term goal. The number of learning units selected as a short-term goal is not particularly limited, and may be any number between one unit and several hundred units, for example. For example, the learning execution module 213 can accept the selection of a learning unit as a short-term goal by the user. Then, the learning execution module 213 counts an event of short-term goal achievement when all learning of the learning units selected as a short-term goal is completed within the set period of the short-term goal. The learning execution module 213 calculates a value by dividing the number of events by the total usage time of this educational service, and stores the value as the number of short-term goal achievements per unit time.

[0113] FIG. 19 is an example of a screen 1900 showing the results of an AB test on whether the feature "number of short-term goals achieved per unit time" contributes to early cancellation (cancellation within 120 days; the same applies below). As shown in FIG. 19, the results of the AB test indicate that there is a significant difference. The number of short-term goals achieved per unit time corresponds to the frequency of reaching a milestone (intermediate target point) in progressing with learning, and it can be considered that the larger this value is, the more the user feels a sense of accomplishment in learning.

[0114] 27 is an example of a screen 2700 showing an ALE plot of early churn for the feature "number of short-term goal achievements per unit time." The number of short-term goal achievements per unit time shows a tendency for early churn to drop sharply when the frequency exceeds a certain threshold. Therefore, the machine learning model generation module 216 may include the number of short-term goals achieved per unit time in the input data.

[0115] - Total top recommendation selection rate The cumulative top recommended selection rate is an index that can evaluate the degree of autonomous learning of a user, and can particularly indicate the tendency of the user to select learning subjects by thinking for themselves. The learning execution module 213 is configured to display (recommend) a list of multiple learning contents that are recommended to be implemented in order of the degree of recommendation according to, for example, the user's learning goals and learning history. When the user selects the learning content to be implemented from the learning content ranked first in the recommended menu, the learning execution module 213 calculates and stores the percentage (%) of selection of such learning content as the cumulative top recommended selection rate.

[0116] Fig. 20 is an example of a screen 2000 showing the results of an AB test regarding whether the feature amount "collection rate of top recommendation selection" contributes to early cancellation. As shown in Fig. 20, the results of the AB test show that there is a significant difference. 28 is an example of a screen 2800 showing an ALE plot of the feature "total recommended top selection rate" for early churn. The number of times of educational material listing execution alerts per unit time generally tends to increase with increasing value, indicating a higher degree of early churn. Therefore, the machine learning model generation module 216 may include the cumulative top recommendation selection rate in the input data.

[0117] -Number of times per unit time to list teaching materials and number of alerts The number of alerts of the number of times the learning material list has been performed per unit time is an index that can evaluate the degree of the user's autonomous learning. The learning execution module 213 is configured to recommend learning content that is desirable to be performed according to the user's learning goals and learning history, for example. When the user selects the learning content to be performed from the content list rather than from the recommended menu, the learning execution module 213 counts the number of times such learning content has been selected as the number of times the learning material list has been performed. For example, the learning execution module 213 stores the value obtained by dividing the number of times the learning material list has been performed by the usage time [hr] of the educational service as the number of times the learning material list has been performed per unit time.

[0118] Fig. 21 is an example of a screen 2100 showing the results of an AB test on whether the feature "Number of times of implementing the educational material list per unit time, number of alerts" contributes to early cancellation (cancellation within 120 days. The same applies below.) As shown in Fig. 21, the results of the AB test show that there is a significant difference. 29 is an example of a screen 2900 showing an ALE plot of the feature "Number of Alerts of the Number of Times of Teaching Material Listings per Unit Time" for early churn. When the number of alerts of the number of times of teaching material listings per unit time becomes larger than a certain threshold, the degree of early churn tends to increase rapidly. Therefore, the machine learning model generation module 216 may include the number of times alerts for the teaching material list are performed per unit time in the input data.

[0119] - Hours of study per active day The study time per active day is an index that can evaluate the degree of the user's autonomous study and the interest in study using the educational service. The study execution module 213 stores, for example, the value obtained by dividing the total time the user used the educational service by the number of days the educational service was used as the study time per active day.

[0120] Fig. 22 is an example of a screen 2200 showing the results of an AB test regarding whether or not the feature amount "learning time per active day" contributes to early cancellation. As shown in Fig. 22, the results of the AB test show that there is a significant difference. 30 is an example of a screen 3000 showing an ALE plot of the feature "learning time per active days" for early churn. The higher the value of the learning time per active days, the lower the degree of early churn. Therefore, the machine learning model generation module 216 may include the learning time per active days in the input data.

[0121] - Study time per non-target unit passed in the last week The study time per number of passed non-target units in the most recent week is the study time required to pass one non-target unit. The study execution module 213 calculates the value by dividing the study time of the non-target unit (e.g., the retroactive unit) in the most recent week by the number of completed studies (number of passed) of the non-target units in the most recent week, and stores the result as the study time per number of passed non-target units in the most recent week.

[0122] 23 is an example of a screen 2300 showing the results of an AB test on whether the feature "study time per number of non-target units passed in the most recent week" contributes to early cancellation. As shown in FIG. 23, the results of the AB test show that there is a significant difference. A low value of the study time per number of non-target units passed in the last week means that the user is being forced to study retroactive units that are too easy for the user. In this case, the study of the retroactive units is considered to be inappropriate for the user, and the user's motivation to study and satisfaction with the study tend to decrease, and the early cancellation rate tends to increase. Therefore, the machine learning model generation module 216 should include the study time per active days in the input data.

[0123] Total Hint View Rate The total hint viewing rate is the rate at which hints for solving a problem were viewed. The learning execution module 213 calculates a value by dividing the total number of views of hints viewed when solving one or more problems by the number of the problems, and stores the value as the total hint viewing rate.

[0124] 31 is an example of a screen 3100 showing an ALE plot of the feature "total hint viewing rate" for early churn. When the total hint viewing rate exceeds a certain threshold, the learning fitness tends to increase rapidly. Therefore, the machine learning model generation module 216 may include the learning time per active day in the input data.

[0125] As described above in detail, the present technology provides a mechanism for predicting the learning suitability of a user who uses an educational service.

[0126] <Additional learning> Fig. 14 is an example of a screen 1400 showing the contribution of each feature in the learning history information of a high-achieving user to early cancellation. And Fig. 15 is an example of a screen 1500 showing the contribution of each feature in the learning history information of a low-achieving user to early cancellation. As is clear from Fig. 14 and Fig. 15, the content of the learning history information that contributes to the learning suitability may be significantly different between high-achieving users and low-achieving users.

[0127] Therefore, the machine learning model generation module 216 may generate a machine learning model optimized for each user attribute by performing additional learning for each user attribute. For example, the machine learning model generation module 216 may perform learning on the machine learning model using the learning history information of the high-scholarship user, and create a machine learning model that can estimate the learning suitability of the high-scholarship user at a higher level. Specifically, the learning execution module 213 may be configured to acquire learning history information including information indicating that the academic ability is higher than a predetermined threshold as information corresponding to the degree of completion of the output problem. Then, the machine learning model generation module 216 may generate a machine learning model (including additional learning) based on the learning history information including the information indicating the high academic ability. Then, the evaluation module 214 may use the machine learning model thus generated, optimized for the high-scholarship user, to evaluate the learning suitability of the user based on the learning history information including the information indicating the high academic ability.

[0128] In this case, examples of the learning history information (input data) used for additional learning include the session completion rate, the net number of units passed on the first try, and the number of answers per unit time. Here, a session refers to the implementation of one learning content (e.g., any one of a lecture learning content, an exercise learning content, or a review learning content, etc.) prepared for learning a certain learning unit (e.g., a learning unit). The session completion rate is the percentage of learning content that has been completed (in other words, excluding learning content that has been in the middle or stopped) among the learning contents that should be implemented when studying one or more learning units. The learning session completion rate may be calculated for all learning contents together, or may be calculated for each predetermined learning content (e.g., for each type of learning content).

[0129] For example, the machine learning model generation module 216 may perform learning on the machine learning model using the learning history information of the low-ability user, and create a machine learning model that can estimate the learning suitability of the low-ability user at a higher level. Specifically, the learning execution module 213 may be configured to acquire learning history information including information indicating that the academic ability is lower than a predetermined threshold as information corresponding to the degree of learning stumbling (for example, the degree of learning stumbling per predetermined time). Then, the machine learning model generation module 216 may generate a machine learning model (including additional learning) based on the learning history information including the information indicating low academic ability. Then, the evaluation module 214 may use the machine learning model thus generated, optimized for the low-ability user, to evaluate the learning suitability of the user based on the learning history information including the information indicating low academic ability.

[0130] In this case, the learning history information used for learning may include, for example, the short-term goal achievement rate per unit time, the number of unit stumbling alerts per unit time, the number of problem stumbling alerts per unit time, and the session completion rate.

[0131] The net number of first-time passed units is the number of times that a learning unit that should be studied is passed on the first try (in other words, the number of learning units that have been passed). Note that in many educational services, for example, the same learning unit can be repeatedly studied multiple times, and it is possible to pass the same learning unit multiple times. Also, depending on the learning content provided (learning content such as confirmation tests and warm-ups that are posed across multiple learning units), a learning unit that does not need to be studied may also be passed. The net number of first-time passed units includes the notation "net" in that it does not count passes for such learning units that do not need to be studied.

[0132] Based on the net number of units passed on the first try, in relation to the study of learning units other than retroactive units, the total number of units passed on the first try in the most recent week (total number of units passed on the first try in the most recent week @ target) may be used as input data for additional study. Also, based on the net number of units passed on the first try, in relation to the study of retroactive units, the total number of units passed on the first try in the most recent week (total number of units passed on the first try in the most recent week @ non-target units) may be used as input data for additional study.

[0133] Furthermore, for example, the total study time for a study unit other than the retroactive unit or for the study of the retroactive unit divided by the number of units that were passed on the first attempt (respectively, the number of units that were passed on the first attempt @ the study time per target unit, or the number of units that were passed on the first attempt @ the study time per non-target unit) may be used as input data. This value represents the study time per passing attempt on the first attempt, and can be an index for evaluating the difficulty of the study unit for the user. For example, the smaller this value, the lower the difficulty of the study unit for the user, and it can be used to evaluate the possibility that the difficulty setting or goal setting of the study unit to be studied is inappropriate.

[0134] <Recommendations for recommended actions> An example of displaying the estimated learning compatibility in the user management system 100 will be described below.

[0135] The advice generating module 217 generates recommended advice for the instructor and the user's learning tendency in natural language based on the learning suitability and learning history information 800 (e.g., feature value) evaluated by the evaluation module 214. The advice generating module 217 generates advice using, for example, large-scale natural language models (LLMs). For example, specifically, the advice generating module 217 connects to the large-scale natural language model via the communication control unit 206, and outputs an instruction to generate advice according to a predetermined format based on the learning suitability and feature value. In addition, the advice generating module 217 acquires the advice generated from the large-scale natural language model and stores it in, for example, the auxiliary storage device 202. The output module 215 displays, for example, a learning suitability display screen 3500 on the display of the educational facility terminal 103 together with the stored advice.

[0136] Here, it is preferable that the advice generation module 217 is configured to ensure that, when generating advice using a large-scale natural language model, for example, (1) a stable response of a certain level or more is received for appropriate input, (2) sensitive information such as the user's personal information is not used, and (3) the generated advice is appropriate.

[0137] (1) In order to receive a certain level of stable response to an appropriate input, for example, commands (prompts) to be output to the large-scale language model may be templated in advance and stored in the auxiliary storage device 202. Then, the advice generation module 217 outputs the appropriate templated command to the large-scale language model based on the learning compatibility and feature values. In addition, the advice generation module 217 may be configured to provide a parameter (generally known as Temperature) to the large-scale language model and further provide examples of responses to commands (generally known as a method called Few-Shot Prompting) so that the generation result does not fluctuate stochastically.

[0138] In addition, large-scale language models are generally not good at numerical calculations in many cases. Therefore, the advice generation module 217 may be configured to improve stability by adjusting the prompts so that the large-scale language model can appropriately handle the feature values ​​by chain-of-thought prompting.

[0139] (2) In order not to use sensitive information such as personal information of a user, the advice generation module 217 may be configured to, for example, control the masking of sensitive data and / or non-use in inference for a data set before it is given to a prompt. For example, if a certain feature value significantly deviates from the overall distribution, there is a risk that the corresponding individual can be inferred from the feature value. Therefore, the advice generation module 217 may be configured to increase safety by calculating in advance the deviation of each feature value to be input into a large-scale language model and generating advice using only data whose deviation falls within a certain range.

[0140] (3) In order to guarantee the quality of the generated advice, the advice generation module 217 may be configured to utilize a large-scale language model that is trained to avoid harmful remarks. The output module 215 may be configured to display, for example, on the educational facility terminal 103 and / or the user terminal 102, only advice whose quality has been checked by a service operator or the like.

[0141] In this embodiment, the advice generating module 217 provides the large-scale language model with user information required for generating advice via a prompt. However, the configuration for providing user information to the large-scale language model is not limited to this example. For example, the advice generating module 217 may be configured to provide information such as the user's learning adaptability and each feature amount in the form of an external data set that the large-scale language model can refer to, and generate advice by having the large-scale language model read the external data set (this can be realized by using a vector store or an agent, etc.). In addition, for example, the advice generating module 217 may be configured to include a large-scale language model and to have the model itself learn necessary information (this can be realized by fine-tuning, etc.). Alternatively, the advice generating module 217 may be configured to generate advice on a rule basis without using a large-scale language model.

[0142] 36 is an example of a learning suitability display screen 3500. The output module 215 displays the learning suitability display screen 3500 on, for example, the educational facility terminal 103. The output module 215 can display, on the learning suitability display screen 3500, information related to the learning suitability as well as recommended actions by the instructor according to the learning suitability.

[0143] 35, the output module 215 displays a user ID selection section 3510, a learning compatibility display section 3520, a learning tendency display section 3530, and a recommended behavior display section 3540 on a learning compatibility display screen 3500. However, the display contents of the learning compatibility display screen 3500 are not limited to this example.

[0144] The user ID selection section 3510 is an element for specifying the user ID of the target user for which information on learning suitability is to be displayed. In this embodiment, the instructor, for example, selects or inputs the user ID of the target user in the user ID selection section 3510. At this point, the learning suitability display section 3520, the learning tendency display section 3530, and the recommended behavior display section 3540 in the learning suitability display screen 3500 are, for example, empty or display information on the target user specified one time ago. Note that the value input to the user ID selection section 3510 is not limited to the user ID, and may be any information that can uniquely identify the user or that can be narrowed down by search. Instead of the user ID, information such as a user display ID or a name may be used.

[0145] The output module 215 acquires information on the learning suitability of the target user and advice on learning tendencies and recommended actions based on the specified user ID. Then, the output module 215 outputs a learning suitability display screen 3500 to the educational facility terminal 103, and displays the acquired information on the learning suitability and advice on learning tendencies and recommended actions in a learning suitability display section 3520, a learning tendency display section 3530, and a recommended action display section 3540 of the learning suitability display screen 3500, respectively.

[0146] The output module 215 displays, for example, a contribution graph of each feature on the learning compatibility display unit 3520. The output module 215 visually displays, for example, the value of each feature used in calculating the learning compatibility in a graph for the target user as the contribution graph of each feature. It is difficult to visually determine what behaviors are strongly correlated and what behaviors should be reviewed based on the learning compatibility alone. However, with this configuration, it is possible to display the value of each feature that is strongly correlated with the learning compatibility, and the relevance of information with advice described later can be visually understood, which can promote a review of quality learning behavior.

[0147] The output module 215 displays the actual state of the user's learning tendency indicated by the learning compatibility and each feature in natural language in the learning tendency display section 3530. The sentences on the learning tendency are generated by the advice generation module 217, and represent the tendency of the user's learning behavior in association with the information on the contribution graph of each feature described above.

[0148] The tendency of learning behavior includes, for example, information on whether the tendency is positive or negative, and information on comparison with the tendency of a standard user. This is expected to have the effect of not only displaying a simple learning suitability, but also understanding the relationship between the recommended advice described below and one's own tendency of learning behavior, which will lead to easier improvement of learning behavior.

[0149] The recommended action display unit 3540 synthesizes the information collected so far and displays advice recommended to the user in natural language. The sentences of the recommended action are generated by the advice generation module 217, and the advice of the action recommended to the user is displayed in association with information such as the contribution graph of each feature amount described above and learning tendency.

[0150] For example, the instructor can use the learning suitability display screen 3600 when acquiring the learning suitability of a specific user in the user management system 100. In this case, along with the learning suitability of the specific user, data on the contribution of each feature amount and advice on recommended actions in natural language can be obtained. This allows the instructor to obtain information that cannot be obtained by simply receiving only the learning suitability information, such as information on what kind of coaching activities should be performed for the user. As a result, the instructor can implement learning support activities that are optimized for each individual user.

[0151] In this embodiment, the learning suitability display screen 3500 is displayed on the educational facility terminal 103, but the present technology is not limited to this. The output module 215 may display the learning suitability display screen 3500 on, for example, the output device 205 of the management server 101, or on the output device 305 of the user terminal 102. In particular, when displayed on the user terminal 102, the user can autonomously reflect on his or her own learning behavior. As a result, the user can improve his or her own learning style to a more effective one, and can fully enjoy the effects of the educational service.

[0152] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0153] In the above embodiment, supervised learning is adopted as the machine learning method applied to generate the machine learning model. However, the machine learning method is not limited to this, and unsupervised learning, reinforcement learning, etc. may be adopted. Furthermore, the machine learning model is not limited to one that evaluates whether or not the contract will be cancelled within a predetermined period as the learning suitability. For example, the machine learning model may be trained to predict other learning suitability such as the contract period.

[0154] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0155] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0156] 100...user management system, 101...management server, 102...user terminal, 103...educational facility terminal, 211...user terminal management module, 212...educational facility terminal management module, 213...learning execution module, 214...answer evaluation module, 214...evaluation module, 215...model information generation module, 215...output module, 216...machine learning model generation module, 217...advice generation module

Claims

1. An acquisition unit that acquires learning history information of a user; an evaluation unit that evaluates whether the user has benefited from the educational service based on the acquired learning history information; an output unit that outputs the obtained evaluation result as to whether or not the learning effect of the educational service has been enjoyed; A user management system comprising:

2. The evaluation unit is a machine learning model that is trained using the learning history information in the educational service as an input and a degree of continuous use of the educational service as an output; inputting the acquired learning history information of the user in the educational service into the machine learning model, and outputting, as an evaluation result, a degree of continuous use of the educational service by the user; The user management system according to claim 1 .

3. The learning history information includes at least one of information regarding learning failure information, non-target unit learning information, and learning efficiency information. The user management system according to claim 1 or 2.

4. The learning history information includes, as the learning failure information, Information regarding the event where the time taken to answer a question exceeds the standard answer time plus a prescribed time; Information regarding an event in which the time required to complete a learning unit exceeds a standard time required to complete a learning unit plus a predetermined time; Information regarding the event of viewing a hint when solving a problem; The information includes information regarding at least one of The user management system according to claim 3.

5. The learning history information includes, as the non-target unit learning information, Information about the amount of time it took to complete the non-target unit during the past week, or Information regarding the number of times a non-target unit was completed on the first attempt per unit of study time for that non-target unit; At least one of The user management system according to claim 3.

6. The learning history information includes, as the learning efficiency information, short-term goal achievement rate, Number of correct answers, Number of completed learning units per unit of time; The information includes information regarding at least one of The user management system according to claim 3.

7. For users whose academic ability is higher than a certain threshold, the acquiring unit acquires the learning history information including information corresponding to a degree of completion of the output question, the evaluation unit evaluates whether the user has enjoyed a learning effect provided by the educational service based on the learning history information including information corresponding to a degree of completion of the questions. The user management system according to claim 2 .

8. For users whose academic ability is below a certain threshold, The acquisition unit acquires the learning history information including information corresponding to a degree of learning difficulty per a predetermined time period, the evaluation unit evaluates whether the user is enjoying the learning effect of the educational service based on the learning history information including information corresponding to the degree of learning difficulty per the predetermined time period. The user management system according to claim 2 .

9. 1. A computer-implemented user management method, comprising: Acquire learning history information of the user in the educational service; Evaluating whether the user has benefited from the educational service based on the acquired learning history information; The user management method outputs the obtained degree of continuous use of the educational service.

10. A program for causing a computer to execute each step of the method according to claim 9.

Images