Method for providing user-customized learning service

Abstract

A method for providing a user-customized learning service comprises: a step of obtaining a user's first response to a first problem of learning content and a first response time taken for the first response; a step of calculating a normal response time range for a problem format to which the first problem belongs, based on a cumulative learning log including whether the answer is correct and the response time for each of a plurality of problems answered by the user; a step of determining the first response to the first problem as a random response or a skill-based response based on the fact that the first response time is shorter than the lower limit of the normal response time range; a step of excluding the first response from data analysis based on the fact that the first response is determined to be a random response; and a step of updating the normal response time range by reflecting the first response time based on the fact that the first response is determined to be a skill-based response.

Description

Technology Field

[0001] The present disclosure relates to a method for providing a user-customized learning service. Background Technology

[0002] With the recent advancement of information and communication technology, various learning services are being provided through user terminals. For example, users can receive learning content problems and input responses to the problems through applications on their user terminals.

[0003] In learning services, learning achievement can be analyzed from user responses. For example, whether the user answered correctly and their response time can be measured and used to analyze the learning status.

[0004] If a user arrives at the correct answer by responding arbitrarily without knowing the correct solution, and the value measured from that response is directly reflected in the analysis of the learning status, the results of the analysis may differ from the user's actual ability. If an arbitrarily reached answer is reflected in the learning analysis, there is a concern that the learning service provided to the user may not correspond to the user's actual learning status. The problem to be solved

[0005] The present disclosure provides a method for providing a user-customized learning service to solve the above-mentioned problems. means of solving the problem

[0006] The present disclosure may be implemented in various ways, including a method, an apparatus (system), or a computer program stored on a readable storage medium.

[0007] A method for providing a user-customized learning service performed by one or more processors according to one embodiment of the present disclosure comprises: obtaining a user's first response to a first problem of learning content and a first response time taken for the first response; calculating a normal response time range for a problem format to which the first problem belongs, based on a cumulative learning log including whether the answer is correct and the response time for each of a plurality of problems answered by the user; determining the first response to the first problem as a random response or a skill response based on the fact that the first response time is shorter than the lower limit of the normal response time range; excluding the first response from data analysis based on the fact that the first response is determined to be a random response; and updating the normal response time range by reflecting the first response time based on the fact that the first response is determined to be a skill response.

[0008] According to one embodiment of the present disclosure, the step of determining a first response to a first problem as a random response or a skill response includes obtaining at least one of whether the first attempt to answer the first problem is correct, the number of attempts to answer the first problem, whether a hint was used for the first problem, or a cumulative correct answer rate by problem type, and determining the first response to the first problem as a random response or a skill response based on at least one obtained.

[0009] According to one embodiment of the present disclosure, the method further comprises the steps of: determining a learning item as a temporary mastery item based on the fact that a proficiency response to a learning item identical to a first problem, including a first response to a first problem, has been received consecutively a predetermined number of times; providing a second problem belonging to the learning item to a user based on the fact that a predetermined period has elapsed after the learning item has been determined as a temporary mastery item; receiving a second response to a second problem and a second response time taken for the second response; determining the second response as a random response or a proficiency response based on the second response time; confirming the learning item as a mastery item based on the fact that the second response is correct and the second response is determined to be a proficiency response; and degrading the learning item to an unmastery item based on the fact that the second response to the second problem is incorrect or the second response is determined to be a random response.

[0010] According to one embodiment of the present disclosure, the step of determining a first response to a first problem as a random response or a skill response includes: a step of calculating a standardized deviation value of the first response time based on a first response time and a representative response time and standard deviation calculated from the response times of a plurality of responses to which the user has previously determined the first problem to be a skill response for a problem format to which the first problem belongs; a step of calculating a reliability score by weightedly combining the standardized deviation value, a first value corresponding to whether the first attempt to answer the first problem is correct or the cumulative correct answer rate for the problem format to which the first problem belongs, and a second value corresponding to whether a hint is used for the first problem; and a step of determining the first response to the first problem as a random response based on the reliability score being less than a preset judgment threshold.

[0011] According to one embodiment of the present disclosure, the step of updating a normal response time range includes, based on determining a first response to a first problem as a skill response, updating a representative response time and dispersion associated with the normal response time range by reflecting the first response time, and updating a lower limit of the normal response time range by subtracting a value obtained by multiplying the updated dispersion by a preset constant from the updated representative response time.

[0012] According to one embodiment of the present disclosure, based on the accuracy rate calculated from a cumulative learning log for each of a plurality of problem formats for a learning item identical to a first problem, the method further comprises the steps of: diagnosing a problem format among a plurality of problem formats in which the accuracy rate is greater than or equal to a first threshold value as the user's strength format and diagnosing a problem format in which the accuracy rate is less than or equal to a second threshold value as the user's weakness format; converting the strength format into an application format or a challenge format; and presenting a plurality of problems corresponding to the weakness format and problems corresponding to the application format or challenge format to the user. Effects of the invention

[0013] According to various embodiments of the present disclosure, fast responses are not uniformly blocked but are processed by distinguishing between random responses and skill responses, thereby preserving fast responses based on skill while reducing the dilution of training data caused by random responses.

[0014] According to various embodiments of the present disclosure, by making a determination by considering multiple learning signals together rather than making a determination based on a single criterion of response time, the misclassification of a user who responds quickly and accurately as an arbitrary response can be reduced.

[0015] According to various embodiments of the present disclosure, by quantitatively weighting and combining the deviation from the normal distribution by individual and problem type with the learning signal to calculate a reliability score, the determination of the response can be performed based on objective numerical values.

[0016] According to various embodiments of the present disclosure, the representative response time and dispersion are updated and the lower limit is recalculated by reflecting only the skill response, so that the judgment criteria can be adaptively corrected to the user's actual skill change.

[0017] According to various embodiments of the present disclosure, the determination of provisional mastery by the continuous arrival of skill responses and reconfirmation with a time delay may reduce the determination of mastery that is accidentally or arbitrarily reached as a mastery item.

[0018] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims. Brief explanation of the drawing

[0019] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto. FIG. 1 is a diagram illustrating an example of a learning screen provided to a user in a method for providing a user-customized learning service according to an embodiment of the present invention. FIG. 2 illustrates a block diagram showing the internal configuration of a user terminal and a server in a user-customized learning service provision system according to various embodiments of the present invention. FIG. 3 is a flowchart illustrating a method for processing training data according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a method for determining an arbitrary response and updating a normal response time range according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating a method for determining a response as a random response or a skill response based on a learning signal according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a method for determining a response as a random response based on a reliability score according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a method for updating a normal response time range by a correction rule according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method for verifying mastery of a learning item according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating a method for automatically configuring a problem format according to an embodiment of the present invention. Specific details for implementing the invention

[0020] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression may include a plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this disclosure. Terms used in this disclosure that are defined in a general dictionary may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this disclosure. In some cases, even terms defined in this disclosure are not to be interpreted to exclude the embodiments of this disclosure.

[0021] In the various embodiments of the present disclosure described below, a hardware-based approach is described as an example. However, since the various embodiments of the present disclosure include techniques using both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0022] Hereinafter, various embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, since the technical concept of the present disclosure can be modified and implemented in various forms, it is not limited to the embodiments described in this specification. In describing the embodiments disclosed in this specification, if it is determined that specifically describing related prior art could obscure the essence of the technical concept of the present disclosure, such specific description of prior art is omitted. Identical or similar components are assigned the same reference number, and redundant descriptions thereof are omitted.

[0023] When an element is described in this specification as being "connected" to another element, this includes not only cases where they are "directly connected" but also cases where they are "indirectly connected" with another element in between. When an element is described as "comprising" another element, this means that, unless specifically stated otherwise, it does not exclude other elements in addition to the other elements but may include additional elements.

[0024] Some embodiments may be described by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. The functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks of the present disclosure may be implemented as algorithms executed on one or more processors. The functions performed by the functional blocks of the present disclosure may be performed by a plurality of functional blocks, or the functions performed by a plurality of functional blocks in the present disclosure may be performed by a single functional block. Additionally, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc.

[0025] Additionally, in this disclosure, expressions such as "greater than" or "less than" have been used to determine whether specific conditions are satisfied or fulfilled; however, this is merely for illustrative purposes and does not exclude descriptions of "greater than" or "less than." Conditions described as "greater than" may be replaced with "greater than," conditions described as "less than" with "less than," and conditions described as "greater than and less than" with "greater than and less than."

[0026] Hereinafter, various embodiments of the present invention will be described with reference to FIGS. 1 through 9. Throughout the specification, the same reference numerals may refer to the same components.

[0027] FIG. 1 is a diagram illustrating an example of a learning screen provided to a user in a method for providing a user-customized learning service according to an embodiment of the present invention.

[0028] Referring to FIG. 1, a user terminal (e.g., user terminal (210) of FIG. 2) may provide a problem of learning content to the user and display a learning screen to obtain the user's response to the problem. The learning screen may include multiple screens that present the same learning item in different problem formats. The first learning screen (110) may be a selection type screen for selecting the meaning of the learning item from a plurality of options, and the second learning screen (120) may be an input type screen for entering the spelling of the learning item.

[0029] A user is a person who uses the learning service and may include children, students, and / or adult learners. A user may include a guardian or teacher who inputs a response on behalf of the user. A user may be a single learner corresponding to a single user account, or it may be a form in which multiple learners sharing a single user terminal are distinguished by their respective identification information.

[0030] Learning content may refer to learning objects including words, vocabulary, concepts, formulas, sentences, and / or images. Learning items may refer to units of content subject to evaluation. A question may be a unit of material presented to a user to evaluate a single learning item. A learning item may correspond to a single word or a single concept, or it may correspond to a unit comprising multiple words or concepts.

[0031] The question format is a presentation format for evaluating the same learning item, and may refer to evaluation forms in which the input method of the response or the type of cognitive processing required differs, such as multiple-choice, input, listening, speaking, fill-in-the-blank, and / or sequence arrangement types. The first learning screen (110) may correspond to a multiple-choice question format, and the second learning screen (120) may correspond to an input question format. Since the type of cognitive processing required of the user may differ when the same learning item is presented as a multiple-choice format versus an input format, the user's response and response time may have different distributions depending on the question format. As the normal response time range is calculated separately for each question format, it can be determined that the same response time falls within the normal range in one question format and falls outside the normal range in another question format.

[0032] The response may refer to an input including selection, text, voice, handwriting, and / or gestures. In the first learning screen (110), the response may be an input for selecting one of a plurality of options, and in the second learning screen (120), the response may be a text input for typing. The learning screen may include a problem presentation area, a view area or an input area, a hint provision means and / or a confirmation means. A learning item and / or an image corresponding to the learning item may be displayed in the problem presentation area. The confirmation means may receive input to confirm the user's response.

[0033] Response time may be the time measured from the point when the problem is presented until the point when the response is confirmed. The point when the response is entered may be when the user selects an option or begins typing, and the point when the response is confirmed may be when input is received through a confirmation means. The measurement interval of response time may be defined as the interval from the point when the problem is presented until the point when the confirmation input is received. In measuring response time, the method of handling periods during which the solution is paused or periods during which the same problem is retried may be defined differently depending on the embodiment. For example, periods during which the solution is paused may be excluded from the measurement of response time, and when a retry is performed, the time from the initial presentation point to the final confirmation input point may be measured as the response time, or the response time may be measured individually for each attempt.

[0034] In the first learning screen (110) and the second learning screen (120), in addition to the user's response and response time, a learning signal including whether the first attempt at the problem was correct, the number of attempts at the problem, and / or whether a hint was used for the problem may be observed. The learning signal may be used to determine whether the user's response is a random response or a skill-based response. The operation of determining whether a response is a random response or a skill-based response using the learning signal will be described in detail later in FIGS. 4 and 5.

[0035] The problem format may not be limited to the selection and input types illustrated in FIG. 1. The problem format may include any evaluation type including listening and selecting, pronunciation speaking through a microphone, handwriting spelling input through a stylus, sequencing, matching, and / or filling in blanks. A response to a single problem may be a multi-modal response combining selection input, text input, voice input, and / or handwriting input. In a problem format that receives voice input, voice is received through the microphone of the user terminal, and the result of voice recognition may be converted into a response. In a problem format that receives handwriting input, handwriting is received through the stylus or touchscreen of the user terminal, and the result of handwriting recognition may be converted into a response.

[0036] The hint provision means may provide hints in the form of disclosing information about the learning item in stages or in parts. The learning screen may further include a difficulty display means for displaying the difficulty of the problem or learning item. The learning screen may further include a screen for a guardian or teacher. The screen for the guardian or teacher may display the user's learning progress, mastered items, and / or unmastered items. The user terminal may detect the user's gaze at the screen or screen departure through a front camera. The period during which screen departure is detected may be excluded from the measurement of response time or may be used as a learning signal to determine whether the response is a random response. The user terminal may receive the solution to the problem while offline without a connection to the network, and synchronize the result of the solution with a server (e.g., server (230) of FIG. 2) after the connection to the network is restored.

[0037] The response, response time, and learning signal obtained through the first learning screen (110) and the second learning screen (120) can be obtained in a state where the data values ​​are preserved in their original form without dilution. Since the response time is preserved as an original value in milliseconds, random responses and skill responses can be distinguished based on minute differences in response time. The acquisition of the response and learning signal through the learning screen can contribute to improving the accuracy of learning data analysis by reflecting response characteristics that differ for each user, and reducing resources required for storing and computing the accumulated learning log by excluding random responses from the data analysis target.

[0038] Operations for determining random responses and skill responses using response times and learning signals obtained through the first learning screen (110) and the second learning screen (120), updating the normal response time range, and configuring the problem format will be described in detail later in FIGS. 4 to 9. Operations for diagnosing strengths and weaknesses by problem format for the same learning item will be described later in FIG. 9. The configuration of the learning screen, learning items, and problem formats shown in FIG. 1 may be examples, and the learning screen may be changed to include any user interface capable of obtaining a user's response.

[0039] FIG. 2 illustrates a block diagram showing the internal configuration of a user terminal (210) and a server (230) in a user-customized learning service provision system according to various embodiments of the present invention.

[0040] Referring to FIGS. 1 and 2, the user terminal (210) may correspond to a learner's terminal displaying the first learning screen (110) and the second learning screen (120) of FIG. 1. The user terminal (210) may refer to any computing device capable of wired and / or wireless communication and capable of running an application. For example, the user terminal (210) may include a smartphone, a mobile phone, a tablet PC, a wearable device, and / or a PDA (personal digital assistant). The user terminal (210) may be a general-purpose smartphone with a learning application for a child or learner installed, and in other embodiments, it may be a kids' tablet or a terminal dedicated to education.

[0041] The user terminal (210) may include memory (211), a processor (212), a communication module (213), and an input / output interface (214). The server (230) may include memory (231), a processor (232), a communication module (233), and an input / output interface (234). The processor (212, 232) may include an AI (artificial intelligence) dedicated accelerator such as a CPU (central processing unit), GPU (graphics processing unit), NPU (neural processing unit), or TPU (tensor processing unit).

[0042] A user terminal (210) and a server (230) can communicate information and / or data through a network (220) using their respective communication modules (213, 233). The network (220) may include any communication network that enables communication between the user terminal (210) and the server (230). Depending on the installation environment, the network (220) may be configured as a wired network including Ethernet, a wired home network and / or RS-serial communication, a mobile communication network, a wireless local area network (WLAN), a wireless network including Wi-Fi, Bluetooth and / or ZigBee, or a combination of a wired network and a wireless network. The network (220) may include an LTE (long term evolution) network, a 5G (5th generation) network and / or a private wireless network. The private wireless network may include a dedicated wireless network installed in a home, school, or academy, and multiple types of networks may be used in combination depending on the scale of the installation environment.

[0043] Memory (211, 231) may include any non-transitory computer-readable recording medium. Memory (211, 231) may include a permanent mass storage device including ROM (read-only memory), a disk drive, an SSD (solid-state drive), and / or flash memory. The permanent mass storage device may be included in the user terminal (210) or server (230) as a separate permanent storage device distinct from memory (211, 231). An operating system and at least one program code may be stored in memory (211, 231). Memory (231) of the server (230) may store learning content, problem format information, cumulative learning logs, representative response times and dispersions associated with normal response time ranges by user and problem format, judgment thresholds, and weights and / or mastery reference values ​​used in calculating confidence scores.

[0044] Software components may be loaded from a computer-readable recording medium separate from memory (211, 231). The separate computer-readable recording medium may include a floppy drive, disk, tape, DVD / CD-ROM drive and / or memory card. Software components may also be loaded into memory (211, 231) via a communication module (213, 233) rather than a computer-readable recording medium.

[0045] The processor (212, 232) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (212, 232) by memory (211, 231) or a communication module (213, 233). The processor (212, 232) can execute instructions received according to program code stored in memory (211, 231).

[0046] The processor (232) of the server (230) can obtain a response to a problem, the response time taken for the response, and a learning signal from the user terminal (210). Based on the accumulated learning logs for multiple problems answered by the user, the processor (232) can calculate the normal response time range of the user for the problem format. Based on the response time, the processor (232) can determine whether the response is a random response or a skill-based response. Based on determining the response as a random response, the processor (232) can exclude the response from data analysis. Based on determining the response as a skill-based response, the processor (232) can update the normal response time range. The processor (232) can verify mastery of the learning item and configure the problem format according to the strengths and weaknesses of each user to provide customized problems to the user. The specific details of each step performed by the processor (232) will be described in detail below in FIG. 3.

[0047] The processor (232) can support parallel processing of response data received from multiple users. The processor (232) may include a GPU or NPU for performing inference operations. The processor (232) can perform inference of a model for calculating a confidence score and recommending a problem format. The processor (232) can perform operations including preprocessing of response data, calculation of a normal response time range, determination of random and skill responses, calculation of a confidence score, and configuration of a problem format in a pipeline manner.

[0048] The communication module (213, 233) may provide a configuration or function for the user terminal (210) and the server (230) to communicate with each other via the network (220). The user terminal (210) and the server (230) may be provided with a function to communicate with another user terminal or another system using their respective communication modules (213, 233). The processor (212) of the user terminal (210) may generate requests or data according to program code stored in memory (211). The generated requests or data may be transmitted to the server (230) via the communication module (213) and the network (220). Control signals, commands, and / or custom issues provided by the processor (232) of the server (230) may be received by the user terminal (210) via the communication module (233), the network (220), and the communication module (213) of the user terminal (210).

[0049] The input / output interface (214, 234) may be a means for interfacing with an input / output device (not shown). The input device may include a keyboard, a mouse, a microphone, and / or a touchscreen, and the output device may include a display, a speaker, and / or a haptic feedback device. The input / output interface (214, 234) may also be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output interface (214) of the user terminal (210) may be used to receive input for the user to solve a problem and input a response, request a hint, and check the result.

[0050] In FIG. 2, the input / output interface (214, 234) is shown as an element configured separately from the processor (212, 232), but an embodiment in which the input / output interface (214, 234) is included in the processor (212, 232) is also possible. The user terminal (210) and the server (230) may include more components than those shown in FIG. 2. The user terminal (210) may further include a microphone, a stylus, and / or a front camera as auxiliary input means. The user terminal (210) may further include a global positioning system (GPS) module, an accelerometer, and / or a gyroscope. The server (230) may further include a learning content providing module for providing learning content and synchronizing learning data.

[0051] The processor (212) of the user terminal (210) can receive a user response (e.g., selection of an option, input of a spelling and / or confirmation input) input through the input / output interface (214) while the learning application is running. The processor (212) can transmit data corresponding to the received user response to the server (230) via the communication module (213) and the network (220). The data corresponding to the user response may include the content of the response, the response time, and / or a learning signal. The processor (212) can display the customized problem or evaluation result received from the server (230) through an output device including a display. For specific examples of a user interface (UI) regarding the display of the customized problem or evaluation result, the first learning screen (110) and the second learning screen (120) of FIG. 1 may be referenced.

[0052] An embodiment in which the server (230) is configured as a cloud-based distributed computing system may be possible. In a cloud-based distributed computing environment, the memory (231) and processor (232) of the server (230) may be distributed across a plurality of physically separated server devices and may be operated in an integrated manner as a virtual instance. Inference operations for the model for judgment are performed in a GPU cluster, operations for recommending problem formats are performed in a separate computation node, and communication with the user terminal (210) may be managed through an API (application programming interface) gateway.

[0053] An embodiment may be possible in which a local server at the school or institution level is placed near the user terminal (210) to process learning data locally. The local server is located in close proximity to the user terminal (210) to process at least a portion of the learning data locally and transmit the processed results to the server (230). By processing the learning data locally at the local server, the extent to which the child's learning data is taken outside the institution may be reduced. The operation of processing locally to protect the child's learning data will be described in detail later in FIG. 3.

[0054] An embodiment may be possible in which the user terminal (210) is equipped with an on-device AI function to perform some processing locally. If the processor (212) of the user terminal (210) includes an NPU, the user terminal (210) may primarily determine whether the response is a random response based on the response time and the learning signal, or perform the measurement of the response time locally. The result processed locally at the user terminal (210) is transmitted to the server (230) and can be used as an auxiliary tool for the server (230)'s determination or calculation of the normal response time range.

[0055] The configuration of the user terminal (210) and server (230) shown in FIG. 2 may be an example, and the user terminal (210) and server (230) may be modified to include more or fewer components. By distributing the processing of the response, response time, and learning signal across the server (230), local server, and / or user terminal (210), the level of protection of the child learning data and the utilization of computational resources can be considered together.

[0056] FIG. 3 is a flowchart illustrating a method for processing training data according to an embodiment of the present invention.

[0057] Hereinafter, FIG. 3 is described with reference to FIG. 2, and the description above with reference to FIG. 1 and FIG. 2 is omitted. The method of processing the learning data of FIG. 3 may be performed by a processor (232). The method of processing the learning data may be performed by a processor (212) of a user terminal (210) or by a local server at the school or institution level. The learning data is data obtained from the user's learning performance and may include measurement values, identification information, and / or time information. Measurement values ​​may refer to values ​​measured from the learning performance, such as whether the answer is correct, response time, and / or the number of attempts. Identification information may refer to information that identifies the user or can be used for user identification, and time information may refer to the absolute time or date when the learning was performed.

[0058] The processor (232) can classify the training data according to the nature of the items (S310). The nature of the items may include identification information, visual information, measurements, and / or external transmission risks. The processor (232) classifies the items included in the training data into one of identification information, visual information, and measurements according to the nature of the items, and may apply different processing to each item according to the result of the classification. The classification of items may be performed by rules based on the name, data type, or range of values ​​of the items, or may be performed automatically by a model trained to infer the nature of the items.

[0059] The processor (232) can preserve the measurement value without loss (S320). Lossless preservation of the measurement value may refer to maintaining the original value of the measurement value at its original resolution without diluting it. The processor (232) can preserve the response time as an original value in milliseconds. The processor (232) verifies whether the preserved measurement value matches the original value, and if the preserved measurement value does not match the original value, it may refuse to save the preserved measurement value.

[0060] The processor (232) can block connections between learning units by generating pseudonym identifiers for each learning unit (S330). A learning unit is a minimum unit meaningful for analysis and may include chapters, sessions, dates, and / or subjects. A pseudonym identifier may refer to an identifier that replaces identification information to prevent direct identification of the user. The processor (232) can generate different pseudonym identifiers for the same user if the learning units are different. By generating different pseudonym identifiers for each learning unit, the learning process within a single learning unit remains in a state where it can be analyzed, while long-term connections or backtracking across learning units can be blocked. Pseudonym identifiers may be generated based on hash operations on identification information, the addition of a salt, the separate storage of keys, and / or the periodic rotation of keys.

[0061] The processor (232) can process the time information in an isolation zone to extract only the derived value (S340). The isolation zone may refer to a processing area where access to the original time information is restricted. The derived value is a value calculated from the original time information and may include the time interval between learning or the time period in which learning was performed. The processor (232) can extract the derived value from the time information in the isolation zone, provide the extracted derived value outside the isolation zone, and discard the original time information. An embodiment may also be possible in which feature extraction from the time information is performed at the user terminal (210) and the original time information is not transmitted outside the user terminal (210).

[0062] The processor (232) may limit the output for items with a sample size less than the minimum sample size (S350). The minimum sample size may refer to the minimum number of samples required to allow the output of statistics or analysis results for an item. The processor (232) may limit the output of statistics or analysis results for an item if the number of samples corresponding to the item is less than the minimum sample size. The minimum sample size may be set larger when data is transmitted externally than when data is used internally. The minimum sample size may be dynamically set according to the sensitivity of the item or the size of the user group.

[0063] The processor (232) can store the processed training data (S360). The processed training data can be accumulated to form a cumulative training log. The cumulative training log is a log containing the correct answer status and response time for multiple questions answered by the user, and can be used as an input for calculating the normal response time range in FIG. 4 and below. The processor (232) can block the storage or transmission of items containing identification information based on a one-way gateway and a prohibited key list that prevent identification information from being mixed into the measured value or derived value.

[0064] Alternatively, the order of the operation of generating a pseudonym identifier per training unit (S330) and the operation of processing visual information in an isolation zone (S340) may be interchanged. The operation of limiting the output for items with fewer than the minimum sample size in step S350 may be performed at the time when the processed training data is output externally.

[0065] The processor (232) can add noise for differential privacy to the training data or generalize the training data to satisfy k-anonymity. Processing of the training data is performed in the edge environment or on-device environment of FIG. 2, so that the extent to which the child's training data is taken outside the institution can be reduced. The processor (232) manages the scope of collection, retention period, and deletion of training data based on the guardian's consent, and can delete training data upon a request for deletion. Based on federated learning, where the model is trained locally on multiple user terminals and only the model's parameters are aggregated, the analysis model can be updated without concentration of the original training data.

[0066] By preserving measurements without loss and blocking connections between learning units, the validity of the analysis of the learning data can be maintained while reducing the risk of re-identification and long-term profiling. Since the protection of the learning data is achieved not by diluting data values ​​but by structurally blocking identifiability, the accuracy of discrimination and correction based on minute differences in response time can be maintained. The steps and sequence of processing illustrated in Fig. 3 may be examples, and the method of processing the learning data may be changed to any combination of steps including the preservation of measurements and the blocking of connections between learning units. The operation of analyzing response time and determining random responses using the accumulated learning log formed in Fig. 3 will be described in detail below in Fig. 4.

[0067] FIG. 4 is a flowchart illustrating a method for determining an arbitrary response and updating a normal response time range according to an embodiment of the present invention.

[0068] Hereinafter, FIG. 4 is described together with FIG. 2, and the description above with reference to FIG. 1 to 3 is omitted. The method of determining a random response and updating the normal response time range of FIG. 4 can be performed by a processor (232). The normal response time range is a range in which the normal response time of the user is determined to belong, and can be calculated from the distribution of response times by user and by problem type. The normal response time range can be calculated based on statistical calculation of the distribution of response times, a pre-set rule, or a learned model. A random response is a response not based on the user's skill, and may include a random response or a response based on guesswork, and may be referred to as a first type of response. A skill response may include a response based on the user's understanding or mastery, and may be referred to as a second type of response.

[0069] The processor (232) can obtain a response to a problem and the response time taken for the response (S410). For example, the processor (232) can receive the response and response time input through the first learning screen (110) or the second learning screen (120) of FIG. 1 from the user terminal (210).

[0070] The processor (232) can calculate the normal response time range for a user for a problem type to which the problem belongs, based on the accumulated learning log (S420). The accumulated learning log used in step S420 may be formed by accumulating the learning data processed in FIG. 3. The processor (232) can calculate the normal response time range for each problem type from the correct answer status and response time for each of the multiple problems answered by the user. In the initial stage where the number of samples included in the accumulated learning log is insufficient, a population standard calculated from the response time distribution of the user group may be used as the normal response time range.

[0071] The processor (232) can determine whether a response is a random response or a skill response based on whether the response time is shorter than the lower limit of the normal response time range (S430). The lower limit may refer to the lower boundary of the normal response time range. A response time shorter than the lower limit of the normal response time range may mean that the user responded faster than the time typically required to solve the problem. The operation of determining whether a response is a random response or a skill response using a learning signal will be described in detail later in FIG. 5. The operation of determining a response based on the standardized deviation value and confidence score of the response time will be described in detail later in FIG. 6.

[0072] The processor (232) may exclude a response from data analysis based on determining that the response is a random response (S440). The processor (232) may update the normal response time range by reflecting the response time based on determining that the response is a competent response (S450).

[0073] The data analysis target may refer to a set of data used for the analysis or diagnosis of learning. In step S440, the operation of excluding a response from the data analysis target may include at least one of the following: assigning a weight of 0 to the response, adding a flag indicating exclusion to the response, storing the response in a separate storage area, or excluding the response from the retraining of the model. By excluding random responses from the data analysis target, the dilution of the analysis value of the training data by randomly correct responses may be reduced.

[0074] The operation of updating the normal response time range in step S450 may be an operation of adjusting the normal response time range for each user and problem type to suit the user by reflecting the response time of the response determined to be a skill response. The operation of updating the representative response time and dispersion associated with the normal response time range by the correction rule will be described in detail later in FIG. 7.

[0075] Alternatively, the order of the operation to obtain a response to the problem and the response time (S410) and the operation to calculate the normal response time range (S420) may be interchanged. An embodiment may also be possible in which the normal response time range is calculated and stored in advance before the response is obtained, and is called and used in step S430.

[0076] The trigger that initiates the action of determining whether a response is a random response or a skill-based response may not be limited to cases where the response time is shorter than the lower limit. An embodiment in which the determination of the response is initiated based on the occurrence of consecutive incorrect answers or immediately after the question format is changed may also be possible. The normal response time range may be calculated by distinguishing between users, question formats, learning items, or time zones. The normal response time range may include an upper limit in addition to the lower limit, and separate processing of the response may be performed based on the response time being longer than the upper limit. The processor (232) may aggregate the results of judgments for multiple users to produce learning analysis information at the class or group level.

[0077] By classifying and processing fast responses into random and skill-based responses rather than uniformly blocking them, fast responses based on skill are preserved, while the dilution of training data caused by random responses can be reduced. Since random responses are excluded from data analysis and the normal response time range is updated based on skill-based responses, judgment criteria can be personalized for each user and adaptively corrected. By utilizing the response time measurement for the identification of random responses and the adaptive correction of criteria, the quality of data used for learning analysis is improved, and resources required for storage and computation can be reduced through the exclusion of random responses.

[0078] The sequence and branching of the steps illustrated in FIG. 4 may be examples, and the method of determining random responses and updating the normal response time range may be changed to a combination of any steps including judgment based on response time and updating of criteria. The operation of verifying mastery of a learning item through repeated attainment of a skill response and reconfirmation of the time difference is described later in FIG. 8, and the operation of configuring the problem format according to the strength of each user is described later in FIG. 9.

[0079] FIG. 5 is a flowchart illustrating a method for determining a response as a random response or a skill response based on a learning signal according to an embodiment of the present invention.

[0080] Hereinafter, FIG. 5 is described with reference to FIG. 2 and FIG. 4 together, and the description above with reference to FIG. 1 to FIG. 4 is omitted. FIG. 5 may be a detailed representation of step S430 of FIG. 4, that is, the operation of determining a response as a random response or a skill response, based on a learning signal. The method of determining the response of FIG. 5 may be performed by a processor (232). The determination of FIG. 5 may be performed after step S420 of FIG. 4, which calculates the normal response time range.

[0081] A learning signal is an indicator of the performance or behavior of learning observed independently of the response itself, and may include whether the answer is correct on the first attempt, the number of attempts, whether a hint was used, and / or the cumulative correct answer rate by problem type. The types of learning signals are not limited to the exemplified signals and may include any indicators that can be observed from the performance of learning in addition to the response. For the observation points of each learning signal, refer to FIG. 1.

[0082] The processor (232) may obtain at least one of whether the answer was correct on the first attempt, the number of attempts, whether a hint was used, or the cumulative correct answer rate by problem type (S431). Whether the answer was correct on the first attempt may be a signal indicating whether the user answered the problem correctly on the first attempt. The number of attempts may be a signal indicating the number of times the user attempted to answer until reaching the correct answer. Whether a hint was used may be a signal indicating whether the user used a hint to solve the problem. The cumulative correct answer rate by problem type may be a signal indicating the ratio of correct answers achieved cumulatively by the user for the same problem type.

[0083] The processor (232) can determine whether a response is a random response or a skill response based on at least one obtained in step S431 (S432). The processor (232) can observe multiple learning signals together to determine whether a response that cannot be distinguished by response time alone is a random response or a skill response. For a response with a response time shorter than the lower limit, if signals are observed together such that the first attempt is correct, the number of attempts is small, no hints are used, and the cumulative correct answer rate by question type is high, the response may be determined as a skill response. For a response with a response time shorter than the lower limit, if signals are observed such that the first attempt is incorrect, the number of attempts is large, hints are used, or the cumulative correct answer rate by question type is low, the response may be determined as a random response.

[0084] If the response is determined to be a random response, step S440, which excludes the response from data analysis, may be performed. If the response is determined to be a competent response, step S450, which updates the normal response time range of FIG. 4, may be performed.

[0085] The learning signal may further include variability in response time, patterns of consecutive correct or incorrect answers, bias in view position in multiple-choice types, screen deviation, rhythm of input, and / or the result of the previous problem. The processor (232) may perform a judgment by combining multiple learning signals according to preset rules, by weighting combination according to signal-specific weights, or by combining them according to a learned model. The weights assigned to the learning signals may be set differently depending on the problem format. The processor (232) may perform a judgment using only a single learning signal or by combining multiple learning signals.

[0086] By making a judgment by considering multiple learning signals together rather than relying solely on the single criterion of response time, the misclassification of a user who responds quickly and accurately as a random response can be reduced. The accuracy of distinguishing between random responses and skill responses can be improved through the combination of multiple learning signals. The operation of quantitatively scoring learning signals and weightedly combining them to calculate a reliability score will be described in detail later in FIG. 6. The types of learning signals and judgment methods illustrated in FIG. 5 may be examples, and the method of judging a response based on learning signals may be changed to any judgment method that utilizes at least one indicator observed in addition to the response.

[0087] FIG. 6 is a flowchart illustrating a method for determining a response as a random response based on a reliability score according to an embodiment of the present invention.

[0088] Hereinafter, FIG. 6 is described with reference to FIG. 2 and FIG. 4 together, and the description above with reference to FIG. 1 to FIG. 5 is omitted. FIG. 6 may be a quantitatively detailed version of step S430 of FIG. 4 based on a reliability score. The method for determining the response of FIG. 6 may be performed by a processor (232). The determination of FIG. 6 may be performed after step S420, which calculates the normal response time range of FIG. 4.

[0089] The processor (232) can calculate a standardized deviation value of the response time based on the response time, the representative response time for the problem format, and the standard deviation (S610). The representative response time is a value representing the user's response time for the problem format and may include an average value, a median value, or a weighted average value. The standard deviation may be a value representing the dispersion of the user's response time for the problem format. The standardized deviation value may be a value representing the degree to which the response time deviates from the representative response time by a standard deviation. The standardized deviation value (z) can be calculated by subtracting the representative response time (μ) from the response time (t) and dividing the result by the standard deviation (s), i.e., z = (t - μ) / s. If the response time is shorter than the representative response time, the standardized deviation value becomes negative, and as the response is faster, the standardized deviation value becomes smaller, which may act in a direction that lowers the reliability score. The standardized deviation value may also be calculated in the positive form r = (μ - t) / s, which is the result of subtracting the response time from the representative response time and dividing it by the standard deviation.

[0090] The processor (232) can calculate a reliability score by weightedly combining standardized deviation values ​​and learning signal values ​​(S620). The reliability score may be a score indicating the degree of confidence that the response is based on the user's ability. The processor (232) can calculate a reliability score by weightedly combining standardized deviation values ​​and learning signal values ​​by weights and transforming the result of the weighted combination by a logistic function. The reliability score (C) can be calculated by the following mathematical formula 1.

[0091] [Mathematical Formula 1]

[0092] C = 1 / (1 + exp(-(w1z + w2a - w3h + b)))

[0094] In mathematical formula 1, C represents the confidence score, z represents the standardized deviation value, a represents the learning signal value, h represents whether a hint is used, w1, w2, and w3 represent the weights, and b represents the bias. The learning signal value (a) may be a value of a signal indicating skill, such as whether the answer is correct on the first attempt or the cumulative correct answer rate by question type, and whether a hint is used (h) may have a value of 1 if a hint is used and a value of 0 if a hint is not used. When the answer is correct on the first attempt is used as the learning signal value (a), the learning signal value (a) may have a value of 1 corresponding to the correct answer on the first attempt and a value of 0 corresponding to everything else. When the cumulative correct answer rate by question type is used as the learning signal value (a), the learning signal value (a) may have a value corresponding to the corresponding cumulative correct answer rate, and as the cumulative correct answer rate increases, the learning signal value (a) increases, which may act in a direction that increases the confidence score (C).

[0095] As the response time decreases, the standardized deviation value (z) increases in the negative direction, so the confidence score (C) may decrease when the weight (w1) is positive. As the learning signal value (a), such as whether the first attempt is correct or the cumulative correct answer rate by question type, increases, the confidence score (C) may increase. If a hint is used and the hint usage status (h) has a value of 1, the confidence score (C) may decrease by the product of the weight (w3) and 1. Since the confidence score (C) is calculated as a value between 0 and 1 by the logistic function, the confidence score approaches 1 as the result of the weighted combination increases, and the confidence score approaches 0 as the result of the weighted combination decreases.

[0096] In one example, when weights w1, w2, and w3 are each set to 1 and bias b is set to 0, if the standardized deviation value (z) is -2, the learning signal value (a) is 0, and the hint usage (h) is 0, the result of the weighted combination is -2 and the confidence score (C) can be calculated as approximately 0.12. If the learning signal value (a) is observed to be 1 under the same weight and bias conditions, the result of the weighted combination is -1 and the confidence score (C) can be increased to approximately 0.27.

[0097] The processor (232) may determine that the response is a random response based on whether the reliability score is below the judgment threshold (S630). The judgment threshold may be a value that serves as a criterion for determining whether to determine the response as a random response. If the reliability score is below the judgment threshold, the response is determined to be a random response, and if the reliability score is above the judgment threshold, the response may be determined to be a competent response. If the response is determined to be a random response, step S440, which excludes the response from data analysis, may be performed.

[0098] The function combining standardized deviation values ​​and learning signal values ​​is not limited to a logistic function and can be implemented by a linear function, a tree-based model, or a neural network. The calculation of the confidence score may further incorporate a term corresponding to the number of attempts or a term corresponding to the cumulative correct answer rate by question type. An embodiment in which the confidence score is calculated based on the ratio obtained by dividing the difference between the representative response time and the response time by the standard deviation plus a stabilization term may also be possible. The judgment threshold can be adaptively set differently for each user, question type, or difficulty level. The processor (232) can use a first judgment threshold and a second judgment threshold greater than the first judgment threshold to determine the confidence score as a random response if it is below the first judgment threshold, as a pending response if it is above the first judgment threshold but below the second judgment threshold, and as a skill response if it is above the second judgment threshold. After the confidence score is calculated, it can be corrected by calibration to correspond to the actual probability.

[0099] By quantitatively weighting and combining the deviation from the normal distribution and the learning signal for each individual and problem type to calculate a reliability score, the judgment of the response can be performed based on objective numerical values. By calculating the reliability score as a value between 0 and 1 using a logistic function and adjusting the judgment threshold, the sensitivity of the judgment can be controlled to suit the user or the problem type. Through a process of calculating standardized deviation values ​​and reliability scores from response times and learning signals and comparing the reliability score with the judgment threshold, the judgment of random responses can be performed based on quantitative criteria, and the quality of the data used for learning analysis can be improved. The operation of updating the representative response time and dispersion associated with the normal response time range using a correction rule is described in detail later in FIG. 7. The calculation steps, mathematical formulas, and functions illustrated in FIG. 6 may be examples, and the method of judging the response based on the reliability score can be changed to any calculation method that combines the deviation of the response time and the learning signal.

[0100] FIG. 7 is a flowchart illustrating a method for updating a normal response time range by a correction rule according to an embodiment of the present invention.

[0101] Hereinafter, FIG. 7 is described with reference to FIG. 2 and FIG. 4 together, and the description above with reference to FIG. 1 through 6 is omitted. FIG. 7 may be a detailed representation of step S450 of FIG. 4, that is, the operation of updating the normal response time range, based on a correction rule. The method of updating the normal response time range of FIG. 7 may be performed by a processor (232). The update of FIG. 7 may be performed based on the fact that the response is determined to be a valid response in step S430 of FIG. 4, which determines the response.

[0102] The correction rule is a rule that adaptively updates the standard for the normal response time range and may include an exponentially weighted moving average, a moving average, a Bayesian update, and / or a Kalman filter. Dispersion is a measure of the scattering of response times and may include variance, standard deviation, and / or interquartile range.

[0103] Based on the determination that a response is a competent response, the processor (232) can update the representative response time and dispersion associated with the normal response time range by reflecting the response time (S451). Since only the response time of the response determined to be a competent response is reflected and the response time of the random response is not reflected, the standard can be prevented from being contaminated by the random response. The representative response time and dispersion can be updated by an exponentially weighted moving average. The representative response time and dispersion can be updated by the following mathematical formula 2.

[0104] [Mathematical Formula 2]

[0105] μ' = (1 - α)μ + αt

[0106] v' = (1 - α)v + α(t - μ')²

[0108] In mathematical formula 2, μ' represents the updated representative response time, μ represents the representative response time before the update, t represents the response time of the response judged as a skill response, α represents the learning rate, v' represents the updated dispersion, and v represents the dispersion before the update. The learning rate (α) can be a value between 0 and 1, and the larger the learning rate, the more the recent response time is reflected in the representative response time and dispersion. The smaller the learning rate (α), the longer the past response time is maintained, and the representative response time and dispersion may change slowly. In one example, if the representative response time before the update (μ) is 2000 milliseconds, the learning rate (α) is 0.2, and the response time of the skill response (t) is 1500 milliseconds, the updated representative response time (μ') can be calculated as 1900 milliseconds.

[0109] The processor (232) can update the lower limit of the normal response time range by subtracting the value obtained by multiplying the square root of the updated dispersion by a constant from the updated representative response time (S452). The lower limit (L) can be updated by the value obtained by subtracting the product of the constant (c) and the square root of the updated dispersion from the updated representative response time (μ'), i.e., by the following mathematical formula 3.

[0110] [Mathematical Formula 3]

[0111] L = μ'- c·sqrt(v')

[0112] In mathematical formula 3, L represents the lower limit of the normal response time range, μ' represents the updated representative response time, c represents a constant, and v' represents the updated dispersion. As the constant (c) increases, the lower limit decreases, and the range of candidates for random responses that are shorter than the lower limit may be narrowed. As the updated dispersion (v') increases, that is, as the dispersion of response times increases, the lower limit decreases, and the criteria for determining a random response may be relaxed for users with large fluctuations in response time.

[0113] The correction rule may further include a time-lapse weighting that applies forgetting to past statistics over time. The forgetting weighting applied to past statistics that have elapsed without being updated can be set by the following mathematical formula 4.

[0114] [Mathematical Formula 4]

[0115] γ = exp(-delta(t) / tau)

[0116] In mathematical equation 4, γ represents the forgetting weight, delta(t) represents the elapsed time since the last update, and tau represents the time constant. As the elapsed time (delta(t)) becomes longer, the forgetting weight (γ) approaches 0, so that past statistics are less reflected, and as the elapsed time (delta(t)) becomes shorter, the forgetting weight (γ) approaches 1, so that past statistics can be maintained.

[0117] Representative response times and dispersions can be updated individually by problem type or learning item. In the initial stage when there is a lack of accumulated samples, the population prior distribution calculated from the response time distribution of the user group can be used as the initial value for representative response times and dispersions. The processor (232) can update representative response times and dispersions by excluding outliers from the response times of the skill responses or by reducing the influence of outliers. The processor (232) can update the upper limit along with the lower limit of the normal response time range. Asymmetric correction can be performed in which different constants are applied to directions shorter and longer than the representative response time.

[0118] By reflecting only the skill response, the representative response time and dispersion are updated and the lower limit is recalculated, thereby allowing the judgment criteria to be adaptively corrected to match changes in the user's actual skill level. By adjusting the learning rate and constants, the speed of adaptation of the criteria and the range judged by random responses can be controlled to suit the user or the problem format. By processing to update the representative response time, dispersion, and lower limit by reflecting the response time of the skill response, the judgment criteria can be automatically personalized and the quality of the data used for learning analysis can be improved. The operation for verifying the mastery of the learning item will be described in detail later in FIG. 8. The update steps and mathematical formulas illustrated in FIG. 7 may be examples, and the method of updating the normal response time range by the correction rule can be changed to any method that adaptively updates the criteria by reflecting the skill response.

[0119] FIG. 8 is a flowchart illustrating a method for verifying mastery of a learning item according to an embodiment of the present invention.

[0120] Hereinafter, FIG. 8 is described with reference to FIG. 2 and FIG. 4, and the description above with reference to FIG. 1 to FIG. 7 is omitted. A method for verifying the mastery of a learning item in FIG. 8 can be performed by a processor (232). A temporary mastery item may refer to a learning item in which mastery is temporarily recognized, a mastery item may refer to a learning item in which mastery is confirmed, and an unmastery item may refer to a learning item in which mastery is not recognized.

[0121] The processor (232) may determine a learning item as a temporary mastery item based on the fact that a skill response for the same learning item has been reached consecutively a preset number of times (S810). The preset number of times may be the number of consecutive skill responses required to determine the learning item as a temporary mastery item.

[0122] The processor (232) may provide a re-verification problem belonging to a learning item based on the fact that a preset period has elapsed after the learning item is determined to be a temporary mastery item (S820). The re-verification problem may be a problem that re-evaluates the same learning item after a time interval. The re-verification problem may be provided through the learning screen of FIG. 1. The format of the re-verification problem may be the same or different from the format of the problem when it is determined to be a temporary mastery item.

[0123] The processor (232) can receive a reconfirmation response and a response time for a reconfirmation problem (S830). The processor (232) can receive the reconfirmation response and a response time from the user terminal (210).

[0124] The processor (232) can determine whether the reconfirmation response is a random response or a skill response based on the reconfirmation response time (S840). The determination of whether the reconfirmation response is a random response or a skill response can be performed in the same manner as the determination in step S430 of FIG. 4.

[0125] The processor (232) can determine the learning item as a mastery item based on the fact that the reconfirmation response is correct and determined to be a skill response (S850). The processor (232) can degrade the learning item to an unmastery item based on the fact that the reconfirmation response is incorrect or determined to be a random response (S860). Step S850 is performed when the reconfirmation response is correct and simultaneously determined to be a skill response, and Step S860 is performed when the reconfirmation response is incorrect or determined to be a random response.

[0126] The preset period can be set according to the reconfirmation point of spaced repetition calculated based on the forgetting curve. The reconfirmation question is provided in a question format different from the question format when it is determined as a temporary mastery item, so that the transition between question formats can be verified. Learning items that have been demoted to non-mastery items may be included in the re-learning path and re-issued. The processor (232) can manage the degree of mastery by scoring it as partial mastery and mastery. The processor (232) can provide information regarding mastery items and non-mastery items as a report for guardians or teachers. Alternatively, the provision of reconfirmation questions for multiple temporary mastery items (S820) and the reception of reconfirmation responses (S830) may be performed at different times for each learning item.

[0127] By determining provisional mastery through the continuous attainment of skill responses and reconfirming with a time delay, the likelihood of mastery achieved by chance or randomly being confirmed as a mastery item can be reduced. By performing confirmation as a mastery item or demotion to an unmastery item based on the result of reconfirmation, mastery can be managed in a verified state rather than as a fixed result. Through the process of confirming a mastery item or demotion to an unmastery item via continuous attainment, time-delayed reconfirmation, and re-evaluation, the mastery status of learning is managed through verification, and the reliability of learning analysis can be improved. The question setting strategy and the transition of question formats after demotion to an unmastery item will be described in detail later in Fig. 9. The steps and sequence of verification illustrated in Fig. 8 may be examples, and the method of verifying the mastery of a learning item may be changed to any combination of steps including repeated attainment and time-delayed reconfirmation.

[0128] FIG. 9 is a flowchart illustrating a method for automatically configuring a problem format according to an embodiment of the present invention.

[0129] Hereinafter, FIG. 9 is described together with FIG. 2, and the description above with reference to FIG. 1 to FIG. 8 is omitted. The method of configuring the problem format of FIG. 9 can be performed by a processor (232).

[0130] The processor (232) can diagnose the strengths and weaknesses of a learning item by question type based on the accumulated learning log (S910). The strength type may refer to a question type in which the user shows strength regarding the learning item, and the weakness type may refer to a question type in which the user shows weakness. The processor (232) can diagnose the strengths and weaknesses by question type using the learning signal of FIG. 5 and the confidence score of FIG. 6. The processor (232) can diagnose a question type in which the correct answer rate by question type is greater than or equal to a first threshold value as a strength type, and a question type in which the correct answer rate by question type is less than or equal to a second threshold value as a weakness type. The first threshold value and the second threshold value may be set to the same value, or the first threshold value may be set to a value greater than the second threshold value. Even for the same learning item, strengths and weaknesses may be diagnosed differently depending on the question type.

[0131] The processor (232) can configure the evaluation by focusing on weak formats and converting strong formats into application / challenge formats (S920). Focused questioning refers to repeatedly presenting questions of the weak format, while application / challenge format refers to a format converted into an application or challenge format instead of repeating the strong format at the same difficulty level. Instead of adjusting the difficulty level, the processor (232) can configure the evaluation by converting question formats according to the strengths and weaknesses of each question format of the learning item. The processor (232) can present multiple questions corresponding to the weak format and questions corresponding to the application format or challenge format converted from the strong format to the user.

[0132] The processor (232) can branch into a reinforcement mode, a stabilization mode, or a reconstruction mode based on the accuracy rate, response speed, retries, and hint signals (S930). The reinforcement mode may refer to a strategy for reinforcing weak parts, the stabilization mode to a strategy for maintaining acquired parts, and the reconstruction mode to a strategy for performing relearning. The processor (232) may branch into a reinforcement mode or a reconstruction mode if the accuracy rate is low or many retries or hints are observed, and may branch into a stabilization mode if the accuracy rate is high and the response speed is stable.

[0133] The processor (232) may demote a mastered item to an unmastered item based on the fact that it was later found to be incorrect (S940). A learning item demoted to an unmastered item may be included again in the composition of the evaluation and presented. Refer to FIG. 8 for the mastery and demotion of learning items.

[0134] The processor (232) can configure the evaluation using a model that recommends question formats based on strengths and weaknesses of question formats. The processor (232) can adaptively adjust the difficulty level along with the question formats. The composition of the evaluation may reflect diversity in question types and a dispersion of formats to reduce user boredom. Gamification elements may be reflected in the composition of the evaluation. The processor (232) can automatically generate a curriculum based on the strengths and weaknesses of question formats of learning items. The processor (232) can provide a dashboard for class organization or for teachers by comparing the diagnostic results for multiple users.

[0135] Instead of a fixed test being provided identically to all users, the composition of the evaluation is configured differently for each user based on the strengths and weaknesses of each question format of the learning items, thereby allowing weak formats to be intensively reinforced and strong formats to be converted into application and challenge formats. The composition of the evaluation can be adjusted to the user's learning status through branching of modes based on the correct answer rate, response speed, retry, and hint signals. Through processing that includes diagnosing strengths and weaknesses by question format, intensively presenting weak formats and converting to strong formats, and branching of modes based on signals, the composition of the evaluation can be automatically adjusted to the user's learning status, and learning efficiency can be improved. The steps and modes of the composition illustrated in FIG. 9 may be examples, and the method of configuring the question formats may be changed in any way that adjusts the composition of the evaluation based on strengths and weaknesses by question format.

[0136] The various embodiments described above may be exemplary, and those skilled in the art may practice the invention in modified forms without departing from the essential characteristics of the invention. The steps, configurations, and numerical values ​​described in each figure may be illustrative, and the order of steps or combination of configurations may be changed according to the embodiment.

[0137] Methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software. Methods according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium, or may be implemented as a computer program stored on a computer-readable recording medium in combination with hardware.

[0138] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0139] Such programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic disc storage devices, compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, magnetic cassettes. Alternatively, they may be stored in memory composed of some or all of these. Additionally, each constituent memory may include multiple units.

[0140] Additionally, the program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WAN (wide area network), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0141] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0142] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0143] 110: First learning screen 120: Second learning screen

Claims

Claim 1 A method for providing a user-customized learning service, performed by one or more processors, comprising: a step of obtaining a user's first response to a first problem of learning content and a first response time taken for said first response; a step of calculating a normal response time range of said user for a problem format to which said first problem belongs, based on a cumulative learning log including whether the answer is correct and the response time for each of a plurality of problems answered by said user; a step of determining said first response to said first problem as a random response or a skill response based on the fact that said first response time is shorter than the lower limit of said normal response time range; and a step of excluding said first response from data analysis based on determining said first response as a random response. The method includes the step of updating the normal response time range by reflecting the first response time based on determining the first response as the skill response, and the step of determining the learning item as a temporary mastery item based on the fact that the skill response for the same learning item as the first problem, including the first response to the first problem, has been reached consecutively a preset number of times; the step of providing the user with a second problem belonging to the learning item based on the fact that a preset period has elapsed after the learning item has been determined as the temporary mastery item; the step of receiving a second response to the second problem and a second response time taken for the second response; the step of determining the second response as the random response or the skill response based on the second response time; and the step of confirming the learning item as a mastery item based on the fact that the second response is the correct answer and determining the second response as the skill response. A method for providing a user-customized learning service, further comprising the step of demoting the learning item to an unskilled item based on the second response to the second problem being incorrect or the second response being determined to be the random response. Claim 2 A method for providing a user-customized learning service according to claim 1, wherein the step of determining the first response to the first problem as a random response or a skill response comprises: a step of obtaining at least one of whether the first attempt to the first problem was correct, the number of attempts to the first problem, whether a hint was used for the first problem, or a cumulative correct answer rate by problem type; and a step of determining the first response to the first problem as the random response or the skill response based on the at least one obtained. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete

Images