Information processing method, information processing system, information terminal, and computer program

Abstract

An information processing method for executing a process for determining a learning level of a user or presenting the determined learning level is provided. The information processing method includes: an input step for inputting time-series media information representing an action or behavior of the user in learning; a first determination step for determining a learning level of the user based on the time-series media information; and an output step for outputting a portion in which the action or behavior of the user in the time-series media information is different from a reference action or behavior based on the learning level of the user determined in the first determination step.

Description

Technical Field

[0001] The technology disclosed in this specification (hereinafter referred to as "this disclosure") relates to information processing methods, information processing systems, information terminals, and computer programs for performing processing that assists in learning performed by a user. Background Technology

[0002] In recent years, information technology has been increasingly used to assist user-driven learning in areas such as language learning, musical instrument learning, and sports training (golf, baseball, soccer, etc.). For example, a sound learning system for learning a second language through sound has been proposed. In this system, a computer-executed level determination program determines the learning level based on the learner's vocalization data and adjusts the reproduction speed of second language sentences to match the learner's level (see Patent Document 1). Furthermore, an information processing apparatus has been proposed that acquires sensor information representing information related to a first user playing golf from sensor information obtained using sensors attached to a golf club, acquires feedback information from a second user's terminal as feedback to first generated information based on the sensor information, and transmits this feedback information to the first user's terminal (see Patent Document 2).

[0003] [List of cited references]

[0004] [Patent Literature]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-113904

[0006] [Patent Document 2] PCT Patent Publication No. WO2018 / 220948 Summary of the Invention

[0007] [Technical Issues]

[0008] The purpose of this disclosure is to provide information processing methods, information processing systems, information terminals, and computer programs for performing processing that assists in learning performed by a user.

[0009] [Solution to the problem]

[0010] This disclosure is made in view of the aforementioned problems, and its first aspect is an information processing method comprising: an input step, wherein the input step is: inputting time-series media information representing a user's actions or behaviors during learning; a first determination step, wherein the first determination step is: determining the user's learning level based on the time-series media information; and an output step, wherein the output step is: based on the user's learning level determined in the first determination step, outputting the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior. Here, in the first determination step, after processing the time-series media information using a self-trained first machine learning model, the user's learning level is determined using a supervised-trained second machine learning model. Furthermore, in the first determination step, an attention mechanism included in the second machine learning model is used to determine, based on the time-series media information, whether the user's learning level is insufficient or whether the user needs to learn.

[0011] In the output step, the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior is output to the presentation device. Furthermore, the information processing method according to the first aspect also includes a first presentation step, which involves presenting the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior to the user via the presentation device. In the first presentation step, the portion of the user's actions or behaviors in the time-series information media that differs from a reference action or behavior is presented visually.

[0012] Furthermore, the information processing method according to the first aspect also includes a second determining step, which involves determining distance information representing the difference between a user's action or behavior and a reference action or behavior; and a second presentation step, which involves outputting the determination result from the second determining step to a presentation device and presenting the result to the user. In the second presentation step, the distance information is visually presented in an N-dimensional space, in which the reference action or behavior is arranged in the center.

[0013] Furthermore, a second aspect of this disclosure is an information processing system comprising: an input unit that inputs time-series media information representing a user's actions or behaviors during learning; a first determination unit that determines the user's learning level based on the time-series media information; and an output unit that, based on the user's learning level determined by the first determination unit, outputs the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior. The information processing system according to the second aspect may further include: a sensor unit that detects the user's actions or behaviors during learning and acquires the time-series media information; and a presentation device used by the output unit to output the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior.

[0014] It should be noted that the term "system" as used herein refers to an object that is a logical collection of multiple devices (or functional modules that perform a specific function), and it is not particularly important whether each device or functional module is in a single housing. In other words, a device that includes multiple components or functional modules and a collection of multiple devices are both equivalent to a "system".

[0015] Furthermore, a third aspect of this disclosure is an information terminal, which includes: a sensor unit that detects user actions or behaviors during learning and acquires time-series media information; a communication unit that transmits the time-series media information to an external device and receives from the external device a determination result regarding the user's learning level and the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior; and a presentation unit that presents the received information.

[0016] Furthermore, a fourth aspect of this disclosure is a computer program written in a computer-readable format, which enables a computer to function as: an input unit that inputs time-series media information representing the actions or behaviors of a user in learning; a first determining unit that determines the user's learning level based on the time-series media information; and an output unit that, based on the user's learning level determined by the first determining unit, outputs the portion of the user's actions or behaviors in the time-series media information that differs from a reference action or behavior.

[0017] The computer program according to the fourth aspect of this disclosure defines a computer program written in a computer-readable format that enables predetermined processing to be performed on a computer. In other words, by installing the computer program according to the fourth aspect of this disclosure on a computer, a synergistic effect is exhibited on the computer, and similar effects and advantages as the information processing method according to the first aspect of this disclosure can be obtained.

[0018] [Beneficial effects of the invention]

[0019] This disclosure provides information processing methods, information processing systems, information terminals, and computer programs for performing processes to determine a user's learning level or to present the determined learning level.

[0020] Note that the advantages described in this specification are illustrative only, and the advantages provided by this disclosure are not limited thereto. Furthermore, in some cases, this disclosure achieves additional advantages beyond those described above.

[0021] Other objects, features, and advantages of this disclosure will become apparent from a more detailed description based on the embodiments and accompanying drawings, which will be described later. Attached Figure Description

[0022] [ Figure 1 ] Figure 1 This is a diagram depicting the basic configuration of an information processing system 100 that assists in learning performed by a user.

[0023] [ Figure 2 ] Figure 2 This is a flowchart depicting an operational example of the information processing system 100.

[0024] [ Figure 3 ] Figure 3 This is a flowchart depicting another operational example of the information processing system 100.

[0025] [ Figure 4 ] Figure 4 It is a diagram depicting an example of a definite result presented visually.

[0026] [ Figure 5 ] Figure 5 It is a diagram depicting an example of a definite result presented visually.

[0027] [ Figure 6 ] Figure 6 It is a diagram depicting an example of a definite result presented visually.

[0028] [ Figure 7 ] Figure 7 It is a diagram depicting an example of system configuration.

[0029] [ Figure 8 ] Figure 8 This is a diagram depicting another example of system configuration.

[0030] [ Figure 9 ] Figure 9 This is a diagram depicting yet another example of system configuration.

[0031] [ Figure 10 ] Figure 10 This is a diagram depicting yet another example of system configuration.

[0032] [ Figure 11 ] Figure 11 It is a diagram depicting an example of distance information on a two-dimensional plane that represents the difference between a user's action or behavior and a reference action or behavior.

[0033] [ Figure 12 ] Figure 12 It is a graph that depicts how distance information is presented as learning progresses.

[0034] [ Figure 13 ] Figure 13 It is a diagram depicting an example of distance information in three-dimensional space that represents the difference between a user's action or behavior and a reference action or behavior.

[0035] [ Figure 14 ] Figure 14 This is a diagram depicting an example of the internal configuration of the deterministic unit 103, which includes a DNN.

[0036] [ Figure 15 ] Figure 15 This is a diagram used to illustrate the operation of the attention mechanism.

[0037] [ Figure 16 ] Figure 16 This is a diagram illustrating the method for calculating triplet loss.

[0038] [ Figure 17 ] Figure 17 This is a diagram illustrating an example of the internal configuration of the determination unit 103, which focuses on the differences between training methods.

[0039] [ Figure 18 ] Figure 18 This is a diagram used to illustrate the self-training method of the self-trained model 1701.

[0040] [ Figure 19 ] Figure 19 It is a diagram depicting the relationship between learning content that can be aided by this disclosure and the types of time-series media information.

[0041] [ Figure 20 ] Figure 20 This is a diagram illustrating a configuration example of a UI that presents a definitive result about a user's learning level to the user.

[0042] [ Figure 21 ] Figure 21 This is a diagram illustrating a configuration example of a UI that presents a definitive result about a user's learning level to the user.

[0043] [ Figure 22 ] Figure 22This is a diagram illustrating a configuration example of a UI that presents a definitive result about a user's learning level to the user.

[0044] [ Figure 23 ] Figure 23 This is a diagram depicting a configuration example of the information processing device 2300.

[0045] [ Figure 24 ] Figure 24 This is a diagram depicting a configuration example of information terminal 2400. Detailed Implementation

[0046] The contents of this disclosure will now be described in the following order with reference to the accompanying drawings.

[0047] A. Overview

[0048] B. Basic Configuration

[0049] B-1. Function Block

[0050] B-2. System Operation

[0051] B-3. ​​Determine the method for presenting results

[0052] B-4. Specific System Configuration Example

[0053] B-5. Presentation of Distance Information

[0054] C. Implementation using machine learning models

[0055] C-1. Configuration of Machine Learning Models

[0056] C-2. Training Methods

[0057] D. Application Examples

[0058] E.UI Example

[0059] F. Device Configuration Example

[0060] F-1. Example of Information Processing Device Configuration

[0061] F-2. Information Terminal Configuration Example

[0062] A. Overview

[0063] In recent years, information technology has been increasingly used to assist user-driven learning in areas such as language learning, musical instrument learning, and sports training (golf, baseball, soccer, etc.). For example, computers can be used to determine a learner's level (see Patent Document 1). However, it can be argued that simply presenting a level indicating learning progress is insufficient as feedback to the user. That is, a user may struggle to identify future challenges simply by knowing their current level, and they may not be able to pinpoint the differences between their current level and the model they are trying to emulate, or the extent to which those differences are present.

[0064] In view of this, this disclosure proposes a method for determining a user's learning level regarding an action or behavior performed by a user from time-series media information (e.g., video or audio) representing an action or behavior, presenting the determined result to the user, and, if the determined result indicates that learning is insufficient, also presenting learning progress information indicating which part of the user's action or behavior is insufficiently learned and the degree of that insufficient learning. Furthermore, this disclosure proposes apparatus for implementing this method.

[0065] Note that, unless otherwise specified, this instruction manual assumes that the user is a “learner” attempting to master a language, musical instrument, sport, etc.

[0066] When a user attempts to master, for example, a second language, she / he learns to make the sounds she / he produces or the sentences she / he writes more closely resemble those produced or written by a native speaker. The method according to this disclosure can present the user with a definite result regarding whether the sounds or sentences produced or written by the user closely resemble those produced or written by a native speaker. Furthermore, this disclosure can visually present information about which parts of the sounds or sentences produced or written by the user differ from those produced or written by a native speaker, and the degree of that difference. Therefore, when a user attempts to master a second language, even without a native speaker, private tutor, etc. (i.e., even if she / he is self-studying), the user can understand the difference between her / his current pronunciation or sentences and the pronunciation or sentences she / he should strive to achieve, and can train efficiently to master the language. Needless to say, this disclosure can be applied not only to language learning but also to situations where users attempt to master various actions or behaviors involving sound production (e.g., as seen when a user participates in singing or playing a musical instrument, giving a speech, participating in a performance, or performing stand-up comedy).

[0067] Furthermore, when a user attempts to master a sport (such as ball sports like golf, tennis, football, or baseball; or martial arts like judo, karate, swordsmanship, or boxing), the user trains to make their physical movements (swings, kicks, techniques, catches, etc.) resemble those of a professional athlete or coach. The method according to this disclosure can present the user with a determination of whether their physical movements resemble those of a professional athlete or coach. This determination is based on video capturing images of the user during training or competition, and visually presents information about which parts of the user's physical movements differ from the movements of a professional athlete or the instructions of a coach, and the degree of that difference. Therefore, when performing sports training, even without professional athletes or coaches around, the user can understand the difference between their current physical movements and the movements they should strive to achieve, and perform training efficiently. Needless to say, this disclosure can be applied not only to sports but also to situations where users attempt to master various physical movements (e.g., when a user participates in playing a musical instrument, practicing calligraphy, cooking, giving a speech, participating in a performance, or performing stand-up comedy or humorous skits).

[0068] B. Basic Configuration

[0069] B-1. Function Block

[0070] Figure 1 The basic configuration of an information processing system 100 to which this disclosure is applied and to assist learning performed by a user is schematically depicted.

[0071] The sensor unit 101 includes an image sensor, such as a camera device, that detects video, sound, etc., representing actions or behaviors performed by a user, and a sound sensor, such as a microphone. The sensor unit 101 outputs time-series media information (e.g., video or sound) representing actions or behaviors performed by the user.

[0072] The determination unit 103 receives input from the sensor unit 103 via the input unit 102, representing time-series media information (e.g., video or audio) indicating an action or behavior performed by the user. The determination unit 103 then determines the learning level of the user's action or behavior within the time-series media information and presents the determined result to the user. For example, in the case of a user attempting to master a second language conversation, the determination unit 103 determines whether the user's pronunciation is similar to that of a native speaker (i.e., whether the user's pronunciation is at a native speaker level and requires no further learning, or whether the user's pronunciation differs from a native speaker's and therefore requires continued learning), and outputs a determination result regarding whether further learning is needed. Furthermore, if it is determined that the user needs to continue learning, the determination unit 103 determines which part of the time-series media information is insufficiently learned and the degree of inadequacy, and outputs a determination result. The determination unit 103 performs processing to determine the user's learning level using a trained machine learning model, which will be described in detail later.

[0073] The presentation unit 104 presents the result output from the determination unit 103 to the user, and this result is a determination of whether the user needs to learn or not. The presentation unit 104 also presents to the user the portions of the time-series media information that are determined to be insufficiently learned (note that this applies if it is determined that the user needs to continue learning). The presentation unit 104 includes a display that visually presents the determination result made by the determination unit 103, and may also include a sound output device such as a speaker to enable the information to also be presented as an audio broadcast. In particular, if the relevant portions of the time-series media information are visually presented on the display screen, the user can easily understand which part of their action or behavior is insufficiently learned and the degree of that insufficient learning.

[0074] B-2. System Operation

[0075] Figure 2 An operational example of the information processing system 100 is depicted in the form of a flowchart. The activation of this operation is triggered by a user in the learning process, for example, by giving instructions to determine the learning level.

[0076] The sensor unit 101 detects video or sound representing actions or behaviors performed by the user in the learning process using an image sensor or a sound sensor, and outputs the video or sound as time-series media information (step S201).

[0077] The determination unit 103 receives input of time-series media information via the input unit 102 and determines the learning level of the action or behavior performed by the user (step S202). Furthermore, if there are insufficiently learned portions, in step S202, the determination unit 103 determines which portion of the time-series media information is insufficiently learned and the degree of insufficiency. The determination unit 103 performs the following process: using a trained machine learning model, it determines the user's learning level based on the time-series media information. Then, if the user's action or behavior is close to a reference action or behavior, the determination unit 103 determines that the user's learning is sufficient (yes in step S203); however, if the user's action or behavior is not close to the reference action or behavior, the determination unit 103 determines that the user's learning is insufficient (no in step S203).

[0078] For example, when applying this disclosure to the acquisition of a second language, if the user's pronunciation or sentences are at the level of a native speaker, determination unit 103 determines that the user's learning is sufficient; otherwise, it determines that the user's learning is insufficient. Furthermore, when applying this disclosure to sports training, if the user's physical movements closely resemble those of a professional athlete or coach, determination unit 103 determines that the user's learning is insufficient.

[0079] If the determination unit 103 determines that the user's learning is sufficient (Yes in step S203), the determination unit 103 presents the determination result (that the learning is sufficient or that the learning is about to end) to the user through the presentation unit 104 (step S204), and ends the process. Alternatively, even if the user's learning is determined to be sufficient, if there are partially insufficient learning portions, the determination unit 103 may present the insufficiently learned portions of the time-series media information, and allow the user to continue learning if desired.

[0080] On the other hand, if the determination unit 103 determines that the user's learning is insufficient (no in step S203), the determination unit 103 presents the determination result of insufficient learning or needing to continue learning to the user through the presentation unit 104 (step S205), and presents the portion of the time-series media information where learning is insufficient (step S206), and ends the process.

[0081] Figure 3 Another operational example of the information processing system 100 is depicted in the form of a flowchart. This operation is activated by a user in the learning process, for example, by giving instructions to determine the learning level.

[0082] Sensor unit 101 detects video or audio representing actions or behaviors performed by a user during learning using an image sensor or an audio sensor, and outputs the video or audio as time-series media information (step S301). Then, determination unit 103 receives input of the time-series media information via input unit 102 and determines the learning level of the actions or behaviors performed by the user (step S302). Furthermore, in the case of insufficient learning, in step S302, determination unit 103 determines which part of the time-series media information is insufficiently learned and the degree of insufficient learning in that part.

[0083] Here, if the determination unit 103 determines that the user's learning is sufficient (Yes in step S303), the determination unit 103 presents the determination result that the learning is sufficient or that the learning is about to end to the user through the presentation unit 104 (step S304), and ends the process. Furthermore, even if the user's learning is determined to be sufficient, if there are partially insufficient learning portions, the determination unit 103 can present the insufficiently learned portions of the time-series media information, and allow the user to continue learning if desired.

[0084] On the other hand, if the determination unit 103 determines that the user's learning is insufficient (no in step S303), the determination unit 103 presents the determination result of insufficient learning or needing to continue learning to the user through the presentation unit 104 (step S305), and presents the portion of the time-series media information where learning is insufficient (step S306). Thereafter, the process returns to step S301, and the learning performed by the user and the detection and determination of the time-series media information representing the user's actions or behaviors during learning are repeatedly executed until it is determined that the user's learning is sufficient (yes in step S303).

[0085] B-3. ​​Determine the method for presenting results

[0086] Figure 4 An example is depicted where the presentation unit 104 visually presents information about which parts of the time-series media information are underlearned and to what extent. The example here assumes that when a user attempts to learn English pronunciation, the sound waveform signal generated when the user says the phrase "This was easy for us" is input to system 100 as time-series media information. Note, for example, that the phrase "This was easy for us" could be a phrase read aloud by the user when the English learning program presents the phrase on the screen, or it could be a phrase spoken freely by the user.

[0087] If it is determined that the phrase "This was easy for us" spoken by the user differs from the phrase spoken by a native speaker and that there are parts of the user's learning that are insufficient, the determination unit 103 determines which parts of the sound waveform signal are insufficiently learned and to what extent those parts are insufficiently learned. Then, as... Figure 4 The depicted presentation unit 104 highlights the portions of the sound waveform signal that differ from the native speaker's sound waveform signal. Additionally, as... Figure 4 As depicted, the presentation unit 104 displays the phrase "This was easy for us" spoken by the user as text along with the audio waveform signal, and highlights the words or strings "This," "eas," and "for" that are determined to be pronounced differently from the native speaker's pronunciation. Note that the highlighting method is not particularly limited to any particular method. For example, other possible methods besides highlighting the relevant words or strings (or other than highlighting the relevant words or strings) include increasing the font size, displaying the relevant words or strings in bold, changing the font, wrapping the letters, etc.

[0088] Therefore, users not only more easily recognize that their pronunciation differs from that of native speakers, but also more easily understand the parts (words or strings) in which their pronunciation differs. Then, for example, users can efficiently perform language learning by reinforcing the pronunciation of the pointed parts "This," "eas," and "for" in the same phrases, and by carefully correcting the pronunciation of the pointed parts.

[0089] Figure 5 and Figure 6 Each describes a visual representation example of a situation where, as... Figure 3 As depicted in the operational example, the detection and determination of time-series media information are repeatedly performed until it is determined that the user has learned sufficiently.

[0090] Based on such Figure 4 The depicted visual presentation involves the user pronouncing the phrase "This was easy for us," with particular attention paid to the pronunciation of the words or strings "This," "eas," and "for," which are identified as having a different pronunciation from that of a native speaker. The assumption is that the user's pronunciation improves, and the determination unit 103 determines that only the word "This" differs from the native speaker's pronunciation. In this case, as... Figure 5 As depicted, the presentation unit 104 highlights the part related to the word "This" in the input sound waveform signal, and only highlights the word "This" in the text display of the phrase "This was easy for us".

[0091] Therefore, the user can understand that their pronunciation has become closer to that of a native speaker than before, and they should be especially careful to pronounce the word "This" next time. As a result, the user's pronunciation improves further, and the user can pronounce the entire phrase "This was easy for us" at a native speaker level. Consequently, determination unit 103 can determine that the learning is sufficient and conclude the learning of the pronunciation of the phrase "This was easy for us." Additionally, as... Figure 6 The depicted presentation section 104 no longer presents any part of the sound waveform signal that differs from the native speaker's sound waveform signal in the input sound waveform signal and text display of the phrase "This was easy for us".

[0092] Note that, although in Figures 4 to 6 While omissions have been made, the text not only highlights insufficiently learned portions of the time-series media information (e.g., parts of the audio waveform signal that differ from a native speaker's), but also numerically expresses and presents the degree of inadequacy or the basis for identifying it as inadequate. Furthermore, the level of highlighting can be adjusted based on the numerical value of the aforementioned basis (e.g., increasing brightness and / or font size as the value increases). Additionally, for example, instructions on specific methods for improvement (e.g., "'this' is pronounced 'zis'. Pronounce 'th' by pressing the tip of your tongue between your upper and lower teeth.") can be displayed as pop-up messages on the screen, along with a video demonstrating the mouth movements and audio guidance. By visually representing the learning level in this way, users can understand the areas requiring reinforcement.

[0093] B-4. Specific System Configuration Example

[0094] Figure 7An example system configuration is depicted in which the sensor unit 101, input unit 102, determination unit 103, and presentation unit 104 are all mounted on a single device 700. The single device 700 described herein can be, for example, a user-carried multifunction information terminal such as a smartphone or tablet, a personal computer, or a device specifically manufactured for learning assistance. It should be noted that in other possible configurations, some parts of device 700, such as the sensor unit 101 and / or the presentation unit 104, may not be built into device 500, but can be externally connected to device 700. For example, wired interfaces such as USB (Universal Serial Bus) or HDMI (High Definition Multimedia Interface) or wireless interfaces such as Bluetooth or Wi-Fi can be used to externally connect some components to device 500.

[0095] Figure 8 An example system configuration is depicted where a first device 801, on which a sensor unit 101 is mounted, and a second device 802, including an input unit 102, a determination unit 103, and a presentation unit 104, are separated from each other. The first device 801 and the second device 802 are interconnected via a wireless or wired interface. The first device 801 includes a camera, microphone, etc., mounted at a location where actions or behaviors performed by a user in learning can be easily detected. A sensor attached to a tool used by the user in a sports competition (e.g., a swing sensor attached to a golf club) can be the first device 801. Meanwhile, the second device 802 is, for example, a multi-functional information terminal such as a smartphone or tablet computer, or a personal computer carried by the user. The second device 802 determines the user's learning level, which part of the user's actions or behaviors is not sufficiently learned, and the degree of that insufficient learning, based on time-series information acquired from the first device 801 via the wireless or wired interface, and presents the determined results visually.

[0096] Figure 9An example system configuration is depicted, in which a first device 901 with a sensor unit 101, a second device 902 with an input unit 102 and a determination unit 103, and a third device 903 with a presentation unit 104 are separated from each other. The first device 901 includes a camera, microphone, etc., mounted at a location where actions or behaviors performed by the user during learning can be easily detected. A sensor attached to a tool used by the user in a sports competition (e.g., a swing sensor attached to a golf club) can be the first device 901. The second device 902 includes a device with high computing power, such as a personal computer or cloud computer. The third device 903 includes, for example, a multi-functional information terminal carried by the user, such as a smartphone or tablet, and primarily performs only the processing of receiving determination results from the second device 902 and presenting those results to the user. In the case where the second device 902 is a cloud computer, the system can be configured to provide learning level determination processing services to multiple users or many users.

[0097] Figure 10 An example system configuration is described, in which a first device 1001, on which a sensor unit 101 and a presentation unit 104 are mounted, and a second device 1002, on which an input unit 102 and a determination unit 103 are mounted, are separated from each other. The first device 1001 includes, for example, a multi-functional information terminal such as a smartphone or tablet computer, or a personal computer carried by a user. The sensor unit 101 may be configured to be externally connected to the sensor unit of the first device 1001 rather than being built into it. Meanwhile, the second device 1002 is, for example, a cloud computer. The second device 1002 receives time-series media information from the first device 1001 and sends back to the first device 1001 a determination result about the user's learning level based on the time-series media information. Therefore, the first device 1001 obtains time-series media information from the user who is learning, sends (uploads) the time-series media information to the second device 1002, receives (downloads) the determination result about the time-series media information from the second device 1002, and presents the determination result to the user. In the case where the second device 1002 is a cloud computer, the system can be configured to provide learning level determination processing services to multiple users or many users.

[0098] B-5. Presentation of Distance Information

[0099] In the description provided so far, the determination unit 103 determines the user's learning level based on time-series media information (e.g., video or audio) representing actions or behaviors performed by the user. In cases where there are parts with insufficient learning levels, it determines which part of the time-series media information is insufficiently learned and the degree of that insufficiency. The presentation unit 104 then presents the determined results to the user. As an extension, the determination unit 103 can determine the degree of difference between the user's overall actions or behaviors and reference actions or behaviors (specifically, ideal actions or behaviors performed by native speakers, professional athletes, coaches, etc.) as N-dimensional (two-dimensional or three-dimensional) distance information, and the presentation unit 104 can present this N-dimensional distance information. The determination unit 103 can determine the distance information using a distance learning model, which will be described in detail later.

[0100] Figure 11 An example is depicted in which the difference between a user's action or behavior and a reference action or behavior is represented as distance information on a two-dimensional plane, and the presentation unit 104 presents this distance information. Figure 11 In the depicted examples, graphic 1101, representing a reference action or behavior, is displayed at the center of the two-dimensional plane 1100, while graphic 1102, representing an action or behavior performed by the user, is displayed at a location away from the center. In the case of a user attempting to learn the pronunciation of a second language, graphic 1101 in the center represents the pronunciation made by a native speaker, while graphic 1102 displayed around the center represents the current pronunciation made by the user. The distance from the center to the location where graphic 1102 is placed represents the user's current pronunciation level. Similarly, in the case of a user practicing their baseball swing, graphic 1101 in the center represents the ideal swing (or the swing of a professional baseball player), while graphic 1102 displayed around the center represents the user's current swing. The distance from the center to the location where graphic 1102 is placed represents the user's current swing skill. Furthermore, in the case of a user practicing their golf swing, graphic 1101 in the center represents the ideal golf swing (or the golf swing of a professional golfer), while graphic 1102 displayed around the center represents the user's current golf swing. The distance from the center to the location where graphic 1102 is placed indicates the user's current golf skill level.

[0101] Figure 12 This depicts how distance information is presented as it changes on a two-dimensional plane 1100 as the user progresses through learning. In the initial stages of learning, such as... Figure 12 The graph 1102, depicted in (A), representing the user's current learning level, is significantly farther away from graph 901 located at the center of the two-dimensional plane 900. Subsequently, as the user continues learning (e.g., correcting pronunciation), as... Figure 12As depicted in (B) to (D), graph 1102, representing the user's current learning level, gradually approaches graph 1101 at the center. During training, the orientation of graph 1102 relative to graph 1101 at the center changes. For example, this represents the phenomenon where, for instance, the portion of a user's action or behavior in time-series media information that differs from a reference action or behavior changes between these instances. Then, as learning continues, graph 1102 approaches graph 1101 at the center further, while changing its orientation, and as... Figure 12 As depicted in (E), graph 1102 eventually overlaps with graph 1101. This indicates that the difference between the user's action or behavior and the reference action or behavior has been reduced to a sufficiently small level and learning has been completed.

[0102] Based on such Figure 11 The visual representation of distance information allows users to understand whether their actions or behaviors are similar to or significantly different from reference actions or behaviors. Furthermore, by visually observing... Figure 12 The changes in distance information described during the training process allow users to determine, without the help of a coach or other means, whether the training has improved their movements or behaviors, i.e., whether the training is in the right direction.

[0103] In addition, when based on such Figure 11 When the visual representation of the distance information depicted determines that her / his action or behavior is significantly different from the reference action or behavior, the user can use methods such as... Figures 4 to 6 The described visual representation is used to examine the parts of a user's action or behavior in time-series media information that differ from a reference action or behavior, to understand why her / his action or behavior differs from the ideal action or behavior, and to use this understanding as a reference for future training methods.

[0104] Note that, although Figure 11 and Figure 12 An example of representing distance information on a two-dimensional plane 1100 is depicted, but it can also be done as follows: Figure 13 The depiction represents distance information in three-dimensional space. Figure 13 In the depicted example, graphic 1301 representing a reference action or behavior is displayed in the center of three-dimensional space 1300, while graphic 1302 representing a user's action or behavior is displayed at a location away from the center. Although Figure 12 It depicts how distance information is presented as it changes on a two-dimensional plane 1100 as the user learns, but it can also be presented as... Figure 13 The depiction uses a 3D space of 1300 to represent distance information in a richer way as the user learns.

[0105] C. Implementation using machine learning models

[0106] C-1. Configuration of Machine Learning Models

[0107] Item B described above indicates that the determination unit 103 has the following functions: determining the user's learning level in the learning process based on time-series media information representing actions or behaviors performed by the user; and determining the parts of the time-series media information where the actions or behaviors performed by the user differ from reference actions or behaviors. For example, such functions in the determination unit 103 can be implemented using a trained machine learning model such as a DNN (deep neural network).

[0108] Classification models using machine learning (e.g., DNNs) typically include: a feature extraction unit that extracts features from input data such as time-series media information; and a classification unit that classifies output labels based on the extracted features. In this embodiment, the classification unit classifies the user's learning into labels of sufficient learning or insufficient learning. Specifically, the feature extraction unit includes, for example, a CNN (Convolutional Neural Network), and the classification unit includes affine layers. Furthermore, by incorporating an attention mechanism into the classification unit, the portion of the time-series media information used as the basis for labeling the user's learning as insufficient can be indicated. Note that attention is one implementation method of XAI (Interpretable AI) technology for illustrating the decision-making basis of machine learning models, and is well known in the art as a technique for incorporating a mechanism for indicating points of interest in the input data (i.e., the attention mechanism) into a machine learning model.

[0109] Figure 14 An example of the internal configuration of the determination unit 103, including the DNN 1400, is schematically depicted. The following description... Figure 14 The internal configuration of the defined section 103 is depicted.

[0110] The feature extraction unit 1410 includes multiple CNNs 1411-1, 1411-2, ... and 1411-T (in... Figure 14 The example depicted uses T CNNs. Each CNN 1411-1, 1411-2, ..., or 1411-T receives time-series media information (e.g., sound waveform signals of a user's voice) 1401 divided into time periods P1, P2, P3, ..., PT of predetermined lengths. TThe input is the time-segment data. It should be noted that the time-series media information is divided into time-segment data such that each pair of adjacent time segments includes overlapping regions. Each of CNNs 1411-1, 1411-2, ..., and 1411-T extracts features from the input data using feature filters on convolutional layers, ultimately downsampling the features on pooling (global average pooling: GAP) layers and encoding these features into the input data to the downstream classification unit 1420. Figure 14 In the example depicted, the feature extraction unit 1410 outputs feature quantities z1', z2', ... and z... T ', and the characteristic quantities z1', z2', ... and z T The output is to the downstream classification unit 1420 and the distance learning model (described later) that estimates the distance information, wherein the features z1', z2', ... and z... T It is generated by combining the outputs of a predetermined number of CNNs that have received inputs of multiple consecutive time periods of data in CNN 1411-1, 1411-2, ...

[0111] As described later, a self-trained model is used in the feature extraction unit 1410. However, its purpose is not only to make it easier for the downstream classification unit 1420 to perform classification, but also to overcome the difficulty of collecting data on actions or behaviors performed by users. When the sound signal is considered as time-series media information, such as in training user pronunciation for language learning, wav2vec or wav2vec2.0 can be used as the feature extraction unit 1410. Furthermore, when video is considered as time-series media information for training user body movements such as sports, CVRL (Contrastive Video Representation Learning) or PCL (Surrogate Contrastive Learning) can be used as the feature extraction unit 1410.

[0112] The classification unit 1420 classifies user actions or behaviors based on the feature quantities of user actions or behaviors extracted by the feature extraction unit 1410, determining whether the user's actions or behaviors closely resemble reference actions or behaviors (i.e., whether the user's learning has been sufficiently performed). Figure 14 In the depicted example, the classification unit 1420 includes a Bi-LSTM layer 1421 and a classifier layer 1423, and has an attention mechanism 1422 contained therein.

[0113] Bi-LSTM layer (bidirectional LSTM) layer 1421 is an improved neural network called bidirectional LSTM (long short-term memory) RNN (recurrent neural network), and integrates the results obtained by performing LSTM from left to right and the results obtained by performing LSTM from right to left.

[0114] Here, we briefly explain neural networks. A neural network consists of three layers: an input layer, intermediate layers (or hidden layers), and an output layer. Each layer has the necessary number of unit elements called neurons. Neurons in the input layer are coupled to each other by assigning weights to them. Similarly, neurons in the intermediate layers are coupled to each other by assigning weights to them. Then, for example, data such as features are input, and the neural network is trained by updating the weighting factors through error backpropagation, which backpropagates the error signal, so that the correct recognition result can be output. The error signal described here represents the difference between the output signal of the output layer and the teacher signal. An RNN is a neural network with loops in it, and it can dynamically update its internal state while storing past information. An LSTM is configured by replacing the intermediate layers of an RNN with LSTM blocks to create a model for long-term data. An LSTM block has three gates: an input gate, a forget gate, and an output gate, as well as storage units. The storage units represent the internal state and can retain long-term information. The input and output gates are used to adjust the input and output data, respectively. The forget gate is used to adjust the memory units input from previous times.

[0115] Refer again Figure 14 Continuing with the explanation of DNN 1400, classifier layer 1423 determines whether the user's learning is sufficient based on the inference results of the Bi-LSTM layer 1421 on the time-series media information, and outputs the determination result. For example, in the case of language learning, classifier layer 1423 determines whether the user's pronunciation is at the level of a native speaker. Additionally, in the case of sports training, classifier layer 1423 determines whether the user's body movements are at the level of a professional athlete or coach.

[0116] If the classifier layer 1423 determines that the user's learning is insufficient, the attention mechanism 1422 detects the portion of the input time-series media information that serves as the basis for determining that the user's learning is insufficient. Attention is one method for implementing XAI technology, and it is well-known in the art as a technique for incorporating points of interest in the input data into a machine learning model when the model performs classification. For example, in the case of language learning, when the classifier layer 1423 determines that the user's pronunciation differs from that of a native speaker, the attention mechanism 1422 can, as follows: Figure 4The described method identifies portions of the sound waveform signal that differ from those of a native speaker. Furthermore, given a known correspondence between the sound waveform signal and a string (e.g., “This was easy for us”), the attention mechanism 1422 can also identify words or strings that are determined to be pronounced differently from the native speaker's pronunciation.

[0117] Reference Figure 15 Explain the operation of attention mechanism 1422. It should be noted that... Figure 15 In the described scenario, it is assumed that the input raw audio waveform signal 1501 is time-series media information. The feature extraction unit 1410, configured using wav2vec, outputs feature quantities z1', z2', z3', ..., z... extracted from the audio waveform signal 1501. T Then, classification 1423 (not in) Figure 15 (Depicted in the middle) Based on feature quantities z1', z2', z3', ... and z T 'It is determined that the user's voice waveform signal 1501 differs from the reference voice waveform signal (or the ideal voice waveform signal of a native speaker). At this time, the attention mechanism 1422 indicates which time periods of the input voice waveform signal the classification unit 1423 has noticed in order to make a determination. In Figure 15 In the depicted example, the attention mechanism 1422 uses values ​​ranging from 0 to 1—0.38, 0.71, 0.42, ..., 0.92—to represent the contribution of each time segment in the sound waveform signal as a determination that "the sound waveform signal is different from the ideal sound waveform signal." It should be noted that larger values ​​indicate a greater contribution to the determination result. Based on the output of the attention mechanism 1422, the presentation unit 104 can highlight the portions of the sound waveform signal that differ from the native speaker's sound waveform signal, as indicated by reference numeral 1502. Furthermore, the presentation unit 104 adjusts the level of highlighting for the multiple portions of the sound waveform signal that differ from the native speaker's sound waveform signal based on the calculated values.

[0118] In addition, although not in Figure 14 As described in section B-5 above, the determining unit 103 determines distance information representing the degree of difference between the overall action or behavior performed by the user and a reference action or behavior by using a distance learning model. The distance learning model is trained such that it is based on feature quantities z1', z2', z3', ... and z... extracted by the feature extraction unit 1410 from the time-series media information that is the input data. T 'Using feature vectors as element representations to estimate the distance between a user's action or behavior and a reference action or behavior.'

[0119] Distance learning models can be trained using basic loss functions such as contrastive loss or triplet loss. Here, contrastive loss is the loss calculated based on the distance between two points. Additionally, as... Figure 16 The described triplet loss is a loss calculated for a set of three features: an anchor feature used as a reference feature, a positive feature with the same label as the anchor, and a negative feature with a different label than the anchor. Furthermore, if the anchor, positive, and negative features are arranged as vectors in space, the distance between the anchor and the positive feature is defined as d. p And the distance between the anchor and the negative is defined as d. n L can be triplet =[d p -d n +α]+ is defined as the triplet loss (note that α is a hyperparameter representing the margin). Figure 16 The upper part describes a method for calculating triplet loss when the professional (or ideal) is considered positive and the learner is considered negative. Furthermore, Figure 16 The lower part of the text describes a method for calculating triplet loss when learners are considered positive and professional (or ideal) is considered negative.

[0120] The determination unit 103 can determine distance information about the distance between a user's action or behavior and a reference action or behavior by using a trained distance learning model, based on time-series media information obtained by sensing actions or behaviors performed by the user. Then, as... Figure 11 As depicted, the presentation unit 104 can visualize and present the difference between the user's action or behavior and a reference action or behavior as distance information on a two-dimensional plane. Therefore, the user can, based on... Figure 11 The visual information depicted is used to understand whether her / his actions or behaviors are close to or very different from the reference actions or behaviors.

[0121] C-2. Training Methods

[0122] Next, the training method of the machine learning model used by the determination unit 103 will be explained.

[0123] Deep learning requires a large amount of training data. If attempting to perform supervised training of a DNN 1400 to make determinations about actions or behaviors performed by a user, the workload of collecting large amounts of user action data (sound, video, etc.) and performing annotations to label each data point is excessive. Without sufficient training, problems such as DNN 1400 instability or determination errors can occur.

[0124] For example, supervised training of the taxonomy 1420 downstream of the DNN 1400 is performed using motion data from users (or beginners at the same level as the users) and ideal motion data from professionals. However, collecting user motion data is often difficult. For instance, in the case of language learning, native speaker voice data can be collected relatively easily through various types of media such as television or radio broadcasts or video distribution services on the Internet, but it is difficult to collect voice data of learners' pronunciations.

[0125] In view of this, according to the recommendations of this disclosure, the feature extraction unit 1410 of the DNN 1400 used at the determination unit 103 is self-trained using a large amount of data collected through broadcasting, distribution services, etc., and the classification unit 1420 of the downstream side is supervised training is performed.

[0126] Figure 17 An example of the internal configuration of the determination unit 103, which focuses on the differences between training methods, is depicted. Figure 17 In the depicted example, the determination unit 103 includes a self-trained model 1701, a supervised classification model 1702, and a distance learning model 1703. Additionally, the supervised classification model 1702 includes an attention mechanism.

[0127] Self-trained model 1701 corresponds to Figure 14 The feature extraction unit 1410 in the model. A self-trained model 1701 is created by self-training using a well-represented representation from time-series media information, such as sound or video, that the user wishes to learn. A well-represented representation means a representation that can be easily classified by a downstream supervised classification model 1702. In cases where language learning is to be assisted, for example, a self-trained model for sound, such as wav2vec or wav2vec2.0, is used; and in cases where training for physical movements, such as sports, is to be assisted, for example, a self-trained model for video, such as CVRL or PCL, can be used.

[0128] In addition to making classification easier to perform at the downstream supervised classification model 1702, the self-trained model 1701 also overcomes the difficulty of collecting data on actions or behaviors performed by users (time-series media information). The downstream supervised classification model 1702 is trained using user action data and ideal action data from professionals, etc., but it is generally difficult to collect action data from users (or beginners at a similar level to users). Meanwhile, ideal action data from professionals (such as native speaker vocalization data) can be collected in large quantities through television or radio broadcasts, video distribution services via the Internet, etc. By performing self-training using this readily available data, the self-trained model 1701 can obtain representations that can be easily classified at the downstream supervised classification model 1702.

[0129] Supervised classification model 1702 corresponds to Figure 14 The classification unit 1420 includes a time-series DNN, such as an RNN or an LSTM, which is an improved version of an RNN. The supervised classification model 1702 classifies the user's action data and reference action data based on the representations obtained from the upstream self-trained model 1701.

[0130] By including it in supervised classification model 1702, the attention mechanism visualizes information about which parts of the user's action data the supervised classification model 1702 has noticed in order to perform classification (see, for example, see...). Figures 4 to 6 When the supervised classification model 1702 classifies a user's action as different from the ideal action of a professional, its focus is equivalent to the portion of the action or behavior performed by the user that differs from the ideal action or behavior of a professional. For each time segment in the time-series media information, the attention mechanism outputs a value in the range of 0 to 1 as the degree of attention already paid during classification. Then, the portion with large values ​​is visualized and presented on the time-series media information, or the values ​​of time segments are displayed in the time-series media information (see...). Figure 15 This makes it easier for users to understand which part(s) of their action or behavior differs from the ideal action or behavior, and to what extent that part(s) differs.

[0131] The distance learning model 1703 calculates distance information representing the degree of difference between the overall action or behavior performed by the user and a reference action or behavior, based on the representation obtained through the self-trained model 1701 on the upstream side. The distance learning model 1703 can be trained using basic loss functions such as contrastive loss or triplet loss (as described above). Then, as... Figure 11 As depicted, the presentation unit 104 can visualize the difference between the user's action or behavior and a reference action or behavior and present it as distance information on a two-dimensional plane.

[0132] Next, refer to Figure 18 Explain the self-training method of self-training model 1701.

[0133] Self-trained model 1701 corresponds to Figure 14 The feature extraction unit 1410 includes a CNN. When the audio signal is considered as time-series media information, for example, wav2vec or wav2vec2.0 can be used as a self-trained model 1701. Alternatively, when the video is considered as time-series media information for training the user's physical movements, such as those during sports activities, CVRL or PCL can be used as a self-trained model 1701. Figure 18 Taking the use of wav2vec2.0 as an example, a sound recognition framework using a transformer is described. The sound recognition framework includes: an encoding unit 1801, which includes a CNN that convolves the sound signal into a latent representation; and a transformer unit 1802, which obtains a contextual representation from the latent representation.

[0134] Each CNN in the encoding unit 1801 convolves the time-segment data obtained by dividing the sound waveform signal into data for each time segment, and outputs a latent representation Z. The transformer unit 1802 receives the input of the quantized representation Q of the latent representation Z for each time segment and obtains the context representation C. Then, by using the contrastive loss of the latent representation Z and the context representation C for each time segment as a loss function, the self-training model 1701 (i.e., the entire sound recognition framework) is performed, such that the context representation C for each time segment approximates the latent representation Z for the corresponding time segment, but the approximation to the latent representation Z for other time segments is reduced.

[0135] In the self-trained sound recognition framework, the encoding part 1801 of a CNN is used as the self-training model 1701. According to... Figure 14 The description includes that the encoding part 1801 of the CNN is used as the feature extraction part 1410.

[0136] Then, when training the supervised classification model 1702, training of the self-trained model 1701 is stopped, and training data (i.e., labeled time-series media information) is input into the self-trained model 1701, which performs convolution on the training data. The extracted features are then input into the supervised classification model 1702. Supervised training of the supervised classification model 1702 is then performed using error backpropagation, minimizing the loss function based on the error between the training data and the classification data output from the supervised classification model 1702.

[0137] Additionally, when training the distance learning model 1703, training of the self-trained model 1701 is stopped, training data is input into the self-trained model 1701, and convolution is performed on the training data at the self-trained model 1701. The extracted features are then input into the distance learning model 1703. Distance training of the distance learning model 1703 is then performed via error backpropagation using a basic loss function such as contrastive loss or triplet loss.

[0138] D. Application Examples

[0139] The information processing system 100, which utilizes the present disclosure, can assist a user's learning by using time-series media information representing actions or behaviors performed by the user. Examples of time-series media information include information such as video, sound, or sentences that can be identified based on sensor data or signals that can be sensed by the sensor unit 101. For example, sensor data acquired by a swing sensor attached to a golf club or bat, biosignals acquired by an IMU (Inertial Measurement Unit) or biosensors attached to a user training in a sport can also be used as time-series media information.

[0140] Figure 19 This document summarizes the relationship between the learning content that can be supplemented by this disclosure and the types of time-series media information.

[0141] When applying this disclosure to language learning, the information processing system 100 can be used to assist learning performed by the user by using sound signals from the user's voice, sentences obtained through sound recognition, sentences written by the user, etc., as time-series media information.

[0142] When applying this disclosure to musical instrument performance, the information processing system 100 can be used to assist the user in mastering musical instrument performance by using sound signals of the instrument being played by the user or video of images of the user captured during the performance as time-series media information.

[0143] When this disclosure is applied to a speech or presentation, the information processing system 100 can be used to assist a user in improving her / his speaking or presentation skills by using sound signals from the user's voice, sentences or transcripts obtained by sound recognition of the user's voice, or videos capturing images of the user during the speech or presentation as time-series media information.

[0144] When applying this disclosure to training in golf, baseball, or other sports, the information processing system 100 can be used to assist user training by using video capturing images of the user during training as time-series media information. Additionally, although in Figure 19While omitted, sensor data acquired by swing sensors attached to golf clubs or clubs, and biosignals acquired by IMUs or biosensors attached to users during sports training, can also be used as time-series media information.

[0145] When applying this disclosure to cooking, the information processing system 100 can be used to assist the user in cooking by using video of images captured of the user during cooking as time-series media information.

[0146] When this disclosure is applied to various types of treatments such as surgery, other medical practices, and additional massage, the information processing system 100 can be used to assist users in improving their medical care or surgical skills by using video of images of the user captured during surgery, medical examination, or surgical procedure as time-series media information.

[0147] When this disclosure is applied to writing activities such as writing novels, screenplays, or translations performed by a user, the information processing system 100 can be used to assist the user's writing skills by treating sentences written by the user as time-series media information.

[0148] When applying this disclosure to performances or stand-up comedy in films or plays, the information processing system 100 can be used to assist the user's performance by using video obtained by capturing images of the user during the performance, sounds made by the user, sentences obtained by sound recognition, or sentences from the script as time-series media information.

[0149] E.UI Example

[0150] According to the information processing system 100 that applies the present disclosure, the determining unit 103 can determine the learning level of the user's actions or behaviors based on time-series media information (e.g., video or audio) representing the actions or behaviors performed by the user, and, if it is determined that there are insufficiently learned parts, determine which part of the user's actions or behaviors is insufficiently learned and the degree of that insufficient learning. Then, as feedback to the user, the presentation unit 104 can visually present the parts of the user's actions or behaviors in the time-series media information that differ from reference actions or behaviors. Furthermore, if the user has pre-set information about reference actions or behaviors, he / she can perform learning that is preferred for him / her.

[0151] Section E describes a configuration example of a UI (user interface) used to present a determination of a user's learning level to the user. For example, it is assumed that the screen displaying the UI is the screen of a personal computer, smartphone, etc., equipped with at least some components of the information processing system 100.

[0152] Figure 20 This describes a configuration example of a UI screen that presents a definitive result of a user's vocalization when the user is learning a second language. The UI screen displays the sound waveform signal obtained when the user says the phrase "This was easy for us," along with the string "This was easy for us."

[0153] If it is determined that the phrase "This was easy for us" spoken by the user differs from the phrase spoken by a native speaker and that there are parts of the user's learning that are insufficient, the determination unit 103 determines which parts of the sound waveform signal are insufficiently learned and the degree of insufficiency in these parts. Then, as... Figure 20 The depicted presentation unit 104 highlights the portions of the sound waveform signal that differ from the native speaker's sound waveform signal, and highlights words or strings identified as pronounced differently from the native speaker's pronunciation, such as "This," "eas," and "for." Furthermore, since a video of the native speaker's face saying "This was easy for us" is displayed on the UI screen, the user can grasp mouth movements that approximate the native speaker's pronunciation. Needless to say, if information about the degree of difference between the user's sound waveform signal and the native speaker's pronunciation, and information about words or strings in phrases whose pronunciation differs from the native speaker's pronunciation, can be presented, then a method with [specific features] can be employed. Figure 20 The UI depicts different screen layouts. Additionally, for example, users can preset the ideal pronunciation for themselves by considering the native speaker's local location (e.g., British or American English in the case of English), age, social class, etc.

[0154] Figure 21 This describes a configuration example of a UI screen that presents a definitive result of a user's violin playing while the user is practicing. The UI screen displays the sound waveform signal produced by the violin played by the user, along with the sheet music.

[0155] If it is determined that the user's violin playing differs from that of a professional violinist and that there are areas where the user's learning is insufficient, the determination unit 103 determines which parts of the sound waveform signal are insufficiently learned and the degree of insufficiency in these parts. Then, as... Figure 21 The depicted presentation unit 104 highlights the portions of the sound waveform signal that differ from the professional violinist's sound waveform signal, and also highlights the notes on the score that differ from the professional violinist's sound waveform signal. Needless to say, it is possible to use... Figure 21The UI, regardless of the screen layout, can display information such as the degree of difference between the user's playing and a professional violinist's playing, as represented by the sound waveform signal of the violin's sound, and information about the different notes played on the user's sheet music compared to those played by a professional violinist. Furthermore, for example, users can preset the instruments, systems, and playing styles used by professional violinists they consider as references to set their ideal playing style.

[0156] Figure 22 This describes a configuration example of a UI screen that presents a definite result about a user's swing while the user is practicing a baseball swing. On this UI screen, multiple still images of the user are arranged in a time sequence, segmented at predetermined time intervals from a video of the user's swing, and waveform signals representing the amount of change in each of major body parts such as the forearm, hand, knee, or toes are displayed.

[0157] If it is determined that the user's swing differs from that of a professional baseball player and that there are areas where the user's swing practice is insufficient, the determination unit 103 determines which body parts of the user move differently from those in a professional baseball player's swing and the degree of difference in the movements of these body parts. Then, as... Figure 22 The depicted presentation unit 104 highlights body parts that move differently from those in a professional baseball player's swing in each of the still images arranged in a time sequence, and also highlights the time periods in the change signals of each body part that differ between the user's swing and that of a professional baseball player. Needless to say, it can employ [a method with...]. Figure 22 The UI, regardless of the screen layout, can visually represent which parts of the user's swing differ from a professional baseball player's swing and the degree of these differences. Additionally, for example, users can preset the age, physique, and hitting style of a professional baseball player they consider as a reference to set their ideal swing.

[0158] F. Device Configuration Example

[0159] F-1. Example of Information Processing Device Configuration

[0160] Figure 23 A configuration example of the information processing device 2300 is depicted. For example, the information processing device 2300 is equivalent to a general-purpose personal computer. The information processing device 2300 can be used as... Figure 7 The device 700 depicted in the text Figure 8 The second device 802 depicted in the text Figure 9 The second device 902 and depicted in Figure 10The second device 1002 depicted herein shall be used for operation. Each element of the information processing device 2300 shall be described below.

[0161] The CPU (Central Processing Unit) 2301 is interconnected via bus 2310 with various components—ROM (Read-Only Memory) 2302, RAM (Random Access Memory) 2303, Hard Disk Drive (HDD) 2304 and Input / Output Interface 2305.

[0162] CPU 2301 can execute programs loaded from ROM 2302 or HDD 2304 onto RAM 2303 and perform various processes, while temporarily storing working data during execution on RAM 2303. The programs to be executed by CPU 2301 include basic input / output programs stored on ROM 2302 and an operating system (OS) and applications installed on HDD 1604. The OS provides the execution environment for the applications. Additionally, it is assumed that the applications include learning assistance applications that determine the learning level of the user, etc., based on sensor information (time-series media information).

[0163] ROM 2302 is a read-only memory that permanently stores basic input / output programs, device information, etc. RAM 2303 includes volatile memory such as DRAM (Dynamic RAM) and serves as the working area for CPU 2301. HDD 2304 is a high-capacity storage device that uses one or more disks fixed within a cell as recording media and stores programs and data in file formats thereon. SSD (Solid State Drive) can be used instead of HDD.

[0164] Input / output interface 2305 connects to various input / output devices such as output unit 2311, input unit 2312, communication unit 2313, and driver 2314. Output unit 2311 includes display devices such as LCDs (liquid crystal displays), speakers, printers, and other output devices, and outputs, for example, the results of a program executed by CPU 1601. The results of a determination regarding the user's learning level can be presented using the display device. Input unit 2312 includes a keyboard, mouse, touch panel, and other input devices, and accepts commands from the user. Additionally, input unit 2312 includes a microphone, camera device, and other sensors, and acquires time-series media information (e.g., video or sound) related to actions or behaviors performed by the user. Furthermore, output unit 2311 and input unit 2312 may include USB, HDMI (registered trademark), or other interfaces for external connection to externally attached output and input devices.

[0165] The communication unit 2313 includes a wired or wireless communication interface conforming to a predetermined communication standard and performs data communication with an external device. Examples of wired communication interfaces include an Ethernet (registered trademark) communication interface. Examples of wireless communication interfaces include Wi-Fi (registered trademark) and Bluetooth (registered trademark) communication interfaces. When the information processing device 2300 operates as a second device, the communication unit 2313 communicates with the first device.

[0166] In addition, the communication unit 2313 is connected to a wide area network such as the Internet. For example, an application (as described above) can be downloaded from a download site on the Internet using the communication unit 2313 and installed on the information processing device 2300.

[0167] When the removable recording medium 1615 is loaded into the drive 2314, the drive 2314 performs processes for reading from and writing to the removable recording medium 2315 (note that this applies when the removable recording medium 2315 is a rewritable recording medium). The removable recording medium 2315 contains programs, data, etc., in file formats recorded thereon. For example, the removable recording medium 2315 containing packaged software such as an application (as described above) can be loaded into the drive 2314, and this packaged software can be installed on the computer 2300. Examples of removable recording media 2315 include floppy disks, CD-ROMs (Compact Disc Read-Only Memory), MO (Magneto-Optical) discs, DVDs (Digital Multifunction Discs), magnetic disks, semiconductor memory, etc.

[0168] F-2. Information Terminal Configuration Example

[0169] Figure 24 A configuration example of information terminal 2400 is depicted. For example, information terminal 2400 is equivalent to a multi-functional information terminal such as a smartphone or tablet computer. Information terminal 2400 can be used as... Figure 7 The device 700 described Figure 8 The first device 801 depicted Figure 9 The first device 901 and depicted Figure 10 The first device 1001 is described and operated.

[0170] The information terminal 2400 includes a built-in antenna 2401, a mobile communication processing unit 2402, a microphone 2403, a speaker 2404, a storage unit 2405, an operation unit 2406, a display unit 2407, a control unit 2408, a control line 2409, a data line 2410, a WLAN communication antenna 2411, a WLAN communication control unit 2412, a BLE (Bluetooth Low Energy) communication antenna 2413, a BLE communication control unit 2414, an infrared transmitter / receiver unit 2415, a contactless communication antenna 2416, a contactless communication control unit 2417, a GNSS (Global Navigation Satellite System) receiving antenna 2418, a GNSS positioning unit 2419, a camera unit 2420, a memory slot 2421, and a sensor unit 2423. Each component of the information terminal 2400 will be described below.

[0171] The built-in antenna 2401 is configured to perform the following operations: receive signals transmitted through a mobile phone network such as an LTE network or an NR network, and transmit signals to be transmitted from the information terminal 2400 to the mobile phone network. The mobile communication processing unit 2402 performs demodulation and decoding processing on the signals received through the built-in antenna 2401, and performs encoding and modulation processing on the transmission data to be transmitted to the mobile phone network via the built-in antenna 2401.

[0172] Microphone 2403 collects sound, converts the sound into electrical signals, and also performs AD conversion. The sound signal digitized by microphone 103 is provided to mobile communication processing unit 2402 via data line 2410, undergoes encoding and modulation processing, and is then transmitted to the mobile phone network via built-in antenna 2401. Microphone 2403 primarily functions as a microphone, but in this embodiment, it also functions as a sensor unit 101 that collects the user's voice and acquires sound waveform signals (time-series media information).

[0173] The speaker 2404 mainly functions as an earpiece, performing DA conversion on the digital audio signal provided from the mobile communication processing unit 2402 via the data line 2410, amplifying the digital audio signal, and then emitting sound.

[0174] For example, storage unit 2405 includes non-volatile memory such as ROM, RAM, EEPROM (Electrically Erasable Programmable ROM), or flash memory.

[0175] ROM stores and retains important data, such as various types of program code to be executed by the CPU (Central Processing Unit) included in the control unit 2408 (described later), program code for email processing operations such as editing emails, various types of programs (applications) such as programs for processing images captured by the camera unit 120, identification information (ID) of the mobile phone terminal, or data required for various types of processing. For example, RAM is mainly used as a working area to temporarily store intermediate results of processing while the CPU performs various types of processing.

[0176] Non-volatile memory stores and retains data non-volatilely, ensuring that the data is retained even if the power to the information terminal 2400 is cut off. Examples of data stored and retained in non-volatile memory include address book data, email data, image data from images captured by the camera unit 2420, various types of web page data such as image data or text data downloaded via the Internet, various types of setting parameters, dictionary information, and additional programs.

[0177] For example, the operation unit 2406 includes a touch panel superimposed on the screen of the display unit 2407 (described later), a numeric keypad, several symbol keys, several function keys, and a so-called jog-dial key capable of performing rotation and pressing operations. Furthermore, the operation unit 2406 accepts operation input from the user of the information terminal 2400, converts the operation input into electrical signals, and provides the electrical signals to the control unit 2408 via the control line 2409. Therefore, the control unit 2408 can control each unit according to the instructions from the user and perform processing according to the user's instructions.

[0178] For example, display unit 2407 includes a flat panel display element such as an organic EL (electroluminescent) or LCD and its control circuitry, and displays various types of information provided via control line 2409. For example, it can display various types of image data or email data input via built-in antenna 2401 and mobile communication processing unit 2402, text data input via operation unit 2406, pre-prepared operation guidance or various types of message information, or image data input via camera unit 2420. When operation unit 2406 includes a touch panel superimposed on the screen of display unit 2407, the user can perform direct input operations on objects displayed on the screen.

[0179] The control unit 2408 is the main controller of the comprehensive control information terminal 2400. Specifically, the control unit 2408 includes a CPU that loads programs stored in the ROM or the like in the memory unit 2405 into RAM, executes the programs, generates control signals to be provided to each unit, and transmits the control signals to each unit via control lines 2409. For example, the program executed by the control unit 108 includes a program (application) that performs processing related to determining the user's learning level. In addition, by providing information from each unit to the control unit 2408, the control unit 2408 generates new control signals based on the information and provides the control signals via control lines 2409.

[0180] Control line 2409 is a bus primarily used for transmitting control signals and various types of information associated with control. Additionally, data line 2410 is a bus used for transmitting various types of data as transceivers (e.g., voice data, image data, or email data) and various types of data as processors.

[0181] For example, the WLAN communication antenna 2411 is configured to perform the following operations: receive signals transmitted via WLAN using unlicensed frequency bands such as the 2.4 GHz band or the 5 GHz band, and transmit signals to be transmitted from the information terminal 2400 to the WLAN. The WLAN communication control unit 2412 controls the WLAN communication operation using unlicensed frequency bands, performs demodulation and decoding processing on the received signals received by the WLAN communication antenna 2411, and performs encoding and modulation processing on the transmitted data to be transmitted to the WLAN via the WLAN communication antenna 2411. The WLAN communication control unit 112 controls one-to-one wireless communication in ad-hoc mode and wireless communication connected to the WLAN via connecting to adjacent access points in infrastructure mode.

[0182] The BLE communication antenna 2413 is configured to transmit and receive BLE signals. The BLE communication control unit 2414 controls the BLE communication operation, performs demodulation and decoding processing on the received signals received through the BLE communication antenna 2411, and performs encoding and modulation processing on the transmitted data to be transmitted through the BLE communication antenna 2411.

[0183] The infrared transmitter / receiver unit 2415 includes an LED (light-emitting diode) for emitting infrared light and a photodetector for receiving infrared light, and performs signal transmission and reception using infrared light in a frequency band slightly lower than the visible light band. Through this infrared transmitter / receiver unit 2415, for example, the exchange of data such as email addresses or images can be performed by bringing the information terminal 2400 close to another terminal to transmit and receive infrared light. Since communication is performed between mobile phone terminals that are very close to each other, infrared communication can be performed while maintaining security.

[0184] The contactless communication antenna 2416 is configured to use electromagnetic induction to transmit, receive, or transmit and receive contactless signals. For example, the contactless communication control unit 2417 uses contactless communication technology such as FeliCa (registered trademark) to control contactless communication operations. Specifically, the contactless communication control unit 2417 controls operations as a card, reader, or reader / writer in a contactless communication system.

[0185] The GNSS receiving antenna 2418 and the GNSS positioning unit 2419 identify the current location of the information terminal 2400 by analyzing GNSS signals received from GNSS satellites. Specifically, the GNSS receiving antenna 2418 receives GNSS signals from multiple GNSS satellites, and the GNSS positioning unit 2419 synchronously demodulates and analyzes the GNSS signals received by the GNSS receiving antenna 2418 and calculates location information. For example, the information about the current location calculated by the GNSS positioning unit 2419 is used for navigation functions or metadata, which represents the image capture location added to the image data captured by the camera unit 2420 (described later).

[0186] Note that, although not in Figure 24 As depicted, the information terminal 2400 also includes a clock circuit that provides the current year and date, current day of the week, and current time. The current date / time obtained from this clock circuit is added as metadata representing the date / time of image capture to the image data captured by the camera unit 2420 (described later).

[0187] The imaging device 2420 includes an objective lens, a shutter mechanism, and an image capture element such as a CMOS (complementary metal-oxide-semiconductor) (none of which are in the image sensor). Figure 24(As depicted in the image). Furthermore, when an electrical signal representing an image of a subject is received by the image capture element, the electrical signal can be converted into digital data, provided to the storage unit 2405 via data line 2410, and recorded on the storage unit 2405. In this embodiment, the camera unit 2420 also functions as a sensor unit 101 that captures images of actions or behaviors performed by the user during learning and acquires video (time-series media information).

[0188] For example, memory slot 2421 is a device that loads an external memory 2422, such as a microSD card, which is configured to be attached to and detachable from memory slot 2421. For example, when the available storage capacity of storage unit 2405 is insufficient, the user can use external memory 2422 as user storage, or external memory 2422 containing programs (applications) for implementing new functions can be loaded into memory slot 2421 to add new functions to information terminal 2400.

[0189] Sensor unit 2423 may include other sensor elements such as an illuminance sensor, an IMU (inertial measurement unit), a TOF (time of flight) sensor, a temperature sensor, or a humidity sensor. Note that microphone 2403 can be considered as a sound sensor, GNSS communication control unit 2419 can be considered as a positioning sensor, and camera unit 2420 can be considered as an image sensor, and thus they can be processed as part of sensor unit 2423.

[0190] [Industrial Applicability]

[0191] The present disclosure has now been described in detail with reference to specific embodiments. However, it is apparent that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present disclosure.

[0192] Although this specification primarily describes implementations of this disclosure for speech learning that mainly uses voice input by the user, the scope of this disclosure is not limited thereto. For example, this disclosure can also be applied to body movement learning using user video captured by a camera device, and to learning using a combination of sound and video. Furthermore, this disclosure can be applied not only to second language speech but also to sentence learning.

[0193] Furthermore, the methods described in this specification for language learning that aim to use pronunciation made by native speakers as a reference can be implemented as methods to reduce the difference between the user's pronunciation and the reference pronunciation. For example, processing based on the difference between the standard accent of the language and the regional accent of the language (the regional accent is spoken by the user) can be implemented.

[0194] Furthermore, the application of this disclosure is not limited to language learning. For example, this disclosure can also be similarly applied to learning to play musical instruments, learning to speak or give public speeches (learning sound, video (body movements), and sentences), learning various types of sports using videos, learning to cook using videos, learning various types of treatments using videos, such as surgery, other medical practices, and massage, learning sentences in writing activities (novels, scripts, translations, etc.), and learning performance skills such as acting, comedy duos, etc. (pronunciation, sentences, body movements, etc.).

[0195] In summary, this disclosure has been described with reference to exemplary form, and the description herein should not be construed as limiting. The scope of this disclosure should be determined by considering the claims.

[0196] It should be noted that this disclosure may also have the following configuration. (1)

[0198] An information processing method, comprising:

[0199] The input step is as follows: input time-series media information representing the user's actions or behaviors during learning;

[0200] The first determining step is: determining the user's learning level based on time-series media information; and

[0201] The output step is as follows: based on the user's learning level determined in the first determining step, output the part of the user's action or behavior in the time series media information that is different from the reference action or behavior. (2)

[0203] According to the information processing method described in (1) above, wherein,

[0204] In the output step, the portion of the user's action or behavior in the time-series media information that differs from the reference action or behavior is output to the presentation device, and

[0205] The information processing method further includes a first presentation step, which is to present the part of the user's action or behavior in the time-series media information that is different from the reference action or behavior to the user through the presentation device. (3)

[0207] According to the information processing method described in (2) above, in the first presentation step, the part of the user's action or behavior in the time-series information media that is different from the reference action or behavior is presented in a visual manner. (4)

[0209] According to the information processing method described in (3) above, in the first presentation step, words or characters in a phrase spoken by the user whose pronunciation differs from the reference pronunciation (or the ideal pronunciation of a native speaker) are presented visually. (5)

[0211] According to the information processing method described in (3) above, in the first presentation step, the user’s body parts that exhibit body movements that differ from reference body movements (or ideal body movements of professional athletes or coaches) are presented visually. (6)

[0213] The information processing method according to any one of (1) to (5) above further includes:

[0214] The second determining step comprises: determining distance information representing the difference between a user's action or behavior and a reference action or behavior; and

[0215] The second presentation step is as follows: outputting the determined result from the second determination step to the presentation device and presenting the result to the user. (7)

[0217] According to the information processing method described in (6) above, in the second presentation step, the distance information is presented visually in an N-dimensional space, in which the reference action or behavior is arranged in the center. (8)

[0219] According to any one of (1) to (7) above, the information processing method wherein the input step and the first determination step are performed continuously until the user's learning level is determined to be sufficient. (9)

[0221] According to any one of (1) to (8) above, the information processing method wherein, in the first determining step, the determining is performed by using a trained machine learning model. (10)

[0223] According to the information processing method described in (9) above, in the first determining step, after processing time-series media information by using a self-trained first machine learning model, the user's learning level is determined by using a supervised-trained second machine learning model. (11)

[0225] According to the information processing method described in (10) above, in the first determining step, the basis for determining whether a user's learning level is insufficient or whether a user needs to learn is determined based on time-series media information by using an attention mechanism included in the second machine learning model. (12)

[0227] According to the information processing method described in (9) above, the machine learning model includes:

[0228] The feature extraction unit extracts the feature quantities of time-series media information, and

[0229] The classification department classifies the learning level based on the extracted features. (13)

[0231] According to the information processing method described in (12) above, the feature extraction unit is trained by self-training, and the classification unit is trained by supervised training using the trained feature extraction unit. (14)

[0233] According to the information processing method described in (12) or (13) above, in the first determining step, the basis for determining whether a user's learning level is insufficient or whether a user needs to learn is determined based on time-series media information by using an attention mechanism included in the classification section. (15)

[0235] The information processing method according to any one of (12) to (14) above further includes:

[0236] The second determining step is to determine distance information representing the difference between a user's action or behavior and a reference action or behavior based on the feature quantity of the time-series media information extracted by the feature extraction unit. (16)

[0238] According to the information processing method described above (15), in the second determining step, distance information is determined by using a distance learning model, in which contrast loss or triplet loss is used as the loss function. (17)

[0240] An information processing system, comprising:

[0241] The input section receives time-series media information representing the user's actions or behaviors during the learning process.

[0242] The first determining unit, which determines the user's learning level based on time-series media information; and

[0243] The output unit, based on the user's learning level determined by the first determining unit, outputs the portion of the user's actions or behaviors in the time-series media information that differ from a reference action or behavior. (18)

[0245] The information processing system described in (17) above further includes:

[0246] The sensor unit detects the user's actions or behaviors during learning and acquires time-series media information; and

[0247] A presentation device, used by the output unit, to output the portion of a user's action or behavior in time-series media information that differs from a reference action or behavior. (19)

[0249] An information terminal, comprising:

[0250] The sensor unit detects the user's actions or behaviors during learning and acquires time-series media information;

[0251] The communication unit transmits time-series media information to an external device and receives from the external device results regarding the user's learning level and the parts of the user's actions or behaviors in the time-series media information that differ from reference actions or behaviors; and

[0252] The presentation department presents the information it receives. (20)

[0254] A computer program written in a computer-readable format that enables a computer to:

[0255] The input section receives time-series media information representing the user's actions or behaviors during the learning process.

[0256] The first determining unit, which determines the user's learning level based on time-series media information; and

[0257] The output unit, based on the user's learning level determined by the first determining unit, outputs the portion of the user's actions or behaviors in the time-series media information that differ from a reference action or behavior.

[0258] [List of reference numerals]

[0259] 100: Information Processing System

[0260] 101: Sensor Department

[0261] 102: Input Section

[0262] 103: Determination Department

[0263] 104: Presentation Department

[0264] 1400: DNN

[0265] 1410: Feature Extraction Department

[0266] 1411: CNN

[0267] 1420: Classification Department

[0268] 1421: Bi-LSTM layer

[0269] 1422: Attention Mechanisms

[0270] 1423: Classification Department

[0271] 1701: Self-trained model

[0272] 1702: Supervised Classification Model

[0273] 1703: Distance Learning Model

[0274] 1801: Coding Department

[0275] 1802: Converter Section

[0276] 2300: Information Processing Device

[0277] 2301: CPU

[0278] 2302: ROM

[0279] 2303: RAM

[0280] 2304: HDD

[0281] 2305: Input / Output Interface

[0282] 2310: Bus

[0283] 2311: Output Department

[0284] 2312: Input Department

[0285] 2313: Ministry of Communications

[0286] 2314: Driver

[0287] 2315: Removable Recording Media

[0288] 2400: Information Terminal

[0289] 2401: Built-in antenna

[0290] 2402: Mobile Communication Processing Department

[0291] 2403: Microphone

[0292] 2404: Speaker

[0293] 2405: Storage Department

[0294] 2406: Operations Department

[0295] 2407: Display Department

[0296] 2408: Control Department

[0297] 2409: Control Line

[0298] 2410: Data cable

[0299] 2411: WLAN communication antenna

[0300] 2412: WLAN Communication Control Department

[0301] 2413: BLE communication antenna

[0302] 2414: BLE Communication Control Department

[0303] 2415: Infrared Transmitter / Receiver

[0304] 2416: Non-contact communication antenna

[0305] 2417: Contactless Communication Control Unit

[0306] 2418: GNSS Receiving Antenna

[0307] 2419: GNSS Positioning Department

[0308] 2420: Camera Equipment Department

[0309] 2421: Memory slot

[0310] 2422: External memory

[0311] 2423: Sensor Department

Claims

1. An information processing method, comprising: Time-series media information representing user behavior is fed into a trained machine learning model; The trained machine learning model performs the following: Based on a self-trained learning model, feature quantities of the input time-series media information are extracted; and Based on a supervised training learning model, the user's learning level is classified according to the extracted features. The trained machine learning model receives the classified learning level of the user as its output; and Based on the learning level of the classified users, the parts of the user's behavior in the time-series media information that differ from the reference behavior are output.

2. The information processing method according to claim 1 further includes, The portion of the time-series media information is output to the presentation device, and The time-series media information is presented by the presentation device.

3. The information processing method according to claim 2 further includes displaying the portion of the time-series media information on a display screen.

4. The information processing method according to claim 3, wherein The portion of the time-series media information includes at least one word or character segment in a phrase spoken by the user whose pronunciation differs from a reference pronunciation or the ideal pronunciation of a native speaker.

5. The information processing method according to claim 3, wherein The portion of the time-series media information includes at least the body parts of the user exhibiting body movements that differ from reference body movements, which correspond to ideal body movements of professional athletes or coaches.

6. The information processing method according to claim 1, further comprising: Determine distance information representing the difference between the user's behavior and the reference behavior; as well as The determined distance information is output to the presentation device, and The determined distance information is presented through the presentation device.

7. The information processing method according to claim 6 further includes visually presenting the determined distance information in an N-dimensional space, wherein the reference behavior is located at the center of the N-dimensional space.

8. The information processing method according to claim 1 further includes continuously performing the input of the time-series media information and the reception of classified learning levels until the user's learning level is sufficient.

9. The information processing method according to claim 1 further includes, after extracting the time-series media information based on the self-trained learning model, determining the user's learning level based on the supervised training learning model.

10. The information processing method according to claim 9 further includes determining whether the user's learning level is insufficient based on the supervised training learning model, wherein the supervised training learning model includes an attention mechanism.

11. The information processing method according to claim 1, further comprising determining whether the user's learning level is insufficient based on the self-trained learning model, wherein the self-trained learning model includes an attention mechanism.

12. The information processing method according to claim 1, further comprising: Distance information representing the difference between a user's behavior and a reference behavior is determined based on the feature quantities of the time-series media information.

13. The information processing method according to claim 12 further includes determining the distance information based on a distance learning model, wherein a contrastive loss or a triplet loss is used as a loss function in the distance learning model.

14. An information processing system, comprising: At least one processor is configured to perform the following: Receive time-series media information representing user behavior as input; The received time-series media information is input into a trained machine learning model; The trained machine learning model performs the following: Based on a self-trained learning model, feature quantities of the input time-series media information are extracted; and Based on a supervised training learning model, the user's learning level is classified according to the extracted features. The trained machine learning model receives the classified learning level of the user as its output; and Based on the learning level of the categorized users, the portion of the user's behavior in the time-series media information that differs from the reference behavior is output.

15. The information processing system according to claim 14, further comprising: A sensor is configured to detect the user's behavior and acquire the time-series media information; as well as A display screen is configured to show the portion of the user's behavior in the time-series media information that differs from the reference behavior.

16. An information terminal, comprising: A sensor configured to detect user behavior and acquire time-series media information representing said user behavior; The communication interface is configured to perform the following: The time-series media information is sent to a trained machine learning model, wherein the trained machine learning model extracts features of the input time-series media information based on a self-trained learning model, and classifies the user's learning level based on the extracted features based on a supervised training learning model. The user's learning level is received from the trained machine learning model after classification; and Receive, from an external device, the portion of the user's behavior in the time-series media information that differs from reference behavior, wherein the portion of the time-series media information is based on the user's classified learning level; and A display screen is configured to display the portion of the received time-series media information.

17. A non-transitory computer-readable medium having stored thereon computer-executable instructions, which, when executed by a processor, implement the information processing method according to any one of claims 1-13.

Images