Learning management system for predicting concentration on basis of data generated from metaverse teaching environment

Abstract

Disclosed in the present invention is a learning management system for predicting concentration on the basis of data generated from a metaverse teaching environment. The present invention may provide a learning management system based on data generated from a metaverse teaching environment, which includes a sequence model for predicting a learner's concentration from sequential data regarding the learner's viewing posture. The learning management system is capable of maintaining the learner's concentration by switching, replacing, or reproducing video content for which a relatively low concentration is predicted with video content that is capable of arousing the learner's interest. For example, by transmitting image information or numerical information related to the learner's viewing posture to a terminal of a controller, appropriate learning guidance, such as switching video content for which a relatively low concentration is predicted, may be provided, thereby maintaining the learner's concentration and improving learning achievement.

Description

A learning management system for predicting concentration based on data generated from the metaverse teaching environment.

[0001] The present invention relates to a learning management system (LMS) for predicting concentration based on data generated from a metaverse teaching environment.

[0002] As technological advancements usher in the full-fledged metaverse era, extensive research is being conducted to provide metaverse-related services. Interest is growing in building metaverse platforms and metaverse ecosystems, and the establishment or transformation of specialized metaverse companies is underway.

[0003] One embodiment of the present invention includes a learning management system based on data generated from a teaching environment of a metaverse, which can maintain the concentration of the instructor through video content that can arouse interest in the instructor by switching, replacing, or reproducing video content with a predicted relatively low concentration, including a sequence model for predicting the concentration of the instructor from sequential data on the instructor's viewing posture, and which can maintain the concentration of the instructor through appropriate learning guidance, such as switching the video content with a predicted relatively low concentration, by transmitting video information or numerical information on the instructor's viewing posture toward a terminal of a controller, and can increase learning achievement.

[0004] In order to achieve the above-mentioned and other purposes of the present invention, a learning management system based on data generated from the teaching environment of the metaverse of the present invention,

[0005] An instructor's terminal that provides a metaverse teaching environment through the playback of video content; and

[0006] A sequence model for predicting the instructor's concentration level from the instructor's viewing posture taken during playback of the above video content is included.

[0007] According to the present invention, the concentration of the instructor can be maintained through video content that can arouse interest in the instructor by including a sequence model for predicting the concentration of the instructor from sequential data on the instructor's viewing posture, and by switching, replacing or reproducing video content for which relatively low concentration is predicted, for example, by transmitting video information or numerical information on the instructor's viewing posture to a terminal of a controller, the concentration of the instructor can be maintained and learning achievement can be increased through appropriate learning guidance, such as switching video content for which relatively low concentration is predicted.

[0008] FIG. 1 is a diagram illustrating an ASR model for extracting text from speech uttered by an instructor in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and is a diagram illustrating a speech-to-text task.

[0009] FIG. 2 is a diagram for explaining an ASR model for extracting text from speech uttered by an instructor in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and a diagram for explaining a configuration for combining predictions of an Acoustic model (see FIGS. 1 to 3) and a Language model (see FIGS. 1 to 3) each provided in the ASR model for a speech-to-text task.

[0010] FIG. 3 is a diagram for explaining an ASR model for predicting the accuracy of utterance in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and for the task of predicting the accuracy of utterance, a diagram for explaining a configuration for predicting the accuracy of utterance according to the degree of mutual matching between the prediction of an Acoustic model (see FIGS. 1 to 3) and the prediction of a Language model (see FIGS. 1 to 3) each provided in the ASR model is shown.

[0011] FIG. 4 is a drawing for explaining an exemplary architecture of a Language model (see FIGS. 1 to 3) included in an ASR model in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and is a drawing for explaining the architecture of a Language model (see FIGS. 1 to 3) including a stack of multiple decoder blocks.

[0012] FIG. 5 is a diagram illustrating a configuration that operates with word tokens or word embeddings forming a vocabulary to predict the next word in the Language model illustrated in FIG. 4 (see FIGS. 1 to 3).

[0013] Figure 6 shows an example of a spectrogram, which is divided into unit times along the time axis and expressed with different frequency intensities for each time unit, by applying a Fourier transform to decompose the voice forming the speech into different frequency components for each time unit by dividing it into fixed time units.

[0014] Figure 7 shows an example of a Mel-spectrogram generated through Mel scaling, which gives higher weight to low-frequency sounds in consideration of auditory characteristics sensitive to low frequencies.

[0015] FIG. 8 is a diagram illustrating a configuration for extracting voice features from speech uttered by an instructor as illustrated in FIGS. 1 to 3, and for exemplarily explaining contrastive predictive coding (CPC) learning for extracting voice features shared from voice features extracted from previous time steps.

[0016] FIG. 9 is a diagram illustrating a webRTC connection that forms a communication channel between an instructor's terminal and a terminal of a controller having control authority over video content that implements a metaverse teaching environment on the instructor's terminal, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0017] FIG. 10 illustrates a drawing for explaining the basic configuration of a three-dimensional graphic library for forming a digital space of the metaverse in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0018] FIG. 11 illustrates a drawing for explaining a configuration in which a video frame sequence or video frame is captured from a camera set in a scene in which a 3D object is configured to form video content.

[0019] FIG. 12 is a diagram illustrating a configuration in which first to third image frames are captured from camera angles set to different left-eye viewing angles, right-eye viewing angles, and a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle from a scene in which a 3D object is configured, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0020] FIG. 13 is a drawing illustrating a configuration in which video frames are updated to different views according to a camera angle reflecting the instructor's head rotation position in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0021] FIGS. 14a to 14c are drawings for explaining a configuration in which video frames of different views are provided according to the instructor's head rotation position in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and different drawings are shown for explaining a configuration in which video frames of views are provided according to the instructor's head rotation position looking in the front direction and the instructor's head rotation position deviating from the front direction to one side and the other side, respectively.

[0022] FIG. 15 is a diagram illustrating an angular deviation of an instructor's head direction away from the front detected by a sensor mounted on a head-mounted display (HMD) that provides a geared metaverse environment in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0023] FIGS. 16 and 17 illustrate different diagrams for explaining the architecture of an RNN network applied to a sequence model for predicting a learner's concentration level from time-series inputs regarding the learner's viewing posture in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0024] FIG. 18 is a diagram illustrating a sequence model for predicting a learner's concentration level from time-series input regarding the learner's viewing posture in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0025] FIG. 19 is a drawing showing an example of a screen capturing a stream of video content that implements a metaverse teaching environment on an instructor's terminal in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0026] FIG. 20 is a drawing showing an example of a screen captured from a stream of a monitoring video in which a metaverse teaching environment is provided from the video content shown in FIG. 19 on the terminals of two different instructors and a controller who is connected in a 2:1 communication manner, and in which video frames are provided differentially according to the positions of the playback points of the video content played on the terminals of two different instructors, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0027] FIGS. 21 and 22 are drawings showing, by way of example, one screen of a captured stream of a monitoring video in which a different video frame is provided according to the position of the playback point of the video content played on the terminals of two different instructors, each of which provides a metaverse teaching environment from different video contents, on the terminals of a controller who is connected in a 2:1 communication manner, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0028] In order to achieve the above-mentioned and other purposes of the present invention, a learning management system based on data generated from the teaching environment of the metaverse of the present invention,

[0029] An instructor's terminal that provides a metaverse teaching environment through the playback of video content; and

[0030] A sequence model for predicting the instructor's concentration level from the instructor's viewing posture taken during playback of the above video content is included.

[0031] For example, the above sequence model can input sequential data about the instructor's viewing posture at each time step while advancing the time step from the start of playback of the video content toward the end of playback of the video content.

[0032] For example, the above sequence model can predict the concentration for the entire video content as an output at the final time step.

[0033] For example, the sequence model is installed on a processing server that is communicatively connected to the instructor's terminal,

[0034] Data regarding the instructor's viewing posture may be collected from the instructor's terminal and transmitted to a processing server equipped with the sequence model.

[0035] For example, the processing server may evaluate video content collected based on the instructor's viewing posture input into the sequence model based on the concentration predicted from the sequence model, and may evaluate whether the corresponding video content needs to be replaced or reproduced.

[0036] For example, the instructor's terminal includes a plurality of instructor's terminals that are communicatively connected to the processing server,

[0037] The sequence model installed in the above processing server predicts the concentration level of each instructor by inputting the viewing posture of each instructor obtained from each of the terminals of the multiple instructors.

[0038] The above processing server can implement an evaluation of video content in which the instructor's viewing posture is collected from the results of collecting the concentration of each instructor predicted from the viewing posture of each instructor obtained from the terminals of multiple instructors.

[0039] For example, the above sequence model can predict the instructor's concentration level by inputting the instructor's viewing posture, the angular deviation of the instructor's gaze direction or visual field direction away from the object of concentration.

[0040] For example, the above focus target is,

[0041] A non-player character (NPC) that is the counterpart of the instructor's interactions, including the instructor's dialogue and actions; and

[0042] It may include surrounding objects with which interaction with the above NPC is set.

[0043] For example, the above-mentioned focus target may be expressed in the frontal position of the instructor on the video frame forming the video content.

[0044] For example, the above sequence model can predict the instructor's concentration level by inputting the angular deviation of the instructor's head direction away from the front as the instructor's viewing posture.

[0045] For example, the instructor's terminal includes a sensor for detecting the instructor's head rotation position,

[0046] The angle of the camera for capturing a video frame for forming video content from a scene composed of 3D objects is reset based on the instructor's head rotation position detected from the above sensor, and the video frame can be updated based on the reset camera angle.

[0047] For example, the instructor's terminal can track the instructor's head rotation position so that video frames for forming video content are updated according to the instructor's head rotation position detected by the sensor.

[0048] For example, the sequence model may include parameters learned from labeled data to which target labels are assigned, and may include parameters learned to reduce an error or loss function between predicted concentrations using sequential data about the instructor's viewing posture during playback of video content as input as the learning data and target labels classified into classes of different levels with respect to concentrations as the target labels assigned to the learning data.

[0049] For example, the target labels assigned to the learning data of the above sequence model may be assigned to classes of different levels of concentration depending on the degree of interest arousal of the video content itself, but may not be assigned by labeling by a labeler from sequential data regarding the instructor's viewing posture during playback of the video content as learning data.

[0050] For example, in learning the above sequence model, the degree of interest inducing of each video content can be directly correlated with the target label assigned to the learning data of the sequence model.

[0051] For example, the target label assigned to the learning data of the above sequence model may be assigned from viewing impressions or reviews collected from web crawling for each video content.

[0052] For example, the target label assigned to the training data of the above sequence model is

[0053] As classes with different levels of concentration,

[0054] For video content that is relatively interesting, a relatively high level class is given.

[0055] Video content that is relatively uninteresting may be given a relatively low level class.

[0056] For example, a learning management system based on data generated from the above metaverse teaching environment,

[0057] The controller's terminal may further include a terminal of the controller that is communicatively connected to the instructor's terminal and obtains and displays instantaneous data on the instructor's viewing posture generated from the instructor's terminal.

[0058] For example, instantaneous data on the viewing posture of the above instructor,

[0059] The video content may include a video frame captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's gaze direction or field of view according to the instructor's viewing posture.

[0060] For example, the terminal of the controller can acquire and display a third image frame captured at a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle, synchronized with the first and second image frames of the left-eye image and the right-eye image captured at different camera angles of the scene composed of the 3D object from the terminal of the instructor.

[0061] For example, instantaneous data on the viewing posture of the above instructor,

[0062] It may include numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal.

[0063] For example, instantaneous data on the viewing posture of the above instructor,

[0064] i) a video frame of video content captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's line of sight or viewing direction according to the instructor's viewing posture; and

[0065] ii) Including numerical data on the instructor's head rotation position detected from a sensor mounted on the instructor's terminal;

[0066] The above i) video frames of the video content and ii) numerical data may be synchronized with each other with information about the viewing posture of the same instructor.

[0067] For example, the sequence model can predict the instructor's concentration level by inputting data regarding the instructor's viewing posture, such as the position of the instructor's two hands or the motion of the two hands.

[0068] For example, as the instructor's viewing posture, data regarding the position of the instructor's two hands or the motion of the two hands can be generated by detecting position information of the controllers held in the instructor's two hands from the instructor's terminal.

[0069] For example, as the instructor's viewing posture, data regarding the position of the instructor's two hands or the motion of the two hands can be generated from the reception sensitivity of the controller that is communicatively connected to a head-mounted display (HMD) as the instructor's terminal or from an output image captured by the controller from a vision sensor mounted on the head-mounted display.

[0070] Hereinafter, with reference to the attached drawings, a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention will be described.

[0071] Predicting Speech Accuracy and Language Fluency

[0072] Below, the prediction of accuracy of speech and fluency of language use according to one aspect of the present invention will be described.

[0073] A learning management system (LMS) based on data generated from a metaverse teaching environment according to one aspect of the present invention is:

[0074] As a tutor's terminal for providing a tutoring environment in the metaverse, the tutor's terminal generates data regarding speech uttered by the tutor; and

[0075] An ASR (automatic speech recognition) model for extracting text about a voice from a speech uttered by a tutor, which may include an ASR model that predicts the accuracy of a speech accompanying speech-to-text (STT).

[0076] For example, the above ASR model,

[0077] An acoustic model (see Figures 1 to 3) that includes parameters learned from training data in which speech and text are aligned with each other; and

[0078] It may include a Language model (see FIGS. 1 to 3) that includes parameters learned from training data consisting of text without speech.

[0079] For example, the above ASR model,

[0080] i) In the task of speech-to-text (STT), the prediction of the Acoustic model (see Figs. 1 to 3) based on the voice forming the speech and the prediction of the Language model (see Figs. 1 to 3) based on the context formed by the text extracted from the speech are combined,

[0081] ii) In a task of predicting the accuracy of speech, the degree of mutual matching can be analyzed by comparing the prediction of an Acoustic model (see Figs. 1 to 3) based on the voice forming the speech with the prediction of a Language model (see Figs. 1 to 3) based on the context formed by the text extracted from the speech.

[0082] For example, the above ASR model,

[0083] i) In the task of speech-to-text (STT), a predetermined number of candidate words having a relatively high probability of appearing as the next word are input into a Language model (see Figs. 1 to 3) based on the context formed by the text extracted from the speech, from an Acoustic model based on the voice forming the speech (see Figs. 1 to 3), and the probability of appearing as the next word is predicted for each of the candidate words.

[0084] The above Acoustic model (see FIGS. 1 to 3) and Language model (see FIGS. 1 to 3) can implement different phonetic-based and context-based predictions for each of the same candidate words.

[0085] For example, the above ASR model,

[0086] i) In the task of speech-to-text (STT), the data flow can be controlled so that the input and output of the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) are connected in series.

[0087] For example, the above ASR model,

[0088] ii) In the task of predicting the accuracy of speech, prediction of an Acoustic model based on the voice forming the speech (see Figs. 1 to 3) and prediction of a Language model based on the context of text extracted from the speech (see Figs. 1 to 3) can be performed independently of each other.

[0089] For example, the above ASR model,

[0090] ii) In the task for predicting the accuracy of utterance, the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) can independently predict the probability that each word or word embedding forming the vocabulary will appear as the next word from an operation (transpose matrix multiplication or inner product) between a feature vector of a voice extracted from speech and a context vector regarding a context extracted from speech, and a word or word embedding forming the vocabulary.

[0091] For example, the above ASR model,

[0092] The accuracy of speech uttered by the tutor can be evaluated based on the degree of mutual matching between the voice-based prediction that forms speech from the Acoustic model (see FIGS. 1 to 3) and the context-based prediction that forms text extracted from the speech from the Language model (see FIGS. 1 to 3).

[0093] For example, the above ASR model,

[0094] Depending on the degree of mutual matching between the voice-based prediction that forms speech from the Acoustic model (see FIGS. 1 to 3) and the context-based prediction that forms text extracted from speech from the Language model (see FIGS. 1 to 3),

[0095] If the degree of mutual matching between the prediction from the above Acoustic model (see Figs. 1 to 3) and the prediction from the above Language model (see Figs. 1 to 3) is relatively high, the accuracy of the speech uttered by the instructor is evaluated as relatively high.

[0096] If the degree of mutual matching between the prediction from the above Acoustic model (see FIGS. 1 to 3) and the prediction from the above Language model (see FIGS. 1 to 3) is relatively low, the accuracy of the speech uttered by the instructor can be evaluated as relatively low.

[0097] For example, if the ASR model predicts that a word or word embedding predicted as the next word with a relatively high probability from an Acoustic model based on the voice forming the speech (see FIGS. 1 to 3) will appear as the next word with a relatively low probability from a Language model based on the context of the text extracted from the speech (see FIGS. 1 to 3),

[0098] The accuracy of the speech in question can be assessed as low.

[0099] For example, the above ASR model, i) in the task of speech-to-text (STT),

[0100] By inputting speech uttered by a tutor, text about the speech is extracted, and text formed by a sequence of the next word predicted for each time step from the input of the previous time step and the current time step can be extracted.

[0101] For example, the above ASR model, i) in the task of speech-to-text (STT),

[0102] The next word can be inferred based on a weighted sum of a prediction based on the voice forming the speech from the Acoustic model (see FIGS. 1 to 3) and a prediction based on the context forming the text extracted from the speech from the Language model (see FIGS. 1 to 3).

[0103] For example, the above ASR model,

[0104] For the prediction of the acoustic model based on the voice forming the above speech (see Figs. 1 to 3), a relatively higher weight can be given than the prediction based on the context formed by the text extracted from the speech.

[0105] For example, the above ASR model,

[0106] To extract the features of the voice that forms the speech,

[0107] An encoder of the convolution neural network (CNN) series for extracting speech features for the input of each time step; and

[0108] It includes an aggregator of the RNN (recurrent neural network) series for extracting shared voice features from the voice features extracted from the encoder up to the current time step;

[0109] The above acoustic model (see FIGS. 1 to 3) can predict the next word from the voice features extracted from the aggregator.

[0110] For example, the Acoustic model (see FIGS. 1 to 3) can predict the probability that each word or word embedding forming the vocabulary will appear as the next word from an operation (transpose matrix multiplication or inner product) between the feature vector of the voice extracted from the speech uttered by the instructor and the words or word embedding forming the vocabulary.

[0111] For example, the Language model (see FIGS. 1 to 3) can predict the next word at the current time step from a preceding word sequence that includes the next word predicted up to the previous time step by combining the prediction of the Acoustic model (see FIGS. 1 to 3) and the prediction of the Language model (see FIGS. 1 to 3).

[0112] The above Language model (see Figs. 1 to 3) extracts a context vector that includes context information regarding the relationship between a word or word embedding input in the current time step and other words or word embeddings input up to the previous time step,

[0113] From the operation (transpose matrix multiplication or inner product) between the above context vector and the word or word embedding forming the vocabulary, the probability that each word or word embedding forming the vocabulary will appear as the next word can be predicted.

[0114] For example, a learning management system (LMS) based on data generated from the metaverse teaching environment,

[0115] The ASR model may further include a level classification model for classifying language fluency into different level classes based on text extracted from the ASR model that takes the instructor's speech as input.

[0116] For example, the above-mentioned level classification model can classify the language fluency of each tutor's speech into different levels of classes based on the number of words or word embeddings forming the text extracted from the tutor's speech from the ASR model and the class classification according to the vocabulary level of each word forming the text.

[0117] Hereinafter, a learning management system (LMS) based on data generated from a metaverse teaching environment according to one aspect of the present invention will be described in more detail.

[0118] FIG. 1 is a diagram illustrating an ASR model for extracting text from speech uttered by an instructor in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and is a diagram illustrating a speech-to-text task.

[0119] FIG. 2 is a diagram for explaining an ASR model for extracting text from speech uttered by an instructor in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and a diagram for explaining a configuration for combining predictions of an Acoustic model (see FIGS. 1 to 3) and a Language model (see FIGS. 1 to 3) each provided in the ASR model for a speech-to-text task.

[0120] FIG. 3 is a diagram for explaining an ASR model for predicting the accuracy of utterance in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and for the task of predicting the accuracy of utterance, a diagram for explaining a configuration for predicting the accuracy of utterance according to the degree of mutual matching between the prediction of an Acoustic model (see FIGS. 1 to 3) and the prediction of a Language model (see FIGS. 1 to 3) each provided in the ASR model is shown.

[0121] FIG. 4 is a drawing for explaining an exemplary architecture of a Language model (see FIGS. 1 to 3) included in an ASR model in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and is a drawing for explaining the architecture of a Language model (see FIGS. 1 to 3) including a stack of multiple decoder blocks.

[0122] FIG. 5 is a diagram illustrating a configuration that operates with word tokens or word embeddings forming a vocabulary to predict the next word in the Language model illustrated in FIG. 4 (see FIGS. 1 to 3).

[0123] In one aspect of a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention, the accuracy of speech can be predicted based on speech uttered by an instructor. For example, in one aspect of a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention, in evaluating the accuracy of speech according to one aspect, an ASR (automatic speech recognition) model for extracting text related to speech using speech uttered by an instructor as input can be applied, and the ASR model can include an Acoustic model (see FIGS. 1 to 3) trained with training data in which speech and text are aligned with each other, and a Language model (see FIGS. 1 to 3, language model, LM) trained only with text.For example, in one embodiment of the present invention, the ASR model can perform i) a task of extracting text from speech uttered by a tutor (see FIG. 2) of speech-to-text (STT), and ii) a task of predicting the accuracy of utterance for speech uttered by a tutor (see FIG. 3), and for example, in one embodiment of the present invention, the ASR model can perform different tasks such as i) a task of speech-to-text (STT) (see FIG. 2) and ii) a task of predicting the accuracy of utterance (see FIG. 3), while the task of predicting the accuracy of utterance (see FIG. 3) can be implemented concurrently with the task of i) speech-to-text (see FIG. 2), and more specifically, while sharing a network for performing the task of i) speech-to-text (see FIG. 2), at least a part of the computational processing for performing the task of i) speech-to-text (see FIG. 2) can be shared, for example, The accuracy of speech (see Fig. 3, task of predicting accuracy of speech) can be predicted from the result value of the computational processing for performing the speech-to-text task (see Fig. 2).However, in one embodiment of the present invention, the two different tasks of the ASR model, namely, the speech-to-text task (see FIG. 2) and the task of predicting the accuracy of utterance (see FIG. 3), may include different computational processing, and for example, as described below, the ASR model may form different input / output flows of data between the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) and their surrounding configurations for the different tasks of the speech-to-text task (see FIG. 2) and the accuracy prediction of utterance (see FIG. 3), and for example, in the speech-to-text task (see FIG. 2), the probability of the next word appearing may be predicted from the Language model for a group of candidate words predicted with a high probability from the Acoustic model, and in the task of predicting the accuracy of utterance (see FIG. 3), the predictions of the Acoustic model and the Language model may be performed independently of each other. In addition, the ASR model can perform different tasks by combining predictions of the Acoustic model and the Language model (weighting them with different weights) or comparing predictions of the Acoustic model and the Language model to perform the tasks of speech-to-text (see FIG. 2) and the task of predicting the accuracy of utterance (see FIG. 3).

[0124] In one embodiment of the present invention, in the ASR model, the next word can be predicted at each time step from the forward Language process that predicts the next word at each time step to generate a word sequence forming the speech uttered by the tutor, and in the task of predicting the accuracy of speech (see FIG. 3), the probability of the next word can be predicted from the Acoustic model by inputting voice data regarding the speech uttered by the tutor or the features of the voice extracted through voice preprocessing, and the accuracy of the speech uttered by the tutor can be predicted according to the degree of mutual matching between the probability of a group of candidate words (first candidate words) predicted with a relatively high probability as the next word from the Acoustic model and a group of candidate words (second candidate words) predicted with a relatively high probability as the next word from the Language model (see FIGS. 1 to 3) (the task of predicting the accuracy of speech, see FIG. 2), and further, in the task of speech-to-text (see FIG. 2), the probability of the group of candidate words predicted with a high probability as the next word from the Acoustic model is obtained from the Language model. The next word can be inferred from the prediction obtained by comprehensively summing the predictions of the Acoustic model and the Language model from the weighted sum obtained by summing the predictions of these different Acoustic models and the Language model with different weights (speech-to-text task, see Fig. 2), and the text forming the tutor's speech can be inferred from the word sequence including the word or word token predicted as the next word at each time step (speech-to-text task, see Fig. 2), and in this way, the fluency of the tutor's language ability can be evaluated based on the text extracted from the tutor's speech.

[0125] In one aspect of a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention, the accuracy of speech uttered by an instructor can be evaluated. To this end, the learning management system (LMS) of the present invention may include an ASR model that extracts text from the speech forming the instructor's speech, taking the speech uttered by the instructor as input.

[0126] In one embodiment of the present invention, a learning management system (LMS) that takes speech uttered by a tutor as input may include an ASR model that includes an Acoustic model (see FIGS. 1 to 3) for prediction based on the voice forming the speech uttered by the tutor and a Language model (see FIGS. 1 to 3) for prediction based on the overall context of the speech uttered by the tutor, in order to extract text forming the speech (speech-to-text, see FIG. 2) by taking the speech uttered by the tutor as input as raw data for evaluating the accuracy of the speech (see FIG. 3) and the fluency of the language use (see FIG. 1).In an ASR model according to one embodiment of the present invention, by inputting speech uttered by a tutor, the accuracy of prediction regarding text forming the speech uttered by the tutor can be increased from voice-based prediction and context-based prediction, respectively (speech-to-text task, see FIG. 2). For example, in a task of utterance accuracy prediction (see FIG. 3), since the utterance accuracy regarding the voice forming the speech uttered by the tutor is relatively low, if the prediction probability from a Language model that implements context-based prediction for the next word predicted at each time step to predict a word sequence predicted from an Acoustic model learned from aligned learning data of voice and text or a word sequence from an Acoustic model is lower than a threshold probability set in advance, for example, if the Language model is predicted with a probability that the frequency is significantly lower in normal language use, the accuracy of utterance regarding the speech uttered by the tutor can be evaluated as low (speech-to-text task, see FIG. 3). That is, if the speech uttered by the tutor is input, the accuracy of utterance regarding the speech uttered by the tutor can be evaluated as low. Thus, the accuracy of the speech forming the speech uttered by the tutor can be predicted based on the degree of mutual matching between the voice-based prediction and the context-based prediction. For example, if the degree of mutual matching between the voice-based prediction and the context-based prediction is relatively low, the accuracy of the speech forming the speech uttered by the tutor can be predicted to be relatively low (task of predicting accuracy of speech, see Fig. 3). In one embodiment of the present invention, if the degree of mutual matching between the voice-based prediction forming the speech uttered by the tutor and the prediction based on the overall context of the speech uttered by the tutor is high, the accuracy of the speech forming the speech uttered by the tutor can be predicted to be high (task of predicting accuracy of speech, see Fig. 3).

[0127] In a task for predicting the speech accuracy of a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention (see FIG. 3), the speech accuracy of the voice of the speech uttered by the instructor can be numerically predicted according to the degree of mutual matching between the prediction from the Acoustic model and the Language model forming the ASR model as described below (task for predicting the accuracy of the speech, see FIG. 3), and also the fluency of the language speech can be predicted from the text extracted from the instructor's speech (task for predicting the fluency of the language speech after the task of the speech-to-text, see FIG. 1, level classification model). For example, in one embodiment of the present invention, the task for predicting the speech accuracy (see FIG. 3) is accompanied by a task of the speech-to-text (STT) of the ASR model for extracting text from the instructor's speech (see FIG. 2), and the task for predicting the speech accuracy is performed based on the voice forming the instructor's speech and the text or context forming the instructor's speech, respectively. By comparing the predictions based on each other, the accuracy of the speech of the voice forming the speech of the teacher can be inferred according to the degree of mutual matching between the predictions based on different voices and the predictions based on the context, and based on the text extracted from the ASR model (speech-to-text task, see Fig. 2), for example, the number of word tokens forming the word sequence forming the text (e.g., the number of words or word tokens forming the text, which corresponds to the length of the sentence) and the fluency of the language can be inferred according to the class classification of the vocabulary level of each word forming the word sequence (see Fig. 1, level classification model), and thus, in one embodiment of the present invention, as a task accompanying the ASR model that extracts text by inputting the speech uttered by the teacher,The tutor's speech can be subjected to two different tasks, each of which involves making predictions about different evaluation items: 1) accuracy of speech (see Figure 3) and 2) fluency of language use (see Figure 1, level classification model).

[0128] For example, the output of an ASR model for a speech-to-text (STT) task (see Fig. 2) that extracts text from speech by inputting speech uttered by a tutor may be output in the form of text regarding the input speech, and such an ASR model output may not imply a prediction regarding the accuracy and fluency of the input speech. However, in the learning management system (LMS) of the present invention, as a result of different predictions incidentally produced in the speech-to-text task of the ASR model (see Fig. 2), the accuracy of the utterance that forms the speech can be calculated based on the degree of mutual matching between different predictions of a voice-based prediction (Acoustic model, see Fig. 3) and a context-based prediction (Language model, see Fig. 3) regarding the next word, and the next word is inferred based on the result of a weighted summation (see Fig. 2) of these different voice-based predictions and context-based predictions (speech-to-text task, see Fig. 2). (See Fig. 1, level classification model) In this way, the fluency of the instructor's speech can be predicted based on the number of word tokens or word embeddings forming each text or word sequence from the word sequence or text generated from the predicted next word at each time step, that is, from the text generated from the speech-to-text task of the ASR model (see Fig. 2), and the class classification of the sentence length forming the text and the vocabulary of the words forming the text.

[0129] For example, in the task of speech-to-text (see FIG. 2) and the task of predicting the accuracy of utterance (see FIG. 3), the speech-based prediction (Acoustic model) can extract the features of the voice from the same encoder and aggregator targeting the same speech uttered by the instructor, and input the features of the voice extracted from these encoders and aggregators, and from the Acoustic model including the parameters (weights) learned from training data in which the voice and the text are aligned with each other, the output of the last hidden layer and the word token or word embedding that forms the vocabulary are computed (transpose synthesis or inner product) and the probability of each word (word token or word embedding, or tokenized phoneme or syllable, etc.) appearing as the next word (or next phoneme or syllable, etc.), and in one embodiment of the present invention, as different tasks assigned to the ASR, the task of speech-to-text (see FIG. 2) and the task of predicting the accuracy of utterance (see FIG. 3) In the above-described Acoustic model, the prediction of the Acoustic model can be implemented with substantially the same computational processing that predicts the next word (next phoneme or syllable, etc.) by inputting the same speech uttered by the instructor (for example, in the task of predicting the accuracy of the utterance, the result value of the computational processing of the speech-to-text task can be used without performing redundant computational processing as in the speech-to-text task), so in one embodiment of the present invention, it can be said that the task of predicting the accuracy of the utterance (see FIG. 3) can be performed in conjunction with the speech-to-text task (see FIG. 2), but unlike the Acoustic model described above, in the Language model, in the speech-to-text task (see FIG. 2), a group of candidate words predicted with a relatively high probability from the Acoustic model are targeted, and the next word (for example,The Language model can be understood as performing different computational processing in the speech-to-text task (see Fig. 2) and the task of utterance accuracy prediction (see Fig. 3) in that it predicts the probability of each candidate word appearing as a phoneme or syllable (the same candidate words are predicted in the speech-to-text task and the task of utterance accuracy prediction), but in the task of utterance accuracy prediction, it predicts the probability of each word token or word embedding (or tokenized phoneme or syllable, etc.) forming the vocabulary.

[0130] In one embodiment of the present invention, the ASR model can comprehensively mean a network including an architecture of any structure for extracting text from a tutor's speech, that is, for implementing a speech-to-text task (see FIG. 2), and for example, can comprehensively mean any network that can substantially implement a speech-to-text task (see FIG. 2), regardless of the name such as ASR (auto speech recognition) or STT (speech to text). And, the ASR model that can be applied to one embodiment of the present invention may include an Acoustic model (see FIGS. 1 to 3) and a Language model (see FIGS. 1 to 3) that include parameters (weights) learned with different learning data for speech-based prediction and context-based prediction, respectively. In other words, in one embodiment of the present invention, the task of predicting the accuracy of speech for calculating the accuracy of speech uttered by a tutor (see FIG. 3) may include prediction of the next word based on the voice forming the speech (prediction of the Acoustic model) and prediction of the next word based on the overall context of the speech (prediction of the Language model), and in this way, the accuracy of the utterance of the voice forming the speech can be predicted based on the degree of mutual matching between different predictions regarding the next word.

[0131] In one embodiment of the present invention, an Acoustic model implementing text extraction based on voice forming speech (prediction of the next word, speech-to-text task, see FIG. 2) and a Language model implementing text extraction based on the context of speech (prediction of the next word) can be connected in parallel to each other in a task for predicting the accuracy of utterance (see FIG. 3), and for example, in a task for predicting the accuracy of utterance (see FIG. 3), the Acoustic model is configured to operate (e.g., transpose vector product or inner product) on each word (word embedding) forming the vocabulary and each word or word token (e.g., word embedding or tokenized phoneme or syllable, etc.) to determine whether the next word (phoneme or syllable, etc.) will appear in the task for predicting the accuracy of utterance (see FIG. 3). The probability can be predicted, and in the task of predicting the accuracy of the utterance (see Fig. 3), the Language model can extract a context vector containing context information about the relationship or attention between different words by using a word input in each time step as a query and another word input in a time step preceding the current time step as a key, and can calculate the probability that each word (e.g., a phoneme or a syllable, etc.) forming the vocabulary will appear as the next word through an operation (transpose vector product or inner product) between the extracted context vector and each word token (e.g., word embedding or tokenized phoneme or syllable, etc.) forming the vocabulary.

[0132] In one embodiment of the present invention (a task of predicting the accuracy of utterance, see FIG. 3), for each word forming a vocabulary from a speech-based Acoustic model and a context-based Language model, the probability of the word appearing as the next word can be predicted, and the prediction of the Acoustic model and the prediction of the Language model can be performed independently or in parallel, and for example, the Acoustic model can be trained to predict the next word from the speech features by inputting speech features, speech feature vectors, or speech shared vectors output from an encoder of the CNN series and an aggregator of the RNN series, as described below, and learning from training data in which speech and text are aligned with each other, and in the inference step, the next word can be predicted from the speech features by inputting speech features (speech feature vectors or speech shared vectors, described below) output from the encoder and the aggregator.

[0133] Referring to FIGS. 4 and 5, in one embodiment of the present invention, the Language model may not include encoder-decoder cross-attention, including only a decoder in a transformer architecture including an encoder and a decoder, and may include a stack of multiple decoder blocks (e.g., 12 decoder blocks) including multi-head attention with multiple stacks of masked self-attention as a forward Language model, and each decoder block may include multi-head attention and a feed forward neural network (FFN). For example, in one embodiment of the present invention, the Language model can calculate the probability that each word will appear as the next word by calculating the output of the last hidden layer, which includes contextual information (e.g., attention information) or association information with the word input in the preceding time step, with each word token forming the vocabulary (task for predicting the accuracy of utterance, see FIG. 3).

[0134] In this way, in one embodiment of the present invention, in a task for predicting the accuracy of utterance (see FIG. 3), an Acoustic model implementing voice-based prediction and a Language model implementing context-based prediction can independently or in parallel predict the next word through an operation (transpose matrix multiplication or inner product) between the output of the last hidden layer and each word or word token (word embedding or tokenized phoneme or syllable, etc.) forming a vocabulary. For example, in a task for predicting the accuracy of utterance (see FIG. 3), the prediction of the Acoustic model and the prediction of the Language model can be implemented independently or in parallel, and the prediction of the Acoustic model and the Language model can be made based on different information, namely, the voice and context forming the instructor's speech.

[0135] In one embodiment of the present invention, the Acoustic model (see FIGS. 1 to 3) can predict the next word by inputting the features of the voice output from the encoder of the CNN series and the aggregator of the RNN series as described below, and the Language model (see FIGS. 1 to 3) can predict the next word by inputting the preceding word sequence including the next word predicted up to the previous time step, and at this time, the next word can be finally predicted by comprehensively considering the predictions of the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) (speech-to-text task, see FIG. 2), and in this way, the next word predicted at the current time step forms a preceding word sequence together with the next word predicted up to the previous time step, and in the speech-to-text task (see FIG. 2), text can be extracted from the instructor's speech in the form of a sequence of next words sequentially predicted by combining the predictions of the Acoustic model and the Language model at the next time step. (Speech-to-text task, see Fig. 2). In a learning management system (LMS) according to one embodiment of the present invention, in a task of predicting the accuracy of speech for predicting the accuracy of speech of a voice forming an instructor's speech (see Fig. 3), the Acoustic model and the Language model can independently or in parallel predict the next word, and in a speech-to-text task (see Fig. 2), the Acoustic model and the Language model can cooperate with each other to ultimately predict the next word.For example, according to one embodiment of the present invention, in a speech-to-text task that extracts text from a teacher's speech (see FIG. 2), the next word can be predicted by synthesizing the predictions of an acoustic model and a language model, and for example, the word that will appear as the next word can be predicted from the result of synthesizing the predictions of an acoustic model and a language model with different weights (speech-to-text task, see FIG. 2). For example, in one embodiment of the present invention, in a speech-to-text task for extracting text from the instructor's speech (see FIG. 2), rather than a task for predicting the accuracy of speech uttered by the instructor (see FIG. 3), the predictions of the Acoustic model and the Language model are not analyzed in comparison with each other (the degree of mutual matching is calculated), but the predictions of each Acoustic model and the Language model are combined (summed with their respective weights) to ultimately predict the next word, and in this way, the next word finally predicted at each time step can form a text converted from the speech as the output of the speech-to-text task (see FIG. 2) together with the next word predicted at the previous time step (speech-to-text task, see FIG. 2), and the Language model can take a preceding text as input and predict the next word based on the context of the preceding text.For example, in one embodiment of the present invention, when finally predicting the next word in a speech-to-text task (see FIG. 2), the predictions of the Acoustic model and the Language model can be combined with different weights to predict the probability that each word will appear as the next word, and in this way, in the speech-to-text task according to one aspect of the present invention (see FIG. 2), when finally predicting the next word, in order to avoid the difficulty of combining the predictions of the Acoustic model and the Language model due to independent or parallel predictions, for example, if a group of words (the first group of words) predicted as the next word with a relatively high probability from the Acoustic model and a group of words (the second group of words) predicted as the next word with a relatively high probability from the Language model do not contain words that match each other, considering that it may not be possible to finally predict a single word as the next word by combining the predictions of the Acoustic model and the Language model, in the speech-to-text task (see FIG. 2), a group of candidate words predicted as the next word with a relatively high probability from the Acoustic model are targeted, and the Language model is used as the next word. It can be input as a model, and for example, in one embodiment of the present invention, a group of candidate words of a preset number (e.g., 5, etc.) predicted with a relatively high probability from an Acoustic model can be taken, and these group of candidate words can be input into a Language model, and the probability of each of the group of candidate words appearing as the next word can be predicted from the Language model.In this way, in one embodiment of the present invention, in a speech-to-text task (see FIG. 2), the probability of a group of candidate words appearing as the next word can be predicted for the same group of candidate words from an Acoustic model and a Language model, and for example, among words predicted as the next word with a relatively high probability from the Acoustic model, a group of candidate words in a predetermined number (e.g., 5, etc.) in the order of predicted probability can be taken and input to the Language model. In this way, in the speech-to-text task (see Fig. 2), by calculating the prediction probability of the next word for the same group of candidate words, the predictions from each acoustic model and the predictions from the language model may deviate from each other, for example, a group of words for which a relatively high prediction probability was calculated from the acoustic model (words of the first group) and a group of words for which a relatively high probability was calculated from the language model (words of the second group) may not match each other, and thus the difficulty of not being able to ultimately infer or confirm a word as the next word by comprehensively combining the predictions of the acoustic model and the language model can be avoided (avoiding the difficulty of predicting the next word in the speech-to-text task, see Fig. 2).For example, in one embodiment of the present invention, in a speech-to-text task (see FIG. 2), different weights can be given to predictions from an acoustic model and predictions from a language model to calculate a weighted sum, and a relatively higher weight can be given to predictions from an acoustic model that extracts voice features from a teacher's speech and uses the extracted voice features as input to predict the next word, and in a speech-to-text task (see FIG. 2), a relatively higher weight can be given to predictions from an acoustic model based on voice than predictions from a language model based on context, so that the next word can be predicted in the speech-to-text task (see FIG. 2) according to the overall probability calculated from the weighted sum.

[0136] A learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention may include an ASR model for performing two different tasks, namely, a speech-to-text task (see FIG. 2) and a task of predicting the accuracy of speech utterance (see FIG. 3), as described above, and may further include a level classification model (see FIG. 1) for predicting language fluency based on text extracted from speech uttered by a tutor from text generated from the speech-to-text task (see FIG. 2) from the ASR model. For example, the above-mentioned level classification model (see FIG. 1) can count, for example, the number of words forming the text, the number of word tokens, or the number of word embeddings as the length of sentences forming the text extracted through the ASR model, and the speech of the teacher from which the text is extracted including a relatively large number of words, word tokens, or word embeddings can be classified into a relatively high-level class, and the speech of the teacher from which the text is extracted including a relatively small number of words, word tokens, or word embeddings can be classified into a relatively low-level class. For example, in one embodiment of the present invention, the level classification model (see FIG. 1) can assign a higher level of fluency in language use to speeches with longer sentence lengths, even if the speeches have the same meaning or speeches that can accomplish the same quest assigned to the instructor, for example, speeches expressed differently with the same meaning, and more specifically, in the same quest assigned to the instructor, such as greeting an NPC (non-player character), a higher level of fluency can be assigned to speeches uttered with longer sentence lengths.

[0137] The above level classification model (see Fig. 1) can map each word or word embedding forming a vocabulary to a differential level of the look-up table (e.g., a look-up table or code book that classifies each word embedding into a differential level according to the level of vocabulary) for each word or word embedding forming a text extracted from speech, and can classify the corresponding speech into a class with a different level of language fluency by collecting the differential vocabulary levels mapped to each word or word embedding forming a text extracted from speech uttered by an instructor. For example, the above-mentioned level classification model (see Fig. 1) can remove stop words such as conjunctions or articles from text extracted from speech, and can assign a differential level of vocabulary to each word or word embedding remaining after the stop words are removed by referring to a look-up table or a code book, and can classify text formed by the corresponding word sequence or speech from which the text is extracted into classes with different levels of fluency in language use according to the differential level of vocabulary assigned to each word or word embedding.

[0138] In one embodiment of the present invention, the level classification model (see FIG. 1) synthesizes the length of a sentence (such as the number of word tokens or word embeddings) and the vocabulary level of each word forming a sentence as different evaluation items, and applies different weights to these different evaluation items, or inputs the length of a sentence (such as the number of word tokens or word embeddings) and the vocabulary level of each word forming a sentence corresponding to these different evaluation items, and applies a network including learned parameters so as to be able to output classes of different levels of fluency of speech, so as to obtain a more refined decision boundary for the corresponding speech from the length of the sentence (such as the number of word tokens or word embeddings) and the vocabulary level of each word forming a sentence as different evaluation items, that is, the length of the sentence (such as the number of word tokens or word embeddings) and the vocabulary level of each word forming a sentence as different evaluation items, as factors or dimensions for different characteristics expressing fluency of speech, and distinguishes between classes of different levels. A more sophisticated decision boundary can be formed to distinguish between the two. In one embodiment of the present invention, the different classes classified from the above-mentioned level classification model (see Fig. 1) can be assigned according to grade classification according to the educational system, and by classifying language fluency into classes of grade classification according to the educational system, a measure or standard for language fluency that is easier to understand from the perspective of a student as a teacher can be provided.For example, a level classification model according to one embodiment of the present invention (see FIG. 1) can predict language fluency by referencing a look-up table or code book that classifies each word forming the vocabulary as described above into different levels according to the level of vocabulary, and by compiling the level of vocabulary for each word. At this time, in one embodiment of the present invention, the fluency of language fluency and the level of vocabulary of words can be classified into the same class, and for example, the same class classification such as classification of grades according to the education system can be implemented, and accordingly, both the fluency of language fluency and each vocabulary level can be predicted according to the same class classification.

[0139] Extracting voice features from the instructor's speech

[0140] Figure 6 shows an example of a spectrogram, which is divided into unit times along the time axis and expressed with different frequency intensities for each time unit, by applying a Fourier transform to decompose the voice forming the speech into different frequency components for each time unit by dividing it into fixed time units.

[0141] Figure 7 shows an example of a Mel-spectrogram generated through Mel scaling, which gives higher weight to low-frequency sounds in consideration of auditory characteristics sensitive to low frequencies.

[0142] Figure 8 illustrates a diagram for exemplifying contrastive predictive coding (CPC) learning to extract shared voice features from voice features extracted from previous time steps.

[0143] In one embodiment of the present invention, the ASR model divides the voice data or utterance data of the speech uttered by the instructor into a certain time unit (e.g., 15 msec, etc.) and applies a Fourier transform to decompose the voice data or utterance data into different frequency components for each time unit, thereby generating a spectrogram in which the voice data or utterance data is divided into unit time along the time axis and in which the intensity (or strength) of each frequency is expressed for each time unit (see FIG. 6). In this way, the ASR model can perform pre-processing of voice data to convert the voice data or utterance data of the speech into a vector form or a two-dimensional matrix form suitable for DNN (deep neural network) processing. In various embodiments of the present invention, in the pre-processing of the voice data of the speech, a Mel-spectrogram may be generated through Mel scaling that gives a higher weight to the voice data of low frequencies in consideration of the auditory characteristics that are sensitive to low frequencies (see FIG. 7).

[0144] In one embodiment of the present invention, the ASR model may include an encoder (see FIG. 8) that takes as input speech data or utterance data or a spectrogram (see FIGS. 6 and 7) preprocessed from the speech data or utterance data as described above and outputs a compressed vector representation. More specifically, the ASR model may include an encoder (see FIG. 8) of the convolution neural network (CNN) family and an aggregator (see FIG. 8) of the recurrent neural network (RNN) family.

[0145] For the aggregator of the above ASR model (see FIG. 8), considering that the voice data or utterance data of the speech may include information corresponding to the extraction target from the ASR model as noise components that change rapidly along the time axis and information that changes slowly along the time axis and is widely distributed along the time axis and shared, the aggregator (see FIG. 8) may apply CPC (contrastive predictive coding, see FIG. 8) to the feature vector (Zt, see FIG. 8) of the voice output from the encoder (see FIG. 8) for each time step in order to extract the shared information of the voice that is widely distributed along the time axis of the speech, and may be trained to extract the shared vector (Ct) of the voice that is generally shared up to the previous time step. For example, in the above CPC, the feature vector (Zt) of the voice output from the encoder (see FIG. 8) up to the previous time step can be learned to predict the feature vector (Zt+1) of the voice output from the encoder in the subsequent time step, and through such CPC (contrastive predictive coding), the aggregator (see FIG. 8) can be learned to be able to extract the shared vector (Ct) of the voice shared in the previous time step and the subsequent time step well.More specifically, in the above CPC, it can be learned according to a loss function that is configured as a positive pair for the shared vector (Ct) of the voice of the aggregator (see FIG. 8) at the current time step and the feature vector (Zt+1) of the voice output from the encoder at a future time step close to the current time step according to the time position, and as a negative pair for the shared vector (Ct) of the voice of the aggregator (see FIG. 8) at the current time step and the feature vector (e.g., Zt-3, Zt+3) of the voice output from the encoder at a future time step relatively far from the current time step.

[0146] For example, in one embodiment of the present invention, in the CPC, the weights of the transformation matrix can be learned so that the matrix product between the output vector (Zt) of the encoder (see FIG. 8) and the transformation matrix (not shown) at each time step and the shared vector (Ct) of the aggregator (see FIG. 8) have relatively high or relatively low similarity depending on the distance according to the time step, and the shared vector (Ct) of the voice that well extracts the shared features from the feature vector (Zt) of the voice output from the encoder (see FIG. 8) can be produced from the product of the transformation matrix having the weights learned in this way. Referring to FIG. 2, in the speech-to-text task of the ASR model (see FIG. 2), the shared vector (Ct) of the voice extracted from the aggregator can be input to the Acoustic model, and the probability of the next word appearing among the words forming the vocabulary can be predicted from the Acoustic model including parameters learned using voices and texts aligned with each other as training data, for example, the ASR model can include a Language model including parameters learned only from text, together with the Acoustic model learned using voices and texts aligned with each other as training data, and the probability of each candidate word appearing as the next word is calculated by considering the overall context for about 5 candidate words predicted with a relatively high probability from the Acoustic model, and the next word can be predicted based on the result of adding the predictions of the Acoustic model and the predictions of the Language model with different weights (w1, w2, see FIG. 2), and as a result, text can be extracted from the voice forming the speech uttered by the instructor (speech-to-text task, see FIG. 2).For example, in a speech-to-text task (see Fig. 2), the Acoustic model and the Language model can calculate the probability that the same set of candidate words will appear as the next word, and as shown in Fig. 2, they can predict the conditional probability (P, P`, see Fig. 2) that each different word (a, b, c, see Fig. 2) will appear as the next word, given the preceding word or word embedding (S, word token or tokenized phoneme or syllable, etc., see Fig. 2). For reference, when the probability of the next word appearing is predicted from the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) through this specification, the next word may not only mean a unit of meaning, but may also include, for example, predicting a phoneme or syllable that will appear after the phoneme or syllable predicted in the previous time step as a unit of sound, such as a phoneme or syllable, and for example, the Acoustic model (see FIGS. 1 to 3) and the Language model (see FIGS. 1 to 3) may predict the next word, the next syllable, etc. predicted in the previous time step, and thus, the prediction of the next word through this specification may be interpreted in a comprehensive sense.

[0147] Referring to FIG. 2, in one embodiment of the present invention, the ASR model can perform discretization or quantization to extract a discrete integer sequence from a continuous vector, such as speech data or utterance data, and for example, a discrete integer sequence (quantization, see FIG. 2) can be extracted from speech data or utterance data through quantization, and a continuous vector (Z) can be extracted from the discrete integer sequence. ∧After extracting the feature vector (Zt, see Fig. 2), as described above, by applying the CPC loss function, a transformation matrix (not shown) that is multiplied by the matrix with the feature vector (Zt) output from the encoder can be trained to extract the shared vector (Ct), and in the inference step, a shared vector (Ct) of voice that well extracts the shared features from the feature vector (Zt) of voice can be extracted from the matrix multiplication of the feature vector (Zt) of voice output from the encoder and the transformation matrix (not shown) of the learned weights.

[0148] For example, the ASR model can take speech data or utterance data as input, add some noise to the feature vector of the speech (Zt, see FIG. 2) output from the encoder (see FIG. 2), and quantize it (Gumbel-softmax) or apply clustering (K-means clustering) to quantize it, and convert it into a discrete array through such quantization. That is, in one embodiment of the present invention, speech data or utterance data can be taken as input, and a feature vector of the speech (Zt, see FIG. 2) can be output by compressing the speech data to a certain size from the encoder (see FIG. 2), and Gumbel-softmax can be applied to discretize or quantize the feature vector of the speech (Zt, see FIG. 2) output from the encoder. In the above Gumbel-softmax, noise (noise) that is sampled from a uniform distribution u(0, 1) and then a log function is taken is added to the feature vector of the voice output from the encoder (Zt, see Fig. 2), and softmax is taken to derive the probability for each element forming the feature vector of the voice (Zt, see Fig. 2), and the temperature function is applied as a scaling factor, so that a discrete representation similar to a one-hot vector can be produced, and if the embedding matrix is ​​vector-multiplied to this one-hot vector, a continuous vector (Zt) is generated again. ∧ t, see Fig. 2) can be produced, and the continuous vector produced in this way (Z ∧By applying the CPC loss function to the feature vector (Zt) of the voice output from the encoder (see Fig. 2), a transformation matrix can be learned in which a matrix product is calculated with the feature vector (Zt) of the voice output from the encoder to extract a shared vector (Ct) of the voice, and in the inference step, a shared vector (Ct) of the voice that well extracts shared features from the feature vector (Zt) of the voice can be extracted from the matrix product of the feature vector (Zt) of the voice output from the encoder and the transformation matrix of the learned weights.

[0149] In one embodiment of the present invention, a speech feature vector (Zt, see FIG. 2) that compresses the speech data to a certain size from an encoder can be output by inputting speech data or utterance data, and K-means clustering can be applied to discretize or quantize the speech feature vector (Zt, see FIG. 2) output from the encoder. For example, in the K-means clustering, a discrete representation can be produced by assigning an index that is most closely mapped to the speech feature vector (Zt, see FIG. 2) output from the encoder among the indices of an embedding matrix or a code book, and a continuous vector (Zt) is generated again from the index of this discrete representation and the embedding matrix of the code book. ∧ t, see Fig. 2) can be produced, and the continuous vector produced in this way (Z ∧ By applying the CPC loss function to the feature vector (Zt) of the voice output from the encoder (see Fig. 2), a transformation matrix can be learned in which a matrix product is calculated with the feature vector (Zt) of the voice output from the encoder to extract a shared vector (Ct) of the voice, and in the inference step, a shared vector (Ct) of the voice that well extracts shared features from the feature vector (Zt) of the voice can be extracted from the matrix product of the feature vector (Zt) of the voice output from the encoder and the transformation matrix of the learned weights.

[0150] <Controller's Monitoring>

[0151] Hereinafter, as a monitoring by a controller according to one aspect of the present invention, the monitoring by the controller of the location of the playback point of video content that implements a metaverse teaching environment on the instructor's terminal or the instructor's learning progress will be described.

[0152] A learning management system (LMS) based on data generated from a metaverse teaching environment according to one aspect of the present invention is:

[0153] An instructor's terminal that provides a metaverse teaching environment by playing back a first image frame of a left-eye image captured from the left-eye viewing angle of the same subject and a second image frame of a right-eye image captured from the right-eye viewing angle; and

[0154] A terminal of a controller that is communicatively connected to the terminal of the instructor and obtains and displays a third image frame that captures the same subject from a different viewing angle from the left-eye image and right-eye image played on the terminal of the instructor from the terminal of the instructor;

[0155] For example, the above instructor's terminal,

[0156] From a scene composed of 3D objects, a first image frame captured from a left-eye viewing angle, a second image frame captured from a right-eye viewing angle, and a third image frame captured from a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle are generated.

[0157] While providing a metaverse teaching environment on the instructor's terminal from the first and second video frames, the third video frame can be transmitted to the controller's terminal.

[0158] For example, the first and second video frames may be synchronized with the third video frame according to the position of the playback point of the video content implemented from the sequence of the first and second video frames on the instructor's terminal.

[0159] For example, the terminal of the controller can obtain and display a third video frame of a different level of progress for each playback point position of the video content implemented from the sequence of the first and second video frames on the terminals of each instructor from the terminals of a plurality of instructors that are connected in a one-to-many communication manner.

[0160] For example, the terminal of the controller can obtain third image frames from the terminals of multiple instructors and display them as an array of 2D reduced images.

[0161] For example, the terminal of the controller may transmit a control signal for switching the video content being played toward the terminal of the instructor from which the third video frame of the delayed progress has been obtained, based on the judgment of the controller, with respect to a third video frame of a relatively delayed progress, among the third video frames obtained from the terminals of a plurality of instructors, in which the position of the playback point of the video content implemented from the sequence of the first and second video frames is relatively delayed.

[0162] For example, the terminal of the controller can generate a control signal for video content implemented from a sequence of first and second video frames on the terminal of the instructor and transmit the signal to the terminal of the instructor.

[0163] For example, the control signal for the above video content is:

[0164] It may include at least one control signal among a control signal for starting playback of video content, a control signal for ending playback of video content, a control signal for moving a playback point of video content being played, and a control signal for switching video content being played.

[0165] For example, the terminal of the controller can transmit a control signal for moving the playback point of the video content being played or a control signal for switching the video content being played to at least some of the terminals of the instructors among the terminals of the instructors that are connected in a one-to-many communication manner.

[0166] For example, the terminal of the controller can obtain and display instantaneous data on the viewing posture of the instructor to predict the instructor's concentration state from the terminal of the instructor.

[0167] For example, the terminal of the controller may transmit a control signal for switching video content implemented from a sequence of first and second video frames on the terminal of an instructor predicted to have a relatively low concentration state based on the controller's judgment regarding the viewing posture of the instructor predicted to have a relatively low concentration state from instantaneous data on the viewing posture of each instructor obtained from the terminals of a plurality of instructors.

[0168] For example, instantaneous data on the viewing posture of the above instructor,

[0169] It may include information about the instructor's gaze direction or field of vision outside the focus area.

[0170] For example, instantaneous data on the viewing posture of the above instructor,

[0171] The third image frame may be captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's gaze direction or field of view according to the instructor's viewing posture.

[0172] For example, instantaneous data on the viewing posture of the above instructor,

[0173] It may include numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal.

[0174] For example, the angle of the camera for capturing the first to third image frames from a scene in which a 3D object is configured according to the rotation position of the instructor's head detected by the sensor may be reset, and the first to third image frames may be updated according to the reset angle of the camera.

[0175] For example, instantaneous data on the viewing posture of the above instructor,

[0176] The above-mentioned instructor's conversation and action target, a non-player character (NPC), may include numerical data regarding the angular deviation of the instructor's head rotation direction away from the instructor's front.

[0177] For example, instantaneous data on the viewing posture of the above instructor,

[0178] i) the third image frame captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's gaze direction or field of view according to the instructor's viewing posture; and

[0179] ii) Including numerical data on the instructor's head rotation position detected from a sensor mounted on the instructor's terminal;

[0180] The above i) third video frame and ii) numerical data can be synchronized with each other with information about the viewing posture of the same instructor.

[0181] For example, the instructor's terminal and the controller's terminal can be directly connected to communicate with each other via webRTC (web Real-Time Communication).

[0182] For example, the instructor's terminal can transmit a video stream formed from a sequence of the third video frames to the controller's terminal.

[0183] Hereinafter, one aspect of a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention will be described in more detail.

[0184] FIG. 9 is a diagram illustrating a webRTC connection that forms a communication channel between an instructor's terminal and a terminal of a controller having control authority over video content that implements a metaverse teaching environment on the instructor's terminal, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0185] FIG. 10 illustrates a drawing for explaining the basic configuration of a three-dimensional graphic library for forming a digital space of the metaverse in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0186] FIG. 11 illustrates a drawing for explaining a configuration in which a video frame sequence or video frame is captured from a camera set in a scene in which a 3D object is configured to form video content.

[0187] FIG. 12 is a diagram illustrating a configuration in which first to third image frames are captured from camera angles set to different left-eye viewing angles, right-eye viewing angles, and a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle from a scene in which a 3D object is configured, in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0188] In one embodiment of the present invention, while providing a teaching environment of the metaverse, the terminal of the instructor for generating or collecting data regarding speech uttered by the instructor, and the terminal of the controller having the authority to control the teaching environment of the metaverse, for example, the authority to control video content played on the terminal of the instructor to implement the metaverse teaching environment, can be directly connected to each other through a communication channel set to webRTC (web Real-Time Communication, see FIG. 9), and, for example, a processing server that generates data or code for implementing the teaching environment of the metaverse, for example, rendering data rendered in a 3D view in a digital space where the metaverse is implemented, or animation data for implementing dynamic motion of an NPC (non-player character) rendered in a 3D view, and transmits it to the local area of ​​the instructor (the terminal of the instructor) without the intervention of a processing server (for example, a processing server that generates data or code for implementing a digital space of the metaverse and transmits it to the terminal of the instructor). It can be directly connected to enable communication through a communication channel set to webRTC (see FIG. 9), and as described below, the instructor's terminal can transmit video streaming regarding video content implementing the digital space of the metaverse on the instructor's terminal toward the controller's terminal connected to webRTC (see FIG. 9). For example, applications such as a web browser, PC application, or app running on the instructor's terminal and the controller's terminal can transmit video information (a third video frame as described below) regarding digital content implemented on the digital space of the metaverse, such as video streaming, through webRTC (see FIG. 9).

[0189] In one embodiment of the present invention, the controller or the controller's terminal may have overall control authority related to video digital content implemented on the instructor's terminal, such as starting and ending playback of video content implementing the digital space of the metaverse on the instructor's terminal, moving the playback point, switching the video content, etc. In addition, the controller or the controller's terminal may broadcast text messages to multiple instructors' terminals connected to the controller's terminal (transmitting them to all instructors' terminals or communication hosts connected to the communication network) or provide a chat environment such as transmission of text messages to the terminals of instructors selected from among multiple instructors.

[0190] Referring to FIGS. 10 and 11, in one embodiment of the present invention, a 3D object such as the NPC can be rendered using a webGL (web Graphic Library), which is a graphic library or rasterization engine, and a 3D graphic library (for example, three.js, which is a 3D graphic library of JavaScript), and a 3D mesh of a 3D object composing a scene from a camera that captures a scene in which a 3D object (or subject) such as the NPC is composed can be rendered into a 2D image frame according to the position and angle of the camera, and the 3D mesh here can mean a basic structure of a 3D object including primitive data such as points, lines, vertices, and triangles regarding the geometric shape of a 3D object such as the NPC, and 3D Geometry or texture that can express reflectivity or texture for lighting (see Fig. 10) can be mapped to the surface of the mesh (see Fig. 10).

[0191] Referring to FIG. 12, in one embodiment of the present invention, the instructor's terminal can generate different image frames, for example, first to third image frames as described below, that capture a scene in which a 3D object (corresponding to the subject of the claims) such as an NPC (non-player character) is configured at different camera angles, and for example, to implement a digital space of the metaverse from the binocular difference between the left-eye viewing angle and the right-eye viewing angle on the instructor's terminal, a first image frame that captures a scene in which a 3D object such as an NPC is configured at a camera angle set to the left-eye viewing angle, a second image frame that captures a scene in which a 3D object such as an NPC is configured at a camera angle set to the right-eye viewing angle, and a third image frame that captures a scene in which a 3D object such as an NPC is configured at a camera angle set to the central viewing angle that evenly divides the angle between the left-eye viewing angle and the right-eye viewing angle, respectively. A frame sequence of a first video frame captured from a right-eye camera angle and a frame sequence of a second video frame captured from a left-eye camera angle can be played on an instructor's terminal to create video content that provides a teaching environment in the digital space of the metaverse.

[0192] A learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention may further include a processing server that is communicatively connected to an instructor's terminal and a controller's terminal, wherein the processing server generates rendering data of an NPC to be implemented as a 3D view in a digital space of the metaverse, rendering data of a background to be implemented as a 2D view or a 3D view in a digital space of the metaverse as a background surrounding the NPC, and animation data for implementing dynamic motion of the NPC, and transmits code or data including the generated rendering data and / or animation data to a local area of ​​the instructor (the instructor's terminal), and the instructor's terminal that has obtained the code including the generated rendering data and / or animation data constructs a scene in which a 3D object such as an NPC is constructed from the obtained data and code, and displays the constructed scene from different camera angles, for example, different left-eye camera angles, right-eye camera angles, and left-eye viewing angles. As video frames of different camera angles regarding the same scene captured by a central camera angle that equally divides the angles between the viewing angles of the right eye (a camera angle set to the central viewing angle), first to third video frames providing different views can be generated, and for example, a specified dynamic motion of animation data obtained through an animation implemented from an update of an NPC's movement over time or an update of a movement from animation data of data or code obtained from a processing server can be implemented.The same scene, in which the update of the movement of an NPC over time or the dynamic motion is expressed, can be captured from three different camera angles to generate different first to third image frames. In one embodiment of the present invention, a 3D view can be generated (3D reconstruction) in the digital space of the metaverse from the reproduction of a left-eye image captured from a camera angle set to a left-eye viewing angle and a right-eye image captured from a camera angle set to a right-eye viewing angle of the same object or NPC forming a scene, and for this purpose, a first image frame for the left-eye image and a second image frame for the right-eye image are generated from data or code transmitted from a processing server, and a 3D view (3D reconstruction) can be provided in the digital space of the metaverse from the frame sequence of the generated first and second image frames or the first and second image frames. In one embodiment of the present invention, in addition to the first and second image frames for the left-eye image and the right-eye image for generating a 3D view in the digital space of the metaverse, a third image frame captured from a camera angle set to a central viewing angle that equally divides or halves the angle between the left-eye viewing angle or camera angle and the right-eye viewing angle or camera angle may be additionally generated. That is, in one embodiment of the present invention, the instructor's terminal into which the instructor's speech is input may generate different first to third image frames that capture the same scene from different viewing angles or camera angles, and among the first to third image frames, the first and second image frames may be transmitted to a head-mounted display (HMD) forming the instructor's terminal to generate a 3D view on the instructor's terminal, thereby providing an environment of a geared metaverse. In one embodiment of the present invention, in addition to the first and second image frames, the third image frame may beIt can be generated from the instructor's terminal, but can be transmitted to the terminal of the controller who has control over the video content implemented on the instructor's terminal, and for example, it can be transmitted through webRTC (see FIG. 9) that establishes a communication channel between the instructor's terminal (an application running on the operating system of the terminal, such as a web browser, PC application, or app running on the instructor's terminal) and the controller's terminal (an application running on the operating system of the terminal, such as a web browser, PC application, or app running on the controller's terminal), and for example, it can be transmitted from the instructor's terminal to the controller's terminal in the form of video streaming formed as a sequence of third video frames. In other words, a direct communication channel can be established between the instructor's terminal (the application running on the terminal) and the controller's terminal (the application running on the terminal) through webRTC (see FIG. 9), and the third video frame or the frame sequence of the third video frame generated on the instructor's terminal can be transmitted to the controller's terminal in the form of video streaming through the communication channel established between the instructor's terminal and the controller's terminal. In one embodiment of the present invention, the first video frame for the left-eye image and the second video frame for the right-eye image can be generated to implement the digital space of the metaverse of the 3D view on the instructor's terminal, and unlike the first and second video frames, the third video frame can be transmitted to the controller's terminal to check the location of the playback point of the video content implemented on the instructor's terminal or the progress of learning. For example, the third video frame may correspond to an image captured from a central viewing angle or a central camera angle that equally divides the viewing angle of the left eye and the viewing angle of the right eye, and a one-to-many communication state is established with the terminals of multiple instructors.The video content implemented on the terminals of each of the instructors can be transmitted from the terminals of multiple instructors to the terminal of the controller so as to check the location of the playback point or the progress of learning, and can be displayed in the form of an array of reduced images on the terminal of the controller, for example, in one embodiment of the present invention, it can be provided as a plurality of reduced images arranged in a matrix form on the terminal of the controller, and can be provided in a 2D view rather than a 3D view so that the video content implemented on the terminals of multiple instructors can be checked at once, and in one embodiment of the present invention, rather than taking any one of the first image frame for the left-eye image and the second image frame for the right-eye image generated on the terminal of the instructor and transmitting it to the terminal of the controller, for example, rather than causing inconvenience to visual perception by providing only any one of the 3D left-eye image and the 3D right-eye image where the left-eye image and the right-eye image need to be played together, a central viewing angle different from the left-eye viewing angle and the right-eye viewing angle (for example, the frontal view) By transmitting a third video frame providing a 2D view observed from a viewing angle to the terminal of the controller, the controller can easily and collectively check the video contents implemented on the terminals of multiple instructors without any inconvenience in visual recognition, and can provide a third video frame that captures the scene at a frontal viewing angle that is easy to visually check, and accordingly, the awkwardness or inconvenience in visual recognition recognized from any one of the first and second video frames captured at different viewing angles to provide a 3D view can be avoided, and the video contents implemented on the terminals of multiple instructors can be more easily and collectively recognized.

[0193] In one embodiment of the present invention, the instructor's terminal can obtain, as code or data transmitted from the processing server, rendering data of an NPC to be implemented in 3D on the instructor's terminal, rendering data of a background surrounding the NPC to be implemented in a 2D or 3D view on the instructor's terminal, and animation data to implement the dynamic motion of the NPC, and can configure a scene including an NPC and a background surrounding the NPC, and an animation in which a specified motion is implemented from a temporal update of the motion of the NPC, and can generate different first to third video frames from different camera angles (left eye, right eye, and center or front) for the scene in which a 3D object such as an NPC is configured, and the scene in which a 3D object such as an NPC is configured in this way can generate, for example, first to third video frames that change at a constant frame rate (fps) along a time axis while being temporally updated according to the specified dynamic motion of the NPC. These first to third video frames can be synchronized with each other in time series while capturing the same scene from different camera angles instantaneously. For example, the first and second video frames can be synchronized with each other in time series while generating different left-eye and right-eye images capturing the same subject (object, for example, a 3D object such as an NPC) on the instructor's terminal, and the third video frame can be synchronized with the first and second video frames played on the instructor's terminal so as to be synchronized with the playback point position of the video content implemented on the instructor's terminal or the progress of the instructor's learning, and the third video frame synchronized with the first and second video frames can be transmitted from the instructor's terminal to the controller's terminal in the form of video streaming.In this way, in one embodiment of the present invention, the same scene can be captured from different camera angles along the time axis to generate first to third video frames that are synchronized with each other, and a 3D view can be generated on the instructor's terminal from the first and second video frames among the first to third video frames, and the playback point position of the video content implemented on the instructor's terminal or the instructor's learning progress can be confirmed on the controller's terminal from the third video frame that is synchronized with the first and second video frames implemented on the instructor's terminal. For example, in one embodiment of the present invention, the playback point position of the video content implemented on the instructor's terminal means the playback point position determined through the playback speed of the video content and the elapsed time from the start of playback, but can be understood as the instructor's learning progress that can progress differentially for each instructor. For example, in the metaverse teaching environment implemented on the instructor's terminal, the flow of the entire video content can progress as the next stage is moved depending on whether the quest provided for each stage, which transitions to a different phase according to the playback of the video content implementing the digital space of the metaverse, is achieved. In other words, the metaverse teaching environment according to one embodiment of the present invention can be designed so that the next stage is not moved if the quest assigned to the instructor is not achieved, that is, if the quest given to the instructor in the previous stage is not achieved. According to this design, the playback point position of the video content implemented on each instructor's terminal means the playback point position determined through the playback speed of the video content and the elapsed time from the start of playback, but can be understood as the instructor's learning progress that can progress differentially for each instructor. It can be understood to mean the progress of learning.

[0194] In one embodiment of the present invention, the controller or the terminal of the controller can check the learning progress of the video content implemented on the terminals of the multiple instructors by forming a one-to-many connection with the multiple instructors or the terminals of the multiple instructors, and, for example, can comprehensively control the playback of the video content implemented on the terminals of the multiple instructors as needed, or selectively control the playback of the video content played on the terminals of some instructors among the terminals of the multiple instructors. For example, the controller or the terminal of the controller can implement comprehensive control of the video content implemented on the terminals of the multiple instructors, such as starting or ending playback of the video content played independently on the terminals of the multiple instructors, moving the playback point of the video content being played, switching the video content being played, etc., or can implement control of the terminals of some specific instructors among the terminals of the multiple instructors. For example, the terminal of the controller can establish a communication channel with the terminals of multiple instructors through webRTC (see FIG. 9), and can receive a third video frame regarding digital content implemented on each of the terminals of the multiple instructors through the one-to-many communication channel, and also, through the communication channel established between the terminals of the multiple instructors, a series of control signals related to video content implemented on the terminals of the multiple instructors, such as start of playback of video content, end of playback, movement of playback point of video content, switching of video content, etc., can be transmitted, and also, notifications (e.g., broadcasting to all hosts connected to a local area network) or text messages (chat) that are transmitted collectively from the controller to the terminals of the multiple instructors can be transmitted.For example, in one embodiment of the present invention, the instructor's terminal and the controller's terminal can be connected to each other through a local area network such as a LAN, and an identifier such as a dynamic IP is assigned to each terminal so that they can recognize each other on the communication network (host candidate of webRTC), and a control signal can be transmitted to all of the instructor's terminals connected to the same communication network such as a LAN (broadcasting), or a control signal can be transmitted to some of the instructor's terminals among the plurality of instructor's terminals using the identifier of the communication network.

[0195] FIG. 13 is a drawing illustrating a configuration in which video frames are updated to different views according to a camera angle reflecting the instructor's head rotation position in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0196] FIGS. 14a to 14c are drawings for explaining a configuration in which video frames of different views are provided according to the instructor's head rotation position in one aspect of a learning management system (LMS) according to one embodiment of the present invention, and different drawings are shown for explaining a configuration in which video frames of views are provided according to the instructor's head rotation position looking in the front direction and the instructor's head rotation position deviating from the front direction to one side and the other side, respectively.

[0197] In one embodiment of the present invention, the terminal of the controller can obtain and display instantaneous data on the viewing posture of the instructor for predicting the concentration state (or concentration level described below) of the instructor from the terminal of the instructor, and the terminal of the controller can transmit a control signal for switching a video content layer implemented from a sequence of first and second video frames on the terminal of the instructor whose concentration state is predicted to be relatively low, based on the controller's judgment on the viewing posture of the instructor whose concentration state is predicted to be relatively low from the instantaneous data on the viewing posture of each instructor obtained from the terminals of a plurality of instructors. For example, in one embodiment of the present invention, the instantaneous data regarding the instructor's viewing posture may include information regarding the instructor's gaze direction or field of view direction that is outside the focus (e.g., angular deviation of the instructor's head direction that is outside the front, see FIGS. 13 to 14c), and, for example, as the image information regarding the instructor's viewing posture, may include a third image frame captured from a scene in which a 3D object is configured according to an angle of a camera set to reflect the instructor's gaze direction or field of view direction according to the instructor's viewing posture.For example, referring to FIGS. 13 to 14c, depending on the instructor's head rotation position that can comprehensively express the instructor's gaze direction or field of view, a focused object such as an NPC expressed in the frontal direction of the video frame may be expressed in an eccentric position that is deviated from the frontal direction or the central position of the video frame and is biased toward one side or the other side of the video frame, and in this way, depending on the instructor's head rotation direction, when the angle of the camera is reset from the scene in which the 3D object such as the NPC is configured, the view of the captured video frame is updated according to the reset camera angle, and through the updated view, the instructor's viewing posture, for example, the instructor's head rotation position, can be estimated, and from the position of the focused object such as an NPC that is deviated from the frontal direction or the central position on the video frame (corresponding to the third video frame) transmitted from the instructor's terminal to the controller's terminal, it can be determined that the instructor's viewing posture is a non-focused state that is far from a focused state of looking straight ahead.

[0198] In one embodiment of the present invention, the instantaneous data regarding the instructor's viewing posture may include a third image frame as image information as described above, and separately from the image information, for example, as an alternative to or additional to the third image frame as image information regarding the instructor's viewing posture, may include numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal.

[0199] Referring to FIG. 15, in one embodiment of the present invention, the metaverse teaching environment may be provided as a geared metaverse environment in which a metaverse gear such as a head-mounted display (HMD) is worn, and for example, the angle of a camera for capturing the first to third image frames from a scene in which a 3D object is configured may be reset according to the head rotation position of the instructor detected by a sensor mounted on the head-mounted display (HMD), and the first to third image frames may be updated according to the reset camera angle. For example, the instantaneous data regarding the instructor's viewing posture may include numerical data regarding the angular deviation of the instructor's head rotation direction away from the frontal direction or the center position of the instructor in which an NPC as a focus target, who is the counterpart of the instructor's conversation and actions, is implemented.

[0200] In other words, in one embodiment of the present invention, the instantaneous data regarding the instructor's viewing posture may include i) a third image frame captured from a scene in which a 3D object such as an NPC is configured according to the angle of a camera set to reflect the instructor's gaze direction or field of view direction according to the instructor's viewing posture (for example, a focus target such as an NPC expressed in a frontal direction or expressed in a position biased toward one side or the other side according to the instructor's head rotation position, as shown in FIGS. 13 to 14c), and ii) numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal (head-mounted display, HMD) (as shown in FIG. 15, an angular deviation of the instructor's head direction away from the front), and at this time, with respect to the instructor's viewing posture, the i) third image frame as image information (see FIGS. 13 to 14c) and ii) numerical data (as shown in FIG. 15, an angular deviation of the instructor's head direction away from the front) may be included together with the numerical data (as shown in FIG. 15, an angular deviation of the instructor's head direction away from the front) of the same instructor. They can be synchronized with each other to express information about viewing posture (e.g., expressing viewing postures for the same point in time).

[0201] In one embodiment of the present invention, with respect to the instructor's viewing posture, a viewing posture in which the instructor's gaze direction or field of view is directed toward the front direction in which the instructor's conversation / action counterpart, an NPC or a peripheral object with which physical / digital interaction is set, is expressed as a focus object can be predicted as a focus state of the instructor, and a viewing posture in which the instructor's gaze direction or field of view is directed away from the front direction (or the center position of a third video frame transmitted toward the controller's terminal) in which the focus object (an NPC or a peripheral object with which physical / digital interaction is set) is expressed can be predicted as a non-focus state. In one embodiment of the present invention, the head direction of the instructor can be tracked to estimate the instructor's gaze direction or visual direction from the output of a sensor for detecting the instructor's head rotation position with respect to the instructor's viewing posture, and the instructor's state of concentration or non-concentration can be predicted from the degree to which the instructor's head direction deviates from the frontal direction (or the central position of a third video frame transmitted to the terminal of the controller), for example, the angular deviation of the instructor's head direction deviates from the frontal direction (see FIG. 15).In various embodiments of the present invention, the angular deviation of the instructor's head direction away from the front where the focused object is expressed can be detected from the output of a sensor for detecting the instructor's head rotation position (see FIG. 15), and further, the camera angle is reset according to the instructor's head rotation position output from the sensor, and a 3D object is captured according to the reset camera angle to update the first to third video frames, and the updated third video frame is transmitted from the instructor's terminal to the controller's terminal, thereby detecting the instructor's gaze direction or viewing direction from the controller's terminal (see FIGS. 13 to 14c). For example, the controller who has confirmed the instructor's gaze direction or viewing direction away from the front where the 3D object, such as an NPC as the focused object, is expressed can recognize a non-focused state for the video content being played, and can switch the currently playing video content, which is the instructor's non-focused state, to another video content that can induce the instructor's focused state, that is, can arouse more interest. For example, a controller or the controller's terminal having control over video content played on the instructor's terminal may generate a control signal to switch the video content played on the instructor's terminal to other video content at the controller's discretion, and may transmit the generated control signal toward the instructor's terminal.In this way, in one embodiment of the present invention, a third video frame as video information regarding the instructor's state of concentration (e.g., video information reflecting the instructor's head direction away from the front, see FIGS. 13 to 14c) and numerical information regarding the instructor's state of concentration (e.g., angular deviation of the instructor's head direction away from the front, see FIG. 15) can be transmitted from the instructor's terminal to the controller's terminal, and the instructor's state of concentration or non-concentration can be confirmed from the controller's terminal, and based on the controller's judgment that the instructor's non-concentration state has been confirmed, the controller's terminal can generate a control signal for switching the video content confirmed as the instructor's non-concentration state to another video content that can induce the instructor's state of concentration, for example, can be more interesting, and transmit the control signal to the instructor's terminal.

[0202] In one embodiment of the present invention, a third video frame in which a camera angle is reset according to an output of a sensor that detects the instructor's head rotation position or the instructor's gaze direction or field of view direction as information that can confirm the instructor's concentration state is reflected, and instantaneous data on the instructor's viewing posture that can predict the instructor's concentration state (concentrated state or non-concentrated state) is transmitted from the instructor's terminal that collected the video information or numerical data on the instructor's viewing posture to the controller's terminal, for example, by using the third video frame as video information on the instructor's viewing posture, and also, data processed into information on, for example, an angular deviation (see FIG. 15) of the instructor's head direction away from the front from the output of a sensor mounted on the instructor's terminal or an angular deviation (see FIG. 15) of the instructor's head direction away from the front can be transmitted to the controller's terminal as numerical data on the instructor's viewing posture. For example, as instantaneous data about the instructor's viewing posture transmitted from the instructor's terminal to the controller's terminal as described above, both video information (the third video frame, FIGS. 13 to 14c) and numerical data (the angular deviation of the instructor's head direction away from the front, see FIG. 15) may be transmitted together, or only the video information (the third video frame, FIGS. 13 to 14c) may be transmitted selectively. In an embodiment where both the video information (the third video frame, see FIGS. 13 to 14c) and the numerical information (the angular deviation of the instructor's head direction away from the front, see FIG. 15) are transmitted together, the video information and the numerical information are synchronized with each other (viewing posture at the same point in time in time series), and can express information about the instructor's viewing posture that is identical to each other, and can prevent confusion of the controller caused by transmission of information about the instructor's viewing posture that is differentially different from each other.

[0203] <webRTC, web Real-Time Communication>

[0204] Referring to FIG. 9, in one embodiment of the present invention, the instructor's terminal and the controller's terminal can form a communication channel through webRTC, and for example, the instructor's terminal and the controller's terminal can perform signaling including a request (Offer SDP) and an acceptance (Answer SDP) of an SDP (session description protocol) that negotiates multimedia session parameters as a protocol that performs capability exchange, such as exchange or negotiation of media, network, or codec-related information, under the involvement of a signaling server, and for example, the signaling server can mediate the transmission of an SDP request (Offer SDP) and an SDP acceptance (Answer SDP) between the instructor's terminal and the controller's terminal, and in one embodiment of the present invention, the instructor's terminal can be a peer that generates an SDP request (Offer SDP) that attempts a webRTC connection, and the controller's terminal can be a peer that accepts the webRTC connection, i.e., accepts the SDP request (Answer SDP). As a peer, the above webRTC can connect the controller's terminal and the instructor's terminal peer-to-peer.

[0205] After the above SDP request (Offer SDP) and SDP acceptance (Answer SDP), signaling is terminated by exchanging connection candidates (e.g., host candidates, etc.) through the ICE candidate process, and thereafter, video streaming (e.g., video streaming formed as a sequence of third video frames) can be directly transmitted between the instructor's terminal and the controller's terminal through an optimal communication channel.

[0206] Concentration

[0207] Below, concentration prediction according to one aspect of the present invention will be described.

[0208] A learning management system (LMS) based on data generated from a metaverse teaching environment according to one aspect of the present invention is:

[0209] An instructor's terminal that provides a metaverse teaching environment through the playback of video content; and

[0210] It may include a sequence model for predicting the instructor's concentration level from the instructor's viewing posture taken during playback of the above video content.

[0211] For example, the sequence model may input time-series data on the instructor's viewing posture at each time step while advancing the time step from the start of playback of the video content toward the end of playback of the video content.

[0212] For example, the above sequence model can predict the concentration for the entire video content as an output at the final time step.

[0213] For example, the sequence model is installed on a processing server that is communicatively connected to the instructor's terminal,

[0214] Data regarding the instructor's viewing posture may be collected from the instructor's terminal and transmitted to a processing server equipped with the sequence model.

[0215] For example, the processing server may evaluate video content collected based on the instructor's viewing posture input into the sequence model based on the concentration predicted from the sequence model, and may evaluate whether the corresponding video content needs to be replaced or reproduced.

[0216] For example, the instructor's terminal includes a plurality of instructor's terminals that are communicatively connected to the processing server,

[0217] The sequence model installed in the above processing server predicts the concentration level of each instructor by inputting the viewing posture of each instructor obtained from each of the terminals of the multiple instructors.

[0218] The above processing server can implement an evaluation of video content in which the instructor's viewing posture is collected from the results of collecting the concentration of each instructor predicted from the viewing posture of each instructor obtained from the terminals of multiple instructors.

[0219] For example, the above sequence model can predict the instructor's concentration level by inputting the instructor's viewing posture, the angular deviation of the instructor's gaze direction or visual field direction away from the object of concentration.

[0220] For example, the above focus target is,

[0221] A non-player character (NPC) that is the counterpart of the instructor's interactions, including the instructor's dialogue and actions; and

[0222] It may include surrounding objects with which interaction with the above NPC is set.

[0223] For example, the above-mentioned focus target may be expressed in the frontal position of the instructor on the video frame forming the video content.

[0224] For example, the above sequence model can predict the instructor's concentration level by inputting the angular deviation of the instructor's head direction away from the front as the instructor's viewing posture.

[0225] For example, the instructor's terminal includes a sensor for detecting the instructor's head rotation position,

[0226] The angle of the camera for capturing a video frame for forming video content from a scene composed of 3D objects is reset based on the instructor's head rotation position detected from the above sensor, and the video frame can be updated based on the reset camera angle.

[0227] For example, the instructor's terminal can track the instructor's head rotation position so that video frames for forming video content are updated according to the instructor's head rotation position detected by the sensor.

[0228] For example, the sequence model may include parameters learned from labeled data to which target labels are assigned, and may include parameters learned to reduce an error or loss function between predicted concentrations using time-series data about the instructor's viewing posture during playback of video content as input as the learning data and target labels assigned to the learning data, which are classified into classes with different levels of concentration.

[0229] For example, the target labels assigned to the learning data of the above sequence model may be assigned to classes of different levels of concentration depending on the degree of interest arousal of the video content itself, but may not be assigned by labeling by a labeler from time-series data regarding the instructor's viewing posture during playback of the video content as learning data.

[0230] For example, in learning the above sequence model, the degree of interest inducing of each video content can be directly correlated with the target label assigned to the learning data of the sequence model.

[0231] For example, the target label assigned to the learning data of the above sequence model may be assigned from viewing impressions or reviews collected from web crawling for each video content.

[0232] For example, the target label assigned to the training data of the above sequence model is

[0233] As classes with different levels of concentration,

[0234] For video content that is relatively interesting, a relatively high level class is given.

[0235] Video content that is relatively uninteresting may be given a relatively low level class.

[0236] For example, a learning management system (LMS) based on data generated from a metaverse teaching environment according to one embodiment of the present invention may further include a controller's terminal that is communicatively connected to the instructor's terminal and obtains and displays instantaneous data on the instructor's viewing posture generated from the instructor's terminal.

[0237] For example, instantaneous data on the viewing posture of the above instructor,

[0238] The video content may include a video frame captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's gaze direction or field of view according to the instructor's viewing posture.

[0239] For example, the terminal of the controller can acquire and display a third image frame captured at a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle, synchronized with the first and second image frames of the left-eye image and the right-eye image captured at different camera angles of the scene composed of the 3D object from the terminal of the instructor.

[0240] For example, instantaneous data on the viewing posture of the above instructor,

[0241] It may include numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal.

[0242] For example, instantaneous data on the viewing posture of the above instructor,

[0243] i) a video frame of video content captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's line of sight or viewing direction according to the instructor's viewing posture; and

[0244] ii) Including numerical data on the instructor's head rotation position detected from a sensor mounted on the instructor's terminal;

[0245] The above i) video frames of the video content and ii) numerical data may be synchronized with each other with information about the viewing posture of the same instructor.

[0246] Hereinafter, a learning management system (LMS) based on data generated from a metaverse teaching environment according to one aspect of the present invention will be described in more detail.

[0247] FIGS. 16 and 17 illustrate different diagrams for explaining the architecture of an RNN network applied to a sequence model for predicting a learner's concentration level from time-series inputs regarding the learner's viewing posture in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0248] FIG. 18 is a diagram illustrating a sequence model for predicting a learner's concentration level from time-series input regarding the learner's viewing posture in one aspect of a learning management system (LMS) according to one embodiment of the present invention.

[0249] In one embodiment of the present invention, the learning management system (LMS) can predict the instructor's concentration level. For example, in one embodiment of the present invention, the learning management system (LMS) can predict the instructor's concentration level from data regarding the instructor's viewing posture tracked from the instructor's terminal on which video content is played. For example, in one embodiment of the present invention, the instructor's terminal may include a head-mounted display (HMD) for providing a geared metaverse environment that implements a digital space of the metaverse, and the head-mounted display (HMD) may include a sensor (e.g., a gyro sensor) for recognizing a rotational position of the HMD, that is, a head rotational position of the instructor wearing the HMD.

[0250] In one embodiment of the present invention, the predicted concentration based on data generated from the teaching environment of the metaverse can be calculated as a parameter value expressing an angular deviation from the direction of gaze or the direction of vision toward a focus target requiring concentration in the digital space of the metaverse (for example, an angular deviation of the direction of the instructor's head away from the front as illustrated in FIG. 15). For example, in one embodiment of the present invention, a focus target requiring concentration in the digital space of the metaverse may be a non-player character (NPC) that is the target of interaction, such as a conversation or action with the instructor, as a primary focus target. For example, an NPC as a counterpart of a conversation or action spoken or expressed by the instructor in the digital space of the metaverse may be the primary focus target. In addition, surrounding objects with which interaction (for example, physical interaction and digital interaction) with the instructor or the NPC is set may also be focus targets.

[0251] In one embodiment of the present invention, the degree of concentration may mean an index of a state of concentration predicted from a sequence model (see FIGS. 16 to 18) that inputs sequential data regarding the direction of the instructor's gaze or visual field, which is linked to time information (e.g., discrete time steps having a chronological relationship according to the progression of time steps), with the angular deviation from the direction of the instructor's gaze or visual field, which deviates from the direction of orientation toward a target of concentration, such as an NPC and a surrounding object with which interaction with the NPC is set, as a parameter. For example, staring straight ahead where a target of concentration is expressed may mean a state of concentration with high concentration, and the direction of the instructor's gaze or visual field, which deviates from the front, may mean a state of non-concentration with low concentration.

[0252] In one embodiment of the present invention, the concentration of the instructor can be predicted based on data generated from the teaching environment of the metaverse (data on the instructor's viewing posture), and as described below, in learning a sequence model for predicting concentration in one embodiment of the present invention, video contents that relatively induce the instructor's interest and video contents that relatively hinder the instructor's interest can be collected, and a target label can be assigned to the learning data according to the degree of interest-inducing of each video content itself. For example, for learning data in which the instructor's viewing posture is collected as learning data, the video contents in which the learning data is collected can be classified into classes by distinguishing them into video contents that relatively induce the instructor's interest or video contents that hinder the instructor's interest (for example, video contents that induce interest and make it easy to concentrate based on average concentration and video contents that hinder interest and make it difficult to concentrate), and a target label for the learning data can be assigned from the class-classified video contents in this way, and the error between the concentration predicted by using the learning data as input and the target label assigned to each learning data can be backpropagated (or time-series). In a sequence model suitable for processing input, a sequence model for predicting concentration can be trained in a BPTT (backpropagation through time) manner, and, for example, parameters (or weights) of the sequence model can be updated as described below.

[0253] In one embodiment of the present invention, for video content predicted from the sequence model to have a low level of concentration of the instructor, that is, predicted to be in an unfocused state of the instructor, or video content that is relatively detrimental to the instructor's interest, replacement or switching to video content that is relatively more interesting to induce a focused state from the instructor in an unfocused state, or reproduction of the currently playing video content may be considered (for example, in order to switch the video content being played, a control signal is transmitted from the terminal of the controller to the terminal of the instructor in an unfocused state).

[0254] In one embodiment of the present invention, the degree of interest inducing of each video content can be inferred through the concentration of the instructor predicted from the sequence model, and various factors such as the composition of the video frames forming the entire playback section from the start of playback of the video content to the end of playback, the development of the story, and the 3D view of the 3D object that frequently appears in each video frame, for example, the 3D view of the NPC as the counterpart of the instructor's conversation and action or the designated dynamic motion of the NCP, or the 3D view of the surrounding object performing physical / digital interaction with the NPC or the physical / digital interaction can work in combination, and in this way, the degree of interest inducing of the entire video content including the corresponding video frame can be inferred only from the frame sequence of the video frames that continues from the start of playback of the video content to the end of playback, for example, the instantaneous concentration of the instructor at a certain time step or the instantaneous viewing posture of the instructor for predicting the instantaneous concentration of the instructor for the same. Since evaluation may be difficult, in one embodiment of the present invention, a sequence model may be applied that inputs sequential data regarding the instructor's viewing posture at discrete time steps at regular time intervals along the frame sequence extending from the start of playback of the video content to the end of playback.Referring to FIGS. 16 to 18, in one embodiment of the present invention, the sequence model can input (x) the instructor's viewing posture at each time step while advancing the time step from the start of playback of video content to the end of playback, output the hidden state (hidden state, h) at the current time step from the viewing posture (current input, x) at the current time step and the hidden state (hidden state, h) of the previous time step, and predict the instructor's overall concentration (y) for the entire video content over the entire playback section from the start of playback of the video content to the end of playback at the final time step. For example, in the sequence model, the output or hidden state (h) at the current time step can be derived from the hidden state (h) of the previous time step and the viewing posture of the instructor generated at the current time step as the input (x) of the current time step (for example, the angular deviation of the instructor's head direction away from the front as the sensor output data or data processed from the sensor output at the current time step), and in one embodiment of the present invention, the sequence model can include a network such as a recurrent neural network (RNN) series network or a long short term memory (LSTM) network suitable for processing time-series data sequentially input according to the time step.For example, as a sequence model according to one embodiment of the present invention, an RNN may include a weight (Wh) for a hidden state (h) in a previous time step, a weight (Wx) for an input (x) in a current time step, and a weight (Wy) for an output (y), and may be expressed as an expanded state of a memory cell according to the advancement of a time step, but the weight (Wh) for the hidden state (h) in each previous time step, the weight (Wx) for the input (x) in the current time step, and the weight (Wy) for the output (y) may be learned as a constant value throughout the entire time step regardless of the advancement of the time step, and may be maintained as a constant value even in the inference step. For example, in one embodiment of the present invention, a hidden state (h) at a time step can be calculated by adding a bias (corresponding to an intercept) to a weighted sum to which each weight (Wh, Wx) is applied, and applying an activation function thereto. For example, in one embodiment of the present invention, a hyperbolic tangent can be applied as the activation function.

[0255] Referring to FIG. 18, in one embodiment of the present invention, the output (y) of the sequence model at the final time step corresponding to the end of playback of video content can predict the concentration of the instructor by accumulating the instructor's viewing posture from the start of playback of the video content to the end of playback while accumulating the hidden state (h) up to the previous time step as the time step advances and also inputting data (input x) regarding the viewing posture in time series at each time step. Referring to FIG. 18, the input (x) for each time step may correspond to data processed from the output data of a sensor that detects the instructor's head rotation position or head direction output from a sensor mounted on a head-mounted display (HMD) for each time step, or information about the instructor's head direction that is out of the front from the output of the sensor, and in this way, the input (x) for each time step and the hidden state (h) for each time step derived from the hidden state (h) of the previous time step may correspond to the concentration for each time step predicted from the viewing posture accumulated cumulatively up to the previous time step.For example, in various embodiments of the present invention, the sequence model for predicting concentration may be implemented in a many-to-one input / output manner so that it predicts concentration over the entire playback section of video content as an output (y) at the final time step from a time-series input (x, a time-series input regarding the instructor's viewing posture) at each time step, or the sequence model for predicting concentration may be implemented in a many-to-many input / output manner so that it predicts concentration up to the corresponding time step from a time-series input (x, a time-series input regarding the instructor's viewing posture) at each time step as an output (y) at each time step, cumulatively accumulated viewing postures up to the previous time step, and, for example, it may be implemented so that the hidden state (h) at each time step is output (y) at each time step, and the concentration up to the corresponding time step is output.

[0256] In one embodiment of the present invention, the instructor's viewing posture as an input for predicting the instructor's concentration level may comprehensively mean the instructor's viewing attitude or viewing posture, from which the concentration level can be predicted from the appearance, such as the instructor's viewing direction or gaze direction, the instructor's head direction, the outline of the face including landmarks such as the instructor's eyes, eyebrows, and mouth, and the instructor's hand position, and in a video content that implements the digital space of the metaverse or a sequence of video frames forming the video content, a concentration state or a viewing posture that looks straight ahead at a concentration target (NPC or a surrounding object with which interaction with the NPC is set, see FIGS. 13 to 14c) expressed in the front of a video frame can be evaluated as a concentration state, and in contrast, a viewing posture in which the instructor's gaze direction or viewing angle deviates from the center position or frontal direction of a video frame in which a concentration target such as an NPC is expressed can be evaluated as a non-concentrated state (see FIGS. 13 to 14c).

[0257] In one embodiment of the present invention, the instructor's viewing posture input into the sequence model to predict the instructor's concentration may include an angular deviation between the directional direction (e.g., front direction) toward a focus target (e.g., NPC, etc.) requiring concentration in the digital space of the metaverse and the instructor's gaze direction or viewing direction. For example, in one embodiment of the present invention, the focus target requiring concentration in the digital space of the metaverse may be an NPC that is the counterpart of the instructor's interaction, such as a conversation or action, and may correspond to the main focus target. For example, an NPC that is the counterpart of a conversation or action spoken or expressed by the instructor in the digital space of the metaverse may correspond to the main focus target. In addition, a surrounding object that is set to have a physical / digital interaction with the instructor or the NPC may also correspond to the focus target. For example, quests such as payment or delivery of goods may be assigned to the instructor for each stage, and 3D objects such as banknotes, coins, and goods may correspond to surrounding objects that have physical interactions (contact, collision, etc.) with the instructor and / or NPC.

[0258] In one embodiment of the present invention, as time-series data input into a sequence model to predict concentration, the instructor's viewing posture may mean an angular deviation formed between the directional direction (front direction) toward the object of concentration in the digital space of the metaverse and the instructor's gaze direction or viewing angle, and in various embodiments of the present invention, in most of the video contents for implementing the digital space of the metaverse, an NPC as a target of concentration requiring the instructor's concentration or a surrounding object with which physical or digital interaction is set may be captured from a camera (e.g., the position and angle of the camera) set at the center position (front direction) of the video frame in a sequence of video frames forming the video contents, and for example, in one embodiment of the present invention, the target of concentration may be expressed at the center position (front direction) of the corresponding video frame in a video frame captured from the camera set as described above, and the instructor's posture gazing straight ahead toward the digital space of the metaverse implemented from the video frame as described above may be evaluated as a state of concentration.For example, in one embodiment of the present invention, video content for implementing a metaverse teaching environment has relatively few display elements suitable for the purpose of learning, such as NPCs and surrounding objects with which physical / digital interactions between NPCs are set, and backgrounds surrounding these NPCs and surrounding objects, and the story development is generally simple. Therefore, in video content designed for such a learning purpose, an NPC who is the counterpart of the instructor's conversation and actions can be expressed at the center position (frontal direction) of the video frame, and accordingly, in one embodiment of the present invention, the instructor's concentration state can be predicted from the viewing posture, that is, the angular deviation between the frontal direction and the instructor's gaze direction or viewing direction (for example, the angular deviation of the instructor's head direction away from the front, see FIG. 15). As described below, in a metaverse environment that can provide a relatively wide viewing angle, the instructor's gaze direction or viewing direction can be estimated through the instructor's head direction. Therefore, in one embodiment for predicting concentration, the instructor's viewing posture for predicting concentration means the angle of the instructor's head direction away from the front. It can be simplified as a deviation (see Fig. 15).

[0259] In one embodiment of the present invention, the instructor's viewing posture input into the sequence model to predict concentration may be different from the instructor's head direction, and for example, in order to predict the instructor's gaze direction with higher accuracy, the position or direction of the pupil forming the eye may be confirmed from an image capturing the instructor's eye, and the instructor's head direction may be captured from a detection signal of a sensor capable of detecting rotational position or rotational acceleration (e.g., a gyro sensor for detecting rotational acceleration) mounted on a head-mounted display (HMD) that provides a geared metaverse environment.

[0260] In a comparative example contrasting with the present invention, unlike a general 2D viewing environment that provides a relatively narrow viewing angle with a limited screen area, in one embodiment of the present invention, the digital space of the metaverse can provide images of different views according to the settings of the camera angle that are updated from the rotational position of the head detected by the output of a sensor mounted on a head-mounted display (HMD) that provides the environment of the geared metaverse (see FIGS. 13 to 14c), and can provide a wider viewing angle than a general 2D image, for example, a wider viewing angle than a general 2D image, such as a 360-degree panoramic view, and in the digital space of the metaverse where such a wide viewing angle is provided, the instructor can freely move his / her gaze without any restriction on his / her gaze, away from the object of focus (for example, an NPC or a surrounding object that physically / digitally interacts with the NPC), and when considering the viewing environment of the 3D view in which the digital space of the metaverse is implemented, the instructor's head that deviates from the frontal direction (center position) Direction (see Fig. 15) can be evaluated as a measure indicating the instructor's non-concentration state, and in the digital space of the metaverse, which can provide a wider viewing angle than a 2D image with a normally limited viewing angle (for example, by providing an updated image by resetting the camera angle according to the instructor's head rotation position), the instructor's head direction can be tracked as a viewing posture input into a sequence model to predict the instructor's concentration, and sequential data including the angular deviation (see Fig. 15) between the instructor's head directions deviating from the frontal direction at each time step is input, and the instructor's concentration can be predicted from a sequence model that inputs the temporal data regarding the angular deviation.

[0261] In one embodiment of the present invention, sequential data including an angular deviation (see FIG. 15) regarding the direction of the instructor's head at each time step can be generated, and a prediction regarding concentration can be generated as an output (y) of a sequence model from the sequential data including an angular deviation (see FIG. 15) regarding the direction of the instructor's head at each time step. For example, the sequence model can take as input the sequential data regarding the instructor's posture and predict concentration as an output (y), and for example, can predict a classification regarding different classes regarding concentration.

[0262] In one embodiment of the present invention, sequential data on the viewing postures of each instructor collected from the terminals of each instructor are input for a plurality of instructors, and the concentration of each instructor can be predicted. By compiling the predictions on the concentration of each instructor, the degree of interest inducing of video content implemented in the digital space of the metaverse can be evaluated. For video content for which relatively low concentration is predicted from a plurality of instructors, the necessity of replacement, conversion (conversion of video content currently being played), or reproduction of the video content can be examined. In one embodiment of the present invention, a control signal for switching the video content being played can be generated from the terminal of the controller and transmitted to the instructor's terminal based on the judgment of a controller having control authority for switching the video content being played based on the instructor's concentration on the video content being played by inferring a correlation between the video content being played and the instructor's concentration level, or data or code for implementing the video content, such as rendering data of a 3D object such as an NCP, 2D or 3D rendering data of a background surrounding an NPC, and animation data for implementing a designated motion of a 3D object such as an NPC, can be generated and transmitted to the instructor's terminal, so that a processing server can review the reproduction of new video content (creation of new video content) to replace video content predicted to have a relatively low concentration level, and can also provide recommendations to an operator or manager of the processing server.

[0263] In one embodiment of the present invention, the instructor's gaze direction or viewing direction may refer to the instructor's gaze direction or viewing direction determined from the rotational position of a head-mounted display (HMD) placed on the instructor's head, may refer to the instructor's head direction or the instructor's head rotation position, and may refer to the instructor's head direction or the instructor's head rotation position that is away from the front. For example, in one embodiment of the present invention, in a metaverse environment that provides a relatively wide viewing angle, considering that the movement of the viewing direction can be performed unrestrictedly and freely compared to a 2D viewing environment that provides a relatively narrow viewing angle, the instructor's gaze direction or viewing direction can be determined from the instructor's head direction.

[0264] In one embodiment of the present invention, the sequence model inputs learning data, as learning data, about the viewing posture of each video content, for example, learning data collected in time series about the viewing posture from the playback of each video content, while advancing the time step from the start of playback of the video content to the end of playback, and assigns a target label as a hidden state or output of the final stage according to the accumulation of time steps, so that the output of the sequence model can perform backpropagation through time (BPTT) so that the target label is followed, for example, in the sequence model, the backpropagation (BPTT) can measure the error at each time step and propagate it to the previous time step, for example, the error between the target label and the prediction at each time step can be calculated, and parameters (weights and / or biases) can be updated from this error, for example, in the sequence model, the weights (weights and / or biases) for the hidden state at the previous time step, the weights (weights and / or biases) for the input at the current time step, and the final time step. A sequence model that can update weights (weights and / or biases) for output at a step, and includes weights updated from such learning data, can predict a class of concentration as a final output, using the instructor's posture as sequential data as time-series input at each time step from the start position to the end position of the video content. For example, the sequence model can implement a recursive process including one memory cell, and thus, the weights for the hidden state at the previous time step, the weights for the input at the current time step, and the weights for the output at the final time step can be maintained in common at each time step.

[0265] In one embodiment of the present invention, in collecting training data for training a sequence model for predicting concentration, the angular deviation of the instructor's head direction away from the front is input as a time series as the instructor's viewing posture while advancing the time step according to the playback of the video content, and the target label can be assigned from the video content itself without a separate labeling work by a labeler, using the degree of interest-inducing of the corresponding video content as a measure. For example, in one embodiment of the present invention, in training the sequence model, the degree of interest-inducing of each video content and the target label assigned to the training data of the sequence model can be directly related to each other. For example, in one embodiment of the present invention, the target label assigned to the training data of the sequence model can be assigned from viewing impressions or reviews collected by web crawling for each video content.

[0266] In collecting learning data for learning a sequence model for predicting concentration according to one embodiment of the present invention, for example, video content that arouses interest and video content that discourages interest can be prepared together depending on the degree of interest arousal, and a target label related to each video content can be assigned based on data that can be collected through web crawling, such as, for example, viewing impressions, and a labeling task by a labeler for a separate target label may not be performed. In various embodiments of the present invention, in learning a sequence model for predicting the concentration, the viewing posture of the instructor collected according to the playback of each video content is used as a time-series input (sequential data), and as a target label to be compared with the prediction of the concentration output at the final time step, a target class classified into a specific class among the classes set in advance according to the degree of interest-inducing of each video content is assigned without separate labeling, and the parameters (weights) of the sequence model can be learned by calculating an error between the prediction of the sequence model and the time-series input regarding the instructor's viewing posture and backpropagating the error. In various embodiments of the present invention, the target label may be assigned from the labeling of a labeler based on the viewing posture of the instructor collected at each time step according to the playback of the video content. In this way, in various embodiments of the present invention, the target label is assigned without a direct labeling task, for example, by referring to viewing impressions or reviews on the web, using the degree of interest-inducing of the video content as a measure, or for each time step according to the playback of the video content. It can also be assigned from the labeler's labeling based on the collected instructor's viewing posture.In learning a sequence model according to one embodiment of the present invention, the target label assigned to the learning data of the sequence model may be a class of different levels with respect to concentration, such that a relatively high level class may be assigned to video content that is relatively interesting, and a relatively low level class may be assigned to video content that is relatively uninteresting, and the target label for the learning data may not be assigned from the labeling of the labeler.

[0267] As described above, in one embodiment of the present invention, in a teaching environment of a metaverse where a digital space of a 3D view is implemented, an updated 3D view can be provided according to the instructor's head rotation position by resetting the camera angle that captures a video frame from a scene where a 3D object is configured according to the instructor's head rotation position, thereby providing a wider angle of view than in a viewing environment of a 2D view that provides a limited screen area. In a teaching environment of a metaverse according to one embodiment of the present invention, without going through complex data acquisition such as capturing the position or direction of the instructor's eyes to capture the instructor's gaze direction as the instructor's viewing posture for predicting concentration, time-series input data can be generated that can output a prediction regarding the instructor's concentration from tracking the instructor's head direction based on the measurement values ​​of a sensor (a sensor for measuring rotational position or rotational acceleration) mounted on a head-mounted display (HMD) that provides a geared metaverse environment, from an angular deviation of the instructor's head direction away from the frontal direction. For example, in one embodiment of the present invention, the head direction of the instructor for outputting a prediction regarding concentration may be continuously tracked from a head-mounted display (HMD) providing a geared metaverse environment, and the angle of the camera may be reset according to the head direction of the instructor, and the head direction of the instructor may be continuously detected and tracked as a parameter from the head-mounted set to provide an updated 3D view according to the head direction of the instructor, and the head direction of the instructor may be detected and tracked without attaching a separate sensor (e.g., a gyro sensor for detecting rotational acceleration, etc.) for outputting a prediction regarding concentration (detectable from the output of a sensor provided on the head-mounted display (HMD) for the purpose of providing a view of an updated viewing angle according to the tracking of the head direction of the instructor).

[0268] In one embodiment of the present invention, the head direction of the instructor, which can be detected and tracked without attaching a separate sensor, is set as a viewing posture for predicting concentration, and sequential data for inputting a sequence model can be generated from the viewing posture regarding the head direction of the instructor at each time step. In addition, since the target label can be assigned from the structure of the video content itself, for example, from web crawling for texts on the web regarding viewing impressions or reviews regarding each video content, without the manual work of a labeler for separate labeling for the target label for collecting learning data, the sequence model for predicting concentration can be implemented relatively easily. In learning a sequence model for predicting concentration according to one embodiment of the present invention, by directly associating the video content itself with a target label for learning the sequence model, it is possible to predict concentration that is more general or can approach the average as the scale of the population increases while avoiding sampling bias due to individual differences such as, for example, the personal taste or viewing posture of the instructor, and in forming a decision boundary forming a boundary between each class with respect to concentration, it is possible to form a sophisticated decision boundary that is more general or can approach the average as the scale of the population increases.

[0269] In one embodiment of the present invention, as the instructor's viewing posture input into the sequence model for predicting concentration, the instructor's head direction away from the front can be detected for each time step, and can be detected in time series as the time step advances, but can also be detected instantaneously for each time step. In one embodiment of the present invention, even if the instructor's concentration is predicted at a specific time step from the instructor's head direction that is instantaneously detected at each time step according to the playback of the video content, such instantaneous concentration prediction may not express the degree of interest-inducing of the entire video content or may not express the degree of interest-inducing of the entire video content. For example, it may not be suitable for use as evaluation data for the video content itself that implements a metaverse teaching environment for inducing the instructor's interest and stimulating the desire to achieve learning, for example, as evaluation data for evaluating the necessity of conversion, replacement, or reproduction of the video content.For example, in one embodiment of the present invention, the instructor's concentration is predicted for each video content implemented in the digital space of the metaverse, and, depending on the development of each video content, points that can arouse the instructor's interest are differentially formed, from the start of playback to the end of playback of the video content, the instructor's viewing posture is detected at each time step as the time step advances from the playback of the entire video content, and the concentration for the entire video content is predicted as the final output from a sequence model that takes the instructor's viewing posture detected from each time step as a time-series input, thereby evaluating the necessity of switching, replacing, or reproducing video content for different video contents that can arouse interest at different points, and, rather than predicting an instantaneous concentration at an arbitrarily specified time step, the concentration is predicted based on time-series data accumulated cumulatively throughout the playback of the entire video content, thereby providing a measure of concentration that can be applied to, for example, differential video contents with different story developments based on the content of the entire video content. Rather than predicting instantaneous concentration based on instantaneous viewing posture, concentration for the entire video content can be predicted from time-series input regarding viewing postures accumulated cumulatively over the entire playback period, and for video content predicted to have a relatively low concentration class, the need for switching (generating and transmitting a control signal for switching), replacement, or reproduction of the video content being played can be assessed as high.

[0270] In one embodiment of the present invention, for video content implemented in the digital space of the metaverse, the instructor's concentration level for each video content is predicted, and based on the predicted concentration level, for video content predicted to have a relatively low concentration level, a control signal for switching the video content being played can be generated / transmitted according to the judgment of a controller having control authority to switch to new video content, replace it with other video content, or create new video content. In one embodiment of the present invention, replacement or reproduction of video content can comprehensively mean replacing the corresponding video content with other video content already generated, or replacing or changing the corresponding video content by creating new video content.

[0271] In one embodiment of the present invention, a sequence model for predicting concentration can predict the concentration of an individual instructor with respect to a corresponding video content by inputting time-series data on the viewing posture of each instructor, and can predict the concentration of the same video content for a plurality of instructors by collecting the predicted concentrations of the individual instructors with respect to the corresponding video content, and can make an evaluation of the corresponding video content (e.g., an evaluation of the necessity of replacing or reproducing the corresponding video content) based on the prediction of the concentrations for the plurality of instructors.

[0272] In one embodiment of the present invention, an evaluation of the corresponding video content can be performed by compiling the predicted concentration for each individual instructor. For example, a correlation between the predicted concentration for each individual instructor, in which individual differences regarding the same video content are projected, and the learning achievement of each instructor can be analyzed. For example, in one embodiment of the present invention, a correlation between the predicted concentration for each instructor generated from a teaching environment of a metaverse in which the same video content is implemented and the learning achievement (evaluation score) for each instructor generated from the teaching environment of the metaverse can be analyzed. For example, a network can be trained as a universal approximator capable of approximating the correlation between concentration and learning achievement (evaluation score) from a network capable of analyzing the correlation between individual concentration and individual learning achievement (evaluation score). For example, in one embodiment of the present invention, the network learns the correlation between individual concentration and learning achievement (evaluation score) as input and output, and thus can, for example, provide a numerical analysis of learning achievement (evaluation score) and a direction for improvement to increase learning achievement based on the numerical analysis.In various embodiments of the present invention, the sequence model may be configured to predict the concentration level from the instantaneous viewing posture of the instructor to the current time step, and for example, by training the sequence model to output a hidden state at each time step, a many-to-many (multiple outputs for multiple inputs) input-output relationship is formed for predicting the concentration level from time-series data on the instantaneous viewing posture of the instructor to the current time step, and by predicting the concentration level up to the current time step in this way, the concentration level of the instructor can be estimated in real time, and by transmitting information on the concentration level predicted up to the current time step to the terminal of the controller, the controller can appropriately set a switching point of video content where the concentration level of the instructor is low or has decreased so that the concentration level of the instructor can be maintained. Thus, in one embodiment of the present invention, the sequence model may be designed with a many-to-one (one output for multiple inputs) input-output relationship to predict the concentration of the entire playback section as an output at the final time step from a time-series input about the viewing posture at each time step, or may be designed with a many-to-many (multiple outputs for multiple inputs) input-output relationship to predict the concentration from a time-series input about the viewing posture at each time step to each time step.

[0273] In one embodiment of the present invention, by establishing a one-to-many communication channel with the terminals of multiple instructors through webRTC, the terminal of the controller or the controller who obtains the view implemented in the digital space of the metaverse implemented on each instructor's terminal in the form of video streaming can check the location of the playback point of the video content implemented on each instructor's terminal or the progress of learning, and in addition, in one embodiment of the present invention, the state of concentration of the instructor can be inferred from video information or numerical data transmitted from the instructor's terminal to the controller's terminal, such as video information of a third video frame reflecting the instructor's gaze direction or head direction, or numerical data regarding the angular deviation of the instructor's head direction away from the frontal direction as the instructor's viewing posture transmitted from the instructor's terminal, or from a class regarding concentration predicted from the numerical data or time-series data as described above.

[0274] In one embodiment of the present invention, the head rotation position of the instructor can be confirmed from a head-mounted display (HMD) that provides a geared metaverse environment, and according to the head rotation position of the instructor, a camera angle for capturing a video frame from a scene can be reset, and according to the reset camera angle, a video frame of an updated camera angle from the scene, for example, a first video frame for a left-eye video, a second video frame for a right-eye video, and a third video frame at a central position that is half the viewing angles of the left and right eyes, can be generated. That is, the instructor's terminal can generate a first video frame for a left-eye video, a second video frame for a right-eye video, and a third video frame transmitted to the terminal of a controller, each of which captures the same scene at three different viewing angles, i.e., a left-eye viewing angle, a right-eye viewing angle, and a central viewing angle that equally divides the angle between the left-eye viewing angle and the right-eye viewing angle, from a scene in which a 3D object such as an NPC is configured.In one embodiment of the present invention, the instructor's terminal can receive data or code including rendering data for rendering an NPC and surrounding objects with which a physical / digital interaction with the NPC is set, or a background surrounding the NPC and surrounding objects, and animation data for implementing a designated dynamic motion of the NPC, from a processing server that generates data or code for implementing the rendering data and the animation data on a digital space of the metaverse, and can generate first to third image frames capturing the same scene from different angles according to the execution of the code received from the processing server or according to the execution of an app or application running on the instructor's terminal, that is, according to an instruction according to the execution of the data or code received from the processing server or according to an instruction of an app or application running on the instructor's terminal, and in one embodiment of the present invention, the first and second image frames can provide a 3D view while implementing a left-eye image and a right-eye image on the instructor's terminal, and the third image frame is transmitted to the terminal of the controller. The location of the playback point of video content implemented on multiple instructors' terminals can be confirmed on the controller's terminal, and the progress of each instructor can be confirmed from the location of the playback point of the video content.

[0275] In one embodiment of the present invention, the third image frame implemented on the terminal of the controller is mounted on a head-mounted display (HMD) that provides a geared metaverse environment, and the first to third image frames can be updated according to the reset camera angle by resetting the camera angle according to the instructor's head rotation position (gyro sensor for detecting rotational acceleration), and the 3D view can be updated on the instructor's terminal according to the left-eye image and right-eye image of the updated first and second image frames, and at the same time, the third image frame captured with a camera angle reflecting the instructor's head rotation position in real time can be transmitted toward the terminal of the controller. For example, the first to third video frames are different video frames reflecting different camera angles for the same scene, and while each of the first to third video frames reflects the updated camera angle, for example, the head rotation state of the instructor, for example, the direction of the instructor's gaze or the direction of view that is out of the front, can be captured from the terminal of the controller who has obtained the third video frame, and for example, the direction of the instructor's gaze or the direction of view that is out of the front where a focused object such as an NPC is expressed, can be captured, and accordingly, the concentration state of the instructor can be estimated. For example, in one embodiment of the present invention, the concentration states of the instructors are checked in real time by being connected to multiple terminals of multiple instructors one-to-many, for example, the direction of the instructor's gaze or the direction of view that is out of the front, and a text message or the like can be transmitted to the corresponding instructor to induce concentration in learning, or a control signal for switching the currently playing video content to another video content according to the judgment of the controller can be transmitted from the terminal of the controller to the terminal of the instructor.

[0276] In various embodiments of the present invention, numerical information about the instructor's viewing posture generated from the instructor's terminal, for example, the angle deviation of the instructor's gaze direction or visual direction or the instructor's head direction away from the front, can be transmitted to the terminal of the controller, and the numerical information about the angle deviation of the instructor's gaze direction or visual direction or the instructor's head direction away from the front can be provided together with the instructor's gaze direction or visual direction or the instructor's head direction reflected in the video content (third video frame) implemented on the instructor's terminal, thereby making it possible to easily check the concentration state of each instructor at the same time, and the controller can induce the instructor's concentration state by sending a text message to an instructor who is not concentrated or can switch the currently playing video content to a new video content.

[0277] In one embodiment of the present invention, in order to predict the concentration of the instructor, time-series data is input for each time step, and information about the instructor's viewing posture for each time step can be input as a sequence model, and for example, in one embodiment of the present invention, a head-mounted display (HMD) for providing a geared metaverse environment includes a sensor for detecting the instructor's head rotation position, for example, a gyro sensor for detecting rotational acceleration, and can reset a camera angle for capturing different first to third image frames from a scene in real time according to the instructor's head rotation position, and can update the first to third image frames according to the reset camera angle, and for this purpose, the head-mounted display (HMD) for providing a geared metaverse environment or the instructor's terminal including the HMD can track the instructor's head rotation position detected in real time, and in one embodiment of the present invention, an angular deviation (numerical data) of the instructor's head rotation position away from the front with respect to the front can be transmitted together with the third image frame (image information), and for example, the same data about the instructor's head rotation position The angular deviation at the rotation position and the third image frame can be transmitted together.

[0278] In one embodiment of the present invention, the instructor's head rotation position transmitted from the instructor's terminal to the controller's terminal can be transmitted in the form of an angle with respect to the rotation position measured with respect to the front, and, for example, an angle deviation (numerical data) with respect to the instructor's head rotation position away from the front from the instructor's head rotation position detected by a sensor (e.g., a gyro sensor) mounted on a head-mounted display (HMD) can be calculated, and a camera angle for capturing a video frame from a scene can be reset, and a video frame (third video frame, video information) according to the reset camera angle can be captured, so that, for example, as information providing real-time monitoring of the instructor's concentration state, the third video frame as image-type video information, and the angle deviation of the instructor's head rotation position as numerical data can both include information (synchronized data) about the same posture of the instructor, that is, a state in which the instructor's head rotation position is at the same rotational position.

[0279] In one embodiment of the present invention, a sequence model for predicting the instructor's concentration is deployed on the instructor's terminal, and can predict the instructor's concentration during the entire playback from the start position to the end position of the video content by inputting time-series data on the instructor's viewing posture, and in various embodiments of the present invention, the sequence model is deployed on a processing server connected to the instructor's terminal, and can obtain time-series data on the instructor's viewing posture from the instructor's terminal, and can predict the concentration during the playback of the entire video content by inputting time-series data transmitted from the instructor's terminal to the processing server, and in this way, in various embodiments of the present invention, considering the computational burden of the sequence model for predicting the concentration and the computational resources of the instructor's terminal, the sequence model can be deployed on a processing server having relatively superior computational processing capabilities, and can obtain time-series data (time-series data on the instructor's viewing posture) transmitted from each instructor's terminal to the sequence model, and can predict the corresponding video content for each instructor from the obtained time-series data. The concentration can be predicted, and the processing server can evaluate the corresponding video content by collecting the concentrations of a plurality of instructors predicted for the corresponding identical video content, and, for example, through an evaluation of the necessity of replacing or reproducing the corresponding video content, for example, a video content predicted to have a relatively low concentration of an instructor or a low concentration that falls below a preset threshold can be evaluated as needing to be replaced or reproducible.

[0280] In one embodiment of the present invention, a sequence model that inputs sequential data regarding a viewing posture of an instructor can predict the instructor's concentration, including the instructor's concentration state and non-concentration state, and at this time, in order to predict the instructor's concentration, the instructor's concentration can be predicted by inputting sequential data regarding the instructor's viewing posture, such as the instructor's viewing direction outside of a focus target, such as an NPC or a surrounding object with which a physical interaction with an NPC is set, or the instructor's viewing direction or head direction outside of the front where these focus targets are expressed, as the viewing posture of the instructor input to the sequence model. In various embodiments of the present invention, as input data for predicting the instructor's concentration, the instructor's viewing posture may comprehensively mean various instructor poses or postures, such as the instructor's position of both hands or the instructor's dynamic motion of both hands, in addition to or in addition to the instructor's viewing direction, or alternatively to the instructor's viewing direction, and for example, the position of both hands holding the controller or the dynamic motion of both hands may be captured from a controller that is communicatively connected to a head-mounted display (HMD) worn on a part of the instructor's body (such as a head) as an instructor's terminal for implementing a geared metaverse environment, and for example, the position of both hands holding the controller or the dynamic motion of both hands may be captured according to the reception sensitivity received from the controller connected to the head-mounted display (HMD) through short-range communication, for example, short-range communication such as infrared communication, and in various embodiments of the present invention, the position of both hands holding the controller and the dynamic motion of both hands may also be visually captured from a vision sensor mounted on the head-mounted display (HMD).For example, in various embodiments of the present invention, the head mounted display can capture spatial information about the surrounding environment as a landmark (detecting relative translation and rotation to the landmark) in order to obtain information about the instructor's manipulation of the controller, for example, the spatial manipulation of the controller, such as three-axis translation and three-axis rotation, and accordingly, the head mounted display (HMD) can be equipped with a vision sensor for capturing spatial information about the surrounding environment, and the position of the controller or the two hands holding the controller or the motion of the two hands can be visually captured from the vision sensor equipped in the head mounted display (HMD). In various embodiments of the present invention, the concentration of the instructor can be predicted by inputting time-series data on the positions of the instructor's two hands and the dynamic motions of the two hands into a sequence model, and as the instructor's viewing posture input into the sequence model for predicting the instructor's concentration, different types of instructor's viewing postures (e.g., poses or postures), such as the instructor's viewing direction and / or the positions or motions of the instructor's two hands, can be input in a composite manner into the sequence model, or the instructor's concentration can be predicted by inputting the positions of the two hands and / or the motions of the two hands, which can be relatively easy to observe, as the viewing postures input into the sequence model instead of the instructor's viewing direction, which can be relatively difficult to observe.

[0281] Throughout this specification, video content may comprehensively mean video content including a sequence of continuous video frames for implementing a metaverse environment by using, for example, binocular parallax; however, the technical scope of the present invention is not limited thereto, and may encompass all various forms of content for implementing a metaverse environment, and may broadly encompass all various forms of content video that can implement a metaverse environment, for example, regardless of the term, such as realistic content or interactive content.

[0282] Although the present invention has been described with reference to the embodiments shown in the attached drawings, these are merely exemplary, and those skilled in the art to which the present invention pertains will understand that various modifications and equivalent other embodiments are possible therefrom.

[0283] The present invention can be used in industrial fields where a learning management system (LMS) can be applied.

Claims

An instructor's terminal that provides a metaverse teaching environment through the playback of video content; and A learning management system based on data generated from a metaverse teaching environment, characterized in that it includes a sequence model for predicting the concentration of an instructor from the instructor's viewing posture taken during playback of the above video content. In the first paragraph, The above sequence model is a learning management system based on data generated from a metaverse teaching environment, characterized in that it inputs sequential data regarding the instructor's viewing posture at each time step while advancing the time step from the start of playback of the video content toward the end of playback of the video content. In the first paragraph, A learning management system based on data generated from a metaverse teaching environment, wherein the sequence model is characterized in that it predicts the concentration for the entire video content as an output at the final time step. In the first paragraph, The above sequence model is installed on a processing server that is communicably connected to the instructor's terminal, A learning management system based on data generated from a metaverse teaching environment, characterized in that data regarding the instructor's viewing posture is collected from the instructor's terminal and transmitted to a processing server equipped with the sequence model. In paragraph 4, A learning management system based on data generated from a metaverse teaching environment, characterized in that the processing server evaluates video content collected based on the viewing posture of the instructor input into the sequence model according to the concentration predicted from the sequence model, and evaluates whether the corresponding video content needs to be replaced or reproduced. In paragraph 5, The above instructor's terminal includes a plurality of instructor's terminals that are communicatively connected to the processing server, The sequence model installed in the above processing server predicts the concentration level of each instructor by inputting the viewing posture of each instructor obtained from each of the terminals of the multiple instructors. A learning management system based on data generated from a metaverse teaching environment, characterized in that the processing server implements an evaluation of video content from which the instructor's viewing posture is collected, based on the results of collecting the concentration of each instructor predicted from the viewing posture of each instructor obtained from the terminals of multiple instructors. In the first paragraph, The above sequence model is a learning management system based on data generated from a metaverse teaching environment, characterized in that it predicts the concentration of the instructor by inputting the angle deviation of the instructor's gaze direction or visual direction away from the target of concentration as the instructor's viewing posture. In paragraph 7, The above focus is, A non-player character (NPC) that is the counterpart of the instructor's interactions, including the instructor's dialogue and actions; and A learning management system based on data generated from a metaverse teaching environment, characterized in that it includes a surrounding object with which interaction with the above NPC is set. In paragraph 7, A learning management system based on data generated from a metaverse teaching environment, characterized in that the above-mentioned focus target is expressed in the frontal position of the instructor on a video frame forming the video content. In paragraph 9, The above sequence model is a learning management system based on data generated from a metaverse teaching environment, characterized in that it predicts the concentration of the instructor by inputting the angular deviation of the instructor's head direction away from the front as the instructor's viewing posture. In Article 10, The above instructor's terminal includes a sensor for detecting the instructor's head rotation position, A learning management system based on data generated from a metaverse teaching environment, characterized in that the angle of a camera for capturing a video frame for forming video content from a scene composed of 3D objects is reset according to the rotation position of the instructor's head detected from the above sensor, and the video frame is updated according to the reset camera angle. In Article 11, A learning management system based on data generated from a metaverse teaching environment, characterized in that the instructor's terminal tracks the instructor's head rotation position so that a video frame for forming video content is updated according to the instructor's head rotation position detected by the sensor. In the first paragraph, A learning management system based on data generated from a metaverse teaching environment, wherein the sequence model includes parameters learned from labeled data to which target labels are assigned, and includes parameters learned to reduce an error or loss function between predicted concentrations using sequential data about a tutor's viewing posture during playback of video content as input as learning data and target labels classified into classes of different levels with respect to concentrations as target labels assigned to the learning data. In Article 13, A learning management system based on data generated from a metaverse teaching environment, characterized in that the target labels assigned to the learning data of the above sequence model are assigned to classes of different levels of concentration according to the degree of interest arousal of the video content itself, but are not assigned by labeling of a labeler from sequential data regarding the instructor's viewing posture during playback of the video content as learning data. In Article 14, A learning management system based on data generated from a metaverse teaching environment, characterized in that the learning of the above sequence model directly correlates the level of interest inducing of each video content with the target label assigned to the learning data of the sequence model. In Article 14, A learning management system based on data generated from a metaverse teaching environment, characterized in that the target labels assigned to the learning data of the above sequence model are assigned from viewing impressions or reviews collected from web crawling for each video content. In Article 14, The target label assigned to the training data of the above sequence model is As classes with different levels of concentration, For video content that is relatively interesting, a relatively high level class is given. A learning management system based on data generated from a metaverse teaching environment, characterized by assigning relatively low-level classes to video content that is relatively less interesting. In the first paragraph, A learning management system based on data generated from a metaverse teaching environment, characterized in that it further includes a terminal of the controller that is communicatively connected to the terminal of the instructor, and that obtains and displays instantaneous data on the instructor's viewing posture generated from the terminal of the instructor. In Article 18, The instantaneous data on the above instructor's viewing posture is: A learning management system based on data generated from a metaverse teaching environment, characterized in that it includes a video frame of video content captured from a scene in which a 3D object is configured according to the angle of a camera set to reflect the instructor's gaze direction or field of view according to the instructor's viewing posture. In Article 19, A learning management system based on data generated from a metaverse teaching environment, characterized in that the terminal of the controller acquires and displays a third image frame captured at a central viewing angle that equally divides the left-eye viewing angle and the right-eye viewing angle, synchronized with the first and second image frames of the left-eye image and the right-eye image of the scene composed of a 3D object captured at different camera angles from the terminal of the instructor. In Article 18, The instantaneous data on the above instructor's viewing posture is: A learning management system based on data generated from a metaverse teaching environment, characterized in that it includes numerical data regarding the instructor's head rotation position detected from a sensor mounted on the instructor's terminal. In Article 18, The instantaneous data on the above instructor's viewing posture is: i) a video frame of video content captured from a scene in which a 3D object is configured according to the angle of the camera set to reflect the instructor's line of sight or viewing direction according to the instructor's viewing posture; and ii) Including numerical data on the instructor's head rotation position detected from a sensor mounted on the instructor's terminal; A learning management system based on data generated from a metaverse teaching environment, characterized in that the above i) video frames of video content and ii) numerical data are synchronized with each other with information about the viewing posture of the same instructor. In the first paragraph, A learning management system based on data generated from a metaverse teaching environment, characterized in that the sequence model predicts the concentration of the instructor by inputting data regarding the position of the instructor's two hands or the motion of the two hands as the instructor's viewing posture. In Article 23, A learning management system based on data generated from a metaverse teaching environment, characterized in that data regarding the position or motion of the instructor's two hands as the instructor's viewing posture is generated by detecting position information of controllers held in the instructor's two hands from the instructor's terminal. In Article 24, A learning management system based on data generated from a metaverse teaching environment, characterized in that data regarding the position of the instructor's two hands or the motion of the two hands as the instructor's viewing posture is generated from the reception sensitivity of the controller that is communicatively connected to a head-mounted display (HMD) as the instructor's terminal or from an output image captured by the controller from a vision sensor mounted on the head-mounted display.

Images