system

Abstract

Provide a system. 【Solution means】 Means for providing learning content including a plurality of information formats to a user terminal, Means for analyzing learning data collected from the user terminal in real time and evaluating the learning progress, Means for providing feedback to the learner based on the evaluation result, Means for appropriately recommending the next-stage learning content, Means for displaying a three-dimensional model using augmented reality technology, Means for performing analysis using a machine learning algorithm and providing an optimal learning experience, Means for generating advice based on learning needs using a generative AI model, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional learning systems, it has been difficult to efficiently provide learning content that individual learners need, and there has been a problem that it is impossible to evaluate the progress of learning in real time and give appropriate feedback. Furthermore, since it takes time and effort to appropriately select and provide the next-stage learning content according to the learner's level of understanding, there is a problem that the learning effect deteriorates.

Means for Solving the Problems

[0005] This invention provides means for providing learning content using multimodal digital information on a user terminal. Furthermore, it provides means for analyzing learning data collected from the user terminal in real time and evaluating its progress to provide appropriate feedback to the learner. Based on the evaluation results, it also provides means for dynamically recommending the next stage of learning content according to the learner's past learning history and progress, thereby aiming to realize efficient and effective learning.

[0006] "Multimodal digital information" refers to information that includes multiple formats such as text, audio, images, and videos, and by integrating and using them, it enables an effective learning experience.

[0007] A "user terminal" is a device that learners directly operate to display and manipulate learning content, and includes smartphones, tablets, and personal computers.

[0008] "Learning data" refers to information including the learner's operation history, response data, and progress, and is used for evaluating and providing feedback on learning.

[0009] "Real-time analysis" refers to the process of immediately processing and analyzing collected data and quickly utilizing the results.

[0010] "Feedback" refers to providing advice, additional information, and corrections based on the learner's current learning progress, with the aim of improving their understanding.

[0011] "Next-stage learning content" refers to new learning modules or topics that follow the current learning content, and are provided based on the learner's progress and needs.

[0012] "Dynamic recommendation" means selecting and suggesting appropriate items each time based on the information obtained, rather than relying on fixed patterns. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] As an embodiment of the present invention, a system for realizing a multimodal learning experience received by learners using a terminal will be described.

[0035] 1. Content distribution and display:

[0036] The server stores multimodal content, such as text, audio, images, and videos, required for each learning module. When a user's device starts a learning session, it accesses the server and retrieves content related to the selected learning module via download and streaming. The device then integrates this content and displays it on the screen via an AR application.

[0037] 2. Interactive learning experience:

[0038] Users manipulate visual information that combines the real environment with digital content using their devices. For example, when a user taps the screen, detailed information may be displayed or audio narration may be played. They can also perform operations such as rotating and scaling 3D models.

[0039] 3. Learning assessment and feedback:

[0040] The user's device sends its operation history and learning responses to the server. The server's AI agent analyzes this data in real time. Based on the analysis results, the server evaluates the user's progress, generates feedback to improve weaknesses, and sends it to the user's device. Specifically, it provides explanations for questions the learner answered incorrectly and suggests new practice problems.

[0041] 4. Suggested next learning steps:

[0042] The server suggests the next learning topic based on real-time evaluation results and past learning history. Users receive these suggestions through their terminal and can proceed to new learning modules. This enables flexible learning tailored to individual learning needs.

[0043] This system allows learners to access a learning curriculum tailored to their needs, enabling efficient understanding and skill development. For example, a user can select a module to learn about the solar system, access 3D models of the planets, and deepen their understanding through detailed explanations and quizzes.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The server receives a user request and then prepares multimodal content corresponding to the requested learning module. It then organizes the data to provide it in the optimal format and size for delivery to the user's device.

[0047] Step 2:

[0048] The device receives learning content streamed from the server and launches an AR application on the device. By displaying the received content on the device's screen and overlaying it onto the real environment, an interactive user interface is realized.

[0049] Step 3:

[0050] Users interact with and manipulate the displayed content through their devices. For example, tapping an object on the screen can trigger additional relevant information or audio narration. They can also manipulate 3D models (rotate, zoom, etc.).

[0051] Step 4:

[0052] The device collects user operation history and quiz answer data in real time and sends it to the server. This data reflects the user's interaction flow and level of understanding.

[0053] Step 5:

[0054] The AI ​​agent on the server analyzes the data received from the terminal and evaluates the learner's progress and understanding. Based on this, it identifies the learner's weaknesses and strengths and generates necessary feedback.

[0055] Step 6:

[0056] The server generates feedback and sends it to the user's device. This feedback includes information on areas where understanding is lacking and what additional information would be helpful. Additional explanations and practice problems may also be provided.

[0057] Step 7:

[0058] The server references the learner's progress and past learning records, selects the next learning content, and provides that information to the terminal. Based on this, the user can start a new learning module.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Traditional learning systems have faced challenges in enabling learners to engage in effective learning tailored to their individual needs. In particular, they struggle to analyze learning progress in real time, provide adaptive feedback, and dynamically recommend learning materials based on past history. As a result, learners are unable to effectively improve their weaknesses and do not receive an optimal learning experience.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; data processing means for analyzing learning data collected from the user terminal in real time and evaluating individual learning progress; means for providing user-adapted feedback using a generative model based on the evaluation results; and recommendation means for dynamically recommending the next learning module based on the user's learning history and current progress. This enables learners to obtain an efficient learning experience tailored to their individual learning needs.

[0064] A "user terminal" is an electronic device used by learners that is equipped with hardware and software for displaying and interacting with content.

[0065] "Multimodal content" refers to learning materials that include multiple information formats such as text, audio, images, and videos, and that provide information to learners through various senses.

[0066] "Data processing means" refers to a set of systems and algorithms for analyzing collected training data and evaluating learning progress.

[0067] A "generative model" is a model that uses AI technology to generate specific outputs from data, and is used, for example, to generate feedback.

[0068] "Feedback" refers to information that indicates evaluations and areas for improvement regarding a user's learning, and is provided to enhance the effectiveness of learning.

[0069] "Recommended methods" is a system that intelligently suggests what to learn next based on past learning history and progress.

[0070] A "learning module" is a collection of learning content designed to teach specific knowledge or skills.

[0071] Embodiments of this invention will be described below.

[0072] This learning support system aims to enable users to learn efficiently through multimodal content that integrates various information formats. At the heart of the system are a server and a user terminal, which utilize an analysis agent incorporating a generative AI model.

[0073] The server centrally manages various forms of content for learners, including multimedia data such as text, audio, images, and video. Users can access specific learning modules through their devices and receive relevant content from the server, which is delivered via download or streaming.

[0074] The user's device utilizes the received content to display learning material using AR (Augmented Reality). For example, a 3D model of a planet is displayed on the screen and presented in an interactive state, allowing learners to observe the model from various angles. Users can interact with the content through their device, obtaining detailed information, playing narration, and manipulating the model.

[0075] Furthermore, as the learning process progresses, user operation history and response data are collected and immediately sent to the server. An analysis agent on the server processes the data using a generative AI model to analyze the user's learning patterns. The analysis is performed in real time, and the user's level of understanding is evaluated based on the results. The evaluation is provided as direct feedback, suggesting areas for improvement and further learning.

[0076] For example, when a learner is studying the solar system, the device displays 3D models of the planets, which the user can grab and rotate, or tap information icons to view detailed information about specific planets. Furthermore, they can deepen their knowledge through explanations and quizzes provided by the server, and explanations are immediately provided as feedback for questions they answered incorrectly.

[0077] Examples of prompts for a generative AI model include, "I want to learn about the solar system. Please tell me about the characteristics of the planets in detail." This prompt allows the system to immediately provide appropriate learning content, creating an interesting and engaging learning experience.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user operates the device and selects a specific learning module. This causes the device to send a request to the server to retrieve the required multimodal content. The input is the user's selection, and the output is the content data received from the server. The device receives this data and prepares it for playback via download or streaming.

[0081] Step 2:

[0082] The device displays the received content using an AR application. Here, the content is overlaid on real-world footage. The input is content data from a server, and the output is an interactive screen display that the user can visually confirm. The user can perform actions such as displaying detailed information or playing narration by tapping or swiping the screen.

[0083] Step 3:

[0084] During learning, the user's operation history and response data are recorded on the device. The device sends this data to the server. The input is the user's operation history and responses, and the output is the log data transferred to the server. The data is compressed and encrypted before transmission to ensure security.

[0085] Step 4:

[0086] The server analyzes the received user data. A generative AI model is used here, and the data is the subject of evaluation. The input is user log data, and the output is an evaluation result regarding the user's learning progress and understanding. Machine learning algorithms are used for the analysis, and the accuracy rate and trends in operations are analyzed.

[0087] Step 5:

[0088] Based on the analysis results, the server generates feedback tailored to the user's learning progress. This is done using a generative AI model, which suggests optimal improvements and the next learning content for the learner. The input is the server's evaluation results, and the output is feedback messages and suggested learning modules. The feedback is sent to the user's device in real time.

[0089] Step 6:

[0090] The user initiates a new learning process based on the feedback provided. The device then retrieves the necessary content from the server again for this new learning session, and the learning progresses cyclically. The input is the feedback from the server, and the output is the new learning session initiated by the user. This allows the learner to move on to the next step, and the learning continues.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] Conventional learning support systems have struggled to provide feedback tailored to learners' progress and offer optimal learning experiences based on their learning needs, especially in educational methods using digital information. Furthermore, they lacked sufficient integration of augmented reality technology to fully leverage the visual learning effects of integration with the real environment. Therefore, flexible responses tailored to individual learning styles are required.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; means for analyzing learning data collected from the user terminal in real time and evaluating learning progress; means for providing feedback to the learner based on the evaluation results; means for appropriately recommending the next stage of learning content; means for displaying a three-dimensional model using augmented reality technology; means for performing analysis using machine learning algorithms and providing an optimal learning experience; and means for generating advice based on learning needs using a generative AI model. This enables the integration of diverse information formats and the provision of an optimal learning experience tailored to individual learning styles.

[0096] A "user terminal" is an information processing device used by learners, enabling them to download, display, and manipulate content.

[0097] "Learning content" refers to educational materials that include digital information such as text, audio, images, and videos, intended for educational purposes.

[0098] Augmented reality technology is a technology that overlays digital information onto the real environment to display it visually.

[0099] A "machine learning algorithm" is a mathematical method used to learn patterns from data and make predictions and decisions.

[0100] A "generative AI model" is a type of artificial intelligence technology that generates language and visual information based on input data.

[0101] "Feedback" refers to information and advice provided based on a learner's progress and understanding, with the aim of improving their learning.

[0102] A "three-dimensional model" is a digital representation of a three-dimensional object that can be displayed and manipulated on a computer.

[0103] "Real-time analysis" is an information processing technology that processes input data instantly and outputs the results immediately.

[0104] This system utilizes user terminals, servers, and generative AI models to provide learners with an interactive learning experience.

[0105] The server stores multimodal learning content in a database and streams the content to the user's device upon request. For example, the history learning content stored on the server includes text information, audio guides, and 3D models. This data is displayed on the user's device using augmented reality technology, allowing the user to visually confirm digital information overlaid on reality. The hardware used is a smart device, and the software used is ARKit (assuming iOS).

[0106] Meanwhile, the user terminal performs real-time analysis based on digital information received from the server. An application implementing machine learning algorithms runs, tracking the user's actions and learning history. This allows for the evaluation of individual progress and learning styles, and the server analyzes the results to generate appropriate feedback.

[0107] The generative AI model generates prompts and next learning steps based on the user's learning history. This model can provide learners with specific advice and tasks, improving learning efficiency. An example of a specific prompt is: "Generate interactive AR experience content for learning about Egyptian civilization. Create learning content that includes 3D models, audio guides, and quizzes."

[0108] This system allows users to learn more deeply not just by receiving information, but through hands-on experience. For example, when learning geography, users can interactively learn about the characteristics of each continent while viewing a three-dimensional model of the Earth through smart glasses. This effectively stimulates learners' understanding and interest.

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The user selects a learning module using their device. The user opens the application on their device and chooses a module from the various learning content provided that suits their interests and learning needs. Once this selection is complete, the device sends a corresponding request to the server. The input is the user's selection, and the output is the request sent to the server.

[0112] Step 2:

[0113] The server receives requests from the user's terminal and retrieves multimodal content related to the selected learning module from the database. The retrieved data includes text, audio, images, and 3D models. The server streams this data to the user's terminal. The input is the request from the terminal, and the output is a stream of content data.

[0114] Step 3:

[0115] The user terminal displays multimodal content received from the server. Using an augmented reality application, digital information is overlaid onto the real environment, displaying three-dimensional models. The user interacts with these models using the terminal's touchscreen and eye-tracking controls. Input is content data from the server, and output is the augmented reality display.

[0116] Step 4:

[0117] While a user interacts with content, the device collects interaction data in real time. This includes taps, swipes, and voice commands. This data is recorded as an interaction history and used to track the user's learning progress. The input is the user's interaction data, and the output is the interaction history record.

[0118] Step 5:

[0119] The terminal sends collected user operation data to the server. The data arriving at the server is analyzed in real time by machine learning algorithms. This analysis evaluates the user's learning patterns and level of understanding. The input is the transmission of operation data, and the output is the analysis results.

[0120] Step 6:

[0121] Based on the analysis results, the server uses a generative AI model to generate optimal feedback and next learning steps for the user. The generated feedback and prompts are sent to the user's terminal. Specifically, they indicate areas where learning needs improvement and suggest new tasks or challenges. The input is the analysis results, and the output is the feedback and prompts.

[0122] Step 7:

[0123] The user terminal displays feedback received from the server and suggestions for the next learning step. Based on this, the user can proceed with their learning or select new content. The input is the feedback from the server, and the output is the next learning action.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention relates to a multimodal learning support system that combines an emotion engine. Specific embodiments thereof are described below.

[0126] 1. Content delivery and sentiment recognition:

[0127] The server prepares the necessary learning content based on user requests and sends it to the user's device. The device has an AR application for learning installed and is equipped with an emotion engine that senses the user's facial expressions and voice through the camera and microphone. This allows the user's emotional state during learning to be recognized in real time.

[0128] 2. Emotion-based interactive experiences:

[0129] When a user accesses learning content on their device, the emotion engine analyzes the user's emotions at that moment and sends the results to the server. The server uses this emotion data to determine the user's stress level and concentration level. For example, if the user shows a confused expression, the server can use that information to suggest content with a lower difficulty level.

[0130] 3. Learning assessment and feedback adjustment:

[0131] The server's AI analyzes learning data and emotional information obtained from the user's device. Based on this analysis, the server generates feedback, adjusting the content and method of the feedback according to the user's emotional state. For example, if the user is fatigued, it will provide encouraging messages or notifications prompting them to take a break.

[0132] 4. Recommended next steps:

[0133] The server suggests the optimal next learning content based on real-time sentiment data and learning history. This suggestion is intended to maximize learning efficiency while maintaining the user's curiosity.

[0134] For example, if the emotion engine detects signs of a user's concentration waning while they are learning a history module, the server will insert a visually engaging video and rearrange the content to keep them interested. This helps maintain the user's motivation while promoting a deeper understanding.

[0135] Thus, a learning support system that incorporates an emotional engine provides personalized support to individual learners, creating an efficient and effective learning environment.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The server prepares the necessary content for the learning module selected by the user and sends it to the user's device. This includes multiple information formats such as animations, audio guides, and text.

[0139] Step 2:

[0140] The device activates an emotion engine to capture the user's facial expressions through the camera and collect audio through the microphone. This data is used for real-time emotion analysis.

[0141] Step 3:

[0142] Users use their devices to view learning content and engage in interactive activities, such as answering on-screen quizzes or manipulating 3D models.

[0143] Step 4:

[0144] The device's emotion engine analyzes the user's facial expressions and voice data to determine their emotional state and sends that data to the server.

[0145] Step 5:

[0146] The server integrates user emotion data and learning progress data and performs AI analysis. If the user is experiencing stress, the server uses this data to adjust the difficulty level of the learning content or provide additional hints.

[0147] Step 6:

[0148] The server generates feedback corresponding to the user's emotional state and sends it to the user's device. For example, if the user is feeling anxious, it displays a relaxing message such as "calm down."

[0149] Step 7:

[0150] The server recommends the next learning content based on the user's emotions and learning history, and provides this information to the device. The user can review the recommendations and proceed to the next learning module.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] Current digital learning systems are insufficient in providing personalized learning experiences that take into account learners' emotional states. As a result, many learners lose interest in the material or experience decreased learning efficiency. Furthermore, dynamically adjusting the learning process using real-time emotion recognition is difficult. In this context, there is a need to provide advanced learning environments that maintain learners' interest and concentration and meet their individual needs.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal, means for analyzing learning data and emotional data collected from the user terminal in real time and evaluating learning progress and emotional state, and means for providing feedback to the learner and adjusting the difficulty level of the content based on the evaluation results and emotional state. This enables personalized learning support that takes the learner's emotional state into consideration.

[0156] A "user terminal" is an electronic device used by learners, and includes hardware and software for displaying learning content and collecting sentiment data.

[0157] "Learning content" refers to a collection of digital information provided for educational purposes, including multiple information formats such as text, images, audio, and video.

[0158] "Emotional data" refers to information that represents the user's emotional state, and is analyzed based on data such as facial expressions and voice acquired through sensors on the user's device.

[0159] "Real-time analysis" is a technology that processes data instantly to obtain results, enabling the immediate adaptation of learning content to users' emotional responses.

[0160] "Feedback" refers to information and instructions provided based on a learner's learning progress and emotional state, and includes adjustments and advice to support effective learning.

[0161] "Difficulty level adjustment" is a process that dynamically changes the complexity and challenge level of learning content according to the learner's abilities and emotional state.

[0162] "Learning efficiency" is an indicator that shows how effectively learners use their time and effort to achieve their goals.

[0163] "Maintaining interest" refers to a state in which learners continuously engage in learning activities and maintain their concentration without interruption.

[0164] This invention relates to a learning support system for individually optimizing a user's learning experience. This system is implemented using a user terminal, a server, and an emotion engine. The specific implementation method is described below.

[0165] The server manages digital information and prepares the necessary information based on user requests for learning content. This content is provided in formats such as text, images, audio, and video and delivered to the user's terminal. The user's terminal displays this content through a specific application. The terminal has an integrated emotion engine that uses a camera and microphone to sense the user's facial expressions and voice in real time and acquire emotion data.

[0166] Emotional data acquired by the device's emotion engine is immediately sent to the server. The server analyzes this emotional data using a generative AI model to evaluate the user's emotional state. For example, if the server determines that the user is confused, it adjusts the learning content based on this information, such as automatically lowering the difficulty level of the content or inserting additional explanations. The content and format of the feedback are also adaptively adjusted based on the user's emotions and learning progress.

[0167] For example, when a user is using a history learning module, a decrease in concentration may be detected by the emotion engine. In this case, the server can support maintaining motivation and deeper understanding by including visually engaging videos or rearranging the content to capture the user's interest.

[0168] An example of a prompt might be a request such as, "Suggest the next best learning content based on the user's emotional state." Based on this prompt, the server can use a generated AI model to recommend the most suitable learning content for the user.

[0169] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0170] Step 1:

[0171] The server receives a request from the user, selects the relevant learning content from the database, and prepares it. The input is the user's selected learning theme and content request, and the output is the learning content data to be sent to the user's terminal. Specifically, the server processes the user's request, generates the URL of the corresponding content, and sends it to the user's terminal.

[0172] Step 2:

[0173] The device receives learning content sent from the server and displays it to the user. The input is content data from the server, and the output is the display of content to the user. Specifically, the learning application on the device renders the content and provides a screen that the user can view or interact with.

[0174] Step 3:

[0175] The device uses a camera and microphone to sense the user's facial expressions and voice, collecting emotional data in real time. The input is visual and audio data from the camera and microphone, and the output is analyzed emotional state data. Specifically, the emotion engine on the device utilizes facial recognition technology and voice analysis technology to quantify the user's emotions and generate that data.

[0176] Step 4:

[0177] The server receives emotional data transmitted from the terminal and analyzes it using a generative AI model. The input is emotional state data from the terminal, and the output is the user's emotional evaluation data. Specifically, the AI ​​model on the server processes the emotional data, calculates the user's stress level and concentration level, and saves the results.

[0178] Step 5:

[0179] The server generates appropriate feedback for the user based on sentiment evaluation data and adjusts the difficulty level of the content. Inputs are sentiment evaluation data and learning progress data, while outputs are the adjusted feedback and content. Specifically, the server generates encouraging messages and additional explanations of the learning content according to the learner's state and sends them to the user's terminal.

[0180] Step 6:

[0181] The server suggests the next best learning content to the user based on the feedback received. The input is the user's learning history, progress, and latest sentiment rating data, and the output is recommended new learning content. Specifically, the server searches the database, selects content that will keep the user interested while promoting efficient learning, and processes it according to instructions such as "Suggest the next best learning content based on the user's sentiment state."

[0182] (Application Example 2)

[0183] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0184] Traditional learning support systems have not taken into account feedback or content suggestions based on learners' emotional states, making it difficult to provide personalized learning experiences tailored to individual learners. Therefore, there is a need to provide methods that reduce the emotional burden on learners and improve learning efficiency.

[0185] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0186] In this invention, the server includes functions for providing learning information containing multiple information formats to a user terminal, functions for analyzing the learning information and emotional information acquired from the user terminal in real time and evaluating learning progress and emotional state, and functions for providing feedback according to the learner's emotional state based on the evaluation results. This makes it possible to provide a personalized learning experience based on the learner's emotional state and maximize learning efficiency.

[0187] "Multimodal digital information" refers to information expressed in multiple formats, such as text, audio, and images.

[0188] A "learning support system" refers to a system that provides technical support to help learners progress effectively in their studies.

[0189] A "user terminal" refers to an electronic device or equipment that individual users operate and use to receive and transmit information.

[0190] "Learning information" refers to the knowledge, materials, and data that learners need in the learning process.

[0191] "Emotional information" refers to data that represents the user's emotional state, and this includes things like facial expressions and tone of voice.

[0192] "Real-time analysis" refers to a processing method that allows data to be processed instantly and results to be obtained immediately.

[0193] "Learning progress" is an indicator that shows how far a learner has progressed in their studies.

[0194] "Feedback" refers to the responses and information provided by a system to a learner, which helps improve their learning.

[0195] A "personalized learning experience" refers to a learning process that is customized according to the individual learner's needs and circumstances.

[0196] "Learning efficiency" refers to the efficiency with which learners use their time and effort to acquire the knowledge and skills they aim for.

[0197] To implement this invention, a user terminal and a server are required. The user terminal can be a portable electronic device such as a smartphone or tablet. The user accesses learning information through an application installed on the terminal and uses the terminal's camera and microphone to acquire facial expressions and voice as emotional information. This emotional information is preprocessed within the terminal and transmitted to the server.

[0198] The server analyzes the learning and emotion data received in real time. Machine learning frameworks such as TENSORFLOW® can be used for this analysis. A generative AI model on the server processes the emotion data and recognizes the user's emotional state. For example, if the user shows a confused expression, the server recognizes this and immediately adjusts the difficulty level of the learning content.

[0199] Furthermore, feedback is created and presented to the user based on their emotional state and learning progress. This feedback may include encouraging messages or recommendations for breaks. If a learner is losing interest, visually engaging additional content is provided to maintain their motivation.

[0200] For example, when a user is practicing English with a language learning app, if the emotion engine detects confusion, the app can offer supplementary materials or provide interactive translation quizzes.

[0201] An example of a prompt sentence to input into a generative AI model is, "When the user's facial expression indicates confusion, suggest what kind of visual content would help with learning."

[0202] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0203] Step 1:

[0204] The user terminal provides learning information to the user. Here, the terminal displays multiple information formats, such as text and images, through an application. The input is learning information from the server, and the output is content provided through the user's sight and hearing.

[0205] Step 2:

[0206] The device uses a camera and microphone to sense the user's facial expressions and voice in real time. The input is data of the user's face and voice, which is acquired as emotional information. The output is emotional data that has been preprocessed for analysis.

[0207] Step 3:

[0208] The device sends the acquired emotion data to the server. Here, the device uses a network protocol to securely transmit the data. The input is pre-processed emotion data, and the output is the data sent to the server.

[0209] Step 4:

[0210] The server analyzes the received emotion data. Here, the server uses a generative AI model such as TensorFlow to estimate the user's emotional state. The input is the emotion data received from the device, and the output is the recognized emotional state.

[0211] Step 5:

[0212] The server adjusts the difficulty level of the learning content based on the analysis results. If an emotion of confusion is detected, the server will select lower difficulty content or provide supplementary materials. The input is the recognized emotional state, and the output is the adjusted learning content.

[0213] Step 6:

[0214] The server generates feedback and sends it to the user's terminal. Here, the server considers the user's emotional state and creates feedback that includes encouraging messages and suggestions for rest. The input is the recognized emotional state and preceding learning progress, and the output is the feedback sent to the user's terminal.

[0215] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0216] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0217] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0218] [Second Embodiment]

[0219] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0220] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0221] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0222] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0223] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0225] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0226] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0227] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0228] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0229] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0230] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0231] As an embodiment of the present invention, a system for realizing a multimodal learning experience received by learners using a terminal will be described.

[0232] 1. Content distribution and display:

[0233] The server stores multimodal content, such as text, audio, images, and videos, required for each learning module. When a user's device starts a learning session, it accesses the server and retrieves content related to the selected learning module via download and streaming. The device then integrates this content and displays it on the screen via an AR application.

[0234] 2. Interactive learning experience:

[0235] Users manipulate visual information that combines the real environment with digital content using their devices. For example, when a user taps the screen, detailed information may be displayed or audio narration may be played. They can also perform operations such as rotating and scaling 3D models.

[0236] 3. Learning assessment and feedback:

[0237] The user's device sends its operation history and learning responses to the server. The server's AI agent analyzes this data in real time. Based on the analysis results, the server evaluates the user's progress, generates feedback to improve weaknesses, and sends it to the user's device. Specifically, it provides explanations for questions the learner answered incorrectly and suggests new practice problems.

[0238] 4. Suggested next learning steps:

[0239] The server suggests the next learning topic based on real-time evaluation results and past learning history. Users receive these suggestions through their terminal and can proceed to new learning modules. This enables flexible learning tailored to individual learning needs.

[0240] This system allows learners to access a learning curriculum tailored to their needs, enabling efficient understanding and skill development. For example, a user can select a module to learn about the solar system, access 3D models of the planets, and deepen their understanding through detailed explanations and quizzes.

[0241] The following describes the processing flow.

[0242] Step 1:

[0243] The server receives a user request and then prepares multimodal content corresponding to the requested learning module. It then organizes the data to provide it in the optimal format and size for delivery to the user's device.

[0244] Step 2:

[0245] The device receives learning content streamed from the server and launches an AR application on the device. By displaying the received content on the device's screen and overlaying it onto the real environment, an interactive user interface is realized.

[0246] Step 3:

[0247] Users interact with and manipulate the displayed content through their devices. For example, tapping an object on the screen can trigger additional relevant information or audio narration. They can also manipulate 3D models (rotate, zoom, etc.).

[0248] Step 4:

[0249] The device collects user operation history and quiz answer data in real time and sends it to the server. This data reflects the user's interaction flow and level of understanding.

[0250] Step 5:

[0251] The AI ​​agent on the server analyzes the data received from the terminal and evaluates the learner's progress and understanding. Based on this, it identifies the learner's weaknesses and strengths and generates necessary feedback.

[0252] Step 6:

[0253] The server generates feedback and sends it to the user's device. This feedback includes information on areas where understanding is lacking and what additional information would be helpful. Additional explanations and practice problems may also be provided.

[0254] Step 7:

[0255] The server references the learner's progress and past learning records, selects the next learning content, and provides that information to the terminal. Based on this, the user can start a new learning module.

[0256] (Example 1)

[0257] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0258] Traditional learning systems have faced challenges in enabling learners to engage in effective learning tailored to their individual needs. In particular, they struggle to analyze learning progress in real time, provide adaptive feedback, and dynamically recommend learning materials based on past history. As a result, learners are unable to effectively improve their weaknesses and do not receive an optimal learning experience.

[0259] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0260] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; data processing means for analyzing learning data collected from the user terminal in real time and evaluating individual learning progress; means for providing user-adapted feedback using a generative model based on the evaluation results; and recommendation means for dynamically recommending the next learning module based on the user's learning history and current progress. This enables learners to obtain an efficient learning experience tailored to their individual learning needs.

[0261] A "user terminal" is an electronic device used by learners that is equipped with hardware and software for displaying and interacting with content.

[0262] "Multimodal content" refers to learning materials that include multiple information formats such as text, audio, images, and videos, and that provide information to learners through various senses.

[0263] "Data processing means" refers to a set of systems and algorithms for analyzing collected training data and evaluating learning progress.

[0264] A "generative model" is a model that uses AI technology to generate specific outputs from data, and is used, for example, to generate feedback.

[0265] "Feedback" refers to information that indicates evaluations and areas for improvement regarding a user's learning, and is provided to enhance the effectiveness of learning.

[0266] "Recommended methods" is a system that intelligently suggests what to learn next based on past learning history and progress.

[0267] A "learning module" is a collection of learning content designed to teach specific knowledge or skills.

[0268] Embodiments of this invention will be described below.

[0269] This learning support system aims to enable users to learn efficiently through multimodal content that integrates various information formats. At the heart of the system are a server and a user terminal, which utilize an analysis agent incorporating a generative AI model.

[0270] The server centrally manages various forms of content for learners, including multimedia data such as text, audio, images, and video. Users can access specific learning modules through their devices and receive relevant content from the server, which is delivered via download or streaming.

[0271] The user's device utilizes the received content to display learning material using AR (Augmented Reality). For example, a 3D model of a planet is displayed on the screen and presented in an interactive state, allowing learners to observe the model from various angles. Users can interact with the content through their device, obtaining detailed information, playing narration, and manipulating the model.

[0272] Furthermore, as the learning process progresses, user operation history and response data are collected and immediately sent to the server. An analysis agent on the server processes the data using a generative AI model to analyze the user's learning patterns. The analysis is performed in real time, and the user's level of understanding is evaluated based on the results. The evaluation is provided as direct feedback, suggesting areas for improvement and further learning.

[0273] For example, when a learner is studying the solar system, the device displays 3D models of the planets, which the user can grab and rotate, or tap information icons to view detailed information about specific planets. Furthermore, they can deepen their knowledge through explanations and quizzes provided by the server, and explanations are immediately provided as feedback for questions they answered incorrectly.

[0274] Examples of prompts for a generative AI model include, "I want to learn about the solar system. Please tell me about the characteristics of the planets in detail." This prompt allows the system to immediately provide appropriate learning content, creating an interesting and engaging learning experience.

[0275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0276] Step 1:

[0277] The user operates the device and selects a specific learning module. This causes the device to send a request to the server to retrieve the required multimodal content. The input is the user's selection, and the output is the content data received from the server. The device receives this data and prepares it for playback via download or streaming.

[0278] Step 2:

[0279] The device displays the received content using an AR application. Here, the content is overlaid on real-world footage. The input is content data from a server, and the output is an interactive screen display that the user can visually confirm. The user can perform actions such as displaying detailed information or playing narration by tapping or swiping the screen.

[0280] Step 3:

[0281] During learning, the user's operation history and response data are recorded on the device. The device sends this data to the server. The input is the user's operation history and responses, and the output is the log data transferred to the server. The data is compressed and encrypted before transmission to ensure security.

[0282] Step 4:

[0283] The server analyzes the received user data. Here, a generative AI model is used and the data is the object of evaluation. The input is the user's log data, and the output is the evaluation result regarding the user's learning progress and understanding level. A machine learning algorithm is used for the analysis, and the accuracy rate and operation tendencies are analyzed.

[0284] Step 5:

[0285] Based on the analysis results, the server generates feedback according to the user's learning progress. This is done using a generative AI model, and the optimal improvement measures and the next learning content for the learner are proposed. The input is the evaluation result at the server, and the output is the feedback message and the proposed learning module. The feedback is sent to the user's terminal in real time.

[0286] Step 6:

[0287] The user starts new learning through the provided feedback. The terminal retrieves again from the server the content necessary for this new learning, and the learning proceeds cyclically. The input is the feedback from the server, and the output is the new learning session that the user starts. As a result, the learner can proceed to the next step and the learning continues.

[0288] (Application Example 1)

[0289] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0290] In the conventional learning support system, in the educational method using digital information, it was difficult to provide feedback according to the learner's progress and to provide an optimal learning experience based on the learning needs. Also, the incorporation of augmented reality technology to fully utilize the visual learning effect by integration with the real environment was insufficient. For this reason, flexible correspondence according to individual learning styles is required.

[0291] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0292] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; means for analyzing learning data collected from the user terminal in real time and evaluating learning progress; means for providing feedback to the learner based on the evaluation results; means for appropriately recommending the next stage of learning content; means for displaying a three-dimensional model using augmented reality technology; means for performing analysis using machine learning algorithms and providing an optimal learning experience; and means for generating advice based on learning needs using a generative AI model. This enables the integration of diverse information formats and the provision of an optimal learning experience tailored to individual learning styles.

[0293] A "user terminal" is an information processing device used by learners, enabling them to download, display, and manipulate content.

[0294] "Learning content" refers to educational materials that include digital information such as text, audio, images, and videos, intended for educational purposes.

[0295] Augmented reality technology is a technology that overlays digital information onto the real environment to display it visually.

[0296] A "machine learning algorithm" is a mathematical method used to learn patterns from data and make predictions and decisions.

[0297] A "generative AI model" is a type of artificial intelligence technology that generates language and visual information based on input data.

[0298] "Feedback" refers to information and advice provided based on a learner's progress and understanding, with the aim of improving their learning.

[0299] A "three-dimensional model" is a digital representation of a three-dimensional object that can be displayed and manipulated on a computer.

[0300] "Real-time analysis" is an information processing technology that processes input data instantly and outputs the results immediately.

[0301] This system utilizes user terminals, servers, and generative AI models to provide learners with an interactive learning experience.

[0302] The server stores multimodal learning content in a database and streams the content to the user's device upon request. For example, the history learning content stored on the server includes text information, audio guides, and 3D models. This data is displayed on the user's device using augmented reality technology, allowing the user to visually confirm digital information overlaid on reality. The hardware used is a smart device, and the software used is ARKit (assuming iOS).

[0303] Meanwhile, the user terminal performs real-time analysis based on digital information received from the server. An application implementing machine learning algorithms runs, tracking the user's actions and learning history. This allows for the evaluation of individual progress and learning styles, and the server analyzes the results to generate appropriate feedback.

[0304] The generative AI model generates prompts and next learning steps based on the user's learning history. This model can provide learners with specific advice and tasks, improving learning efficiency. An example of a specific prompt is: "Generate interactive AR experience content for learning about Egyptian civilization. Create learning content that includes 3D models, audio guides, and quizzes."

[0305] With this system, users can learn more deeply through the experience of actual operation rather than just receiving information. For example, when learning geography, users can interactively learn about the characteristics of each continent while viewing a three-dimensional model of the Earth through smart glasses. This can effectively stimulate the learners' understanding and interest.

[0306] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0307] Step 1:

[0308] The user uses the terminal to select a learning module. The user opens the application on the terminal and selects a module according to their interests and learning needs from the various learning contents provided. When this selection is completed, the terminal sends a corresponding request to the server. The input is the user's selection, and the output is the transmission of the request to the server.

[0309] Step 2:

[0310] The server receives the request from the user terminal and retrieves the multimodal content related to the selected learning module from the database. The retrieved data includes text, audio, images, and three-dimensional models. The server streams these data to the user terminal. The input is the request from the terminal, and the output is the streaming of the content data.

[0311] Step 3:

[0312] The user terminal displays the multimodal content received from the server. Using an augmented reality application, digital information is overlaid on the real environment to display three-dimensional models. The user operates these models using the touch screen or eye gaze operation of the terminal. The input is the content data from the server, and the output is the augmented reality display.

[0313] Step 4:

[0314] While a user interacts with content, the device collects interaction data in real time. This includes taps, swipes, and voice commands. This data is recorded as an interaction history and used to track the user's learning progress. The input is the user's interaction data, and the output is the interaction history record.

[0315] Step 5:

[0316] The terminal sends collected user operation data to the server. The data arriving at the server is analyzed in real time by machine learning algorithms. This analysis evaluates the user's learning patterns and level of understanding. The input is the transmission of operation data, and the output is the analysis results.

[0317] Step 6:

[0318] Based on the analysis results, the server uses a generative AI model to generate optimal feedback and next learning steps for the user. The generated feedback and prompts are sent to the user's terminal. Specifically, they indicate areas where learning needs improvement and suggest new tasks or challenges. The input is the analysis results, and the output is the feedback and prompts.

[0319] Step 7:

[0320] The user terminal displays feedback received from the server and suggestions for the next learning step. Based on this, the user can proceed with their learning or select new content. The input is the feedback from the server, and the output is the next learning action.

[0321] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0322] This invention relates to a multimodal learning support system that combines an emotion engine. Specific embodiments thereof are described below.

[0323] 1. Content delivery and sentiment recognition:

[0324] The server prepares the necessary learning content based on user requests and sends it to the user's device. The device has an AR application for learning installed and is equipped with an emotion engine that senses the user's facial expressions and voice through the camera and microphone. This allows the user's emotional state during learning to be recognized in real time.

[0325] 2. Emotion-based interactive experiences:

[0326] When a user accesses learning content on their device, the emotion engine analyzes the user's emotions at that moment and sends the results to the server. The server uses this emotion data to determine the user's stress level and concentration level. For example, if the user shows a confused expression, the server can use that information to suggest content with a lower difficulty level.

[0327] 3. Learning assessment and feedback adjustment:

[0328] The server's AI analyzes learning data and emotional information obtained from the user's device. Based on this analysis, the server generates feedback, adjusting the content and method of the feedback according to the user's emotional state. For example, if the user is fatigued, it will provide encouraging messages or notifications prompting them to take a break.

[0329] 4. Recommended next steps:

[0330] The server suggests the optimal next learning content based on real-time sentiment data and learning history. This suggestion is intended to maximize learning efficiency while maintaining the user's curiosity.

[0331] For example, if the emotion engine detects signs of a user's concentration waning while they are learning a history module, the server will insert a visually engaging video and rearrange the content to keep them interested. This helps maintain the user's motivation while promoting a deeper understanding.

[0332] Thus, a learning support system that incorporates an emotional engine provides personalized support to individual learners, creating an efficient and effective learning environment.

[0333] The following describes the processing flow.

[0334] Step 1:

[0335] The server prepares the necessary content for the learning module selected by the user and sends it to the user's device. This includes multiple information formats such as animations, audio guides, and text.

[0336] Step 2:

[0337] The device activates an emotion engine to capture the user's facial expressions through the camera and collect audio through the microphone. This data is used for real-time emotion analysis.

[0338] Step 3:

[0339] Users use their devices to view learning content and engage in interactive activities, such as answering on-screen quizzes or manipulating 3D models.

[0340] Step 4:

[0341] The device's emotion engine analyzes the user's facial expressions and voice data to determine their emotional state and sends that data to the server.

[0342] Step 5:

[0343] The server integrates user emotion data and learning progress data and performs AI analysis. If the user is experiencing stress, the server uses this data to adjust the difficulty level of the learning content or provide additional hints.

[0344] Step 6:

[0345] The server generates feedback corresponding to the user's emotional state and sends it to the user's device. For example, if the user is feeling anxious, it displays a relaxing message such as "calm down."

[0346] Step 7:

[0347] The server recommends the next learning content based on the user's emotions and learning history, and provides this information to the device. The user can review the recommendations and proceed to the next learning module.

[0348] (Example 2)

[0349] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0350] Current digital learning systems are insufficient in providing personalized learning experiences that take into account learners' emotional states. As a result, many learners lose interest in the material or experience decreased learning efficiency. Furthermore, dynamically adjusting the learning process using real-time emotion recognition is difficult. In this context, there is a need to provide advanced learning environments that maintain learners' interest and concentration and meet their individual needs.

[0351] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0352] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal, means for analyzing learning data and emotional data collected from the user terminal in real time and evaluating learning progress and emotional state, and means for providing feedback to the learner and adjusting the difficulty level of the content based on the evaluation results and emotional state. This enables personalized learning support that takes the learner's emotional state into consideration.

[0353] A "user terminal" is an electronic device used by learners, and includes hardware and software for displaying learning content and collecting sentiment data.

[0354] "Learning content" refers to a collection of digital information provided for educational purposes, including multiple information formats such as text, images, audio, and video.

[0355] "Emotional data" refers to information that represents the user's emotional state, and is analyzed based on data such as facial expressions and voice acquired through sensors on the user's device.

[0356] "Real-time analysis" is a technology that processes data instantly to obtain results, enabling the immediate adaptation of learning content to users' emotional responses.

[0357] "Feedback" refers to information and instructions provided based on a learner's learning progress and emotional state, and includes adjustments and advice to support effective learning.

[0358] "Difficulty level adjustment" is a process that dynamically changes the complexity and challenge level of learning content according to the learner's abilities and emotional state.

[0359] "Learning efficiency" is an indicator that shows how effectively learners use their time and effort to achieve their goals.

[0360] "Maintaining interest" refers to a state in which learners continuously engage in learning activities and maintain their concentration without interruption.

[0361] This invention relates to a learning support system for individually optimizing a user's learning experience. This system is implemented using a user terminal, a server, and an emotion engine. The specific implementation method is described below.

[0362] The server manages digital information and prepares the necessary information based on user requests for learning content. This content is provided in formats such as text, images, audio, and video and delivered to the user's terminal. The user's terminal displays this content through a specific application. The terminal has an integrated emotion engine that uses a camera and microphone to sense the user's facial expressions and voice in real time and acquire emotion data.

[0363] Emotional data acquired by the device's emotion engine is immediately sent to the server. The server analyzes this emotional data using a generative AI model to evaluate the user's emotional state. For example, if the server determines that the user is confused, it adjusts the learning content based on this information, such as automatically lowering the difficulty level of the content or inserting additional explanations. The content and format of the feedback are also adaptively adjusted based on the user's emotions and learning progress.

[0364] For example, when a user is using a history learning module, a decrease in concentration may be detected by the emotion engine. In this case, the server can support maintaining motivation and deeper understanding by including visually engaging videos or rearranging the content to capture the user's interest.

[0365] An example of a prompt might be a request such as, "Suggest the next best learning content based on the user's emotional state." Based on this prompt, the server can use a generated AI model to recommend the most suitable learning content for the user.

[0366] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0367] Step 1:

[0368] The server receives a request from the user, selects the relevant learning content from the database, and prepares it. The input is the user's selected learning theme and content request, and the output is the learning content data to be sent to the user's terminal. Specifically, the server processes the user's request, generates the URL of the corresponding content, and sends it to the user's terminal.

[0369] Step 2:

[0370] The device receives learning content sent from the server and displays it to the user. The input is content data from the server, and the output is the display of content to the user. Specifically, the learning application on the device renders the content and provides a screen that the user can view or interact with.

[0371] Step 3:

[0372] The device uses a camera and microphone to sense the user's facial expressions and voice, collecting emotional data in real time. The input is visual and audio data from the camera and microphone, and the output is analyzed emotional state data. Specifically, the emotion engine on the device utilizes facial recognition technology and voice analysis technology to quantify the user's emotions and generate that data.

[0373] Step 4:

[0374] The server receives emotional data transmitted from the terminal and analyzes it using a generative AI model. The input is emotional state data from the terminal, and the output is the user's emotional evaluation data. Specifically, the AI ​​model on the server processes the emotional data, calculates the user's stress level and concentration level, and saves the results.

[0375] Step 5:

[0376] The server generates appropriate feedback for the user based on sentiment evaluation data and adjusts the difficulty level of the content. Inputs are sentiment evaluation data and learning progress data, while outputs are the adjusted feedback and content. Specifically, the server generates encouraging messages and additional explanations of the learning content according to the learner's state and sends them to the user's terminal.

[0377] Step 6:

[0378] The server suggests the next best learning content to the user based on the feedback received. The input is the user's learning history, progress, and latest sentiment rating data, and the output is recommended new learning content. Specifically, the server searches the database, selects content that will keep the user interested while promoting efficient learning, and processes it according to instructions such as "Suggest the next best learning content based on the user's sentiment state."

[0379] (Application Example 2)

[0380] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0381] Traditional learning support systems have not taken into account feedback or content suggestions based on learners' emotional states, making it difficult to provide personalized learning experiences tailored to individual learners. Therefore, there is a need to provide methods that reduce the emotional burden on learners and improve learning efficiency.

[0382] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0383] In this invention, the server includes functions for providing learning information containing multiple information formats to a user terminal, functions for analyzing the learning information and emotional information acquired from the user terminal in real time and evaluating learning progress and emotional state, and functions for providing feedback according to the learner's emotional state based on the evaluation results. This makes it possible to provide a personalized learning experience based on the learner's emotional state and maximize learning efficiency.

[0384] "Multimodal digital information" refers to information expressed in multiple formats, such as text, audio, and images.

[0385] A "learning support system" refers to a system that provides technical support to help learners progress effectively in their studies.

[0386] A "user terminal" refers to an electronic device or equipment that individual users operate and use to receive and transmit information.

[0387] "Learning information" refers to the knowledge, materials, and data that learners need in the learning process.

[0388] "Emotional information" refers to data that represents the user's emotional state, and this includes things like facial expressions and tone of voice.

[0389] "Real-time analysis" refers to a processing method that allows data to be processed instantly and results to be obtained immediately.

[0390] "Learning progress" is an indicator that shows how far a learner has progressed in their studies.

[0391] "Feedback" refers to the responses and information provided by a system to a learner, which helps improve their learning.

[0392] A "personalized learning experience" refers to a learning process that is customized according to the individual learner's needs and circumstances.

[0393] "Learning efficiency" refers to the efficiency with which learners use their time and effort to acquire the knowledge and skills they aim for.

[0394] To implement this invention, a user terminal and a server are required. The user terminal can be a portable electronic device such as a smartphone or tablet. The user accesses learning information through an application installed on the terminal and uses the terminal's camera and microphone to acquire facial expressions and voice as emotional information. This emotional information is preprocessed within the terminal and transmitted to the server.

[0395] The server analyzes the received learning and emotion data in real time. Machine learning frameworks such as TensorFlow can be used for this analysis. A generative AI model on the server processes the emotion data and recognizes the user's emotional state. For example, if the user displays a confused expression, the server recognizes this and immediately adjusts the difficulty level of the learning content.

[0396] Furthermore, feedback is created and presented to the user based on their emotional state and learning progress. This feedback may include encouraging messages or recommendations for breaks. If a learner is losing interest, visually engaging additional content is provided to maintain their motivation.

[0397] For example, when a user is practicing English with a language learning app, if the emotion engine detects confusion, the app can offer supplementary materials or provide interactive translation quizzes.

[0398] An example of a prompt sentence to input into a generative AI model is, "When the user's facial expression indicates confusion, suggest what kind of visual content would help with learning."

[0399] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0400] Step 1:

[0401] The user terminal provides learning information to the user. Here, the terminal displays multiple information formats, such as text and images, through an application. The input is learning information from the server, and the output is content provided through the user's sight and hearing.

[0402] Step 2:

[0403] The device uses a camera and microphone to sense the user's facial expressions and voice in real time. The input is data of the user's face and voice, which is acquired as emotional information. The output is emotional data that has been preprocessed for analysis.

[0404] Step 3:

[0405] The device sends the acquired emotion data to the server. Here, the device uses a network protocol to securely transmit the data. The input is pre-processed emotion data, and the output is the data sent to the server.

[0406] Step 4:

[0407] The server analyzes the received emotion data. Here, the server uses a generative AI model such as TensorFlow to estimate the user's emotional state. The input is the emotion data received from the device, and the output is the recognized emotional state.

[0408] Step 5:

[0409] The server adjusts the difficulty level of the learning content based on the analysis results. If an emotion of confusion is detected, the server will select lower difficulty content or provide supplementary materials. The input is the recognized emotional state, and the output is the adjusted learning content.

[0410] Step 6:

[0411] The server generates feedback and sends it to the user's terminal. Here, the server considers the user's emotional state and creates feedback that includes encouraging messages and suggestions for rest. The input is the recognized emotional state and preceding learning progress, and the output is the feedback sent to the user's terminal.

[0412] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0413] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0414] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0415] [Third Embodiment]

[0416] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0417] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0418] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0419] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0420] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0421] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0422] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0423] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0424] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0425] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0426] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0427] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0428] As an embodiment of the present invention, a system for realizing a multimodal learning experience received by learners using a terminal will be described.

[0429] 1. Content distribution and display:

[0430] The server stores multimodal content, such as text, audio, images, and videos, required for each learning module. When a user's device starts a learning session, it accesses the server and retrieves content related to the selected learning module via download and streaming. The device then integrates this content and displays it on the screen via an AR application.

[0431] 2. Interactive learning experience:

[0432] Users manipulate visual information that combines the real environment with digital content using their devices. For example, when a user taps the screen, detailed information may be displayed or audio narration may be played. They can also perform operations such as rotating and scaling 3D models.

[0433] 3. Learning assessment and feedback:

[0434] The user's device sends its operation history and learning responses to the server. The server's AI agent analyzes this data in real time. Based on the analysis results, the server evaluates the user's progress, generates feedback to improve weaknesses, and sends it to the user's device. Specifically, it provides explanations for questions the learner answered incorrectly and suggests new practice problems.

[0435] 4. Suggested next learning steps:

[0436] The server suggests the next learning topic based on real-time evaluation results and past learning history. Users receive these suggestions through their terminal and can proceed to new learning modules. This enables flexible learning tailored to individual learning needs.

[0437] This system allows learners to access a learning curriculum tailored to their needs, enabling efficient understanding and skill development. For example, a user can select a module to learn about the solar system, access 3D models of the planets, and deepen their understanding through detailed explanations and quizzes.

[0438] The following describes the processing flow.

[0439] Step 1:

[0440] The server receives a user request and then prepares multimodal content corresponding to the requested learning module. It then organizes the data to provide it in the optimal format and size for delivery to the user's device.

[0441] Step 2:

[0442] The device receives learning content streamed from the server and launches an AR application on the device. By displaying the received content on the device's screen and overlaying it onto the real environment, an interactive user interface is realized.

[0443] Step 3:

[0444] Users interact with and manipulate the displayed content through their devices. For example, tapping an object on the screen can trigger additional relevant information or audio narration. They can also manipulate 3D models (rotate, zoom, etc.).

[0445] Step 4:

[0446] The device collects user operation history and quiz answer data in real time and sends it to the server. This data reflects the user's interaction flow and level of understanding.

[0447] Step 5:

[0448] The AI ​​agent on the server analyzes the data received from the terminal and evaluates the learner's progress and understanding. Based on this, it identifies the learner's weaknesses and strengths and generates necessary feedback.

[0449] Step 6:

[0450] The server generates feedback and sends it to the user's device. This feedback includes information on areas where understanding is lacking and what additional information would be helpful. Additional explanations and practice problems may also be provided.

[0451] Step 7:

[0452] The server references the learner's progress and past learning records, selects the next learning content, and provides that information to the terminal. Based on this, the user can start a new learning module.

[0453] (Example 1)

[0454] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0455] Traditional learning systems have faced challenges in enabling learners to engage in effective learning tailored to their individual needs. In particular, they struggle to analyze learning progress in real time, provide adaptive feedback, and dynamically recommend learning materials based on past history. As a result, learners are unable to effectively improve their weaknesses and do not receive an optimal learning experience.

[0456] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0457] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; data processing means for analyzing learning data collected from the user terminal in real time and evaluating individual learning progress; means for providing user-adapted feedback using a generative model based on the evaluation results; and recommendation means for dynamically recommending the next learning module based on the user's learning history and current progress. This enables learners to obtain an efficient learning experience tailored to their individual learning needs.

[0458] A "user terminal" is an electronic device used by learners that is equipped with hardware and software for displaying and interacting with content.

[0459] "Multimodal content" refers to learning materials that include multiple information formats such as text, audio, images, and videos, and that provide information to learners through various senses.

[0460] "Data processing means" refers to a set of systems and algorithms for analyzing collected training data and evaluating learning progress.

[0461] A "generative model" is a model that uses AI technology to generate specific outputs from data, and is used, for example, to generate feedback.

[0462] "Feedback" refers to information that indicates evaluations and areas for improvement regarding a user's learning, and is provided to enhance the effectiveness of learning.

[0463] "Recommended methods" is a system that intelligently suggests what to learn next based on past learning history and progress.

[0464] A "learning module" is a collection of learning content designed to teach specific knowledge or skills.

[0465] Embodiments of this invention will be described below.

[0466] This learning support system aims to enable users to learn efficiently through multimodal content that integrates various information formats. At the heart of the system are a server and a user terminal, which utilize an analysis agent incorporating a generative AI model.

[0467] The server centrally manages various forms of content for learners, including multimedia data such as text, audio, images, and video. Users can access specific learning modules through their devices and receive relevant content from the server, which is delivered via download or streaming.

[0468] The user's device utilizes the received content to display learning material using AR (Augmented Reality). For example, a 3D model of a planet is displayed on the screen and presented in an interactive state, allowing learners to observe the model from various angles. Users can interact with the content through their device, obtaining detailed information, playing narration, and manipulating the model.

[0469] Furthermore, as the learning process progresses, user operation history and response data are collected and immediately sent to the server. An analysis agent on the server processes the data using a generative AI model to analyze the user's learning patterns. The analysis is performed in real time, and the user's level of understanding is evaluated based on the results. The evaluation is provided as direct feedback, suggesting areas for improvement and further learning.

[0470] For example, when a learner is studying the solar system, the device displays 3D models of the planets, which the user can grab and rotate, or tap information icons to view detailed information about specific planets. Furthermore, they can deepen their knowledge through explanations and quizzes provided by the server, and explanations are immediately provided as feedback for questions they answered incorrectly.

[0471] Examples of prompts for a generative AI model include, "I want to learn about the solar system. Please tell me about the characteristics of the planets in detail." This prompt allows the system to immediately provide appropriate learning content, creating an interesting and engaging learning experience.

[0472] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0473] Step 1:

[0474] The user operates the device and selects a specific learning module. This causes the device to send a request to the server to retrieve the required multimodal content. The input is the user's selection, and the output is the content data received from the server. The device receives this data and prepares it for playback via download or streaming.

[0475] Step 2:

[0476] The device displays the received content using an AR application. Here, the content is overlaid on real-world footage. The input is content data from a server, and the output is an interactive screen display that the user can visually confirm. The user can perform actions such as displaying detailed information or playing narration by tapping or swiping the screen.

[0477] Step 3:

[0478] During learning, the user's operation history and response data are recorded on the device. The device sends this data to the server. The input is the user's operation history and responses, and the output is the log data transferred to the server. The data is compressed and encrypted before transmission to ensure security.

[0479] Step 4:

[0480] The server analyzes the received user data. A generative AI model is used here, and the data is the subject of evaluation. The input is user log data, and the output is an evaluation result regarding the user's learning progress and understanding. Machine learning algorithms are used for the analysis, and the accuracy rate and trends in operations are analyzed.

[0481] Step 5:

[0482] Based on the analysis results, the server generates feedback tailored to the user's learning progress. This is done using a generative AI model, which suggests optimal improvements and the next learning content for the learner. The input is the server's evaluation results, and the output is feedback messages and suggested learning modules. The feedback is sent to the user's device in real time.

[0483] Step 6:

[0484] The user initiates a new learning process based on the feedback provided. The device then retrieves the necessary content from the server again for this new learning session, and the learning progresses cyclically. The input is the feedback from the server, and the output is the new learning session initiated by the user. This allows the learner to move on to the next step, and the learning continues.

[0485] (Application Example 1)

[0486] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0487] Conventional learning support systems have struggled to provide feedback tailored to learners' progress and offer optimal learning experiences based on their learning needs, especially in educational methods using digital information. Furthermore, they lacked sufficient integration of augmented reality technology to fully leverage the visual learning effects of integration with the real environment. Therefore, flexible responses tailored to individual learning styles are required.

[0488] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0489] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; means for analyzing learning data collected from the user terminal in real time and evaluating learning progress; means for providing feedback to the learner based on the evaluation results; means for appropriately recommending the next stage of learning content; means for displaying a three-dimensional model using augmented reality technology; means for performing analysis using machine learning algorithms and providing an optimal learning experience; and means for generating advice based on learning needs using a generative AI model. This enables the integration of diverse information formats and the provision of an optimal learning experience tailored to individual learning styles.

[0490] A "user terminal" is an information processing device used by learners, enabling them to download, display, and manipulate content.

[0491] "Learning content" refers to educational materials that include digital information such as text, audio, images, and videos, intended for educational purposes.

[0492] Augmented reality technology is a technology that overlays digital information onto the real environment to display it visually.

[0493] A "machine learning algorithm" is a mathematical method used to learn patterns from data and make predictions and decisions.

[0494] A "generative AI model" is a type of artificial intelligence technology that generates language and visual information based on input data.

[0495] "Feedback" refers to information and advice provided based on a learner's progress and understanding, with the aim of improving their learning.

[0496] A "three-dimensional model" is a digital representation of a three-dimensional object that can be displayed and manipulated on a computer.

[0497] "Real-time analysis" is an information processing technology that processes input data instantly and outputs the results immediately.

[0498] This system utilizes user terminals, servers, and generative AI models to provide learners with an interactive learning experience.

[0499] The server stores multimodal learning content in a database and streams the content to the user's device upon request. For example, the history learning content stored on the server includes text information, audio guides, and 3D models. This data is displayed on the user's device using augmented reality technology, allowing the user to visually confirm digital information overlaid on reality. The hardware used is a smart device, and the software used is ARKit (assuming iOS).

[0500] Meanwhile, the user terminal performs real-time analysis based on digital information received from the server. An application implementing machine learning algorithms runs, tracking the user's actions and learning history. This allows for the evaluation of individual progress and learning styles, and the server analyzes the results to generate appropriate feedback.

[0501] The generative AI model generates prompts and next learning steps based on the user's learning history. This model can provide learners with specific advice and tasks, improving learning efficiency. An example of a specific prompt is: "Generate interactive AR experience content for learning about Egyptian civilization. Create learning content that includes 3D models, audio guides, and quizzes."

[0502] This system allows users to learn more deeply not just by receiving information, but through hands-on experience. For example, when learning geography, users can interactively learn about the characteristics of each continent while viewing a three-dimensional model of the Earth through smart glasses. This effectively stimulates learners' understanding and interest.

[0503] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0504] Step 1:

[0505] The user selects a learning module using their device. The user opens the application on their device and chooses a module from the various learning content provided that suits their interests and learning needs. Once this selection is complete, the device sends a corresponding request to the server. The input is the user's selection, and the output is the request sent to the server.

[0506] Step 2:

[0507] The server receives requests from the user's terminal and retrieves multimodal content related to the selected learning module from the database. The retrieved data includes text, audio, images, and 3D models. The server streams this data to the user's terminal. The input is the request from the terminal, and the output is a stream of content data.

[0508] Step 3:

[0509] The user terminal displays multimodal content received from the server. Using an augmented reality application, digital information is overlaid onto the real environment, displaying three-dimensional models. The user interacts with these models using the terminal's touchscreen and eye-tracking controls. Input is content data from the server, and output is the augmented reality display.

[0510] Step 4:

[0511] While a user interacts with content, the device collects interaction data in real time. This includes taps, swipes, and voice commands. This data is recorded as an interaction history and used to track the user's learning progress. The input is the user's interaction data, and the output is the interaction history record.

[0512] Step 5:

[0513] The terminal sends collected user operation data to the server. The data arriving at the server is analyzed in real time by machine learning algorithms. This analysis evaluates the user's learning patterns and level of understanding. The input is the transmission of operation data, and the output is the analysis results.

[0514] Step 6:

[0515] Based on the analysis results, the server uses a generative AI model to generate optimal feedback and next learning steps for the user. The generated feedback and prompts are sent to the user's terminal. Specifically, they indicate areas where learning needs improvement and suggest new tasks or challenges. The input is the analysis results, and the output is the feedback and prompts.

[0516] Step 7:

[0517] The user terminal displays feedback received from the server and suggestions for the next learning step. Based on this, the user can proceed with their learning or select new content. The input is the feedback from the server, and the output is the next learning action.

[0518] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0519] This invention relates to a multimodal learning support system that combines an emotion engine. Specific embodiments thereof are described below.

[0520] 1. Content delivery and sentiment recognition:

[0521] The server prepares the necessary learning content based on user requests and sends it to the user's device. The device has an AR application for learning installed and is equipped with an emotion engine that senses the user's facial expressions and voice through the camera and microphone. This allows the user's emotional state during learning to be recognized in real time.

[0522] 2. Emotion-based interactive experiences:

[0523] When a user accesses learning content on their device, the emotion engine analyzes the user's emotions at that moment and sends the results to the server. The server uses this emotion data to determine the user's stress level and concentration level. For example, if the user shows a confused expression, the server can use that information to suggest content with a lower difficulty level.

[0524] 3. Learning assessment and feedback adjustment:

[0525] The server's AI analyzes learning data and emotional information obtained from the user's device. Based on this analysis, the server generates feedback, adjusting the content and method of the feedback according to the user's emotional state. For example, if the user is fatigued, it will provide encouraging messages or notifications prompting them to take a break.

[0526] 4. Recommended next steps:

[0527] The server suggests the optimal next learning content based on real-time sentiment data and learning history. This suggestion is intended to maximize learning efficiency while maintaining the user's curiosity.

[0528] For example, if the emotion engine detects signs of a user's concentration waning while they are learning a history module, the server will insert a visually engaging video and rearrange the content to keep them interested. This helps maintain the user's motivation while promoting a deeper understanding.

[0529] Thus, a learning support system that incorporates an emotional engine provides personalized support to individual learners, creating an efficient and effective learning environment.

[0530] The following describes the processing flow.

[0531] Step 1:

[0532] The server prepares the necessary content for the learning module selected by the user and sends it to the user's device. This includes multiple information formats such as animations, audio guides, and text.

[0533] Step 2:

[0534] The device activates an emotion engine to capture the user's facial expressions through the camera and collect audio through the microphone. This data is used for real-time emotion analysis.

[0535] Step 3:

[0536] Users use their devices to view learning content and engage in interactive activities, such as answering on-screen quizzes or manipulating 3D models.

[0537] Step 4:

[0538] The device's emotion engine analyzes the user's facial expressions and voice data to determine their emotional state and sends that data to the server.

[0539] Step 5:

[0540] The server integrates user emotion data and learning progress data and performs AI analysis. If the user is experiencing stress, the server uses this data to adjust the difficulty level of the learning content or provide additional hints.

[0541] Step 6:

[0542] The server generates feedback corresponding to the user's emotional state and sends it to the user's device. For example, if the user is feeling anxious, it displays a relaxing message such as "calm down."

[0543] Step 7:

[0544] The server recommends the next learning content based on the user's emotions and learning history, and provides this information to the device. The user can review the recommendations and proceed to the next learning module.

[0545] (Example 2)

[0546] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0547] Current digital learning systems are insufficient in providing personalized learning experiences that take into account learners' emotional states. As a result, many learners lose interest in the material or experience decreased learning efficiency. Furthermore, dynamically adjusting the learning process using real-time emotion recognition is difficult. In this context, there is a need to provide advanced learning environments that maintain learners' interest and concentration and meet their individual needs.

[0548] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0549] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal, means for analyzing learning data and emotional data collected from the user terminal in real time and evaluating learning progress and emotional state, and means for providing feedback to the learner and adjusting the difficulty level of the content based on the evaluation results and emotional state. This enables personalized learning support that takes the learner's emotional state into consideration.

[0550] A "user terminal" is an electronic device used by learners, and includes hardware and software for displaying learning content and collecting sentiment data.

[0551] "Learning content" refers to a collection of digital information provided for educational purposes, including multiple information formats such as text, images, audio, and video.

[0552] "Emotional data" refers to information that represents the user's emotional state, and is analyzed based on data such as facial expressions and voice acquired through sensors on the user's device.

[0553] "Real-time analysis" is a technology that processes data instantly to obtain results, enabling the immediate adaptation of learning content to users' emotional responses.

[0554] "Feedback" refers to information and instructions provided based on a learner's learning progress and emotional state, and includes adjustments and advice to support effective learning.

[0555] "Difficulty level adjustment" is a process that dynamically changes the complexity and challenge level of learning content according to the learner's abilities and emotional state.

[0556] "Learning efficiency" is an indicator that shows how effectively learners use their time and effort to achieve their goals.

[0557] "Maintaining interest" refers to a state in which learners continuously engage in learning activities and maintain their concentration without interruption.

[0558] This invention relates to a learning support system for individually optimizing a user's learning experience. This system is implemented using a user terminal, a server, and an emotion engine. The specific implementation method is described below.

[0559] The server manages digital information and prepares the necessary information based on user requests for learning content. This content is provided in formats such as text, images, audio, and video and delivered to the user's terminal. The user's terminal displays this content through a specific application. The terminal has an integrated emotion engine that uses a camera and microphone to sense the user's facial expressions and voice in real time and acquire emotion data.

[0560] Emotional data acquired by the device's emotion engine is immediately sent to the server. The server analyzes this emotional data using a generative AI model to evaluate the user's emotional state. For example, if the server determines that the user is confused, it adjusts the learning content based on this information, such as automatically lowering the difficulty level of the content or inserting additional explanations. The content and format of the feedback are also adaptively adjusted based on the user's emotions and learning progress.

[0561] For example, when a user is using a history learning module, a decrease in concentration may be detected by the emotion engine. In this case, the server can support maintaining motivation and deeper understanding by including visually engaging videos or rearranging the content to capture the user's interest.

[0562] An example of a prompt might be a request such as, "Suggest the next best learning content based on the user's emotional state." Based on this prompt, the server can use a generated AI model to recommend the most suitable learning content for the user.

[0563] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0564] Step 1:

[0565] The server receives a request from the user, selects the relevant learning content from the database, and prepares it. The input is the user's selected learning theme and content request, and the output is the learning content data to be sent to the user's terminal. Specifically, the server processes the user's request, generates the URL of the corresponding content, and sends it to the user's terminal.

[0566] Step 2:

[0567] The device receives learning content sent from the server and displays it to the user. The input is content data from the server, and the output is the display of content to the user. Specifically, the learning application on the device renders the content and provides a screen that the user can view or interact with.

[0568] Step 3:

[0569] The device uses a camera and microphone to sense the user's facial expressions and voice, collecting emotional data in real time. The input is visual and audio data from the camera and microphone, and the output is analyzed emotional state data. Specifically, the emotion engine on the device utilizes facial recognition technology and voice analysis technology to quantify the user's emotions and generate that data.

[0570] Step 4:

[0571] The server receives emotional data transmitted from the terminal and analyzes it using a generative AI model. The input is emotional state data from the terminal, and the output is the user's emotional evaluation data. Specifically, the AI ​​model on the server processes the emotional data, calculates the user's stress level and concentration level, and saves the results.

[0572] Step 5:

[0573] The server generates appropriate feedback for the user based on sentiment evaluation data and adjusts the difficulty level of the content. Inputs are sentiment evaluation data and learning progress data, while outputs are the adjusted feedback and content. Specifically, the server generates encouraging messages and additional explanations of the learning content according to the learner's state and sends them to the user's terminal.

[0574] Step 6:

[0575] The server suggests the next best learning content to the user based on the feedback received. The input is the user's learning history, progress, and latest sentiment rating data, and the output is recommended new learning content. Specifically, the server searches the database, selects content that will keep the user interested while promoting efficient learning, and processes it according to instructions such as "Suggest the next best learning content based on the user's sentiment state."

[0576] (Application Example 2)

[0577] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0578] Traditional learning support systems have not taken into account feedback or content suggestions based on learners' emotional states, making it difficult to provide personalized learning experiences tailored to individual learners. Therefore, there is a need to provide methods that reduce the emotional burden on learners and improve learning efficiency.

[0579] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0580] In this invention, the server includes functions for providing learning information containing multiple information formats to a user terminal, functions for analyzing the learning information and emotional information acquired from the user terminal in real time and evaluating learning progress and emotional state, and functions for providing feedback according to the learner's emotional state based on the evaluation results. This makes it possible to provide a personalized learning experience based on the learner's emotional state and maximize learning efficiency.

[0581] "Multimodal digital information" refers to information expressed in multiple formats, such as text, audio, and images.

[0582] A "learning support system" refers to a system that provides technical support to help learners progress effectively in their studies.

[0583] A "user terminal" refers to an electronic device or equipment that individual users operate and use to receive and transmit information.

[0584] "Learning information" refers to the knowledge, materials, and data that learners need in the learning process.

[0585] "Emotional information" refers to data that represents the user's emotional state, and this includes things like facial expressions and tone of voice.

[0586] "Real-time analysis" refers to a processing method that allows data to be processed instantly and results to be obtained immediately.

[0587] "Learning progress" is an indicator that shows how far a learner has progressed in their studies.

[0588] "Feedback" refers to the responses and information provided by a system to a learner, which helps improve their learning.

[0589] A "personalized learning experience" refers to a learning process that is customized according to the individual learner's needs and circumstances.

[0590] "Learning efficiency" refers to the efficiency with which learners use their time and effort to acquire the knowledge and skills they aim for.

[0591] To implement this invention, a user terminal and a server are required. The user terminal can be a portable electronic device such as a smartphone or tablet. The user accesses learning information through an application installed on the terminal and uses the terminal's camera and microphone to acquire facial expressions and voice as emotional information. This emotional information is preprocessed within the terminal and transmitted to the server.

[0592] The server analyzes the received learning and emotion data in real time. Machine learning frameworks such as TensorFlow can be used for this analysis. A generative AI model on the server processes the emotion data and recognizes the user's emotional state. For example, if the user displays a confused expression, the server recognizes this and immediately adjusts the difficulty level of the learning content.

[0593] Furthermore, feedback is created and presented to the user based on their emotional state and learning progress. This feedback may include encouraging messages or recommendations for breaks. If a learner is losing interest, visually engaging additional content is provided to maintain their motivation.

[0594] For example, when a user is practicing English with a language learning app, if the emotion engine detects confusion, the app can offer supplementary materials or provide interactive translation quizzes.

[0595] An example of a prompt sentence to input into a generative AI model is, "When the user's facial expression indicates confusion, suggest what kind of visual content would help with learning."

[0596] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0597] Step 1:

[0598] The user terminal provides learning information to the user. Here, the terminal displays multiple information formats, such as text and images, through an application. The input is learning information from the server, and the output is content provided through the user's sight and hearing.

[0599] Step 2:

[0600] The device uses a camera and microphone to sense the user's facial expressions and voice in real time. The input is data of the user's face and voice, which is acquired as emotional information. The output is emotional data that has been preprocessed for analysis.

[0601] Step 3:

[0602] The device sends the acquired emotion data to the server. Here, the device uses a network protocol to securely transmit the data. The input is pre-processed emotion data, and the output is the data sent to the server.

[0603] Step 4:

[0604] The server analyzes the received emotion data. Here, the server uses a generative AI model such as TensorFlow to estimate the user's emotional state. The input is the emotion data received from the device, and the output is the recognized emotional state.

[0605] Step 5:

[0606] The server adjusts the difficulty level of the learning content based on the analysis results. If an emotion of confusion is detected, the server will select lower difficulty content or provide supplementary materials. The input is the recognized emotional state, and the output is the adjusted learning content.

[0607] Step 6:

[0608] The server generates feedback and sends it to the user's terminal. Here, the server considers the user's emotional state and creates feedback that includes encouraging messages and suggestions for rest. The input is the recognized emotional state and preceding learning progress, and the output is the feedback sent to the user's terminal.

[0609] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0610] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0611] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0612] [Fourth Embodiment]

[0613] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0614] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0615] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0616] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0617] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0618] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0619] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0620] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0621] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0622] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0623] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0624] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0625] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0626] As an embodiment of the present invention, a system for realizing a multimodal learning experience received by learners using a terminal will be described.

[0627] 1. Content distribution and display:

[0628] The server stores multimodal content, such as text, audio, images, and videos, required for each learning module. When a user's device starts a learning session, it accesses the server and retrieves content related to the selected learning module via download and streaming. The device then integrates this content and displays it on the screen via an AR application.

[0629] 2. Interactive learning experience:

[0630] Users manipulate visual information that combines the real environment with digital content using their devices. For example, when a user taps the screen, detailed information may be displayed or audio narration may be played. They can also perform operations such as rotating and scaling 3D models.

[0631] 3. Learning assessment and feedback:

[0632] The user's device sends its operation history and learning responses to the server. The server's AI agent analyzes this data in real time. Based on the analysis results, the server evaluates the user's progress, generates feedback to improve weaknesses, and sends it to the user's device. Specifically, it provides explanations for questions the learner answered incorrectly and suggests new practice problems.

[0633] 4. Suggested next learning steps:

[0634] The server suggests the next learning topic based on real-time evaluation results and past learning history. Users receive these suggestions through their terminal and can proceed to new learning modules. This enables flexible learning tailored to individual learning needs.

[0635] This system allows learners to access a learning curriculum tailored to their needs, enabling efficient understanding and skill development. For example, a user can select a module to learn about the solar system, access 3D models of the planets, and deepen their understanding through detailed explanations and quizzes.

[0636] The following describes the processing flow.

[0637] Step 1:

[0638] The server receives a user request and then prepares multimodal content corresponding to the requested learning module. It then organizes the data to provide it in the optimal format and size for delivery to the user's device.

[0639] Step 2:

[0640] The device receives learning content streamed from the server and launches an AR application on the device. By displaying the received content on the device's screen and overlaying it onto the real environment, an interactive user interface is realized.

[0641] Step 3:

[0642] Users interact with and manipulate the displayed content through their devices. For example, tapping an object on the screen can trigger additional relevant information or audio narration. They can also manipulate 3D models (rotate, zoom, etc.).

[0643] Step 4:

[0644] The device collects user operation history and quiz answer data in real time and sends it to the server. This data reflects the user's interaction flow and level of understanding.

[0645] Step 5:

[0646] The AI ​​agent on the server analyzes the data received from the terminal and evaluates the learner's progress and understanding. Based on this, it identifies the learner's weaknesses and strengths and generates necessary feedback.

[0647] Step 6:

[0648] The server generates feedback and sends it to the user's device. This feedback includes information on areas where understanding is lacking and what additional information would be helpful. Additional explanations and practice problems may also be provided.

[0649] Step 7:

[0650] The server references the learner's progress and past learning records, selects the next learning content, and provides that information to the terminal. Based on this, the user can start a new learning module.

[0651] (Example 1)

[0652] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0653] Traditional learning systems have faced challenges in enabling learners to engage in effective learning tailored to their individual needs. In particular, they struggle to analyze learning progress in real time, provide adaptive feedback, and dynamically recommend learning materials based on past history. As a result, learners are unable to effectively improve their weaknesses and do not receive an optimal learning experience.

[0654] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0655] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; data processing means for analyzing learning data collected from the user terminal in real time and evaluating individual learning progress; means for providing user-adapted feedback using a generative model based on the evaluation results; and recommendation means for dynamically recommending the next learning module based on the user's learning history and current progress. This enables learners to obtain an efficient learning experience tailored to their individual learning needs.

[0656] A "user terminal" is an electronic device used by learners that is equipped with hardware and software for displaying and interacting with content.

[0657] "Multimodal content" refers to learning materials that include multiple information formats such as text, audio, images, and videos, and that provide information to learners through various senses.

[0658] "Data processing means" refers to a set of systems and algorithms for analyzing collected training data and evaluating learning progress.

[0659] A "generative model" is a model that uses AI technology to generate specific outputs from data, and is used, for example, to generate feedback.

[0660] "Feedback" refers to information that indicates evaluations and areas for improvement regarding a user's learning, and is provided to enhance the effectiveness of learning.

[0661] "Recommended methods" is a system that intelligently suggests what to learn next based on past learning history and progress.

[0662] A "learning module" is a collection of learning content designed to teach specific knowledge or skills.

[0663] Embodiments of this invention will be described below.

[0664] This learning support system aims to enable users to learn efficiently through multimodal content that integrates various information formats. At the heart of the system are a server and a user terminal, which utilize an analysis agent incorporating a generative AI model.

[0665] The server centrally manages various forms of content for learners, including multimedia data such as text, audio, images, and video. Users can access specific learning modules through their devices and receive relevant content from the server, which is delivered via download or streaming.

[0666] The user's device utilizes the received content to display learning material using AR (Augmented Reality). For example, a 3D model of a planet is displayed on the screen and presented in an interactive state, allowing learners to observe the model from various angles. Users can interact with the content through their device, obtaining detailed information, playing narration, and manipulating the model.

[0667] Furthermore, as the learning process progresses, user operation history and response data are collected and immediately sent to the server. An analysis agent on the server processes the data using a generative AI model to analyze the user's learning patterns. The analysis is performed in real time, and the user's level of understanding is evaluated based on the results. The evaluation is provided as direct feedback, suggesting areas for improvement and further learning.

[0668] For example, when a learner is studying the solar system, the device displays 3D models of the planets, which the user can grab and rotate, or tap information icons to view detailed information about specific planets. Furthermore, they can deepen their knowledge through explanations and quizzes provided by the server, and explanations are immediately provided as feedback for questions they answered incorrectly.

[0669] Examples of prompts for a generative AI model include, "I want to learn about the solar system. Please tell me about the characteristics of the planets in detail." This prompt allows the system to immediately provide appropriate learning content, creating an interesting and engaging learning experience.

[0670] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0671] Step 1:

[0672] The user operates the device and selects a specific learning module. This causes the device to send a request to the server to retrieve the required multimodal content. The input is the user's selection, and the output is the content data received from the server. The device receives this data and prepares it for playback via download or streaming.

[0673] Step 2:

[0674] The device displays the received content using an AR application. Here, the content is overlaid on real-world footage. The input is content data from a server, and the output is an interactive screen display that the user can visually confirm. The user can perform actions such as displaying detailed information or playing narration by tapping or swiping the screen.

[0675] Step 3:

[0676] During learning, the user's operation history and response data are recorded on the device. The device sends this data to the server. The input is the user's operation history and responses, and the output is the log data transferred to the server. The data is compressed and encrypted before transmission to ensure security.

[0677] Step 4:

[0678] The server analyzes the received user data. A generative AI model is used here, and the data is the subject of evaluation. The input is user log data, and the output is an evaluation result regarding the user's learning progress and understanding. Machine learning algorithms are used for the analysis, and the accuracy rate and trends in operations are analyzed.

[0679] Step 5:

[0680] Based on the analysis results, the server generates feedback tailored to the user's learning progress. This is done using a generative AI model, which suggests optimal improvements and the next learning content for the learner. The input is the server's evaluation results, and the output is feedback messages and suggested learning modules. The feedback is sent to the user's device in real time.

[0681] Step 6:

[0682] The user initiates a new learning process based on the feedback provided. The device then retrieves the necessary content from the server again for this new learning session, and the learning progresses cyclically. The input is the feedback from the server, and the output is the new learning session initiated by the user. This allows the learner to move on to the next step, and the learning continues.

[0683] (Application Example 1)

[0684] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] Conventional learning support systems have struggled to provide feedback tailored to learners' progress and offer optimal learning experiences based on their learning needs, especially in educational methods using digital information. Furthermore, they lacked sufficient integration of augmented reality technology to fully leverage the visual learning effects of integration with the real environment. Therefore, flexible responses tailored to individual learning styles are required.

[0686] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0687] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal; means for analyzing learning data collected from the user terminal in real time and evaluating learning progress; means for providing feedback to the learner based on the evaluation results; means for appropriately recommending the next stage of learning content; means for displaying a three-dimensional model using augmented reality technology; means for performing analysis using machine learning algorithms and providing an optimal learning experience; and means for generating advice based on learning needs using a generative AI model. This enables the integration of diverse information formats and the provision of an optimal learning experience tailored to individual learning styles.

[0688] A "user terminal" is an information processing device used by learners, enabling them to download, display, and manipulate content.

[0689] "Learning content" refers to educational materials that include digital information such as text, audio, images, and videos, intended for educational purposes.

[0690] Augmented reality technology is a technology that overlays digital information onto the real environment to display it visually.

[0691] A "machine learning algorithm" is a mathematical method used to learn patterns from data and make predictions and decisions.

[0692] A "generative AI model" is a type of artificial intelligence technology that generates language and visual information based on input data.

[0693] "Feedback" refers to information and advice provided based on a learner's progress and understanding, with the aim of improving their learning.

[0694] A "three-dimensional model" is a digital representation of a three-dimensional object that can be displayed and manipulated on a computer.

[0695] "Real-time analysis" is an information processing technology that processes input data instantly and outputs the results immediately.

[0696] This system utilizes user terminals, servers, and generative AI models to provide learners with an interactive learning experience.

[0697] The server stores multimodal learning content in a database and streams the content to the user's device upon request. For example, the history learning content stored on the server includes text information, audio guides, and 3D models. This data is displayed on the user's device using augmented reality technology, allowing the user to visually confirm digital information overlaid on reality. The hardware used is a smart device, and the software used is ARKit (assuming iOS).

[0698] Meanwhile, the user terminal performs real-time analysis based on digital information received from the server. An application implementing machine learning algorithms runs, tracking the user's actions and learning history. This allows for the evaluation of individual progress and learning styles, and the server analyzes the results to generate appropriate feedback.

[0699] The generative AI model generates prompts and next learning steps based on the user's learning history. This model can provide learners with specific advice and tasks, improving learning efficiency. An example of a specific prompt is: "Generate interactive AR experience content for learning about Egyptian civilization. Create learning content that includes 3D models, audio guides, and quizzes."

[0700] This system allows users to learn more deeply not just by receiving information, but through hands-on experience. For example, when learning geography, users can interactively learn about the characteristics of each continent while viewing a three-dimensional model of the Earth through smart glasses. This effectively stimulates learners' understanding and interest.

[0701] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0702] Step 1:

[0703] The user selects a learning module using their device. The user opens the application on their device and chooses a module from the various learning content provided that suits their interests and learning needs. Once this selection is complete, the device sends a corresponding request to the server. The input is the user's selection, and the output is the request sent to the server.

[0704] Step 2:

[0705] The server receives requests from the user's terminal and retrieves multimodal content related to the selected learning module from the database. The retrieved data includes text, audio, images, and 3D models. The server streams this data to the user's terminal. The input is the request from the terminal, and the output is a stream of content data.

[0706] Step 3:

[0707] The user terminal displays multimodal content received from the server. Using an augmented reality application, digital information is overlaid onto the real environment, displaying three-dimensional models. The user interacts with these models using the terminal's touchscreen and eye-tracking controls. Input is content data from the server, and output is the augmented reality display.

[0708] Step 4:

[0709] While a user interacts with content, the device collects interaction data in real time. This includes taps, swipes, and voice commands. This data is recorded as an interaction history and used to track the user's learning progress. The input is the user's interaction data, and the output is the interaction history record.

[0710] Step 5:

[0711] The terminal sends collected user operation data to the server. The data arriving at the server is analyzed in real time by machine learning algorithms. This analysis evaluates the user's learning patterns and level of understanding. The input is the transmission of operation data, and the output is the analysis results.

[0712] Step 6:

[0713] Based on the analysis results, the server uses a generative AI model to generate optimal feedback and next learning steps for the user. The generated feedback and prompts are sent to the user's terminal. Specifically, they indicate areas where learning needs improvement and suggest new tasks or challenges. The input is the analysis results, and the output is the feedback and prompts.

[0714] Step 7:

[0715] The user terminal displays feedback received from the server and suggestions for the next learning step. Based on this, the user can proceed with their learning or select new content. The input is the feedback from the server, and the output is the next learning action.

[0716] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0717] This invention relates to a multimodal learning support system that combines an emotion engine. Specific embodiments thereof are described below.

[0718] 1. Content delivery and sentiment recognition:

[0719] The server prepares the necessary learning content based on user requests and sends it to the user's device. The device has an AR application for learning installed and is equipped with an emotion engine that senses the user's facial expressions and voice through the camera and microphone. This allows the user's emotional state during learning to be recognized in real time.

[0720] 2. Emotion-based interactive experiences:

[0721] When a user accesses learning content on their device, the emotion engine analyzes the user's emotions at that moment and sends the results to the server. The server uses this emotion data to determine the user's stress level and concentration level. For example, if the user shows a confused expression, the server can use that information to suggest content with a lower difficulty level.

[0722] 3. Learning assessment and feedback adjustment:

[0723] The server's AI analyzes learning data and emotional information obtained from the user's device. Based on this analysis, the server generates feedback, adjusting the content and method of the feedback according to the user's emotional state. For example, if the user is fatigued, it will provide encouraging messages or notifications prompting them to take a break.

[0724] 4. Recommended next steps:

[0725] The server suggests the optimal next learning content based on real-time sentiment data and learning history. This suggestion is intended to maximize learning efficiency while maintaining the user's curiosity.

[0726] For example, if the emotion engine detects signs of a user's concentration waning while they are learning a history module, the server will insert a visually engaging video and rearrange the content to keep them interested. This helps maintain the user's motivation while promoting a deeper understanding.

[0727] Thus, a learning support system that incorporates an emotional engine provides personalized support to individual learners, creating an efficient and effective learning environment.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] The server prepares the necessary content for the learning module selected by the user and sends it to the user's device. This includes multiple information formats such as animations, audio guides, and text.

[0731] Step 2:

[0732] The device activates an emotion engine to capture the user's facial expressions through the camera and collect audio through the microphone. This data is used for real-time emotion analysis.

[0733] Step 3:

[0734] Users use their devices to view learning content and engage in interactive activities, such as answering on-screen quizzes or manipulating 3D models.

[0735] Step 4:

[0736] The device's emotion engine analyzes the user's facial expressions and voice data to determine their emotional state and sends that data to the server.

[0737] Step 5:

[0738] The server integrates user emotion data and learning progress data and performs AI analysis. If the user is experiencing stress, the server uses this data to adjust the difficulty level of the learning content or provide additional hints.

[0739] Step 6:

[0740] The server generates feedback corresponding to the user's emotional state and sends it to the user's device. For example, if the user is feeling anxious, it displays a relaxing message such as "calm down."

[0741] Step 7:

[0742] The server recommends the next learning content based on the user's emotions and learning history, and provides this information to the device. The user can review the recommendations and proceed to the next learning module.

[0743] (Example 2)

[0744] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0745] Current digital learning systems are insufficient in providing personalized learning experiences that take into account learners' emotional states. As a result, many learners lose interest in the material or experience decreased learning efficiency. Furthermore, dynamically adjusting the learning process using real-time emotion recognition is difficult. In this context, there is a need to provide advanced learning environments that maintain learners' interest and concentration and meet their individual needs.

[0746] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0747] In this invention, the server includes means for providing learning content containing multiple information formats to a user terminal, means for analyzing learning data and emotional data collected from the user terminal in real time and evaluating learning progress and emotional state, and means for providing feedback to the learner and adjusting the difficulty level of the content based on the evaluation results and emotional state. This enables personalized learning support that takes the learner's emotional state into consideration.

[0748] A "user terminal" is an electronic device used by learners, and includes hardware and software for displaying learning content and collecting sentiment data.

[0749] "Learning content" refers to a collection of digital information provided for educational purposes, including multiple information formats such as text, images, audio, and video.

[0750] "Emotional data" refers to information that represents the user's emotional state, and is analyzed based on data such as facial expressions and voice acquired through sensors on the user's device.

[0751] "Real-time analysis" is a technology that processes data instantly to obtain results, enabling the immediate adaptation of learning content to users' emotional responses.

[0752] "Feedback" refers to information and instructions provided based on a learner's learning progress and emotional state, and includes adjustments and advice to support effective learning.

[0753] "Difficulty level adjustment" is a process that dynamically changes the complexity and challenge level of learning content according to the learner's abilities and emotional state.

[0754] "Learning efficiency" is an indicator that shows how effectively learners use their time and effort to achieve their goals.

[0755] "Maintaining interest" refers to a state in which learners continuously engage in learning activities and maintain their concentration without interruption.

[0756] This invention relates to a learning support system for individually optimizing a user's learning experience. This system is implemented using a user terminal, a server, and an emotion engine. The specific implementation method is described below.

[0757] The server manages digital information and prepares the necessary information based on user requests for learning content. This content is provided in formats such as text, images, audio, and video and delivered to the user's terminal. The user's terminal displays this content through a specific application. The terminal has an integrated emotion engine that uses a camera and microphone to sense the user's facial expressions and voice in real time and acquire emotion data.

[0758] Emotional data acquired by the device's emotion engine is immediately sent to the server. The server analyzes this emotional data using a generative AI model to evaluate the user's emotional state. For example, if the server determines that the user is confused, it adjusts the learning content based on this information, such as automatically lowering the difficulty level of the content or inserting additional explanations. The content and format of the feedback are also adaptively adjusted based on the user's emotions and learning progress.

[0759] For example, when a user is using a history learning module, a decrease in concentration may be detected by the emotion engine. In this case, the server can support maintaining motivation and deeper understanding by including visually engaging videos or rearranging the content to capture the user's interest.

[0760] An example of a prompt might be a request such as, "Suggest the next best learning content based on the user's emotional state." Based on this prompt, the server can use a generated AI model to recommend the most suitable learning content for the user.

[0761] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0762] Step 1:

[0763] The server receives a request from the user, selects the relevant learning content from the database, and prepares it. The input is the user's selected learning theme and content request, and the output is the learning content data to be sent to the user's terminal. Specifically, the server processes the user's request, generates the URL of the corresponding content, and sends it to the user's terminal.

[0764] Step 2:

[0765] The device receives learning content sent from the server and displays it to the user. The input is content data from the server, and the output is the display of content to the user. Specifically, the learning application on the device renders the content and provides a screen that the user can view or interact with.

[0766] Step 3:

[0767] The device uses a camera and microphone to sense the user's facial expressions and voice, collecting emotional data in real time. The input is visual and audio data from the camera and microphone, and the output is analyzed emotional state data. Specifically, the emotion engine on the device utilizes facial recognition technology and voice analysis technology to quantify the user's emotions and generate that data.

[0768] Step 4:

[0769] The server receives emotional data transmitted from the terminal and analyzes it using a generative AI model. The input is emotional state data from the terminal, and the output is the user's emotional evaluation data. Specifically, the AI ​​model on the server processes the emotional data, calculates the user's stress level and concentration level, and saves the results.

[0770] Step 5:

[0771] The server generates appropriate feedback for the user based on sentiment evaluation data and adjusts the difficulty level of the content. Inputs are sentiment evaluation data and learning progress data, while outputs are the adjusted feedback and content. Specifically, the server generates encouraging messages and additional explanations of the learning content according to the learner's state and sends them to the user's terminal.

[0772] Step 6:

[0773] The server suggests the next best learning content to the user based on the feedback received. The input is the user's learning history, progress, and latest sentiment rating data, and the output is recommended new learning content. Specifically, the server searches the database, selects content that will keep the user interested while promoting efficient learning, and processes it according to instructions such as "Suggest the next best learning content based on the user's sentiment state."

[0774] (Application Example 2)

[0775] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0776] Traditional learning support systems have not taken into account feedback or content suggestions based on learners' emotional states, making it difficult to provide personalized learning experiences tailored to individual learners. Therefore, there is a need to provide methods that reduce the emotional burden on learners and improve learning efficiency.

[0777] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0778] In this invention, the server includes functions for providing learning information containing multiple information formats to a user terminal, functions for analyzing the learning information and emotional information acquired from the user terminal in real time and evaluating learning progress and emotional state, and functions for providing feedback according to the learner's emotional state based on the evaluation results. This makes it possible to provide a personalized learning experience based on the learner's emotional state and maximize learning efficiency.

[0779] "Multimodal digital information" refers to information expressed in multiple formats, such as text, audio, and images.

[0780] A "learning support system" refers to a system that provides technical support to help learners progress effectively in their studies.

[0781] A "user terminal" refers to an electronic device or equipment that individual users operate and use to receive and transmit information.

[0782] "Learning information" refers to the knowledge, materials, and data that learners need in the learning process.

[0783] "Emotional information" refers to data that represents the user's emotional state, and this includes things like facial expressions and tone of voice.

[0784] "Real-time analysis" refers to a processing method that allows data to be processed instantly and results to be obtained immediately.

[0785] "Learning progress" is an indicator that shows how far a learner has progressed in their studies.

[0786] "Feedback" refers to the responses and information provided by a system to a learner, which helps improve their learning.

[0787] A "personalized learning experience" refers to a learning process that is customized according to the individual learner's needs and circumstances.

[0788] "Learning efficiency" refers to the efficiency with which learners use their time and effort to acquire the knowledge and skills they aim for.

[0789] To implement this invention, a user terminal and a server are required. The user terminal can be a portable electronic device such as a smartphone or tablet. The user accesses learning information through an application installed on the terminal and uses the terminal's camera and microphone to acquire facial expressions and voice as emotional information. This emotional information is preprocessed within the terminal and transmitted to the server.

[0790] The server analyzes the received learning and emotion data in real time. Machine learning frameworks such as TensorFlow can be used for this analysis. A generative AI model on the server processes the emotion data and recognizes the user's emotional state. For example, if the user displays a confused expression, the server recognizes this and immediately adjusts the difficulty level of the learning content.

[0791] Furthermore, feedback is created and presented to the user based on their emotional state and learning progress. This feedback may include encouraging messages or recommendations for breaks. If a learner is losing interest, visually engaging additional content is provided to maintain their motivation.

[0792] For example, when a user is practicing English with a language learning app, if the emotion engine detects confusion, the app can offer supplementary materials or provide interactive translation quizzes.

[0793] An example of a prompt sentence to input into a generative AI model is, "When the user's facial expression indicates confusion, suggest what kind of visual content would help with learning."

[0794] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0795] Step 1:

[0796] The user terminal provides learning information to the user. Here, the terminal displays multiple information formats, such as text and images, through an application. The input is learning information from the server, and the output is content provided through the user's sight and hearing.

[0797] Step 2:

[0798] The device uses a camera and microphone to sense the user's facial expressions and voice in real time. The input is data of the user's face and voice, which is acquired as emotional information. The output is emotional data that has been preprocessed for analysis.

[0799] Step 3:

[0800] The device sends the acquired emotion data to the server. Here, the device uses a network protocol to securely transmit the data. The input is pre-processed emotion data, and the output is the data sent to the server.

[0801] Step 4:

[0802] The server analyzes the received emotion data. Here, the server uses a generative AI model such as TensorFlow to estimate the user's emotional state. The input is the emotion data received from the device, and the output is the recognized emotional state.

[0803] Step 5:

[0804] The server adjusts the difficulty level of the learning content based on the analysis results. If an emotion of confusion is detected, the server will select lower difficulty content or provide supplementary materials. The input is the recognized emotional state, and the output is the adjusted learning content.

[0805] Step 6:

[0806] The server generates feedback and sends it to the user's terminal. Here, the server considers the user's emotional state and creates feedback that includes encouraging messages and suggestions for rest. The input is the recognized emotional state and preceding learning progress, and the output is the feedback sent to the user's terminal.

[0807] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0808] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0809] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0810] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0811] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0812] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0813] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0814] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0815] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0816] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0817] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0818] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0819] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0820] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0821] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0822] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0823] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0824] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0825] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0826] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0827] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0828] The following is further disclosed regarding the embodiments described above.

[0829] (Claim 1)

[0830] A learning support system that uses multimodal digital information,

[0831] A means for providing learning content containing multiple information formats to a user terminal,

[0832] A means for analyzing the learning data collected from the user terminal in real time and evaluating the learning progress,

[0833] A means of providing feedback to learners based on the evaluation results,

[0834] A means to appropriately recommend the next stage of learning content,

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The system according to claim 1, characterized in that the feedback identifies areas where the learner is lacking in learning and provides additional content.

[0838] (Claim 3)

[0839] The system according to claim 1, characterized in that the recommended next-stage learning content is dynamically selected based on the learner's past learning history and progress.

[0840] "Example 1"

[0841] (Claim 1)

[0842] A means for providing learning content containing multiple information formats to a user terminal,

[0843] A data processing means for analyzing learning data collected from the user terminal in real time and evaluating individual learning progress,

[0844] Based on the evaluation results, a means to provide user-adapted feedback using a generative model,

[0845] A recommendation mechanism for dynamically recommending the next learning module based on the user's learning history and current progress,

[0846] A system that includes this.

[0847] (Claim 2)

[0848] The system according to claim 1, characterized in that the feedback identifies areas where the learner needs improvement and provides more advanced learning resources.

[0849] (Claim 3)

[0850] The system according to claim 1, characterized in that the recommended next-stage learning module is intelligently selected through a generative model based on the user's past learning history and progress.

[0851] "Application Example 1"

[0852] (Claim 1)

[0853] A means for providing learning content containing multiple information formats to a user terminal,

[0854] A means for analyzing the learning data collected from the user terminal in real time and evaluating the learning progress,

[0855] A means of providing feedback to learners based on the evaluation results,

[0856] A means to appropriately recommend the next stage of learning content,

[0857] A means of displaying a three-dimensional model using augmented reality technology,

[0858] A means to perform analysis using machine learning algorithms and provide the optimal learning experience,

[0859] A means of generating advice based on learning needs using a generative AI model,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, characterized in that the feedback identifies areas where the learner is lacking in learning, provides additional content, and is visually displayed through augmented reality technology.

[0863] (Claim 3)

[0864] The system according to claim 1, characterized in that the recommended next-stage learning content is dynamically selected based on the learner's past learning history and progress, and a generative AI model is used to generate prompt sentences for the learner.

[0865] "Example 2 of combining an emotion engine"

[0866] (Claim 1)

[0867] A means of providing learning content containing multiple information formats to a user terminal,

[0868] A means for analyzing learning data and emotional data collected from the user terminal in real time and evaluating learning progress and emotional state,

[0869] A means of providing feedback to learners and adjusting the difficulty level of the content based on evaluation results and emotional state,

[0870] The aforementioned feedback includes means for dynamically adjusting the content and method according to the learner's emotional state, and providing encouraging messages or break notifications.

[0871] To appropriately recommend the next stage of learning content and to maintain learners' learning efficiency and interest,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, characterized in that the feedback identifies areas where the learner is lacking in learning and provides additional content.

[0875] (Claim 3)

[0876] The system according to claim 1, characterized in that the recommended next-stage learning content is dynamically selected based on the learner's past learning history, progress, and emotional state.

[0877] "Application example 2 when combining with an emotional engine"

[0878] (Claim 1)

[0879] A learning support system that uses multimodal digital information,

[0880] A function for providing learning information containing multiple information formats to the user terminal,

[0881] The system includes a function to analyze learning information and emotional information acquired from the user terminal in real time and to evaluate learning progress and emotional state,

[0882] Based on the evaluation results, a function to provide feedback according to the learner's emotional state,

[0883] Features to appropriately suggest the next learning stage,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, characterized in that the feedback identifies areas where the learner is lacking in learning and provides additional information based on the learner's emotional state.

[0887] (Claim 3)

[0888] The system according to claim 1, characterized in that the proposed next stage of learning content is dynamically selected based on the learner's past learning history, progress, and emotional information. [Explanation of symbols]

[0889] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for providing learning content containing multiple information formats to a user terminal, A means for analyzing the learning data collected from the user terminal in real time and evaluating the learning progress, A means of providing feedback to learners based on the evaluation results, A means to appropriately recommend the next stage of learning content, A means of displaying a three-dimensional model using augmented reality technology, A means to perform analysis using machine learning algorithms and provide the optimal learning experience, A means of generating advice based on learning needs using a generative AI model, A system that includes this.

2. The system according to claim 1, characterized in that the feedback identifies areas where the learner is lacking in learning, provides additional content, and is visually displayed through augmented reality technology.

3. The system according to claim 1, characterized in that the recommended next-stage learning content is dynamically selected based on the learner's past learning history and progress, and a generative AI model is used to generate prompt sentences for the learner.

Images