system

Abstract

We provide the system. [Solution] A means for generating content that combines multiple media on a device used by learners and displaying it in an augmented reality space, A means of collecting learner operation logs and analyzing their progress information, A means of generating and providing personalized feedback to learners based on analyzed progress information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional learning support systems, there has been a problem that appropriate feedback according to the progress of individual learners is insufficient, resulting in a decrease in learning efficiency. Also, the inadequacy of digital content has been an obstacle to deepening learners' understanding. As a result, it has been difficult for learners to obtain an effective and confident learning experience.

Means for Solving the Problems

[0005] This invention solves this problem by providing a means for displaying content combining multiple media in an augmented reality space on a terminal used by learners. Furthermore, by including a means for collecting learner operation logs and analyzing progress information, and a means for generating and providing personalized feedback to learners based on the results, it enables appropriate learning support according to each individual's learning progress.

[0006] A "terminal" is a device that learners can carry and use, and that is equipped with the necessary hardware and software to display augmented reality content.

[0007] "Augmented reality" is a virtual environment that overlays computer-generated virtual information onto real-world visual information to provide an interactive experience.

[0008] "Content" refers to a collection of data that provides information to learners, consisting of multiple media such as text, audio, images, and videos.

[0009] An "operation log" is recorded information about operations and actions performed by learners through their devices, and is used as data to analyze their progress and level of understanding.

[0010] "Progress information" refers to indicators that show how far a learner has progressed in the learning process and how much understanding they possess.

[0011] "Feedback" refers to information, including support and advice, generated based on the learner's progress, and is provided to deepen the learner's understanding.

[0012] "Personalized feedback" refers to information that provides appropriate support and advice tailored to each learner's individual progress. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] In this invention, the system consists of a terminal used by learners, a server, and a network connecting them. The terminal has the function of displaying learning content using AR technology and recording user operations. The server is responsible for distributing learning content, analyzing learner operation logs, and generating feedback.

[0035] Specifically, the server provides the device with the latest learning content data, and the device uses this data to display the content to the learner in an augmented reality (AR) space. The user progresses through the learning process, making operations and selections through the AR content presented via the device. The device records these operations as logs and sends them to the server.

[0036] The server uses an AI algorithm to analyze the operation logs sent by the user and generate progress information. This progress information includes the learner's level of understanding and viewing time. Based on this information, the server generates personalized feedback for each learner and sends it back to the terminal.

[0037] The device displays the feedback received to the user and, if necessary, presents additional learning content and practice problems. This allows learners to engage in learning that is optimally tailored to their own learning progress.

[0038] For example, a user studying history can learn by viewing 3D models of historical events in an augmented reality (AR) space through their device. When the user explores points they don't understand, their activity log is analyzed on the server, and if it's determined that their understanding is insufficient, additional explanations and practice problems related to that section are displayed on the device. This allows the user to efficiently progress through their learning while resolving any points of confusion.

[0039] Thus, the present invention is a system that provides learners with intuitive and personalized learning support, enabling improvements in learning efficiency and comprehension.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server loads the learning content data and AI analysis algorithms, preparing them for access by the devices. Necessary updates are then distributed from the server to all devices used by the user.

[0043] Step 2:

[0044] The device launches the application and establishes communication with the server. It downloads content corresponding to the learning module selected by the user from the server and caches it on the device.

[0045] Step 3:

[0046] The user checks the available learning modules on their device and selects the desired module. Based on that selection, the device prepares to display the learning content in the AR space.

[0047] Step 4:

[0048] The device displays augmented reality content by overlaying it onto real-world images using its camera function. Audio guides are played automatically, and text information, images, and videos are visually provided as related information.

[0049] Step 5:

[0050] Users interact with objects displayed in the AR space. For example, they can select specific objects or use zoom functions to view details.

[0051] Step 6:

[0052] The device meticulously records user activity logs. These logs include object selection history and time spent viewing learning content.

[0053] Step 7:

[0054] Operation logs sent from the terminal are aggregated on the server. The server uses an AI analysis algorithm to analyze the received data and generate learner progress information.

[0055] Step 8:

[0056] Based on progress information generated on the server, personalized feedback is created for the learner. This progress information includes details about content that the learner doesn't fully understand and topics that require further study.

[0057] Step 9:

[0058] The server sends the generated feedback back to the terminal. The terminal displays the feedback to the user in an easily understandable format, such as additional exercises or videos with detailed explanations.

[0059] Step 10:

[0060] Users revisit their learning based on feedback provided by their device. This enables efficient and effective learning tailored to individual progress.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] In today's learning environment, the lack of individualized support tailored to each learner's progress and level of understanding hinders efficient learning. Furthermore, existing educational systems often lack sufficient intuitive interfaces for learners to engage in augmented reality learning through concrete actions. This makes it difficult to maintain learners' interest and can lead to a decline in motivation.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for generating information by combining multiple information media and displaying it in an augmented reality space in an information processing device used by learners; means for collecting the learner's operation history and analyzing its progress information; means for generating personalized instructional information based on the analyzed progress information and providing it to the learner; and means for displaying additional exercises and explanations to the learner based on the generated instructional information. This enables optimal learning support according to the learner's progress, improving learning efficiency and maintaining learning motivation.

[0066] An "information processing device" is an electronic device used for inputting, processing, and outputting data, and provides an interface for users to interact with learning content.

[0067] An "information medium" is a means of storing and presenting information in forms such as text, images, audio, and video.

[0068] Augmented reality is a virtual space that provides users with new experiences by overlaying computer-generated information onto the real world environment.

[0069] "Operation history" refers to recorded data about operations performed by a user using an information processing device, and is used to track the progress of learning activities.

[0070] "Progress information" refers to information that indicates the user's learning status and level of understanding, and is an analysis result generated based on records of learning activities.

[0071] "Personalized instructional information" refers to customized advice and explanatory information provided based on the user's progress and tailored to their learning needs.

[0072] "Additional exercises and explanations" are supplementary learning materials and problem sets provided to deepen the user's understanding and complement existing learning content.

[0073] This invention is a learning support system that uses augmented reality technology and consists of a server, terminals, and a network connecting them.

[0074] The server is a central information processing device for managing and distributing learning content and analyzing learners' activity history. The server contains various learning content stored in a database, from which the latest information is sent to the terminal. The server uses AI algorithms to collect activity history and generate progress information. This analysis algorithm employs a generative AI model. Furthermore, based on the analyzed progress information, it creates personalized instructional information and provides it back to the terminal.

[0075] The terminal is an information processing device for displaying learning content received from the server in an augmented reality space. The terminal is equipped with the necessary hardware to build an AR environment, combining a camera and a display. The terminal records user actions and transmits them to the server. Dedicated AR applications and libraries are installed on the terminal, enabling, for example, the display of historical 3D models integrating multiple information sources.

[0076] The user is someone who engages in educational activities through augmented reality content displayed on their device. The user can manipulate 3D models and explore information using their fingers or specific input devices. This allows the user to intuitively learn through the provided content while fulfilling their role as a learner.

[0077] As a concrete example, when a user is learning history, they can access 3D models that recreate historical events in an AR space displayed on their device. For instance, they can visually experience what a city looked like during the Edo period. Furthermore, operations performed through the device are recorded as points of confusion and analyzed on the server. If it is determined that the user's understanding is insufficient, additional explanations or practice problems are displayed on the device.

[0078] Example of a prompt:

[0079] "Please describe how a system using AR technology allows users to manipulate 3D models during history lessons. Include how the learning content adapts to the user's actions."

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The server retrieves the latest content data from the learning content database and delivers this data to the terminal. Specifically, the server extracts the requested information (for example, a 3D model related to a historical topic) and sends it to the terminal via the network. The input in this process is the request, and the output is the corresponding content data.

[0083] Step 2:

[0084] The device uses learning content data received from the server to run an AR application and display the content to the learner in an augmented reality space. For example, a 3D model of a traditional Japanese city is overlaid and displayed on the device's display via the camera. In this process, the input is the content data, and the output is the displayed AR content.

[0085] Step 3:

[0086] Users interact with content displayed in an AR space on their device to advance their learning. Here, users rotate and zoom 3D models using screen touches and gesture input. This allows users to interact with the learning content. The input for this interaction is the user's physical actions, and the output is a visual change within the AR space.

[0087] Step 4:

[0088] The terminal records user actions as an operation log and sends it to the server. The terminal records all user actions along with their time, converts them into a data format, and sends them to the server. The input here is the user's operation data, and the output is the data accumulated as the operation log.

[0089] Step 5:

[0090] The server analyzes the received operation logs using an AI algorithm to generate learner progress information. This analysis process includes, for example, analyzing operation frequency and viewing time for specific content. The input is the operation log, and the output is learner progress information (such as comprehension level and study time).

[0091] Step 6:

[0092] The server generates user-optimized instructional information based on progress data and sends it to the terminal. This process, for example, recommends supplementary materials for areas where understanding is insufficient. The input is progress data, and the output is instructional information and additional learning content.

[0093] Step 7:

[0094] The terminal displays additional learning content and practice problems to the user based on instructional information received from the server. Specific examples include displaying supplementary explanatory videos or practice problems for topics the user doesn't fully understand. Here, the input is instructional information, and the output is new content presented to the learner.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In the field of information learning, there is a need to provide efficient and personalized learning utilizing multidimensional spaces. In particular, a challenge lies in the lack of systems that allow learners to instantly receive feedback and additional content tailored to their level of understanding and progress. Furthermore, it is necessary to create an environment where learners can interact within a multidimensional space and efficiently acquire specific learning tasks.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for presenting information in a multidimensional space, means for collecting the user's operation history and analyzing its progress information, and means for generating and providing personalized responses to the user based on the analyzed progress information. This enables learners to efficiently receive feedback and additional informational content tailored to their own progress.

[0100] An "information processing device" is a general term for electronic devices that process data and provide information to users.

[0101] A "multidimensional space" is a virtual environment that is digitally generated as an extension of reality and presents information visually or interactively.

[0102] "Means of presenting information" refers to a device or method that has the function of interpreting digital data and providing it to the user in a visual, auditory, or other form.

[0103] "User operation history" refers to a record of a series of interactions and actions performed by a user on an information processing device.

[0104] "Progress information" refers to data or indicators that show how far a user has progressed towards a certain goal.

[0105] "Personalized response" refers to information or feedback provided in a way that is tailored to the user's specific circumstances, needs, or characteristics.

[0106] "Interaction" refers to the exchange of data and information between a user and an information processing device, or between users themselves.

[0107] This invention realizes a learning support system using a multidimensional space via an information processing device. The server prepares the latest digital content and provides it to the user's terminal via the network. The user's terminal displays this content in the multidimensional space and records the user's operations in real time.

[0108] The server utilizes AI algorithms to analyze collected operation history and generate progress information. This progress information includes the user's level of understanding and work time, and based on this, it generates personalized responses and sends them to the user's device. This enables learners to learn efficiently at their own pace.

[0109] A concrete example of its use is when students in a history class use their devices to visualize the Egyptian pyramids in a multidimensional space, learning about their structure and historical background. During this process, if a student shows interest in a particular structure within the pyramids, this action is recorded as a log and analyzed on the server side. As a result, additional information and related questions about the areas of high interest are automatically presented.

[0110] Furthermore, by using a generative AI model, the information provided can be refined through prompts such as, "Generate an in-depth explanation of the Egyptian pyramids. The target audience is elementary school students." In this way, users can learn effectively and intuitively.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The terminal displays digital content received from the server in a multidimensional space. The input is content data from the server, which is then visually presented to the user using a display module. As an example, a 3D model of the Egyptian pyramids is displayed.

[0114] Step 2:

[0115] The user operates the device and interacts with information content in a multidimensional space. Input here consists of user actions such as taps and swipes, which the device detects with sensors, converts appropriately, and records as an operation history. For example, the user's action of zooming in on a part of a pyramid is recorded.

[0116] Step 3:

[0117] The terminal sends the collected operation history to the server. The input is the user's entire operation history, which the terminal compresses or encrypts and transfers to the server over the network. The server then receives the user's learned behavior data.

[0118] Step 4:

[0119] The server analyzes the received operation history using an AI algorithm. The input is user operation history data, and the AI ​​algorithm generates user progress information. Specifically, it evaluates the level of understanding based on operation frequency and viewing time.

[0120] Step 5:

[0121] The server generates personalized responses based on the analysis results. The input here is progress information, and it uses a generative AI model and prompts to generate personalized feedback, inserting additional information as needed. For example, if the user shows interest in a particular section, it provides historical context related to that section.

[0122] Step 6:

[0123] The server sends the generated feedback to the terminal. The input is individualized feedback, and the server converts it into a format that the terminal can understand and sends it again.

[0124] Step 7:

[0125] The device displays the received feedback to the user. The input is feedback data from the server, which the device visualizes and presents in a way that suits the user interface, and displays additional content as needed to facilitate further learning.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention relates to a learning support system comprising a terminal used by learners, a server, and a network that connects them. This system incorporates an emotion engine that recognizes the learner's emotional state in real time and has the function of adjusting learning content and feedback based on the obtained emotional information.

[0128] Specifically, the server is responsible for managing the learning content, executing the emotion recognition algorithm, and generating feedback based on the emotion recognition results. The terminal functions as an interface for learners to interact with the system, using its camera and microphone to collect the data required by the emotion engine.

[0129] When a user begins learning, the device displays the provided learning content in an augmented reality (AR) space. At this time, the emotion engine analyzes the user's facial expressions and voice captured by the device's camera to determine the user's emotional state in real time. For example, if the user appears confused, the server will either lower the difficulty level of the learning content or generate feedback that provides more detailed explanations based on the analysis results. Conversely, if the user shows satisfaction or understanding, it may be possible to present more challenging tasks.

[0130] Alternatively, consider a scenario where the user is learning a history module. In this case, the device displays AR content that recreates historical events, while the emotion engine analyzes the user's reactions. If the server determines that the user is interested, it selects and delivers relevant content or additional information to the device to maintain their interest.

[0131] Thus, the present invention is a system that takes into account the emotional state of learners to realize a more personalized learning experience, thereby enabling improvements in learning efficiency and motivation.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The server pre-prepares the learning content data and the algorithms necessary for emotion recognition, and loads all the data so that the device can access it. Any necessary updates are then delivered to the device.

[0135] Step 2:

[0136] The terminal launches the application and displays a list of modules to be offered to the learner. When the user selects a desired module, the terminal sends a request to the server and downloads and prepares the necessary content.

[0137] Step 3:

[0138] The user begins learning using the device. The device displays content combining multiple media formats in the AR space and prepares to utilize its camera and microphone to analyze the user's facial expressions and voice in real time through an emotion engine.

[0139] Step 4:

[0140] The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotional state. For example, it can detect facial expressions indicating decreased attention or satisfaction.

[0141] Step 5:

[0142] The device sends emotional data obtained from the emotion engine to the server. The server uses an AI model to analyze the emotional data and operation logs to understand the user's progress and level of understanding.

[0143] Step 6:

[0144] Based on the analysis results, the server generates feedback tailored to the user's current emotions and learning needs. In particular, if the server determines that understanding is insufficient, it prepares content that includes more detailed explanations.

[0145] Step 7:

[0146] The server sends the generated feedback to the device. The device applies the feedback to the user and displays new content or additional learning materials in the AR space.

[0147] Step 8:

[0148] Based on the feedback provided, users can adjust their learning approach as needed and continue learning further. This allows users to learn efficiently at their own pace.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] In today's learning environment, there is a demand for personalized educational experiences that consider not only learners' progress but also their emotional states. However, conventional systems struggle to effectively incorporate emotional information, and the generation of feedback tailored to each individual learner is not adequately realized. Therefore, solutions are needed to maximize learning efficiency and motivation.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for generating educational materials by combining multiple information media and displaying them in an augmented virtual space on a device used by learners; means for collecting learners' activity records and analyzing their progress and emotional information; and means for generating personalized feedback based on the analyzed progress and emotional information and providing it to the learners. This provides an educational experience that takes into account the learners' progress and emotional state, enabling improvements in learning efficiency and motivation.

[0154] The term "learner" refers to an individual who is receiving education or who is actively working to acquire skills.

[0155] A "device" refers to a collection of hardware and software configured to perform a specific function.

[0156] "Information media" refers to all forms of data used to transmit information, including text, images, audio, and video.

[0157] "Educational materials" refer to teaching materials and resources that learners use to acquire knowledge and skills.

[0158] "Augmented virtual space" refers to a technology that overlays digital information onto the physical real world, providing learners with an interactive experience.

[0159] "Activity records" refer to data that records the history of operations and actions performed by learners.

[0160] "Progress information" refers to data that serves as an indicator of how much content a learner has completed and to what extent they understand it during the learning process.

[0161] "Emotional information" refers to data that indicates the learner's emotional state, and typically refers to psychological state data analyzed from facial expressions, voice, etc.

[0162] "Analysis" refers to the process of handling data, extracting meaningful information, and deriving results.

[0163] "Individualized feedback" refers to responses and suggestions that are tailored to each learner's learning situation and emotional state.

[0164] This learning support system aims to provide a personalized educational experience by taking into account the learner's emotional state. A specific implementation is shown below.

[0165] Server Role

[0166] The server manages the learning content. Specifically, it organizes the diverse educational materials stored in the database and selects content suitable for the user. It also processes emotional data transmitted from the device using an emotion recognition algorithm and analyzes the learner's emotional state in real time. Furthermore, the server generates personalized feedback based on the analysis results. This feedback includes specific advice and supplementary information that takes into account the learner's current situation, generated using a generative AI model.

[0167] Terminal role

[0168] The terminal is a device that learners interact with, displaying educational materials as an augmented virtual space. This is typically achieved using AR technology. The terminal is equipped with a camera and microphone, which are used to record the learner's facial expressions and voice, and send emotional information to a server. The terminal also notifies the learner of the feedback sent from the server and displays adjusted educational tasks based on that feedback.

[0169] User interaction

[0170] Users learn by interacting with content displayed in an augmented virtual space through their devices. For example, they can visually experience AR content that recreates historical events or manipulate objects to solve problems. If a user shows confusion, the server uses analysis results to adjust the difficulty level or provide additional explanations. Also, if the user shows interest, more challenging tasks are presented.

[0171] Specific examples and prompt statements

[0172] As a concrete example, when a user is using the geography learning module, the device displays a terrain model in augmented reality (AR). Simultaneously, based on emotion recognition data, the server selects and presents relevant content to broaden the user's interests. An example of a prompt in this case would be, "If the user is interested in the terrain model, suggest relevant information that will allow him to further his learning."

[0173] This system provides a flexible learning experience that simultaneously considers the learner's emotions and progress, and is expected to improve the efficiency of education and enhance learning motivation.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The server analyzes the learner's profile data and selects appropriate learning content. This process receives progress information and past learning history from the database as input and performs data calculations to output the most suitable learning materials. Specifically, it filters content based on the learner's past performance and level of understanding.

[0177] Step 2:

[0178] The server transmits the selected learning content to the device. This requires converting the data to the format of the selected information medium, and delivering it in a format optimized for each device. The server receives content data as input, and the output is learning material converted to a format suitable for the device. Specifically, this includes packaging the content according to the device's display capabilities.

[0179] Step 3:

[0180] The device displays the received learning content in an augmented virtual space. The learning materials, as input, are processed within the device using AR technology, and a learning environment that the user can visually experience is output. Specifically, the camera and display are used to overlay digital information onto the actual physical space.

[0181] Step 4:

[0182] The device collects emotional information from the user's facial expressions and voice through its camera and microphone. It processes this input data, performs calculations to quantify the emotional state in real time, and generates emotional information as output. Specific operations include the use of facial recognition technology and voice analysis algorithms.

[0183] Step 5:

[0184] The server receives emotional information transmitted from the terminal and analyzes it using an emotional recognition algorithm. The emotional information received as input is used for data calculations to identify the user's psychological state, and the analysis results are generated as output. Specifically, a generative AI model classifies the emotional state.

[0185] Step 6:

[0186] The server generates personalized feedback based on the analysis results. This feedback generation utilizes progress and sentiment information as input and provides user-appropriate learning support messages as output. This includes using a generative AI model to automatically write out advice tailored to specific situations.

[0187] Step 7:

[0188] The server sends the generated feedback to the terminal, which then displays it to the user. Here, the feedback information is received as input and processed to output it in a user-friendly format. Specifically, this includes actions such as visual display on the screen and notifications via the voice assistant.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] There is a need to realize personalized learning that is tailored to each learner's level of understanding and interests. Conventional learning support systems do not adequately consider the learner's emotional state when responding or dynamically adjusting the learning environment, resulting in challenges in improving learning efficiency and maintaining motivation.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for generating complex data and displaying it in an augmented reality environment, means for collecting the learner's action history and analyzing its progress information, and means for evaluating the learner's facial features and acoustic data using an emotion analysis engine to determine the learner's emotional state. This enables dynamic adjustment of the individualized learning environment based on the learner's emotional state.

[0194] "Device" refers to all hardware used by learners for interaction.

[0195] "Materials" refers to data containing information and content for learners to study.

[0196] An "augmented reality environment" refers to an environment that uses technology to overlay virtual information onto the real world.

[0197] "Action history" refers to a record of a learner's operations and actions.

[0198] "Progress information" refers to data related to learners' learning status and progress.

[0199] "Analysis" refers to the process of evaluating collected data and extracting meaningful information.

[0200] "Individualized responses" refer to feedback and instructions that are tailored to the learner's characteristics and circumstances.

[0201] An "emotion analysis engine" refers to an algorithm or system used to recognize a learner's emotional state.

[0202] "Facial features" refer to facial characteristics and changes captured in real time.

[0203] "Acoustic data" refers to data related to the learner's voice and surrounding sounds.

[0204] "Emotional state" refers to the psychological and emotional condition exhibited by the learner.

[0205] The system that realizes this invention mainly consists of a server, a terminal used by learners, and an emotion analysis engine. The server generates and manages learning content and plays a central role in analyzing progress information and emotional states.

[0206] The device provides an interface with the learner and displays materials in an augmented reality environment. It collects the learner's facial features and acoustic data through its camera and microphone and sends it to an emotion analysis engine. The emotion analysis engine evaluates the learner's real-time emotional state and returns that information to the server.

[0207] Based on data received from the emotion analysis engine, the server generates personalized responses tailored to the learner's emotional state and dynamically adjusts the learning environment. Specifically, if a learner is confused, the server understands the situation and provides easier tasks or more detailed explanations. If the learner shows interest, it delivers relevant information and additional content to the device.

[0208] This invention makes it possible to provide learning experiences tailored to the emotional state of individual learners, thereby improving learning efficiency and motivation.

[0209] As a concrete example, consider a scenario where learners use augmented reality during a history lesson to visualize historical events. In this case, by analyzing the learners' emotional state and providing appropriate feedback and supplementary information, it is possible to deepen their understanding while maintaining their interest.

[0210] An example of a prompt for a generative AI model is: "Think about how a robot that supports children's learning should talk to a child who has become bored."

[0211] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0212] Step 1:

[0213] Once the device is powered on and the learner begins learning, it uses its camera and microphone to collect the learner's facial features and audio data in real time. The input data consists of camera video and audio data, which are sent to the emotion analysis engine.

[0214] Step 2:

[0215] The emotion analysis engine analyzes received facial features and acoustic data to determine the learner's current emotional state. A deep learning model is used in this process to output an "emotional state" from the input data. Emotional states include states such as concentration, confusion, and interest.

[0216] Step 3:

[0217] The server adjusts the content and responses provided to the learner based on emotional state data sent from the emotion analysis engine. For example, if the server determines that the learner is confused, it generates easier learning content or additional explanations. In this process, emotional state is used as input, and an adjusted response is generated as output.

[0218] Step 4:

[0219] The device displays pre-configured learning content and feedback sent from the server in an augmented reality environment. In this process, the device uses AR technology based on the provided data to create a composite display. The input is data from the server, and the output is the augmented reality content viewed by the learner.

[0220] Step 5:

[0221] When a user performs a new interaction, the operation log is recorded on the terminal and sent to the server as progress information. The server uses this progress information to prepare for further adjustments to future learning support. Here, the user operation log is used as input, and the analyzed progress information is output.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] In this invention, the system consists of a terminal used by learners, a server, and a network connecting them. The terminal has the function of displaying learning content using AR technology and recording user operations. The server is responsible for distributing learning content, analyzing learner operation logs, and generating feedback.

[0239] Specifically, the server provides the device with the latest learning content data, and the device uses this data to display the content to the learner in an augmented reality (AR) space. The user progresses through the learning process, making operations and selections through the AR content presented via the device. The device records these operations as logs and sends them to the server.

[0240] The server uses an AI algorithm to analyze the operation logs sent by the user and generate progress information. This progress information includes the learner's level of understanding and viewing time. Based on this information, the server generates personalized feedback for each learner and sends it back to the terminal.

[0241] The device displays the feedback received to the user and, if necessary, presents additional learning content and practice problems. This allows learners to engage in learning that is optimally tailored to their own learning progress.

[0242] For example, a user studying history can learn by viewing 3D models of historical events in an augmented reality (AR) space through their device. When the user explores points they don't understand, their activity log is analyzed on the server, and if it's determined that their understanding is insufficient, additional explanations and practice problems related to that section are displayed on the device. This allows the user to efficiently progress through their learning while resolving any points of confusion.

[0243] Thus, the present invention is a system that provides learners with intuitive and personalized learning support, enabling improvements in learning efficiency and comprehension.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The server loads the learning content data and AI analysis algorithms, preparing them for access by the devices. Necessary updates are then distributed from the server to all devices used by the user.

[0247] Step 2:

[0248] The device launches the application and establishes communication with the server. It downloads content corresponding to the learning module selected by the user from the server and caches it on the device.

[0249] Step 3:

[0250] The user checks the available learning modules on their device and selects the desired module. Based on that selection, the device prepares to display the learning content in the AR space.

[0251] Step 4:

[0252] The device displays augmented reality content by overlaying it onto real-world images using its camera function. Audio guides are played automatically, and text information, images, and videos are visually provided as related information.

[0253] Step 5:

[0254] Users interact with objects displayed in the AR space. For example, they can select specific objects or use zoom functions to view details.

[0255] Step 6:

[0256] The device meticulously records user activity logs. These logs include object selection history and time spent viewing learning content.

[0257] Step 7:

[0258] Operation logs sent from the terminal are aggregated on the server. The server uses an AI analysis algorithm to analyze the received data and generate learner progress information.

[0259] Step 8:

[0260] Based on progress information generated on the server, personalized feedback is created for the learner. This progress information includes details about content that the learner doesn't fully understand and topics that require further study.

[0261] Step 9:

[0262] The server sends the generated feedback back to the terminal. The terminal displays the feedback to the user in an easily understandable format, such as additional exercises or videos with detailed explanations.

[0263] Step 10:

[0264] Users revisit their learning based on feedback provided by their device. This enables efficient and effective learning tailored to individual progress.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] In today's learning environment, the lack of individualized support tailored to each learner's progress and level of understanding hinders efficient learning. Furthermore, existing educational systems often lack sufficient intuitive interfaces for learners to engage in augmented reality learning through concrete actions. This makes it difficult to maintain learners' interest and can lead to a decline in motivation.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for generating information by combining multiple information media and displaying it in an augmented reality space in an information processing device used by learners; means for collecting the learner's operation history and analyzing its progress information; means for generating personalized instructional information based on the analyzed progress information and providing it to the learner; and means for displaying additional exercises and explanations to the learner based on the generated instructional information. This enables optimal learning support according to the learner's progress, improving learning efficiency and maintaining learning motivation.

[0270] An "information processing device" is an electronic device used for inputting, processing, and outputting data, and provides an interface for users to interact with learning content.

[0271] An "information medium" is a means of storing and presenting information in forms such as text, images, audio, and video.

[0272] Augmented reality is a virtual space that provides users with new experiences by overlaying computer-generated information onto the real world environment.

[0273] "Operation history" refers to recorded data about operations performed by a user using an information processing device, and is used to track the progress of learning activities.

[0274] "Progress information" refers to information that indicates the user's learning status and level of understanding, and is an analysis result generated based on records of learning activities.

[0275] "Personalized instructional information" refers to customized advice and explanatory information provided based on the user's progress and tailored to their learning needs.

[0276] "Additional exercises and explanations" are supplementary learning materials and problem sets provided to deepen the user's understanding and complement existing learning content.

[0277] This invention is a learning support system that uses augmented reality technology and consists of a server, terminals, and a network connecting them.

[0278] The server is a central information processing device for managing and distributing learning content and analyzing learners' activity history. The server contains various learning content stored in a database, from which the latest information is sent to the terminal. The server uses AI algorithms to collect activity history and generate progress information. This analysis algorithm employs a generative AI model. Furthermore, based on the analyzed progress information, it creates personalized instructional information and provides it back to the terminal.

[0279] The terminal is an information processing device for displaying learning content received from the server in an augmented reality space. The terminal is equipped with the necessary hardware to build an AR environment, combining a camera and a display. The terminal records user actions and transmits them to the server. Dedicated AR applications and libraries are installed on the terminal, enabling, for example, the display of historical 3D models integrating multiple information sources.

[0280] The user is someone who engages in educational activities through augmented reality content displayed on their device. The user can manipulate 3D models and explore information using their fingers or specific input devices. This allows the user to intuitively learn through the provided content while fulfilling their role as a learner.

[0281] As a specific example, when a user studies history, they can access a 3D model that reproduces historical events in the AR space displayed on the terminal. For example, they can visually experience what an Edo-period city was like. Also, if an operation performed through the terminal is recorded as an unclear point and analyzed by the server, and it is determined that the user's understanding is insufficient, additional explanations and practice problems will be displayed on the terminal.

[0282] Example of a prompt sentence:

[0283] "Please explain the operation of a system where a user operates a 3D model during a history lesson using AR technology. Please describe including the adaptation of learning content according to the user's operation."

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The server acquires the latest content data from the learning content database and distributes this data to the terminal. Specifically, the server extracts the information received in the request (for example, a 3D model related to a history topic) and transmits it to the terminal via the network. The input at this time is the request content, and the output is the corresponding content data.

[0287] Step 2:

[0288] The terminal uses the learning content data received from the server to operate the AR app and display the content to the learner in the augmented reality space. For example, it overlays and displays a 3D model of a traditional Japanese city on the terminal's display through the camera. In this process, the input is the content data, and the output is the displayed AR content.

[0289] Step 3:

[0290] Users interact with content displayed in an AR space on their device to advance their learning. Here, users rotate and zoom 3D models using screen touches and gesture input. This allows users to interact with the learning content. The input for this interaction is the user's physical actions, and the output is a visual change within the AR space.

[0291] Step 4:

[0292] The terminal records user actions as an operation log and sends it to the server. The terminal records all user actions along with their time, converts them into a data format, and sends them to the server. The input here is the user's operation data, and the output is the data accumulated as the operation log.

[0293] Step 5:

[0294] The server analyzes the received operation logs using an AI algorithm to generate learner progress information. This analysis process includes, for example, analyzing operation frequency and viewing time for specific content. The input is the operation log, and the output is learner progress information (such as comprehension level and study time).

[0295] Step 6:

[0296] The server generates user-optimized instructional information based on progress data and sends it to the terminal. This process, for example, recommends supplementary materials for areas where understanding is insufficient. The input is progress data, and the output is instructional information and additional learning content.

[0297] Step 7:

[0298] The terminal displays additional learning content and practice problems to the user based on instructional information received from the server. Specific examples include displaying supplementary explanatory videos or practice problems for topics the user doesn't fully understand. Here, the input is instructional information, and the output is new content presented to the learner.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] In the field of information learning, there is a need to provide efficient and personalized learning utilizing multidimensional spaces. In particular, a challenge lies in the lack of systems that allow learners to instantly receive feedback and additional content tailored to their level of understanding and progress. Furthermore, it is necessary to create an environment where learners can interact within a multidimensional space and efficiently acquire specific learning tasks.

[0302] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes means for presenting information in a multidimensional space, means for collecting the user's operation history and analyzing its progress information, and means for generating and providing personalized responses to the user based on the analyzed progress information. This enables learners to efficiently receive feedback and additional informational content tailored to their own progress.

[0304] An "information processing device" is a general term for electronic devices that process data and provide information to users.

[0305] A "multidimensional space" is a virtual environment that is digitally generated as an extension of reality and presents information visually or interactively.

[0306] "Means of presenting information" refers to a device or method that has the function of interpreting digital data and providing it to the user in a visual, auditory, or other form.

[0307] "User operation history" refers to a record of a series of interactions and actions performed by a user on an information processing device.

[0308] "Progress information" refers to data or indicators that show how far a user has progressed towards a certain goal.

[0309] "Individualized response" refers to information or feedback provided in a form tailored to the specific situation, needs, or characteristics of the user.

[0310] "Interaction" refers to the exchange of data and information between the user and the information processing device, or between users.

[0311] This invention realizes a learning support system using a multi-dimensional space by an information processing device. The server is responsible for preparing the latest digital content and providing it to the user's terminal through the network. The user's terminal displays this content in a multi-dimensional space and records the user's operations in real time.

[0312] The server utilizes an AI algorithm to analyze the collected operation history and generate progress information. The user's progress information includes the degree of understanding and working time. Based on this, an individualized response is generated and sent to the user's terminal. As a result, learners can perform efficient learning according to their respective learning paces.

[0313] As a specific example of use, a case can be considered where a learner uses a terminal to visualize the pyramids of Egypt in a multi-dimensional space and learn about their structure and historical background in a history class. In this process, if the learner shows interest in a specific structure inside the pyramid, that operation is recorded as a log and analyzed on the server side. As a result, additional information and related questions about the areas of high interest are automatically presented.

[0314] Furthermore, by using a generative AI model, the information provided can be refined through prompts such as, "Generate an in-depth explanation of the Egyptian pyramids. The target audience is elementary school students." In this way, users can learn effectively and intuitively.

[0315] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0316] Step 1:

[0317] The terminal displays digital content received from the server in a multidimensional space. The input is content data from the server, which is then visually presented to the user using a display module. As an example, a 3D model of the Egyptian pyramids is displayed.

[0318] Step 2:

[0319] The user operates the device and interacts with information content in a multidimensional space. Input here consists of user actions such as taps and swipes, which the device detects with sensors, converts appropriately, and records as an operation history. For example, the user's action of zooming in on a part of a pyramid is recorded.

[0320] Step 3:

[0321] The terminal sends the collected operation history to the server. The input is the user's entire operation history, which the terminal compresses or encrypts and transfers to the server over the network. The server then receives the user's learned behavior data.

[0322] Step 4:

[0323] The server analyzes the received operation history using an AI algorithm. The input is user operation history data, and the AI ​​algorithm generates user progress information. Specifically, it evaluates the level of understanding based on operation frequency and viewing time.

[0324] Step 5:

[0325] The server generates personalized responses based on the analysis results. The input here is progress information, and it uses a generative AI model and prompts to generate personalized feedback, inserting additional information as needed. For example, if the user shows interest in a particular section, it provides historical context related to that section.

[0326] Step 6:

[0327] The server sends the generated feedback to the terminal. The input is individualized feedback, and the server converts it into a format that the terminal can understand and sends it again.

[0328] Step 7:

[0329] The device displays the received feedback to the user. The input is feedback data from the server, which the device visualizes and presents in a way that suits the user interface, and displays additional content as needed to facilitate further learning.

[0330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0331] This invention relates to a learning support system comprising a terminal used by learners, a server, and a network that connects them. This system incorporates an emotion engine that recognizes the learner's emotional state in real time and has the function of adjusting learning content and feedback based on the obtained emotional information.

[0332] Specifically, the server is responsible for managing the learning content, executing the emotion recognition algorithm, and generating feedback based on the emotion recognition results. The terminal functions as an interface for learners to interact with the system, using its camera and microphone to collect the data required by the emotion engine.

[0333] When a user begins learning, the device displays the provided learning content in an augmented reality (AR) space. At this time, the emotion engine analyzes the user's facial expressions and voice captured by the device's camera to determine the user's emotional state in real time. For example, if the user appears confused, the server will either lower the difficulty level of the learning content or generate feedback that provides more detailed explanations based on the analysis results. Conversely, if the user shows satisfaction or understanding, it may be possible to present more challenging tasks.

[0334] Alternatively, consider a scenario where the user is learning a history module. In this case, the device displays AR content that recreates historical events, while the emotion engine analyzes the user's reactions. If the server determines that the user is interested, it selects and delivers relevant content or additional information to the device to maintain their interest.

[0335] Thus, the present invention is a system that takes into account the emotional state of learners to realize a more personalized learning experience, thereby enabling improvements in learning efficiency and motivation.

[0336] The following describes the processing flow.

[0337] Step 1:

[0338] The server pre-prepares the learning content data and the algorithms necessary for emotion recognition, and loads all the data so that the device can access it. Any necessary updates are then delivered to the device.

[0339] Step 2:

[0340] The terminal launches the application and displays a list of modules to be offered to the learner. When the user selects a desired module, the terminal sends a request to the server and downloads and prepares the necessary content.

[0341] Step 3:

[0342] The user begins learning using the device. The device displays content combining multiple media formats in the AR space and prepares to utilize its camera and microphone to analyze the user's facial expressions and voice in real time through an emotion engine.

[0343] Step 4:

[0344] The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotional state. For example, it can detect facial expressions indicating decreased attention or satisfaction.

[0345] Step 5:

[0346] The device sends emotional data obtained from the emotion engine to the server. The server uses an AI model to analyze the emotional data and operation logs to understand the user's progress and level of understanding.

[0347] Step 6:

[0348] Based on the analysis results, the server generates feedback tailored to the user's current emotions and learning needs. In particular, if the server determines that understanding is insufficient, it prepares content that includes more detailed explanations.

[0349] Step 7:

[0350] The server sends the generated feedback to the device. The device applies the feedback to the user and displays new content or additional learning materials in the AR space.

[0351] Step 8:

[0352] Based on the feedback provided, users can adjust their learning approach as needed and continue learning further. This allows users to learn efficiently at their own pace.

[0353] (Example 2)

[0354] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0355] In today's learning environment, there is a demand for personalized educational experiences that consider not only learners' progress but also their emotional states. However, conventional systems struggle to effectively incorporate emotional information, and the generation of feedback tailored to each individual learner is not adequately realized. Therefore, solutions are needed to maximize learning efficiency and motivation.

[0356] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0357] In this invention, the server includes means for generating educational materials by combining multiple information media and displaying them in an augmented virtual space on a device used by learners; means for collecting learners' activity records and analyzing their progress and emotional information; and means for generating personalized feedback based on the analyzed progress and emotional information and providing it to the learners. This provides an educational experience that takes into account the learners' progress and emotional state, enabling improvements in learning efficiency and motivation.

[0358] The term "learner" refers to an individual who is receiving education or who is actively working to acquire skills.

[0359] A "device" refers to a collection of hardware and software configured to perform a specific function.

[0360] "Information media" refers to all forms of data used to transmit information, including text, images, audio, and video.

[0361] "Educational materials" refer to teaching materials and resources that learners use to acquire knowledge and skills.

[0362] "Augmented virtual space" refers to a technology that overlays digital information onto the physical real world, providing learners with an interactive experience.

[0363] "Activity records" refer to data that records the history of operations and actions performed by learners.

[0364] "Progress information" refers to data that serves as an indicator of how much content a learner has completed and to what extent they understand it during the learning process.

[0365] "Emotional information" refers to data that indicates the learner's emotional state, and typically refers to psychological state data analyzed from facial expressions, voice, etc.

[0366] "Analysis" refers to the process of handling data, extracting meaningful information, and deriving results.

[0367] "Individualized feedback" refers to responses and suggestions that are tailored to each learner's learning situation and emotional state.

[0368] This learning support system aims to provide a personalized educational experience by taking into account the learner's emotional state. A specific implementation is shown below.

[0369] Server Role

[0370] The server manages the learning content. Specifically, it organizes the diverse educational materials stored in the database and selects content suitable for the user. It also processes emotional data transmitted from the device using an emotion recognition algorithm and analyzes the learner's emotional state in real time. Furthermore, the server generates personalized feedback based on the analysis results. This feedback includes specific advice and supplementary information that takes into account the learner's current situation, generated using a generative AI model.

[0371] Terminal role

[0372] The terminal is a device that learners interact with, displaying educational materials as an augmented virtual space. This is typically achieved using AR technology. The terminal is equipped with a camera and microphone, which are used to record the learner's facial expressions and voice, and send emotional information to a server. The terminal also notifies the learner of the feedback sent from the server and displays adjusted educational tasks based on that feedback.

[0373] User interaction

[0374] Users learn by interacting with content displayed in an augmented virtual space through their devices. For example, they can visually experience AR content that recreates historical events or manipulate objects to solve problems. If a user shows confusion, the server uses analysis results to adjust the difficulty level or provide additional explanations. Also, if the user shows interest, more challenging tasks are presented.

[0375] Specific examples and prompt statements

[0376] As a concrete example, when a user is using the geography learning module, the device displays a terrain model in augmented reality (AR). Simultaneously, based on emotion recognition data, the server selects and presents relevant content to broaden the user's interests. An example of a prompt in this case would be, "If the user is interested in the terrain model, suggest relevant information that will allow him to further his learning."

[0377] This system provides a flexible learning experience that simultaneously considers the learner's emotions and progress, and is expected to improve the efficiency of education and enhance learning motivation.

[0378] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0379] Step 1:

[0380] The server analyzes the learner's profile data and selects appropriate learning content. This process receives progress information and past learning history from the database as input and performs data calculations to output the most suitable learning materials. Specifically, it filters content based on the learner's past performance and level of understanding.

[0381] Step 2:

[0382] The server transmits the selected learning content to the device. This requires converting the data to the format of the selected information medium, and delivering it in a format optimized for each device. The server receives content data as input, and the output is learning material converted to a format suitable for the device. Specifically, this includes packaging the content according to the device's display capabilities.

[0383] Step 3:

[0384] The device displays the received learning content in an augmented virtual space. The learning materials, as input, are processed within the device using AR technology, and a learning environment that the user can visually experience is output. Specifically, the camera and display are used to overlay digital information onto the actual physical space.

[0385] Step 4:

[0386] The device collects emotional information from the user's facial expressions and voice through its camera and microphone. It processes this input data, performs calculations to quantify the emotional state in real time, and generates emotional information as output. Specific operations include the use of facial recognition technology and voice analysis algorithms.

[0387] Step 5:

[0388] The server receives emotional information transmitted from the terminal and analyzes it using an emotional recognition algorithm. The emotional information received as input is used for data calculations to identify the user's psychological state, and the analysis results are generated as output. Specifically, a generative AI model classifies the emotional state.

[0389] Step 6:

[0390] The server generates personalized feedback based on the analysis results. This feedback generation utilizes progress and sentiment information as input and provides user-appropriate learning support messages as output. This includes using a generative AI model to automatically write out advice tailored to specific situations.

[0391] Step 7:

[0392] The server sends the generated feedback to the terminal, which then displays it to the user. Here, the feedback information is received as input and processed to output it in a user-friendly format. Specifically, this includes actions such as visual display on the screen and notifications via the voice assistant.

[0393] (Application Example 2)

[0394] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0395] There is a need to realize personalized learning that is tailored to each learner's level of understanding and interests. Conventional learning support systems do not adequately consider the learner's emotional state when responding or dynamically adjusting the learning environment, resulting in challenges in improving learning efficiency and maintaining motivation.

[0396] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0397] In this invention, the server includes means for generating complex data and displaying it in an augmented reality environment, means for collecting the learner's action history and analyzing its progress information, and means for evaluating the learner's facial features and acoustic data using an emotion analysis engine to determine the learner's emotional state. This enables dynamic adjustment of the individualized learning environment based on the learner's emotional state.

[0398] "Device" refers to all hardware used by learners for interaction.

[0399] "Materials" refers to data containing information and content for learners to study.

[0400] An "augmented reality environment" refers to an environment that uses technology to overlay virtual information onto the real world.

[0401] "Action history" refers to a record of a learner's operations and actions.

[0402] "Progress information" refers to data related to learners' learning status and progress.

[0403] "Analysis" refers to the process of evaluating collected data and extracting meaningful information.

[0404] "Individualized responses" refer to feedback and instructions that are tailored to the learner's characteristics and circumstances.

[0405] An "emotion analysis engine" refers to an algorithm or system used to recognize a learner's emotional state.

[0406] "Facial features" refer to facial characteristics and changes captured in real time.

[0407] "Acoustic data" refers to data related to the learner's voice and surrounding sounds.

[0408] "Emotional state" refers to the psychological and emotional condition exhibited by the learner.

[0409] The system that realizes this invention mainly consists of a server, a terminal used by learners, and an emotion analysis engine. The server generates and manages learning content and plays a central role in analyzing progress information and emotional states.

[0410] The device provides an interface with the learner and displays materials in an augmented reality environment. It collects the learner's facial features and acoustic data through its camera and microphone and sends it to an emotion analysis engine. The emotion analysis engine evaluates the learner's real-time emotional state and returns that information to the server.

[0411] Based on data received from the emotion analysis engine, the server generates personalized responses tailored to the learner's emotional state and dynamically adjusts the learning environment. Specifically, if a learner is confused, the server understands the situation and provides easier tasks or more detailed explanations. If the learner shows interest, it delivers relevant information and additional content to the device.

[0412] This invention makes it possible to provide learning experiences tailored to the emotional state of individual learners, thereby improving learning efficiency and motivation.

[0413] As a concrete example, consider a scenario where learners use augmented reality during a history lesson to visualize historical events. In this case, by analyzing the learners' emotional state and providing appropriate feedback and supplementary information, it is possible to deepen their understanding while maintaining their interest.

[0414] An example of a prompt for a generative AI model is: "Think about how a robot that supports children's learning should talk to a child who has become bored."

[0415] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0416] Step 1:

[0417] Once the device is powered on and the learner begins learning, it uses its camera and microphone to collect the learner's facial features and audio data in real time. The input data consists of camera video and audio data, which are sent to the emotion analysis engine.

[0418] Step 2:

[0419] The emotion analysis engine analyzes received facial features and acoustic data to determine the learner's current emotional state. A deep learning model is used in this process to output an "emotional state" from the input data. Emotional states include states such as concentration, confusion, and interest.

[0420] Step 3:

[0421] The server adjusts the content and responses provided to the learner based on emotional state data sent from the emotion analysis engine. For example, if the server determines that the learner is confused, it generates easier learning content or additional explanations. In this process, emotional state is used as input, and an adjusted response is generated as output.

[0422] Step 4:

[0423] The device displays pre-configured learning content and feedback sent from the server in an augmented reality environment. In this process, the device uses AR technology based on the provided data to create a composite display. The input is data from the server, and the output is the augmented reality content viewed by the learner.

[0424] Step 5:

[0425] When a user performs a new interaction, the operation log is recorded on the terminal and sent to the server as progress information. The server uses this progress information to prepare for further adjustments to future learning support. Here, the user operation log is used as input, and the analyzed progress information is output.

[0426] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0427] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0428] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0429] [Third Embodiment]

[0430] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0431] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0432] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0433] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0434] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0436] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0437] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0438] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0439] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0440] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0441] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0442] In this invention, the system consists of a terminal used by learners, a server, and a network connecting them. The terminal has the function of displaying learning content using AR technology and recording user operations. The server is responsible for distributing learning content, analyzing learner operation logs, and generating feedback.

[0443] Specifically, the server provides the device with the latest learning content data, and the device uses this data to display the content to the learner in an augmented reality (AR) space. The user progresses through the learning process, making operations and selections through the AR content presented via the device. The device records these operations as logs and sends them to the server.

[0444] The server uses an AI algorithm to analyze the operation logs sent by the user and generate progress information. This progress information includes the learner's level of understanding and viewing time. Based on this information, the server generates personalized feedback for each learner and sends it back to the terminal.

[0445] The device displays the feedback received to the user and, if necessary, presents additional learning content and practice problems. This allows learners to engage in learning that is optimally tailored to their own learning progress.

[0446] For example, a user studying history can learn by viewing 3D models of historical events in an augmented reality (AR) space through their device. When the user explores points they don't understand, their activity log is analyzed on the server, and if it's determined that their understanding is insufficient, additional explanations and practice problems related to that section are displayed on the device. This allows the user to efficiently progress through their learning while resolving any points of confusion.

[0447] Thus, the present invention is a system that provides learners with intuitive and personalized learning support, enabling improvements in learning efficiency and comprehension.

[0448] The following describes the processing flow.

[0449] Step 1:

[0450] The server loads the learning content data and AI analysis algorithms, preparing them for access by the devices. Necessary updates are then distributed from the server to all devices used by the user.

[0451] Step 2:

[0452] The device launches the application and establishes communication with the server. It downloads content corresponding to the learning module selected by the user from the server and caches it on the device.

[0453] Step 3:

[0454] The user checks the available learning modules on their device and selects the desired module. Based on that selection, the device prepares to display the learning content in the AR space.

[0455] Step 4:

[0456] The device displays augmented reality content by overlaying it onto real-world images using its camera function. Audio guides are played automatically, and text information, images, and videos are visually provided as related information.

[0457] Step 5:

[0458] Users interact with objects displayed in the AR space. For example, they can select specific objects or use zoom functions to view details.

[0459] Step 6:

[0460] The device meticulously records user activity logs. These logs include object selection history and time spent viewing learning content.

[0461] Step 7:

[0462] Operation logs sent from the terminal are aggregated on the server. The server uses an AI analysis algorithm to analyze the received data and generate learner progress information.

[0463] Step 8:

[0464] Based on progress information generated on the server, personalized feedback is created for the learner. This progress information includes details about content that the learner doesn't fully understand and topics that require further study.

[0465] Step 9:

[0466] The server sends the generated feedback back to the terminal. The terminal displays the feedback to the user in an easily understandable format, such as additional exercises or videos with detailed explanations.

[0467] Step 10:

[0468] Users revisit their learning based on feedback provided by their device. This enables efficient and effective learning tailored to individual progress.

[0469] (Example 1)

[0470] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0471] In today's learning environment, the lack of individualized support tailored to each learner's progress and level of understanding hinders efficient learning. Furthermore, existing educational systems often lack sufficient intuitive interfaces for learners to engage in augmented reality learning through concrete actions. This makes it difficult to maintain learners' interest and can lead to a decline in motivation.

[0472] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0473] In this invention, the server includes means for generating information by combining multiple information media and displaying it in an augmented reality space in an information processing device used by learners; means for collecting the learner's operation history and analyzing its progress information; means for generating personalized instructional information based on the analyzed progress information and providing it to the learner; and means for displaying additional exercises and explanations to the learner based on the generated instructional information. This enables optimal learning support according to the learner's progress, improving learning efficiency and maintaining learning motivation.

[0474] An "information processing device" is an electronic device used for inputting, processing, and outputting data, and provides an interface for users to interact with learning content.

[0475] An "information medium" is a means of storing and presenting information in forms such as text, images, audio, and video.

[0476] Augmented reality is a virtual space that provides users with new experiences by overlaying computer-generated information onto the real world environment.

[0477] "Operation history" refers to recorded data about operations performed by a user using an information processing device, and is used to track the progress of learning activities.

[0478] "Progress information" refers to information that indicates the user's learning status and level of understanding, and is an analysis result generated based on records of learning activities.

[0479] "Personalized instructional information" refers to customized advice and explanatory information provided based on the user's progress and tailored to their learning needs.

[0480] "Additional exercises and explanations" are supplementary learning materials and problem sets provided to deepen the user's understanding and complement existing learning content.

[0481] This invention is a learning support system that uses augmented reality technology and consists of a server, terminals, and a network connecting them.

[0482] The server is a central information processing device for managing and distributing learning content and analyzing learners' activity history. The server contains various learning content stored in a database, from which the latest information is sent to the terminal. The server uses AI algorithms to collect activity history and generate progress information. This analysis algorithm employs a generative AI model. Furthermore, based on the analyzed progress information, it creates personalized instructional information and provides it back to the terminal.

[0483] The terminal is an information processing device for displaying learning content received from the server in an augmented reality space. The terminal is equipped with the necessary hardware to build an AR environment, combining a camera and a display. The terminal records user actions and transmits them to the server. Dedicated AR applications and libraries are installed on the terminal, enabling, for example, the display of historical 3D models integrating multiple information sources.

[0484] The user is someone who engages in educational activities through augmented reality content displayed on their device. The user can manipulate 3D models and explore information using their fingers or specific input devices. This allows the user to intuitively learn through the provided content while fulfilling their role as a learner.

[0485] As a concrete example, when a user is learning history, they can access 3D models that recreate historical events in an AR space displayed on their device. For instance, they can visually experience what a city looked like during the Edo period. Furthermore, operations performed through the device are recorded as points of confusion and analyzed on the server. If it is determined that the user's understanding is insufficient, additional explanations or practice problems are displayed on the device.

[0486] Example of a prompt:

[0487] "Please describe how a system using AR technology allows users to manipulate 3D models during history lessons. Include how the learning content adapts to the user's actions."

[0488] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0489] Step 1:

[0490] The server retrieves the latest content data from the learning content database and delivers this data to the terminal. Specifically, the server extracts the requested information (for example, a 3D model related to a historical topic) and sends it to the terminal via the network. The input in this process is the request, and the output is the corresponding content data.

[0491] Step 2:

[0492] The device uses learning content data received from the server to run an AR application and display the content to the learner in an augmented reality space. For example, a 3D model of a traditional Japanese city is overlaid and displayed on the device's display via the camera. In this process, the input is the content data, and the output is the displayed AR content.

[0493] Step 3:

[0494] Users interact with content displayed in an AR space on their device to advance their learning. Here, users rotate and zoom 3D models using screen touches and gesture input. This allows users to interact with the learning content. The input for this interaction is the user's physical actions, and the output is a visual change within the AR space.

[0495] Step 4:

[0496] The terminal records user actions as an operation log and sends it to the server. The terminal records all user actions along with their time, converts them into a data format, and sends them to the server. The input here is the user's operation data, and the output is the data accumulated as the operation log.

[0497] Step 5:

[0498] The server analyzes the received operation logs using an AI algorithm to generate learner progress information. This analysis process includes, for example, analyzing operation frequency and viewing time for specific content. The input is the operation log, and the output is learner progress information (such as comprehension level and study time).

[0499] Step 6:

[0500] The server generates user-optimized instructional information based on progress data and sends it to the terminal. This process, for example, recommends supplementary materials for areas where understanding is insufficient. The input is progress data, and the output is instructional information and additional learning content.

[0501] Step 7:

[0502] The terminal displays additional learning content and practice problems to the user based on instructional information received from the server. Specific examples include displaying supplementary explanatory videos or practice problems for topics the user doesn't fully understand. Here, the input is instructional information, and the output is new content presented to the learner.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0505] In the field of information learning, there is a need to provide efficient and personalized learning utilizing multidimensional spaces. In particular, a challenge lies in the lack of systems that allow learners to instantly receive feedback and additional content tailored to their level of understanding and progress. Furthermore, it is necessary to create an environment where learners can interact within a multidimensional space and efficiently acquire specific learning tasks.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for presenting information in a multidimensional space, means for collecting the user's operation history and analyzing its progress information, and means for generating and providing personalized responses to the user based on the analyzed progress information. This enables learners to efficiently receive feedback and additional informational content tailored to their own progress.

[0508] An "information processing device" is a general term for electronic devices that process data and provide information to users.

[0509] A "multidimensional space" is a virtual environment that is digitally generated as an extension of reality and presents information visually or interactively.

[0510] "Means of presenting information" refers to a device or method that has the function of interpreting digital data and providing it to the user in a visual, auditory, or other form.

[0511] "User operation history" refers to a record of a series of interactions and actions performed by a user on an information processing device.

[0512] "Progress information" refers to data or indicators that show how far a user has progressed towards a certain goal.

[0513] "Personalized response" refers to information or feedback provided in a way that is tailored to the user's specific circumstances, needs, or characteristics.

[0514] "Interaction" refers to the exchange of data and information between a user and an information processing device, or between users themselves.

[0515] This invention realizes a learning support system using a multidimensional space via an information processing device. The server prepares the latest digital content and provides it to the user's terminal via the network. The user's terminal displays this content in the multidimensional space and records the user's operations in real time.

[0516] The server utilizes AI algorithms to analyze collected operation history and generate progress information. This progress information includes the user's level of understanding and work time, and based on this, it generates personalized responses and sends them to the user's device. This enables learners to learn efficiently at their own pace.

[0517] A concrete example of its use is when students in a history class use their devices to visualize the Egyptian pyramids in a multidimensional space, learning about their structure and historical background. During this process, if a student shows interest in a particular structure within the pyramids, this action is recorded as a log and analyzed on the server side. As a result, additional information and related questions about the areas of high interest are automatically presented.

[0518] Furthermore, by using a generative AI model, the information provided can be refined through prompts such as, "Generate an in-depth explanation of the Egyptian pyramids. The target audience is elementary school students." In this way, users can learn effectively and intuitively.

[0519] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0520] Step 1:

[0521] The terminal displays digital content received from the server in a multidimensional space. The input is content data from the server, which is then visually presented to the user using a display module. As an example, a 3D model of the Egyptian pyramids is displayed.

[0522] Step 2:

[0523] The user operates the device and interacts with information content in a multidimensional space. Input here consists of user actions such as taps and swipes, which the device detects with sensors, converts appropriately, and records as an operation history. For example, the user's action of zooming in on a part of a pyramid is recorded.

[0524] Step 3:

[0525] The terminal sends the collected operation history to the server. The input is the user's entire operation history, which the terminal compresses or encrypts and transfers to the server over the network. The server then receives the user's learned behavior data.

[0526] Step 4:

[0527] The server analyzes the received operation history using an AI algorithm. The input is user operation history data, and the AI ​​algorithm generates user progress information. Specifically, it evaluates the level of understanding based on operation frequency and viewing time.

[0528] Step 5:

[0529] The server generates personalized responses based on the analysis results. The input here is progress information, and it uses a generative AI model and prompts to generate personalized feedback, inserting additional information as needed. For example, if the user shows interest in a particular section, it provides historical context related to that section.

[0530] Step 6:

[0531] The server sends the generated feedback to the terminal. The input is individualized feedback, and the server converts it into a format that the terminal can understand and sends it again.

[0532] Step 7:

[0533] The device displays the received feedback to the user. The input is feedback data from the server, which the device visualizes and presents in a way that suits the user interface, and displays additional content as needed to facilitate further learning.

[0534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0535] This invention relates to a learning support system comprising a terminal used by learners, a server, and a network that connects them. This system incorporates an emotion engine that recognizes the learner's emotional state in real time and has the function of adjusting learning content and feedback based on the obtained emotional information.

[0536] Specifically, the server is responsible for managing the learning content, executing the emotion recognition algorithm, and generating feedback based on the emotion recognition results. The terminal functions as an interface for learners to interact with the system, using its camera and microphone to collect the data required by the emotion engine.

[0537] When a user begins learning, the device displays the provided learning content in an augmented reality (AR) space. At this time, the emotion engine analyzes the user's facial expressions and voice captured by the device's camera to determine the user's emotional state in real time. For example, if the user appears confused, the server will either lower the difficulty level of the learning content or generate feedback that provides more detailed explanations based on the analysis results. Conversely, if the user shows satisfaction or understanding, it may be possible to present more challenging tasks.

[0538] Alternatively, consider a scenario where the user is learning a history module. In this case, the device displays AR content that recreates historical events, while the emotion engine analyzes the user's reactions. If the server determines that the user is interested, it selects and delivers relevant content or additional information to the device to maintain their interest.

[0539] Thus, the present invention is a system that takes into account the emotional state of learners to realize a more personalized learning experience, thereby enabling improvements in learning efficiency and motivation.

[0540] The following describes the processing flow.

[0541] Step 1:

[0542] The server pre-prepares the learning content data and the algorithms necessary for emotion recognition, and loads all the data so that the device can access it. Any necessary updates are then delivered to the device.

[0543] Step 2:

[0544] The terminal launches the application and displays a list of modules to be offered to the learner. When the user selects a desired module, the terminal sends a request to the server and downloads and prepares the necessary content.

[0545] Step 3:

[0546] The user begins learning using the device. The device displays content combining multiple media formats in the AR space and prepares to utilize its camera and microphone to analyze the user's facial expressions and voice in real time through an emotion engine.

[0547] Step 4:

[0548] The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotional state. For example, it can detect facial expressions indicating decreased attention or satisfaction.

[0549] Step 5:

[0550] The device sends emotional data obtained from the emotion engine to the server. The server uses an AI model to analyze the emotional data and operation logs to understand the user's progress and level of understanding.

[0551] Step 6:

[0552] Based on the analysis results, the server generates feedback tailored to the user's current emotions and learning needs. In particular, if the server determines that understanding is insufficient, it prepares content that includes more detailed explanations.

[0553] Step 7:

[0554] The server sends the generated feedback to the device. The device applies the feedback to the user and displays new content or additional learning materials in the AR space.

[0555] Step 8:

[0556] Based on the feedback provided, users can adjust their learning approach as needed and continue learning further. This allows users to learn efficiently at their own pace.

[0557] (Example 2)

[0558] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0559] In today's learning environment, there is a demand for personalized educational experiences that consider not only learners' progress but also their emotional states. However, conventional systems struggle to effectively incorporate emotional information, and the generation of feedback tailored to each individual learner is not adequately realized. Therefore, solutions are needed to maximize learning efficiency and motivation.

[0560] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0561] In this invention, the server includes means for generating educational materials by combining multiple information media and displaying them in an augmented virtual space on a device used by learners; means for collecting learners' activity records and analyzing their progress and emotional information; and means for generating personalized feedback based on the analyzed progress and emotional information and providing it to the learners. This provides an educational experience that takes into account the learners' progress and emotional state, enabling improvements in learning efficiency and motivation.

[0562] The term "learner" refers to an individual who is receiving education or who is actively working to acquire skills.

[0563] A "device" refers to a collection of hardware and software configured to perform a specific function.

[0564] "Information media" refers to all forms of data used to transmit information, including text, images, audio, and video.

[0565] "Educational materials" refer to teaching materials and resources that learners use to acquire knowledge and skills.

[0566] "Augmented virtual space" refers to a technology that overlays digital information onto the physical real world, providing learners with an interactive experience.

[0567] "Activity records" refer to data that records the history of operations and actions performed by learners.

[0568] "Progress information" refers to data that serves as an indicator of how much content a learner has completed and to what extent they understand it during the learning process.

[0569] "Emotional information" refers to data that indicates the learner's emotional state, and typically refers to psychological state data analyzed from facial expressions, voice, etc.

[0570] "Analysis" refers to the process of handling data, extracting meaningful information, and deriving results.

[0571] "Individualized feedback" refers to responses and suggestions that are tailored to each learner's learning situation and emotional state.

[0572] This learning support system aims to provide a personalized educational experience by taking into account the learner's emotional state. A specific implementation is shown below.

[0573] Server Role

[0574] The server manages the learning content. Specifically, it organizes the diverse educational materials stored in the database and selects content suitable for the user. It also processes emotional data transmitted from the device using an emotion recognition algorithm and analyzes the learner's emotional state in real time. Furthermore, the server generates personalized feedback based on the analysis results. This feedback includes specific advice and supplementary information that takes into account the learner's current situation, generated using a generative AI model.

[0575] Terminal role

[0576] The terminal is a device that learners interact with, displaying educational materials as an augmented virtual space. This is typically achieved using AR technology. The terminal is equipped with a camera and microphone, which are used to record the learner's facial expressions and voice, and send emotional information to a server. The terminal also notifies the learner of the feedback sent from the server and displays adjusted educational tasks based on that feedback.

[0577] User interaction

[0578] Users learn by interacting with content displayed in an augmented virtual space through their devices. For example, they can visually experience AR content that recreates historical events or manipulate objects to solve problems. If a user shows confusion, the server uses analysis results to adjust the difficulty level or provide additional explanations. Also, if the user shows interest, more challenging tasks are presented.

[0579] Specific examples and prompt statements

[0580] As a concrete example, when a user is using the geography learning module, the device displays a terrain model in augmented reality (AR). Simultaneously, based on emotion recognition data, the server selects and presents relevant content to broaden the user's interests. An example of a prompt in this case would be, "If the user is interested in the terrain model, suggest relevant information that will allow him to further his learning."

[0581] This system provides a flexible learning experience that simultaneously considers the learner's emotions and progress, and is expected to improve the efficiency of education and enhance learning motivation.

[0582] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0583] Step 1:

[0584] The server analyzes the learner's profile data and selects appropriate learning content. This process receives progress information and past learning history from the database as input and performs data calculations to output the most suitable learning materials. Specifically, it filters content based on the learner's past performance and level of understanding.

[0585] Step 2:

[0586] The server transmits the selected learning content to the device. This requires converting the data to the format of the selected information medium, and delivering it in a format optimized for each device. The server receives content data as input, and the output is learning material converted to a format suitable for the device. Specifically, this includes packaging the content according to the device's display capabilities.

[0587] Step 3:

[0588] The device displays the received learning content in an augmented virtual space. The learning materials, as input, are processed within the device using AR technology, and a learning environment that the user can visually experience is output. Specifically, the camera and display are used to overlay digital information onto the actual physical space.

[0589] Step 4:

[0590] The device collects emotional information from the user's facial expressions and voice through its camera and microphone. It processes this input data, performs calculations to quantify the emotional state in real time, and generates emotional information as output. Specific operations include the use of facial recognition technology and voice analysis algorithms.

[0591] Step 5:

[0592] The server receives emotional information transmitted from the terminal and analyzes it using an emotional recognition algorithm. The emotional information received as input is used for data calculations to identify the user's psychological state, and the analysis results are generated as output. Specifically, a generative AI model classifies the emotional state.

[0593] Step 6:

[0594] The server generates personalized feedback based on the analysis results. This feedback generation utilizes progress and sentiment information as input and provides user-appropriate learning support messages as output. This includes using a generative AI model to automatically write out advice tailored to specific situations.

[0595] Step 7:

[0596] The server sends the generated feedback to the terminal, which then displays it to the user. Here, the feedback information is received as input and processed to output it in a user-friendly format. Specifically, this includes actions such as visual display on the screen and notifications via the voice assistant.

[0597] (Application Example 2)

[0598] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0599] There is a need to realize personalized learning that is tailored to each learner's level of understanding and interests. Conventional learning support systems do not adequately consider the learner's emotional state when responding or dynamically adjusting the learning environment, resulting in challenges in improving learning efficiency and maintaining motivation.

[0600] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0601] In this invention, the server includes means for generating complex data and displaying it in an augmented reality environment, means for collecting the learner's action history and analyzing its progress information, and means for evaluating the learner's facial features and acoustic data using an emotion analysis engine to determine the learner's emotional state. This enables dynamic adjustment of the individualized learning environment based on the learner's emotional state.

[0602] "Device" refers to all hardware used by learners for interaction.

[0603] "Materials" refers to data containing information and content for learners to study.

[0604] An "augmented reality environment" refers to an environment that uses technology to overlay virtual information onto the real world.

[0605] "Action history" refers to a record of a learner's operations and actions.

[0606] "Progress information" refers to data related to learners' learning status and progress.

[0607] "Analysis" refers to the process of evaluating collected data and extracting meaningful information.

[0608] "Individualized responses" refer to feedback and instructions that are tailored to the learner's characteristics and circumstances.

[0609] An "emotion analysis engine" refers to an algorithm or system used to recognize a learner's emotional state.

[0610] "Facial features" refer to facial characteristics and changes captured in real time.

[0611] "Acoustic data" refers to data related to the learner's voice and surrounding sounds.

[0612] "Emotional state" refers to the psychological and emotional condition exhibited by the learner.

[0613] The system that realizes this invention mainly consists of a server, a terminal used by learners, and an emotion analysis engine. The server generates and manages learning content and plays a central role in analyzing progress information and emotional states.

[0614] The device provides an interface with the learner and displays materials in an augmented reality environment. It collects the learner's facial features and acoustic data through its camera and microphone and sends it to an emotion analysis engine. The emotion analysis engine evaluates the learner's real-time emotional state and returns that information to the server.

[0615] Based on data received from the emotion analysis engine, the server generates personalized responses tailored to the learner's emotional state and dynamically adjusts the learning environment. Specifically, if a learner is confused, the server understands the situation and provides easier tasks or more detailed explanations. If the learner shows interest, it delivers relevant information and additional content to the device.

[0616] This invention makes it possible to provide learning experiences tailored to the emotional state of individual learners, thereby improving learning efficiency and motivation.

[0617] As a concrete example, consider a scenario where learners use augmented reality during a history lesson to visualize historical events. In this case, by analyzing the learners' emotional state and providing appropriate feedback and supplementary information, it is possible to deepen their understanding while maintaining their interest.

[0618] An example of a prompt for a generative AI model is: "Think about how a robot that supports children's learning should talk to a child who has become bored."

[0619] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0620] Step 1:

[0621] Once the device is powered on and the learner begins learning, it uses its camera and microphone to collect the learner's facial features and audio data in real time. The input data consists of camera video and audio data, which are sent to the emotion analysis engine.

[0622] Step 2:

[0623] The emotion analysis engine analyzes received facial features and acoustic data to determine the learner's current emotional state. A deep learning model is used in this process to output an "emotional state" from the input data. Emotional states include states such as concentration, confusion, and interest.

[0624] Step 3:

[0625] The server adjusts the content and responses provided to the learner based on emotional state data sent from the emotion analysis engine. For example, if the server determines that the learner is confused, it generates easier learning content or additional explanations. In this process, emotional state is used as input, and an adjusted response is generated as output.

[0626] Step 4:

[0627] The device displays pre-configured learning content and feedback sent from the server in an augmented reality environment. In this process, the device uses AR technology based on the provided data to create a composite display. The input is data from the server, and the output is the augmented reality content viewed by the learner.

[0628] Step 5:

[0629] When a user performs a new interaction, the operation log is recorded on the terminal and sent to the server as progress information. The server uses this progress information to prepare for further adjustments to future learning support. Here, the user operation log is used as input, and the analyzed progress information is output.

[0630] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0631] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0632] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0633] [Fourth Embodiment]

[0634] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0635] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0636] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0637] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0638] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0639] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0640] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0641] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0642] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0643] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0644] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0645] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0646] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] In this invention, the system consists of a terminal used by learners, a server, and a network connecting them. The terminal has the function of displaying learning content using AR technology and recording user operations. The server is responsible for distributing learning content, analyzing learner operation logs, and generating feedback.

[0648] Specifically, the server provides the device with the latest learning content data, and the device uses this data to display the content to the learner in an augmented reality (AR) space. The user progresses through the learning process, making operations and selections through the AR content presented via the device. The device records these operations as logs and sends them to the server.

[0649] The server uses an AI algorithm to analyze the operation logs sent by the user and generate progress information. This progress information includes the learner's level of understanding and viewing time. Based on this information, the server generates personalized feedback for each learner and sends it back to the terminal.

[0650] The device displays the feedback received to the user and, if necessary, presents additional learning content and practice problems. This allows learners to engage in learning that is optimally tailored to their own learning progress.

[0651] For example, a user studying history can learn by viewing 3D models of historical events in an augmented reality (AR) space through their device. When the user explores points they don't understand, their activity log is analyzed on the server, and if it's determined that their understanding is insufficient, additional explanations and practice problems related to that section are displayed on the device. This allows the user to efficiently progress through their learning while resolving any points of confusion.

[0652] Thus, the present invention is a system that provides learners with intuitive and personalized learning support, enabling improvements in learning efficiency and comprehension.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] The server loads the learning content data and AI analysis algorithms, preparing them for access by the devices. Necessary updates are then distributed from the server to all devices used by the user.

[0656] Step 2:

[0657] The device launches the application and establishes communication with the server. It downloads content corresponding to the learning module selected by the user from the server and caches it on the device.

[0658] Step 3:

[0659] The user checks the available learning modules on their device and selects the desired module. Based on that selection, the device prepares to display the learning content in the AR space.

[0660] Step 4:

[0661] The device displays augmented reality content by overlaying it onto real-world images using its camera function. Audio guides are played automatically, and text information, images, and videos are visually provided as related information.

[0662] Step 5:

[0663] Users interact with objects displayed in the AR space. For example, they can select specific objects or use zoom functions to view details.

[0664] Step 6:

[0665] The device meticulously records user activity logs. These logs include object selection history and time spent viewing learning content.

[0666] Step 7:

[0667] Operation logs sent from the terminal are aggregated on the server. The server uses an AI analysis algorithm to analyze the received data and generate learner progress information.

[0668] Step 8:

[0669] Based on progress information generated on the server, personalized feedback is created for the learner. This progress information includes details about content that the learner doesn't fully understand and topics that require further study.

[0670] Step 9:

[0671] The server sends the generated feedback back to the terminal. The terminal displays the feedback to the user in an easily understandable format, such as additional exercises or videos with detailed explanations.

[0672] Step 10:

[0673] Users revisit their learning based on feedback provided by their device. This enables efficient and effective learning tailored to individual progress.

[0674] (Example 1)

[0675] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] In today's learning environment, the lack of individualized support tailored to each learner's progress and level of understanding hinders efficient learning. Furthermore, existing educational systems often lack sufficient intuitive interfaces for learners to engage in augmented reality learning through concrete actions. This makes it difficult to maintain learners' interest and can lead to a decline in motivation.

[0677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0678] In this invention, the server includes means for generating information by combining multiple information media and displaying it in an augmented reality space in an information processing device used by learners; means for collecting the learner's operation history and analyzing its progress information; means for generating personalized instructional information based on the analyzed progress information and providing it to the learner; and means for displaying additional exercises and explanations to the learner based on the generated instructional information. This enables optimal learning support according to the learner's progress, improving learning efficiency and maintaining learning motivation.

[0679] An "information processing device" is an electronic device used for inputting, processing, and outputting data, and provides an interface for users to interact with learning content.

[0680] An "information medium" is a means of storing and presenting information in forms such as text, images, audio, and video.

[0681] Augmented reality is a virtual space that provides users with new experiences by overlaying computer-generated information onto the real world environment.

[0682] "Operation history" refers to recorded data about operations performed by a user using an information processing device, and is used to track the progress of learning activities.

[0683] "Progress information" refers to information that indicates the user's learning status and level of understanding, and is an analysis result generated based on records of learning activities.

[0684] "Personalized instructional information" refers to customized advice and explanatory information provided based on the user's progress and tailored to their learning needs.

[0685] "Additional exercises and explanations" are supplementary learning materials and problem sets provided to deepen the user's understanding and complement existing learning content.

[0686] This invention is a learning support system that uses augmented reality technology and consists of a server, terminals, and a network connecting them.

[0687] The server is a central information processing device for managing and distributing learning content and analyzing learners' activity history. The server contains various learning content stored in a database, from which the latest information is sent to the terminal. The server uses AI algorithms to collect activity history and generate progress information. This analysis algorithm employs a generative AI model. Furthermore, based on the analyzed progress information, it creates personalized instructional information and provides it back to the terminal.

[0688] The terminal is an information processing device for displaying learning content received from the server in an augmented reality space. The terminal is equipped with the necessary hardware to build an AR environment, combining a camera and a display. The terminal records user actions and transmits them to the server. Dedicated AR applications and libraries are installed on the terminal, enabling, for example, the display of historical 3D models integrating multiple information sources.

[0689] The user is someone who engages in educational activities through augmented reality content displayed on their device. The user can manipulate 3D models and explore information using their fingers or specific input devices. This allows the user to intuitively learn through the provided content while fulfilling their role as a learner.

[0690] As a concrete example, when a user is learning history, they can access 3D models that recreate historical events in an AR space displayed on their device. For instance, they can visually experience what a city looked like during the Edo period. Furthermore, operations performed through the device are recorded as points of confusion and analyzed on the server. If it is determined that the user's understanding is insufficient, additional explanations or practice problems are displayed on the device.

[0691] Example of a prompt:

[0692] "Please describe how a system using AR technology allows users to manipulate 3D models during history lessons. Include how the learning content adapts to the user's actions."

[0693] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0694] Step 1:

[0695] The server retrieves the latest content data from the learning content database and delivers this data to the terminal. Specifically, the server extracts the requested information (for example, a 3D model related to a historical topic) and sends it to the terminal via the network. The input in this process is the request, and the output is the corresponding content data.

[0696] Step 2:

[0697] The device uses learning content data received from the server to run an AR application and display the content to the learner in an augmented reality space. For example, a 3D model of a traditional Japanese city is overlaid and displayed on the device's display via the camera. In this process, the input is the content data, and the output is the displayed AR content.

[0698] Step 3:

[0699] Users interact with content displayed in an AR space on their device to advance their learning. Here, users rotate and zoom 3D models using screen touches and gesture input. This allows users to interact with the learning content. The input for this interaction is the user's physical actions, and the output is a visual change within the AR space.

[0700] Step 4:

[0701] The terminal records user actions as an operation log and sends it to the server. The terminal records all user actions along with their time, converts them into a data format, and sends them to the server. The input here is the user's operation data, and the output is the data accumulated as the operation log.

[0702] Step 5:

[0703] The server analyzes the received operation logs using an AI algorithm to generate learner progress information. This analysis process includes, for example, analyzing operation frequency and viewing time for specific content. The input is the operation log, and the output is learner progress information (such as comprehension level and study time).

[0704] Step 6:

[0705] The server generates user-optimized instructional information based on progress data and sends it to the terminal. This process, for example, recommends supplementary materials for areas where understanding is insufficient. The input is progress data, and the output is instructional information and additional learning content.

[0706] Step 7:

[0707] The terminal displays additional learning content and practice problems to the user based on instructional information received from the server. Specific examples include displaying supplementary explanatory videos or practice problems for topics the user doesn't fully understand. Here, the input is instructional information, and the output is new content presented to the learner.

[0708] (Application Example 1)

[0709] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0710] In the field of information learning, there is a need to provide efficient and personalized learning utilizing multidimensional spaces. In particular, a challenge lies in the lack of systems that allow learners to instantly receive feedback and additional content tailored to their level of understanding and progress. Furthermore, it is necessary to create an environment where learners can interact within a multidimensional space and efficiently acquire specific learning tasks.

[0711] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0712] In this invention, the server includes means for presenting information in a multidimensional space, means for collecting the user's operation history and analyzing its progress information, and means for generating and providing personalized responses to the user based on the analyzed progress information. This enables learners to efficiently receive feedback and additional informational content tailored to their own progress.

[0713] An "information processing device" is a general term for electronic devices that process data and provide information to users.

[0714] A "multidimensional space" is a virtual environment that is digitally generated as an extension of reality and presents information visually or interactively.

[0715] "Means of presenting information" refers to a device or method that has the function of interpreting digital data and providing it to the user in a visual, auditory, or other form.

[0716] "User operation history" refers to a record of a series of interactions and actions performed by a user on an information processing device.

[0717] "Progress information" refers to data or indicators that show how far a user has progressed towards a certain goal.

[0718] "Personalized response" refers to information or feedback provided in a way that is tailored to the user's specific circumstances, needs, or characteristics.

[0719] "Interaction" refers to the exchange of data and information between a user and an information processing device, or between users themselves.

[0720] This invention realizes a learning support system using a multidimensional space via an information processing device. The server prepares the latest digital content and provides it to the user's terminal via the network. The user's terminal displays this content in the multidimensional space and records the user's operations in real time.

[0721] The server utilizes AI algorithms to analyze collected operation history and generate progress information. This progress information includes the user's level of understanding and work time, and based on this, it generates personalized responses and sends them to the user's device. This enables learners to learn efficiently at their own pace.

[0722] A concrete example of its use is when students in a history class use their devices to visualize the Egyptian pyramids in a multidimensional space, learning about their structure and historical background. During this process, if a student shows interest in a particular structure within the pyramids, this action is recorded as a log and analyzed on the server side. As a result, additional information and related questions about the areas of high interest are automatically presented.

[0723] Furthermore, by using a generative AI model, the information provided can be refined through prompts such as, "Generate an in-depth explanation of the Egyptian pyramids. The target audience is elementary school students." In this way, users can learn effectively and intuitively.

[0724] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0725] Step 1:

[0726] The terminal displays digital content received from the server in a multidimensional space. The input is content data from the server, which is then visually presented to the user using a display module. As an example, a 3D model of the Egyptian pyramids is displayed.

[0727] Step 2:

[0728] The user operates the device and interacts with information content in a multidimensional space. Input here consists of user actions such as taps and swipes, which the device detects with sensors, converts appropriately, and records as an operation history. For example, the user's action of zooming in on a part of a pyramid is recorded.

[0729] Step 3:

[0730] The terminal sends the collected operation history to the server. The input is the user's entire operation history, which the terminal compresses or encrypts and transfers to the server over the network. The server then receives the user's learned behavior data.

[0731] Step 4:

[0732] The server analyzes the received operation history using an AI algorithm. The input is user operation history data, and the AI ​​algorithm generates user progress information. Specifically, it evaluates the level of understanding based on operation frequency and viewing time.

[0733] Step 5:

[0734] The server generates personalized responses based on the analysis results. The input here is progress information, and it uses a generative AI model and prompts to generate personalized feedback, inserting additional information as needed. For example, if the user shows interest in a particular section, it provides historical context related to that section.

[0735] Step 6:

[0736] The server sends the generated feedback to the terminal. The input is individualized feedback, and the server converts it into a format that the terminal can understand and sends it again.

[0737] Step 7:

[0738] The device displays the received feedback to the user. The input is feedback data from the server, which the device visualizes and presents in a way that suits the user interface, and displays additional content as needed to facilitate further learning.

[0739] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0740] This invention relates to a learning support system comprising a terminal used by learners, a server, and a network that connects them. This system incorporates an emotion engine that recognizes the learner's emotional state in real time and has the function of adjusting learning content and feedback based on the obtained emotional information.

[0741] Specifically, the server is responsible for managing the learning content, executing the emotion recognition algorithm, and generating feedback based on the emotion recognition results. The terminal functions as an interface for learners to interact with the system, using its camera and microphone to collect the data required by the emotion engine.

[0742] When a user begins learning, the device displays the provided learning content in an augmented reality (AR) space. At this time, the emotion engine analyzes the user's facial expressions and voice captured by the device's camera to determine the user's emotional state in real time. For example, if the user appears confused, the server will either lower the difficulty level of the learning content or generate feedback that provides more detailed explanations based on the analysis results. Conversely, if the user shows satisfaction or understanding, it may be possible to present more challenging tasks.

[0743] Alternatively, consider a scenario where the user is learning a history module. In this case, the device displays AR content that recreates historical events, while the emotion engine analyzes the user's reactions. If the server determines that the user is interested, it selects and delivers relevant content or additional information to the device to maintain their interest.

[0744] Thus, the present invention is a system that takes into account the emotional state of learners to realize a more personalized learning experience, thereby enabling improvements in learning efficiency and motivation.

[0745] The following describes the processing flow.

[0746] Step 1:

[0747] The server pre-prepares the learning content data and the algorithms necessary for emotion recognition, and loads all the data so that the device can access it. Any necessary updates are then delivered to the device.

[0748] Step 2:

[0749] The terminal launches the application and displays a list of modules to be offered to the learner. When the user selects a desired module, the terminal sends a request to the server and downloads and prepares the necessary content.

[0750] Step 3:

[0751] The user begins learning using the device. The device displays content combining multiple media formats in the AR space and prepares to utilize its camera and microphone to analyze the user's facial expressions and voice in real time through an emotion engine.

[0752] Step 4:

[0753] The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotional state. For example, it can detect facial expressions indicating decreased attention or satisfaction.

[0754] Step 5:

[0755] The device sends emotional data obtained from the emotion engine to the server. The server uses an AI model to analyze the emotional data and operation logs to understand the user's progress and level of understanding.

[0756] Step 6:

[0757] Based on the analysis results, the server generates feedback tailored to the user's current emotions and learning needs. In particular, if the server determines that understanding is insufficient, it prepares content that includes more detailed explanations.

[0758] Step 7:

[0759] The server sends the generated feedback to the device. The device applies the feedback to the user and displays new content or additional learning materials in the AR space.

[0760] Step 8:

[0761] Based on the feedback provided, users can adjust their learning approach as needed and continue learning further. This allows users to learn efficiently at their own pace.

[0762] (Example 2)

[0763] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0764] In today's learning environment, there is a demand for personalized educational experiences that consider not only learners' progress but also their emotional states. However, conventional systems struggle to effectively incorporate emotional information, and the generation of feedback tailored to each individual learner is not adequately realized. Therefore, solutions are needed to maximize learning efficiency and motivation.

[0765] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0766] In this invention, the server includes means for generating educational materials by combining multiple information media and displaying them in an augmented virtual space on a device used by learners; means for collecting learners' activity records and analyzing their progress and emotional information; and means for generating personalized feedback based on the analyzed progress and emotional information and providing it to the learners. This provides an educational experience that takes into account the learners' progress and emotional state, enabling improvements in learning efficiency and motivation.

[0767] The term "learner" refers to an individual who is receiving education or who is actively working to acquire skills.

[0768] A "device" refers to a collection of hardware and software configured to perform a specific function.

[0769] "Information media" refers to all forms of data used to transmit information, including text, images, audio, and video.

[0770] "Educational materials" refer to teaching materials and resources that learners use to acquire knowledge and skills.

[0771] "Augmented virtual space" refers to a technology that overlays digital information onto the physical real world, providing learners with an interactive experience.

[0772] "Activity records" refer to data that records the history of operations and actions performed by learners.

[0773] "Progress information" refers to data that serves as an indicator of how much content a learner has completed and to what extent they understand it during the learning process.

[0774] "Emotional information" refers to data that indicates the learner's emotional state, and typically refers to psychological state data analyzed from facial expressions, voice, etc.

[0775] "Analysis" refers to the process of handling data, extracting meaningful information, and deriving results.

[0776] "Individualized feedback" refers to responses and suggestions that are tailored to each learner's learning situation and emotional state.

[0777] This learning support system aims to provide a personalized educational experience by taking into account the learner's emotional state. A specific implementation is shown below.

[0778] Server Role

[0779] The server manages the learning content. Specifically, it organizes the diverse educational materials stored in the database and selects content suitable for the user. It also processes emotional data transmitted from the device using an emotion recognition algorithm and analyzes the learner's emotional state in real time. Furthermore, the server generates personalized feedback based on the analysis results. This feedback includes specific advice and supplementary information that takes into account the learner's current situation, generated using a generative AI model.

[0780] Terminal role

[0781] The terminal is a device that learners interact with, displaying educational materials as an augmented virtual space. This is typically achieved using AR technology. The terminal is equipped with a camera and microphone, which are used to record the learner's facial expressions and voice, and send emotional information to a server. The terminal also notifies the learner of the feedback sent from the server and displays adjusted educational tasks based on that feedback.

[0782] User interaction

[0783] Users learn by interacting with content displayed in an augmented virtual space through their devices. For example, they can visually experience AR content that recreates historical events or manipulate objects to solve problems. If a user shows confusion, the server uses analysis results to adjust the difficulty level or provide additional explanations. Also, if the user shows interest, more challenging tasks are presented.

[0784] Specific examples and prompt statements

[0785] As a concrete example, when a user is using the geography learning module, the device displays a terrain model in augmented reality (AR). Simultaneously, based on emotion recognition data, the server selects and presents relevant content to broaden the user's interests. An example of a prompt in this case would be, "If the user is interested in the terrain model, suggest relevant information that will allow him to further his learning."

[0786] This system provides a flexible learning experience that simultaneously considers the learner's emotions and progress, and is expected to improve the efficiency of education and enhance learning motivation.

[0787] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0788] Step 1:

[0789] The server analyzes the learner's profile data and selects appropriate learning content. This process receives progress information and past learning history from the database as input and performs data calculations to output the most suitable learning materials. Specifically, it filters content based on the learner's past performance and level of understanding.

[0790] Step 2:

[0791] The server transmits the selected learning content to the device. This requires converting the data to the format of the selected information medium, and delivering it in a format optimized for each device. The server receives content data as input, and the output is learning material converted to a format suitable for the device. Specifically, this includes packaging the content according to the device's display capabilities.

[0792] Step 3:

[0793] The device displays the received learning content in an augmented virtual space. The learning materials, as input, are processed within the device using AR technology, and a learning environment that the user can visually experience is output. Specifically, the camera and display are used to overlay digital information onto the actual physical space.

[0794] Step 4:

[0795] The device collects emotional information from the user's facial expressions and voice through its camera and microphone. It processes this input data, performs calculations to quantify the emotional state in real time, and generates emotional information as output. Specific operations include the use of facial recognition technology and voice analysis algorithms.

[0796] Step 5:

[0797] The server receives emotional information transmitted from the terminal and analyzes it using an emotional recognition algorithm. The emotional information received as input is used for data calculations to identify the user's psychological state, and the analysis results are generated as output. Specifically, a generative AI model classifies the emotional state.

[0798] Step 6:

[0799] The server generates personalized feedback based on the analysis results. This feedback generation utilizes progress and sentiment information as input and provides user-appropriate learning support messages as output. This includes using a generative AI model to automatically write out advice tailored to specific situations.

[0800] Step 7:

[0801] The server sends the generated feedback to the terminal, which then displays it to the user. Here, the feedback information is received as input and processed to output it in a user-friendly format. Specifically, this includes actions such as visual display on the screen and notifications via the voice assistant.

[0802] (Application Example 2)

[0803] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0804] There is a need to realize personalized learning that is tailored to each learner's level of understanding and interests. Conventional learning support systems do not adequately consider the learner's emotional state when responding or dynamically adjusting the learning environment, resulting in challenges in improving learning efficiency and maintaining motivation.

[0805] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0806] In this invention, the server includes means for generating complex data and displaying it in an augmented reality environment, means for collecting the learner's action history and analyzing its progress information, and means for evaluating the learner's facial features and acoustic data using an emotion analysis engine to determine the learner's emotional state. This enables dynamic adjustment of the individualized learning environment based on the learner's emotional state.

[0807] "Device" refers to all hardware used by learners for interaction.

[0808] "Materials" refers to data containing information and content for learners to study.

[0809] An "augmented reality environment" refers to an environment that uses technology to overlay virtual information onto the real world.

[0810] "Action history" refers to a record of a learner's operations and actions.

[0811] "Progress information" refers to data related to learners' learning status and progress.

[0812] "Analysis" refers to the process of evaluating collected data and extracting meaningful information.

[0813] "Individualized responses" refer to feedback and instructions that are tailored to the learner's characteristics and circumstances.

[0814] An "emotion analysis engine" refers to an algorithm or system used to recognize a learner's emotional state.

[0815] "Facial features" refer to facial characteristics and changes captured in real time.

[0816] "Acoustic data" refers to data related to the learner's voice and surrounding sounds.

[0817] "Emotional state" refers to the psychological and emotional condition exhibited by the learner.

[0818] The system that realizes this invention mainly consists of a server, a terminal used by learners, and an emotion analysis engine. The server generates and manages learning content and plays a central role in analyzing progress information and emotional states.

[0819] The device provides an interface with the learner and displays materials in an augmented reality environment. It collects the learner's facial features and acoustic data through its camera and microphone and sends it to an emotion analysis engine. The emotion analysis engine evaluates the learner's real-time emotional state and returns that information to the server.

[0820] Based on data received from the emotion analysis engine, the server generates personalized responses tailored to the learner's emotional state and dynamically adjusts the learning environment. Specifically, if a learner is confused, the server understands the situation and provides easier tasks or more detailed explanations. If the learner shows interest, it delivers relevant information and additional content to the device.

[0821] This invention makes it possible to provide learning experiences tailored to the emotional state of individual learners, thereby improving learning efficiency and motivation.

[0822] As a concrete example, consider a scenario where learners use augmented reality during a history lesson to visualize historical events. In this case, by analyzing the learners' emotional state and providing appropriate feedback and supplementary information, it is possible to deepen their understanding while maintaining their interest.

[0823] An example of a prompt for a generative AI model is: "Think about how a robot that supports children's learning should talk to a child who has become bored."

[0824] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0825] Step 1:

[0826] Once the device is powered on and the learner begins learning, it uses its camera and microphone to collect the learner's facial features and audio data in real time. The input data consists of camera video and audio data, which are sent to the emotion analysis engine.

[0827] Step 2:

[0828] The emotion analysis engine analyzes received facial features and acoustic data to determine the learner's current emotional state. A deep learning model is used in this process to output an "emotional state" from the input data. Emotional states include states such as concentration, confusion, and interest.

[0829] Step 3:

[0830] The server adjusts the content and responses provided to the learner based on emotional state data sent from the emotion analysis engine. For example, if the server determines that the learner is confused, it generates easier learning content or additional explanations. In this process, emotional state is used as input, and an adjusted response is generated as output.

[0831] Step 4:

[0832] The device displays pre-configured learning content and feedback sent from the server in an augmented reality environment. In this process, the device uses AR technology based on the provided data to create a composite display. The input is data from the server, and the output is the augmented reality content viewed by the learner.

[0833] Step 5:

[0834] When a user performs a new interaction, the operation log is recorded on the terminal and sent to the server as progress information. The server uses this progress information to prepare for further adjustments to future learning support. Here, the user operation log is used as input, and the analyzed progress information is output.

[0835] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0836] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0837] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0838] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0839] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0840] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0841] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0842] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0843] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0844] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0845] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0846] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0847] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0848] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0849] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0850] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0851] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0852] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0853] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0854] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0855] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0856] The following is further disclosed regarding the embodiments described above.

[0857] (Claim 1)

[0858] A means for generating content that combines multiple media on a device used by learners and displaying it in an augmented reality space,

[0859] A means of collecting learner operation logs and analyzing their progress information,

[0860] A means of generating and providing personalized feedback to learners based on analyzed progress information,

[0861] A system that includes this.

[0862] (Claim 2)

[0863] The system according to claim 1, which automatically selects and provides additional learning content and materials to learners based on analyzed progress information.

[0864] (Claim 3)

[0865] The system according to claim 1, which enables learners to access specific learning tasks through interaction in an augmented reality space.

[0866] "Example 1"

[0867] (Claim 1)

[0868] In an information processing device used by learners, a means for generating information by combining multiple information media and displaying it in an augmented reality space,

[0869] A means for collecting learners' operation history and analyzing their progress information,

[0870] A means of generating personalized instructional information based on analyzed progress data and providing it to learners,

[0871] A means of displaying additional exercises and explanations to learners based on the generated instructional information,

[0872] ...

[0873] A system that includes this.

[0874] (Claim 2)

[0875] The system according to claim 1, which automatically selects and provides additional educational content and materials to learners based on analyzed progress information.

[0876] (Claim 3)

[0877] The system according to claim 1, which enables learners to access specific learning tasks through user interaction within an augmented reality space.

[0878] "Application Example 1"

[0879] (Claim 1)

[0880] In an information processing device, means for presenting information in a multidimensional space,

[0881] A means for collecting user operation history and analyzing the progress information,

[0882] A means for generating and providing personalized responses to users based on analyzed progress information,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, which automatically selects and provides additional information content and materials to the user based on the analyzed progress information.

[0886] (Claim 3)

[0887] The system according to claim 1, which enables users to access specific informational tasks through interaction in a multidimensional space.

[0888] "Example 2 of combining an emotion engine"

[0889] (Claim 1)

[0890] A device used by learners provides means for generating educational materials that combine multiple information media and displaying them in an augmented virtual space,

[0891] A means for collecting learners' activity records and analyzing their progress and emotional information,

[0892] A means of generating and providing personalized feedback to learners based on analyzed progress and emotional information,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, which automatically selects and provides additional educational materials and information to learners based on analyzed progress information and emotional information.

[0896] (Claim 3)

[0897] The system according to claim 1, which enables learners to access specific educational tasks through interaction within an augmented virtual space.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A device used by learners includes means for generating complex materials and displaying them in an augmented reality environment,

[0901] A means of collecting learners' action history and analyzing their progress information,

[0902] A means of generating and providing individualized responses to learners based on analyzed progress information,

[0903] A means of evaluating the learner's facial features and acoustic data using an emotion analysis engine to determine the learner's emotional state,

[0904] A means of dynamically adjusting the learning environment and responses based on the determined emotional state,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, which automatically selects and provides to the learner supplementary learning materials and information based on analyzed progress information and determined emotional state.

[0908] (Claim 3)

[0909] The system according to claim 1, which enables learners to connect to specific learning tasks through interaction within an augmented reality environment. [Explanation of Symbols]

[0910] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for generating content that combines multiple media on a device used by learners and displaying it in an augmented reality space, A means of collecting learner operation logs and analyzing their progress information, A means of generating and providing personalized feedback to learners based on analyzed progress information, A system that includes this.

2. The system according to claim 1, which automatically selects and provides additional learning content and materials to learners based on analyzed progress information.

3. The system according to claim 1, which enables learners to access specific learning tasks through interaction in an augmented reality space.

Images