system

Abstract

We provide the system. [Solution] Means for providing an interface for the user to select a specific historical event, A means of sending a data request to a server regarding a selected historical event, A means of obtaining information related to the relevant historical event from a database, A means of processing acquired information, colorizing it, visualizing it, and creating a VR version, A means of streaming processed data to a terminal in real time, A means for users to experience and interact with historical events through VR goggles, A means of sending user feedback to the server, A system that includes a server to analyze feedback and implement measures to improve the learning experience.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] History and past events are important elements of education, but in conventional methods, understanding often does not deepen simply by reading textbooks and materials. Furthermore, many of those materials are monochrome images or old videos, lacking visual appeal and making it difficult to attract the interest of students and the general public. As a result, there is a problem that the importance of history and the awareness of peace cannot be fully cultivated. There is a need for a system that can utilize modern technology to learn history more intuitively and experientially.

Means for Solving the Problems

[0005] To solve this problem, the present invention provides the following means: means for providing an interface for a user to select a specific historical event; means for sending a data request to a server regarding the selected historical event; means for obtaining information related to the relevant historical event from a database; means for processing the obtained information, colorizing it, visualizing it, and creating a VR version; means for streaming the processed data to a terminal in real time; means for the user to experience and interact with the historical event through VR goggles; means for sending user feedback to the server; and means for the server to analyze the feedback and improve the learning experience. This makes it possible to learn history visually and interactively and to cultivate an awareness of peace.

[0006] An "interface" is a screen or means of operation that allows a user to select a specific historical event.

[0007] A "server" is a computer system that receives data requests, retrieves related information from a database, and processes it.

[0008] A "database" is an information management system used to structure and store information related to historical events.

[0009] "Colorization" is a process that adds color to monochrome images and videos, and it is a technology that allows for the creation of visually appealing content.

[0010] "Visualization" is a method of converting abstract information or data into a format that is easy to understand visually.

[0011] "VR conversion" is a method that uses virtual reality technology to allow users to experience historical events of their choice in a three-dimensional virtual environment.

[0012] "Streaming" is a technology that transmits data processed in real time to a device and displays it continuously.

[0013] A "terminal" is a device used by a user, and is a piece of equipment that provides interface operation and VR experiences.

[0014] A "VR headset" is a device used by users to visually experience a virtual reality environment.

[0015] "Feedback" refers to the opinions and impressions that users provide after experiencing a system, and is used to improve the system.

[0016] "Analysis" is a means of analyzing collected feedback data, extracting insights, and improving the learning experience.

[0017] Artificial intelligence is a technology that learns from large amounts of data and generates colorized images and videos. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Further, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention provides a system that allows users to learn about historical events intuitively and experientially. The program for this system operates in the following steps.

[0040] First, the application is launched on the user's device. The application displays a screen through its user interface where the user can select a specific historical event. For example, the user might select "World War II" and then "The Normandy landings."

[0041] Next, the terminal receives the user's selection and sends that information to the server as a data request. The request includes event identification information and the required data type (e.g., text information, image, video, 3D model, etc.).

[0042] The server receives a request and retrieves information related to the relevant historical event from its internal database. The retrieved information includes text, images, videos, and 3D model data. The server also uses AI technology to colorize monochrome images and videos and convert them into a visually recognizable format.

[0043] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server streams the generated data package to the device in real time.

[0044] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. Users wear VR goggles and "present" on the beach where the Normandy landings took place, experiencing historical events in real time. Interactive elements allow users to explore different perspectives and details. For example, users can "approach" specific tanks or soldiers and view their details.

[0045] After the experience, users enter feedback within the application. This feedback includes their impressions of the experience, suggestions for improvement, and information they would like to see added.

[0046] The device sends user feedback to the server. The server receives this feedback, analyzes the data, and extracts insights to improve the next learning experience.

[0047] As a concrete example, consider the case where a user selects "the Normandy landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings by wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[0048] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0049] The following describes the processing flow.

[0050] Step 1:

[0051] The user launches the application on their device and checks the home screen. The application then performs the necessary initial setup.

[0052] Step 2:

[0053] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0054] Step 3:

[0055] The terminal receives the user's selection information and sends a data request containing that information to the server.

[0056] Step 4:

[0057] The server receives a data request. The request includes event identification information and the required data type.

[0058] Step 5:

[0059] The server searches its internal database for and retrieves information related to the relevant historical event. This information includes text, images, videos, and 3D models.

[0060] Step 6:

[0061] The server uses AI technology to colorize images and videos it has acquired. Specifically, it analyzes monochrome video frame by frame and adds color to it.

[0062] Step 7:

[0063] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[0064] Step 8:

[0065] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[0066] Step 9:

[0067] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0068] Step 10:

[0069] The server streams the generated data package to the terminal in real time.

[0070] Step 11:

[0071] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0072] Step 12:

[0073] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[0074] Step 13:

[0075] After the user finishes their experience, a screen will appear where they can provide feedback, including their thoughts and suggestions for improvement.

[0076] Step 14:

[0077] The device receives user feedback and sends it to the server.

[0078] Step 15:

[0079] The server analyzes user feedback data to extract insights for improving the next learning experience.

[0080] Step 16:

[0081] The server updates the system based on the analysis results, improving the quality of the learning experience. For example, improvements such as adding new interactive elements may be made.

[0082] (Example 1)

[0083] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] Traditional history education systems rely solely on text information and static images, failing to provide users with sufficient opportunities to learn about past events visually and experientially. This limits learners' interest and understanding, making it particularly difficult for younger generations to deepen their appreciation for the importance of history and their awareness of peace. Furthermore, static data alone is insufficient to meet the diverse learning needs of users, preventing the provision of interactive learning experiences.

[0085] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0086] In this invention, the server includes means for providing an interface for a user to select a specific past event; means for sending a data request to the server regarding the selected past event; means for obtaining information related to the relevant past event from an information aggregate; means for processing, colorizing, visualizing, and virtualizing the obtained information; means for sequentially transmitting the processed data to an information processing device in real time; means for the user to experience and interact with the past event through a virtual reality device; means for sending user feedback to the server; and means for the server to analyze the feedback and improve the learning experience. As a result, the user can learn about past events visually and experientially, deepen their interest in and understanding of learning, and deepen their awareness of the importance of history and peace through an interactive learning experience.

[0087] "Specific past events" refer to historically significant events or occurrences that users can learn about or experience.

[0088] "Interface" refers to the means by which a user interacts with a system, and includes menus, buttons, and other input devices on the screen.

[0089] A "data request" refers to the process of a user requesting information about a past event from a server.

[0090] An "information repository" refers to a database or other information system that stores data related to past events.

[0091] "Colorization" refers to the process of adding color to monochrome images or videos, making them easier to recognize visually.

[0092] "Visualization" refers to the process of displaying acquired information in the form of charts, graphs, 3D models, and other similar formats.

[0093] "Virtualization" refers to the process of transforming information into a virtual reality environment, allowing users to experience it in an immersive way.

[0094] "Information processing device" refers to a terminal used for processing and displaying data, and includes computers, smartphones, tablets, etc.

[0095] "Sequential transmission" refers to the process of continuously transmitting data in real time.

[0096] "Virtual reality equipment" refers to devices used by users to experience a virtual reality environment, and includes VR goggles, headsets, and the like.

[0097] "Interactive manipulation" refers to users interacting with objects and scenes within a virtual environment.

[0098] "Feedback" refers to the opinions and impressions that users provide regarding their experiences or learning content.

[0099] "Analysis" refers to the process of compiling and evaluating collected feedback to extract insights for improving the next learning experience.

[0100] This invention is a system for allowing users to visually and experientially learn about important past events. This system is based on the interaction between a server, a terminal, and a user, and uses virtual reality technology to recreate historical events. The specific method for carrying out this invention is described below.

[0101] Hardware and software to be used

[0102] terminal

[0103] A terminal is a device that a user uses to launch and operate applications. Typically, a smartphone, tablet, or personal computer is used. Applications running on a terminal allow the user to select specific past events through a user interface.

[0104] server

[0105] A server is a central information processing unit that processes data requests from users and uses the acquired data to generate virtual reality experiences. Servers are equipped with high-performance processors and can process large amounts of data in real time.

[0106] Virtual reality device

[0107] Virtual reality devices are devices used by users to experience a virtual reality environment. Typical examples include VR goggles and headsets. This allows users to become visually immersed and engage in interactive activities.

[0108] Data processing and data calculation

[0109] Colorization and visualization

[0110] The server colorizes the acquired monochrome images and videos and converts them into a visually recognizable format. This is done using a generative AI model based on deep learning technology. For example, image processing libraries such as OpenCV or PIL can be used.

[0111] Virtual reality

[0112] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. High-performance game engines such as Unity or Unreal Engine are used to generate the 3D models.

[0113] Real-time streaming

[0114] The server streams the generated data package to the terminal in real time. This is achieved using streaming protocols such as WebRTC and RTMP.

[0115] Specific example

[0116] For example, if the user selects "The Normandy landings"

[0117] 1. The user launches the application and selects "World War II" and "Normandy landings".

[0118] 2. The server collects data about the selected event and generates colorized images, videos, and 3D models as needed.

[0119] 3. The server streams the generated data package to the terminal in real time.

[0120] 4. Users wear VR goggles and "be present" on the beach where the Normandy landings took place, experiencing the historical event in real time.

[0121] 5. Users provide feedback after completing the experience, and the server analyzes this information to improve the quality of the user experience.

[0122] Examples of prompt statements for a generative AI model are as follows:

[0123] Please provide detailed information about the Normandy landings during World War II. This should include text, colorized images, videos, and 3D models.

[0124] In this way, the present invention provides users with an opportunity to learn history intuitively and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0125] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0126] Step 1:

[0127] The user launches the application on their device and selects a specific past event.

[0128] Specific operation: The user launches the app by tapping it from the home screen of their smartphone or computer, and selects "World War II" and "Normandy landings" from a dropdown menu or list.

[0129] Input: A past event selected by the user (for example, "The Normandy landings").

[0130] Output: The application retrieves identification information for the selected event.

[0131] Step 2:

[0132] The terminal sends a data request to the server regarding selected past events.

[0133] Specific operation: The application sends a request to the server containing the identification ID of the selected event and the required data type (text information, image, video, 3D model, etc.).

[0134] Input: Identification information for the event selected by the user.

[0135] Output: Data request sent to the server.

[0136] Step 3:

[0137] The server retrieves information related to the relevant past event from its internal database.

[0138] Specific operation: The server queries the database and retrieves relevant text, images, videos, and 3D models.

[0139] Input: Identification information for the event included in the data request.

[0140] Output: Acquired text, images, videos, and 3D models.

[0141] Step 4:

[0142] The server takes monochrome images and videos, colors them, and converts them into a visually identifiable format.

[0143] Specific operation: The server uses a generative AI model employing deep learning techniques (e.g., OpenCV or PIL) to colorize a monochrome image.

[0144] Input: Acquired monochrome images or videos.

[0145] Output: Colorized images or videos.

[0146] Step 5:

[0147] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience.

[0148] Specific operation: The server generates 3D models using game engines such as Unity or Unreal Engine, and integrates them with text information, voice narration, and interactive elements.

[0149] Input: Colorized images, 3D models, text information, audio data.

[0150] Output: Data package optimized for VR experience.

[0151] Step 6:

[0152] The server streams the generated data package to the terminal in real time.

[0153] Specific operation: Data packages are transmitted sequentially in real time using streaming protocols such as WebRTC and RTMP.

[0154] Input: A data package optimized for VR experiences.

[0155] Output: Data transmitted sequentially.

[0156] Step 7:

[0157] The device displays the received data package on the VR goggles, allowing the user to experience past events in a VR environment.

[0158] Specific operation: The device displays the received data on the VR goggles, and the user interacts with it while wearing the VR goggles.

[0159] Input: Data transmitted sequentially.

[0160] Output: User experience through a VR environment.

[0161] Step 8:

[0162] Users enter feedback within the application after completing the experience.

[0163] Specific actions: Enter your thoughts, suggestions for improvement, and information you'd like to see added into the application's feedback form.

[0164] Input: User feedback.

[0165] Output: Input feedback information.

[0166] Step 9:

[0167] The device sends user feedback to the server.

[0168] Specific action: Send the data entered in the feedback form to the server.

[0169] Input: User feedback information.

[0170] Output: Sent feedback data.

[0171] Step 10:

[0172] The server analyzes the feedback and extracts insights to improve the next learning experience.

[0173] Specific actions: Use analytical algorithms to evaluate feedback data and extract insights for improving the next learning experience.

[0174] Input: Submitted feedback data.

[0175] Output: Extracted insights.

[0176] (Application Example 1)

[0177] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0178] Traditional history education has primarily relied on providing visual and auditory information through textbooks and documentaries. However, these methods struggle to maintain user engagement and foster a visceral understanding of historical events. Furthermore, they lack the ability to customize the learning experience based on individual learners' comprehension levels and interests, resulting in limited learning effectiveness.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0180] In this invention, the server includes means for providing an interface for a user to select a specific historical event; means for sending a data request to the server regarding the selected historical event; means for obtaining information related to the relevant historical event from a database; means for processing the obtained information, colorizing it, visualizing it, and creating a VR version of it; means for streaming the processed data to a terminal in real time; means for the user to experience and interact with the historical event through VR goggles; means for sending user feedback to the server; means for the server to analyze the feedback and improve the learning experience; means for recording and saving user experience information on the terminal in real time; and means for reflecting the recorded information in the next experience and providing a customized learning experience. This makes it possible for the user to experience a customized historical event in real time and enhance the learning effect.

[0181] A "user" is an entity that uses the system to experience historical events.

[0182] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[0183] A "server" is a central processing unit that receives user requests and provides the relevant data.

[0184] A "data request" is a request for information about a historical event selected by the user, which is sent to the server.

[0185] A "database" is a collection of information that stores historical events for a system to access.

[0186] "Colorization" refers to the process of converting monochrome images or videos into color using artificial intelligence technology.

[0187] "Visualization" refers to the process of displaying acquired data in a way that is easy for users to understand.

[0188] "VR conversion" is the process of transforming historical events into a format that allows users to experience them with a sense of presence, using virtual reality technology.

[0189] "Streaming" is a technology that transmits processed data to the user's device in real time and plays it back continuously.

[0190] "Device" refers to devices owned by the user, such as smartphones, smart glasses, and head-mounted displays.

[0191] "VR goggles" are virtual reality devices worn by users to experience historical events.

[0192] "Feedback" refers to information that users enter after experiencing a product or service, including their impressions and suggestions for improvement.

[0193] "Analysis" is the process by which the server analyzes feedback to improve the next learning experience.

[0194] "Experience information" refers to data such as operation logs and viewpoint information recorded when a user experiences a historical event.

[0195] "Customization" refers to adjusting the content of the next experience based on the individual user's interests and level of understanding.

[0196] This invention relates to a system that allows users to learn about historical events intuitively and experientially. The entire system operates through the coordinated efforts of the user's terminal, a server, and devices such as VR goggles.

[0197] First, the application is launched on the user's device. The application provides an interface for the user to select a specific historical event. Through this interface, the user selects, for example, "The Normandy Landings." Once this selection is made, the device sends a data request to the server.

[0198] The server receives this request, searches its internal database for information related to the relevant historical event, and retrieves the necessary data. This data includes text information, images, videos, and 3D models. The server also uses AI technology to colorize monochrome images and videos and prepare them in a visually identifiable format.

[0199] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server then streams the generated data package to the device in real time.

[0200] The device displays the received streaming data on VR goggles, allowing users to experience historical events in a VR environment. By wearing the VR goggles, users can "be present" in the Normandy landings through a realistic experience, experiencing historical events in real time. Interactive elements also allow users to explore different perspectives and detailed information. For example, users can "approach" specific tanks or soldiers to view their details.

[0201] Once the experience ends, users enter feedback within the application. This feedback includes their impressions of the experience, areas for improvement, and additional information they would like to see. The device sends this feedback to the server. The server analyzes the feedback and extracts insights to improve the learning experience. Based on the results of the feedback analysis, the next experience is customized, providing a learning experience tailored to each individual user.

[0202] As a concrete example, consider the case where the user selects "The Normandy Landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings while wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[0203] This application allows users to learn history in an engaging way, enhancing their learning effectiveness. Furthermore, because the experience is customized based on each user's individual history, it can foster a deeper understanding and greater interest.

[0204] Example of a prompt:

[0205] Write code to retrieve data on historical events based on user input and display it on a VR device. The code should display a 3D model, text information, audio narration, and interactive elements for the event selected by the user. After the experience, collect user feedback and send it to the server.

[0206] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0207] Step 1:

[0208] The user launches the application on their device. The user interacts with the interface and selects a specific historical event. The input is the name of the historical event selected by the user, and the output is data related to that selection.

[0209] Step 2:

[0210] The terminal sends a data request to the server regarding the selected historical event. The input is the user's selection, and the output is the result of sending the data request. Specifically, the terminal issues an HTTP request to the server, which includes identification information for the selected historical event.

[0211] Step 3:

[0212] The server receives a data request and retrieves information related to the relevant historical event from its internal database. The input is the identification information of the selected historical event, and the output is a dataset of related information (text, images, videos, 3D models). Specifically, the server issues queries to the database and collects the necessary data.

[0213] Step 4:

[0214] The server processes the acquired information, colorizing, visualizing, and creating VR versions of monochrome images and videos. The input is the acquired dataset, and the output is the visualized dataset. Specifically, it uses a generative AI model to colorize images and videos and render 3D models.

[0215] Step 5:

[0216] The server integrates the processed data and generates a data package optimized for the user's VR experience. The input is a visualized dataset, and the output is a data package. Specifically, it packages various types of data and prepares them for streaming.

[0217] Step 6:

[0218] The server generates data packages and streams them to the terminal in real time. The input is the data package, and the output is the streaming data sent to the terminal. Specifically, the data is compressed and sent to the terminal over the network.

[0219] Step 7:

[0220] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. The input is streaming data, and the output is the experience scene displayed on the VR goggles. Specifically, the data is rendered using the VR goggles' API.

[0221] Step 8:

[0222] The user interacts with the experience, performing actions such as approaching specific tanks or soldiers. Input is the user's actions, and output is visual and auditory feedback based on those actions. Specifically, information is displayed in response to the user's viewpoint movements and clicks.

[0223] Step 9:

[0224] After the experience ends, users enter feedback through the application. This feedback consists of user impressions and suggestions for improvement, while the output is the feedback data saved on the device. Specifically, feedback is collected using an input form.

[0225] Step 10:

[0226] The device sends user feedback to the server. The input is the feedback data, and the output is the feedback sent to the server. Specifically, the feedback data is sent to the server via an HTTP request.

[0227] Step 11:

[0228] The server receives feedback, analyzes it, and extracts insights to improve the learning experience. The input is feedback data, and the output is improvement suggestions. Specifically, a feedback analysis algorithm is used to generate data for customizing the next experience.

[0229] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0230] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[0231] First, the user launches the application on their device and checks the home screen. The application then performs the necessary initial setup and prepares to run the emotion engine.

[0232] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0233] The terminal receives the selection information and sends a data request containing that information to the server.

[0234] The server receives a data request and searches its internal database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[0235] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[0236] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[0237] The server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, it displays detailed information.

[0238] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0239] The server streams the generated data package to the terminal in real time.

[0240] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0241] While users experience historical events in a VR environment and interact with interactive elements, the emotion engine collects and recognizes emotional data from the user's facial expressions and voice in real time.

[0242] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[0243] After the user finishes their experience, a screen is displayed for them to provide feedback, allowing them to enter their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[0244] The device sends user feedback and sentiment data to the server.

[0245] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. This includes customizing content based on the user's emotional state.

[0246] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0247] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their awareness of peace, and using an emotion engine to deliver a personalized learning experience.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The user launches the application on their device and checks the home screen. The application performs the necessary initial setup and prepares the emotion engine.

[0251] Step 2:

[0252] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0253] Step 3:

[0254] The terminal receives the user's selection information and sends a data request containing that information to the server.

[0255] Step 4:

[0256] The server receives a data request. The request includes event identification information and the required data type (text information, images, videos, 3D models, etc.).

[0257] Step 5:

[0258] The server searches the database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[0259] Step 6:

[0260] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[0261] Step 7:

[0262] The server generates or optimizes 3D models and places the necessary objects (such as tanks and soldiers) for the historical scene.

[0263] Step 8:

[0264] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[0265] Step 9:

[0266] The server integrates all the data and generates a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0267] Step 10:

[0268] The server streams the generated data package to the terminal in real time.

[0269] Step 11:

[0270] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0271] Step 12:

[0272] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[0273] Step 13:

[0274] The emotion engine collects and recognizes emotional data in real time from the user's facial expressions and voice.

[0275] Step 14:

[0276] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[0277] Step 15:

[0278] After the user finishes their experience, a screen is displayed where they can provide feedback, including their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[0279] Step 16:

[0280] The device sends user feedback and sentiment data to the server.

[0281] Step 17:

[0282] The server analyzes the user's feedback and sentiment data and extracts insights to improve the next learning experience. For example, customize the content according to the user's emotional state.

[0283] Step 18:

[0284] Based on the analysis results, the server updates the system to improve the quality of the learning experience. Specific examples include adding new interactive elements or adjusting the progress speed of the scene.

[0285] As a result, the user can learn history visually and experientially, deepen their sense of peace, and achieve a learning experience that is customized individually using the emotion engine.

[0286] (Example 2)

[0287] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0288] In a conventional history learning system, there is a problem that it is difficult for users to learn intuitively and experientially. Also, since the technology for collecting sentiment data in real time and providing feedback to the learning experience is insufficient, the learning effect of the user cannot be maximized. Furthermore, there is a lack of means to effectively utilize the feedback provided by the user to customize the next learning experience.

[0289] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotional state in real time and adjusting the display content, means for an emotion recognition engine to collect and analyze emotional data from the user's facial expressions and voice, and means for utilizing a generative AI model to generate colorized images and videos. This enables users to learn about historical events intuitively and experientially, and to provide a individually customized learning experience by utilizing emotional data.

[0290] An "interface" is a means for a user to select a specific historical event.

[0291] An "information processing device" is a computer system that receives data requests and retrieves and analyzes related information.

[0292] A "storage device" refers to a database system or storage device used to store related information.

[0293] "Colorization" is the process of adding color to monochrome images or videos.

[0294] "Visualization" is a means of displaying data visually.

[0295] "Virtualization" is a method of recreating elements of the real world and past events in a virtual reality environment.

[0296] A "display device" refers to a device that allows a user to experience a virtual reality environment, and more specifically, virtual reality goggles.

[0297] "Streaming" is the process of continuously transmitting data to a device in real time.

[0298] "An interactive means of manipulation" refers to a function that allows users to select and manipulate objects within a virtual reality environment.

[0299] "Feedback" refers to the impressions and improvement information provided by the user after experiencing.

[0300] "Emotion recognition engine" is software for collecting and analyzing emotion data from the user's expressions and voices.

[0301] "Generative AI model" is an algorithm that uses artificial intelligence to color monochrome images or perform other data generation.

[0302] "Interaction log" is data that records the user's operation history.

[0303] Adjusting "display content" is a process of changing the progress speed and display content of the experience according to the user's emotional state. <00,00957>

[0304] This invention is a system that enables users to intuitively and experientially learn historical events. This system includes an emotion engine for recognizing the user's emotions in real time and providing feedback to the learning experience.

[0305] The system is mainly composed of the following components: a terminal, a server, an emotion recognition engine, and a communication network connecting them. The terminal is a device for the user to connect, and the server is a central device for data processing, storage, and analysis. The emotion recognition engine is software for collecting and analyzing emotion data.

[0306] Specifications of Hardware and Software

[0307] Terminal: For example, a general tablet terminal or a personal computer (e.g., iPad (registered trademark) Pro, MICROSOFT (registered trademark) SURFACE (registered trademark)), a virtual reality headset (e.g., Oculus Rift).

[0308] Servers: For cloud servers, we will use Amazon Web Services (AWS®) or other cloud service providers.

[0309] Emotion Recognition Engine: Uses NVIDIA's DeepStream SDK to analyze emotional data from the user's facial expressions and voice.

[0310] Generative AI Model: Uses OpenAI® APIs and other AI models to colorize images and videos.

[0311] Database: Relational databases such as MySQL® and PostgreSQL are used for storing and managing information.

[0312] 3D modeling software: Use Blender or Unity to generate 3D models related to historical events.

[0313] Program Processing Overview

[0314] Application startup and initial setup

[0315] When a user launches the application on their device, the home screen is displayed. The application internally starts its emotion recognition engine and waits for the user to be ready.

[0316] Selection of Historical Events

[0317] The user selects a specific historical event from the home screen. For example, they might select "World War II" and then "The Normandy Landings."

[0318] Sending a data request

[0319] The terminal generates a data request based on the user's selection information and sends it to the server.

[0320] Searching and retrieving data

[0321] The server receives a data request and retrieves the corresponding data (text information, images, videos, 3D models) from its internal database.

[0322] Colorization of data and generation of 3D models

[0323] The server processes the acquired data using a generation AI model to colorize monochrome images. It also optimizes 3D models and places objects such as tanks and soldiers.

[0324] Data integration and preparation for VR experiences

[0325] The server integrates text information, audio narration, and interactive elements to generate a data package for the VR experience.

[0326] Providing streaming and VR experiences

[0327] The server streams the data package to the terminal in real time, and the user puts on VR goggles to begin the experience.

[0328] Collection and analysis of emotional data

[0329] The emotion recognition engine collects emotional data in real time from the user's facial expressions and voice, and adapts it to the user's experience.

[0330] Providing feedback and improving the learning experience

[0331] After the experience ends, the user provides feedback, and the device sends that data to the server. The server analyzes the feedback and sentiment data to extract insights for improving the next learning experience.

[0332] Examples of specific cases and prompt statements

[0333] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. This data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion recognition engine analyzes the user's emotions in real time and adjusts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0334] Examples of prompts to input into a generative AI model:

[0335] "Please provide detailed information regarding the Normandy landings."

[0336] "Please analyze the emotional data of users when they selected the Normandy landings."

[0337] "Please integrate the data for the relevant historical event into the VR experience."

[0338] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0339] Step 1: Launching and initial setup of the application

[0340] The user launches the "HistoricalLearn" application on their device.

[0341] Input: Application launched by user action.

[0342] The device displays the application's home screen and initializes the emotion recognition engine.

[0343] Output: Home screen displayed, emotion recognition engine ready.

[0344] Specific operation: The application calls the Emotion Recognition API in the backend to complete the initial setup.

[0345] Step 2: Selecting Historical Events

[0346] The user selects a specific historical event, "World War II," from the application's menu, and then chooses "The Normandy Landings."

[0347] Input: User selection operation.

[0348] The device collects the selected information.

[0349] Output: Selected information "World War II" and "Normandy landings".

[0350] Specific operation: The application retrieves and prepares the selection information from the interface.

[0351] Step 3: Sending the Data Request

[0352] The device creates a data request based on the user's selection information and sends it to the server.

[0353] Input: User's selected information.

[0354] The terminal generates a data request and sends it to the server over the network.

[0355] Output: Data request sent to the server.

[0356] Specific operation: The data request includes selection information and is sent to the server via the API.

[0357] Step 4: Search and retrieve data

[0358] The server receives a data request and searches for and retrieves the relevant information from its internal database.

[0359] Input: Data request.

[0360] The server queries the database (MySQL) to retrieve relevant text information, images, videos, and 3D models.

[0361] Output: Acquired data (text information, images, videos, 3D models).

[0362] Specific operation: Use SQL queries to retrieve necessary information from the database and load it into memory.

[0363] Step 5: Colorizing the data and generating a 3D model

[0364] The server processes the acquired data using an AI model to colorize it and generate a 3D model.

[0365] Input: Acquired data (monochrome images, 3D models).

[0366] The server uses a generated AI model (OpenAI API) to colorize the image. It also uses 3D modeling software (Blender, Unity) to optimize the 3D model.

[0367] Output: Colorized image and optimized 3D model.

[0368] Specific operation: Colorize the image using an image processing algorithm, then position and adjust the model using a 3D modeling tool.

[0369] Step 6: Data Integration and Preparing for the VR Experience

[0370] The server integrates text information, audio narration, and interactive elements to generate a VR experience package.

[0371] Input: Colorized images, optimized 3D models, text information, audio narration, and interactive elements.

[0372] The server integrates the data and generates a data package for the VR experience.

[0373] Output: Data package for VR experience.

[0374] Specific action: Use data integration software to combine each element into one.

[0375] Step 7: Providing streaming and VR experiences

[0376] The server streams the generated data package to the terminal in real time.

[0377] Input: VR experience data package.

[0378] The server transmits data in real time using a streaming protocol.

[0379] The device processes the received data and displays it on the VR goggles.

[0380] Output: VR experience started.

[0381] Specific operation: Data is transmitted via a streaming server and displayed on the display device.

[0382] Step 8: Collecting and analyzing emotional data

[0383] While the user experiences historical events within the VR environment, the emotion engine collects the user's emotional data.

[0384] Input: User's facial expressions and voice.

[0385] The emotion recognition engine analyzes emotion data using facial expression recognition algorithms.

[0386] Output: Real-time sentiment data.

[0387] Specific operation: Emotional data is collected through the camera and microphone, and the emotion recognition engine analyzes the data.

[0388] Step 9: Providing feedback and improving the learning experience

[0389] Users provide feedback after completing the experience.

[0390] Input: User feedback information.

[0391] The device sends feedback information to the server.

[0392] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience.

[0393] Output: Analysis results and insights.

[0394] Specific actions: Analyze feedback data using natural language processing and data analysis tools to identify areas for improvement.

[0395] (Application Example 2)

[0396] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0397] In modern education, methods for learning historical events visually and experientially are limited. As a result, students and learners find it difficult to achieve the deep immersion necessary to understand history. Furthermore, traditional methods cannot individually customize the learning experience according to the learner's emotions and level of understanding. Therefore, new methods are needed to enhance the effectiveness of history learning.

[0398] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0399] This invention includes a server that recognizes the user's emotions in real time and adjusts the experience content, a server that analyzes the user's emotions using a generative AI model and dynamically adjusts the experience content using prompt sentences, and a server that analyzes feedback and emotion data to improve the learning experience. This enables learners to gain a deeply immersive, emotion-based experience while learning history in a individually customized way.

[0400] A "user" refers to a person who uses the system to experience and learn about historical events.

[0401] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[0402] A "server" refers to a computer system that receives data requests and retrieves, processes, and distributes information related to the relevant historical event.

[0403] A "data request" refers to a communication in which a user requests information about a historical event selected by the user from the server.

[0404] A "database" refers to data storage that accumulates data such as text information, images, videos, and 3D models related to historical events.

[0405] "Colorization" refers to the process of adding color to monochrome images or videos using artificial intelligence.

[0406] "Visualization" refers to the process of converting acquired data into a format that can be visually observed by the user.

[0407] "VR conversion" refers to the process of converting data into a format that can be applied to a virtual reality environment.

[0408] A "terminal" refers to a device used by a user to operate the system and experience VR.

[0409] "Streaming" refers to a method of continuously delivering data in real time.

[0410] "VR goggles" refers to a headset device used by users to experience virtual reality.

[0411] "Interactive" refers to the ability of users to interact with the system.

[0412] "Recognizing emotions in real time" refers to instantly reading emotions from the user's facial expressions, voice, and other cues.

[0413] "Adjusting the experience" refers to changing the content displayed and the progression of scenes according to the user's emotions and situation.

[0414] "Feedback" refers to the impressions and evaluations that users provide after completing an experience.

[0415] "Emotional data" refers to data about the user's emotional state collected during their experience.

[0416] A "generative AI model" refers to a model generated using artificial intelligence technology, and is primarily used for image colorization and sentiment analysis.

[0417] A "prompt" refers to a text instruction used to give instructions to a generative AI model.

[0418] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[0419] First, the user launches the application on their device (e.g., a smartphone or VR goggles) and checks the home screen. The application performs the necessary initial setup and prepares to activate the emotion engine. The user then selects a specific historical event through the interface.

[0420] The terminal receives the selection information and sends a data request containing that information to the server. The server receives the data request, searches its internal database for information related to the relevant historical event (text information, images, videos, 3D models), and retrieves it. The retrieved data is colorized using artificial intelligence (e.g., generative AI models). Monochrome footage is analyzed frame by frame, and color is added.

[0421] Next, the server generates or optimizes 3D models and places objects necessary for the historical scene (e.g., tanks and soldiers). Furthermore, the server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, detailed information is displayed. The server integrates all the data to generate a data package for the virtual reality experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements. The server streams the generated data package to the terminal in real time.

[0422] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience. During the experience, the emotion engine collects and analyzes emotional data from the user's facial expressions and voice in real time. The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the speed of the scene progression and the displayed content.

[0423] After the experience ends, users enter their thoughts and suggestions for improvement on a feedback screen. The user's feedback and emotional data are sent from the device to the server. The server analyzes the feedback and emotional data to extract insights for improving the next learning experience. This includes customizing the content to match the user's emotional state.

[0424] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0425] Example of a prompt:

[0426] "Start a learning session that simulates the Normandy landings of World War II. Adjust the experience based on user emotional feedback."

[0427] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0428] Step 1:

[0429] The device launches the application. The user launches the application and checks the home screen. The application performs initial setup and prepares the emotion engine to operate. The input data is the user's login information and initial setup information, and the output data is the system's initialization status.

[0430] Step 2:

[0431] The user selects a specific historical event using an interface. The input data is the event selection information from the user, and the output data is the identification information of the selected event. The terminal receives this information and sends a data request to the server.

[0432] Step 3:

[0433] The server receives a data request and retrieves information related to the relevant historical event from the database. The input data is the identification information of the selected event, and the output data includes text information, images, videos, and 3D models related to the event. The server executes database queries to collect the necessary data.

[0434] Step 4:

[0435] The server processes the acquired information. First, monochrome images and videos are colorized using a generative AI model. The input data is monochrome images and videos, and the output data is colorized images and videos. The generative AI model is used to analyze each frame and add color.

[0436] Step 5:

[0437] The server generates or optimizes 3D models and places the necessary objects for the historical scene. The input data is the initial data of the acquired 3D model, and the output data is the optimized 3D model. The server uses dedicated modeling software to place objects and construct the scene.

[0438] Step 6:

[0439] The server integrates text information and audio narration, and adds interactive elements. Input data consists of text and audio data, while output data is interactive educational content. Detailed information displayed when the user selects a specific object is configured.

[0440] Step 7:

[0441] The server integrates all the data to generate a data package for the virtual reality experience. Input data includes colorized video, optimized 3D models, integrated text, and audio; output data is a unified data package for VR. This package is streamed to the user in real time.

[0442] Step 8:

[0443] The device processes the received streaming data and displays it on the VR goggles. The input data is a data package streamed from the server, and the output data is the video and interactive content displayed on the VR goggles. The user puts on the VR goggles and begins the experience.

[0444] Step 9:

[0445] During the experience, the emotion engine analyzes the user's facial expressions and voice in real time and collects emotional data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional data. The emotion engine recognizes the user's real-time emotional state.

[0446] Step 10:

[0447] The emotion engine adjusts the displayed content according to the user's emotional state. The input data is analyzed emotion data, and the output data is the adjusted content. For example, if the user feels excited or anxious, the speed of the scene progression and the displayed content will be changed.

[0448] Step 11:

[0449] After the experience ends, users enter their feedback and suggestions for improvement on a feedback screen. The input data is the user's feedback, and the output data is the feedback information. This information is sent from the device to the server.

[0450] Step 12:

[0451] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. Input data consists of user feedback and sentiment data, while output data provides insights into the improved learning experience. The server then uses this information to customize the learning content.

[0452] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0453] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0454] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0455] [Second Embodiment]

[0456] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0457] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0458] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0459] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0460] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0462] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0463] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0464] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0465] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0466] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0467] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0468] This invention provides a system that allows users to learn about historical events intuitively and experientially. The program for this system operates in the following steps.

[0469] First, the application is launched on the user's device. The application displays a screen through its user interface where the user can select a specific historical event. For example, the user might select "World War II" and then "The Normandy landings."

[0470] Next, the terminal receives the user's selection and sends that information to the server as a data request. The request includes event identification information and the required data type (e.g., text information, image, video, 3D model, etc.).

[0471] The server receives a request and retrieves information related to the relevant historical event from its internal database. The retrieved information includes text, images, videos, and 3D model data. The server also uses AI technology to colorize monochrome images and videos and convert them into a visually recognizable format.

[0472] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server streams the generated data package to the device in real time.

[0473] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. Users wear VR goggles and "present" on the beach where the Normandy landings took place, experiencing historical events in real time. Interactive elements allow users to explore different perspectives and details. For example, users can "approach" specific tanks or soldiers and view their details.

[0474] After the experience, users enter feedback within the application. This feedback includes their impressions of the experience, suggestions for improvement, and information they would like to see added.

[0475] The device sends user feedback to the server. The server receives this feedback, analyzes the data, and extracts insights to improve the next learning experience.

[0476] As a concrete example, consider the case where a user selects "the Normandy landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings by wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[0477] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0478] The following describes the processing flow.

[0479] Step 1:

[0480] The user launches the application on their device and checks the home screen. The application then performs the necessary initial setup.

[0481] Step 2:

[0482] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0483] Step 3:

[0484] The terminal receives the user's selection information and sends a data request containing that information to the server.

[0485] Step 4:

[0486] The server receives a data request. The request includes event identification information and the required data type.

[0487] Step 5:

[0488] The server searches its internal database for and retrieves information related to the relevant historical event. This information includes text, images, videos, and 3D models.

[0489] Step 6:

[0490] The server uses AI technology to colorize images and videos it has acquired. Specifically, it analyzes monochrome video frame by frame and adds color to it.

[0491] Step 7:

[0492] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[0493] Step 8:

[0494] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[0495] Step 9:

[0496] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0497] Step 10:

[0498] The server streams the generated data package to the terminal in real time.

[0499] Step 11:

[0500] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0501] Step 12:

[0502] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[0503] Step 13:

[0504] After the user finishes their experience, a screen will appear where they can provide feedback, including their thoughts and suggestions for improvement.

[0505] Step 14:

[0506] The device receives user feedback and sends it to the server.

[0507] Step 15:

[0508] The server analyzes user feedback data to extract insights for improving the next learning experience.

[0509] Step 16:

[0510] The server updates the system based on the analysis results, improving the quality of the learning experience. For example, improvements such as adding new interactive elements may be made.

[0511] (Example 1)

[0512] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0513] Traditional history education systems rely solely on text information and static images, failing to provide users with sufficient opportunities to learn about past events visually and experientially. This limits learners' interest and understanding, making it particularly difficult for younger generations to deepen their appreciation for the importance of history and their awareness of peace. Furthermore, static data alone is insufficient to meet the diverse learning needs of users, preventing the provision of interactive learning experiences.

[0514] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0515] In this invention, the server includes means for providing an interface for a user to select a specific past event; means for sending a data request to the server regarding the selected past event; means for obtaining information related to the relevant past event from an information aggregate; means for processing, colorizing, visualizing, and virtualizing the obtained information; means for sequentially transmitting the processed data to an information processing device in real time; means for the user to experience and interact with the past event through a virtual reality device; means for sending user feedback to the server; and means for the server to analyze the feedback and improve the learning experience. As a result, the user can learn about past events visually and experientially, deepen their interest in and understanding of learning, and deepen their awareness of the importance of history and peace through an interactive learning experience.

[0516] "Specific past events" refer to historically significant events or occurrences that users can learn about or experience.

[0517] "Interface" refers to the means by which a user interacts with a system, and includes menus, buttons, and other input devices on the screen.

[0518] A "data request" refers to the process of a user requesting information about a past event from a server.

[0519] An "information repository" refers to a database or other information system that stores data related to past events.

[0520] "Colorization" refers to the process of adding color to monochrome images or videos, making them easier to recognize visually.

[0521] "Visualization" refers to the process of displaying acquired information in the form of charts, graphs, 3D models, and other similar formats.

[0522] "Virtualization" refers to the process of transforming information into a virtual reality environment, allowing users to experience it in an immersive way.

[0523] "Information processing device" refers to a terminal used for processing and displaying data, and includes computers, smartphones, tablets, etc.

[0524] "Sequential transmission" refers to the process of continuously transmitting data in real time.

[0525] "Virtual reality equipment" refers to devices used by users to experience a virtual reality environment, and includes VR goggles, headsets, and the like.

[0526] "Interactive manipulation" refers to users interacting with objects and scenes within a virtual environment.

[0527] "Feedback" refers to the opinions and impressions that users provide regarding their experiences or learning content.

[0528] "Analysis" refers to the process of compiling and evaluating collected feedback to extract insights for improving the next learning experience.

[0529] This invention is a system for allowing users to visually and experientially learn about important past events. This system is based on the interaction between a server, a terminal, and a user, and uses virtual reality technology to recreate historical events. The specific method for carrying out this invention is described below.

[0530] Hardware and software to be used

[0531] terminal

[0532] A terminal is a device that a user uses to launch and operate applications. Typically, a smartphone, tablet, or personal computer is used. Applications running on a terminal allow the user to select specific past events through a user interface.

[0533] server

[0534] A server is a central information processing unit that processes data requests from users and uses the acquired data to generate virtual reality experiences. Servers are equipped with high-performance processors and can process large amounts of data in real time.

[0535] Virtual reality device

[0536] Virtual reality devices are devices used by users to experience a virtual reality environment. Typical examples include VR goggles and headsets. This allows users to become visually immersed and engage in interactive activities.

[0537] Data processing and data calculation

[0538] Colorization and visualization

[0539] The server colorizes the acquired monochrome images and videos and converts them into a visually recognizable format. This is done using a generative AI model based on deep learning technology. For example, image processing libraries such as OpenCV or PIL can be used.

[0540] Virtual reality

[0541] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. High-performance game engines such as Unity or Unreal Engine are used to generate the 3D models.

[0542] Real-time streaming

[0543] The server streams the generated data package to the terminal in real time. This is achieved using streaming protocols such as WebRTC and RTMP.

[0544] Specific example

[0545] For example, if the user selects "The Normandy landings"

[0546] 1. The user launches the application and selects "World War II" and "Normandy landings".

[0547] 2. The server collects data about the selected event and generates colorized images, videos, and 3D models as needed.

[0548] 3. The server streams the generated data package to the terminal in real time.

[0549] 4. Users wear VR goggles and "be present" on the beach where the Normandy landings took place, experiencing the historical event in real time.

[0550] 5. Users provide feedback after completing the experience, and the server analyzes this information to improve the quality of the user experience.

[0551] Examples of prompt statements for a generative AI model are as follows:

[0552] Please provide detailed information about the Normandy landings during World War II. This should include text, colorized images, videos, and 3D models.

[0553] In this way, the present invention provides users with an opportunity to learn history intuitively and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0554] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0555] Step 1:

[0556] The user launches the application on their device and selects a specific past event.

[0557] Specific operation: The user launches the app by tapping it from the home screen of their smartphone or computer, and selects "World War II" and "Normandy landings" from a dropdown menu or list.

[0558] Input: A past event selected by the user (for example, "The Normandy landings").

[0559] Output: The application retrieves identification information for the selected event.

[0560] Step 2:

[0561] The terminal sends a data request to the server regarding selected past events.

[0562] Specific operation: The application sends a request to the server containing the identification ID of the selected event and the required data type (text information, image, video, 3D model, etc.).

[0563] Input: Identification information for the event selected by the user.

[0564] Output: Data request sent to the server.

[0565] Step 3:

[0566] The server retrieves information related to the relevant past event from its internal database.

[0567] Specific operation: The server queries the database and retrieves relevant text, images, videos, and 3D models.

[0568] Input: Identification information for the event included in the data request.

[0569] Output: Acquired text, images, videos, and 3D models.

[0570] Step 4:

[0571] The server takes monochrome images and videos, colors them, and converts them into a visually identifiable format.

[0572] Specific operation: The server uses a generative AI model employing deep learning techniques (e.g., OpenCV or PIL) to colorize a monochrome image.

[0573] Input: Acquired monochrome images or videos.

[0574] Output: Colorized images or videos.

[0575] Step 5:

[0576] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience.

[0577] Specific operation: The server generates 3D models using game engines such as Unity or Unreal Engine, and integrates them with text information, voice narration, and interactive elements.

[0578] Input: Colorized images, 3D models, text information, audio data.

[0579] Output: Data package optimized for VR experience.

[0580] Step 6:

[0581] The server streams the generated data package to the terminal in real time.

[0582] Specific operation: Data packages are transmitted sequentially in real time using streaming protocols such as WebRTC and RTMP.

[0583] Input: A data package optimized for VR experiences.

[0584] Output: Data transmitted sequentially.

[0585] Step 7:

[0586] The device displays the received data package on the VR goggles, allowing the user to experience past events in a VR environment.

[0587] Specific operation: The device displays the received data on the VR goggles, and the user interacts with it while wearing the VR goggles.

[0588] Input: Data transmitted sequentially.

[0589] Output: User experience through a VR environment.

[0590] Step 8:

[0591] Users enter feedback within the application after completing the experience.

[0592] Specific actions: Enter your thoughts, suggestions for improvement, and information you'd like to see added into the application's feedback form.

[0593] Input: User feedback.

[0594] Output: Input feedback information.

[0595] Step 9:

[0596] The device sends user feedback to the server.

[0597] Specific action: Send the data entered in the feedback form to the server.

[0598] Input: User feedback information.

[0599] Output: Sent feedback data.

[0600] Step 10:

[0601] The server analyzes the feedback and extracts insights to improve the next learning experience.

[0602] Specific actions: Use analytical algorithms to evaluate feedback data and extract insights for improving the next learning experience.

[0603] Input: Submitted feedback data.

[0604] Output: Extracted insights.

[0605] (Application Example 1)

[0606] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0607] Traditional history education has primarily relied on providing visual and auditory information through textbooks and documentaries. However, these methods struggle to maintain user engagement and foster a visceral understanding of historical events. Furthermore, they lack the ability to customize the learning experience based on individual learners' comprehension levels and interests, resulting in limited learning effectiveness.

[0608] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0609] In this invention, the server includes means for providing an interface for a user to select a specific historical event; means for sending a data request to the server regarding the selected historical event; means for obtaining information related to the relevant historical event from a database; means for processing the obtained information, colorizing it, visualizing it, and creating a VR version of it; means for streaming the processed data to a terminal in real time; means for the user to experience and interact with the historical event through VR goggles; means for sending user feedback to the server; means for the server to analyze the feedback and improve the learning experience; means for recording and saving user experience information on the terminal in real time; and means for reflecting the recorded information in the next experience and providing a customized learning experience. This makes it possible for the user to experience a customized historical event in real time and enhance the learning effect.

[0610] A "user" is an entity that uses the system to experience historical events.

[0611] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[0612] A "server" is a central processing unit that receives user requests and provides the relevant data.

[0613] A "data request" is a request for information about a historical event selected by the user, which is sent to the server.

[0614] A "database" is a collection of information that stores historical events for a system to access.

[0615] "Colorization" refers to the process of converting monochrome images or videos into color using artificial intelligence technology.

[0616] "Visualization" refers to the process of displaying acquired data in a way that is easy for users to understand.

[0617] "VR conversion" is the process of transforming historical events into a format that allows users to experience them with a sense of presence, using virtual reality technology.

[0618] "Streaming" is a technology that transmits processed data to the user's device in real time and plays it back continuously.

[0619] "Device" refers to devices owned by the user, such as smartphones, smart glasses, and head-mounted displays.

[0620] "VR goggles" are virtual reality devices worn by users to experience historical events.

[0621] "Feedback" refers to information that users enter after experiencing a product or service, including their impressions and suggestions for improvement.

[0622] "Analysis" is the process by which the server analyzes feedback to improve the next learning experience.

[0623] "Experience information" refers to data such as operation logs and viewpoint information recorded when a user experiences a historical event.

[0624] "Customization" refers to adjusting the content of the next experience based on the individual user's interests and level of understanding.

[0625] This invention relates to a system that allows users to learn about historical events intuitively and experientially. The entire system operates through the coordinated efforts of the user's terminal, a server, and devices such as VR goggles.

[0626] First, the application is launched on the user's device. The application provides an interface for the user to select a specific historical event. Through this interface, the user selects, for example, "The Normandy Landings." Once this selection is made, the device sends a data request to the server.

[0627] The server receives this request, searches its internal database for information related to the relevant historical event, and retrieves the necessary data. This data includes text information, images, videos, and 3D models. The server also uses AI technology to colorize monochrome images and videos and prepare them in a visually identifiable format.

[0628] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server then streams the generated data package to the device in real time.

[0629] The device displays the received streaming data on VR goggles, allowing users to experience historical events in a VR environment. By wearing the VR goggles, users can "be present" in the Normandy landings through a realistic experience, experiencing historical events in real time. Interactive elements also allow users to explore different perspectives and detailed information. For example, users can "approach" specific tanks or soldiers to view their details.

[0630] Once the experience ends, users enter feedback within the application. This feedback includes their impressions of the experience, areas for improvement, and additional information they would like to see. The device sends this feedback to the server. The server analyzes the feedback and extracts insights to improve the learning experience. Based on the results of the feedback analysis, the next experience is customized, providing a learning experience tailored to each individual user.

[0631] As a concrete example, consider the case where the user selects "The Normandy Landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings while wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[0632] This application allows users to learn history in an engaging way, enhancing their learning effectiveness. Furthermore, because the experience is customized based on each user's individual history, it can foster a deeper understanding and greater interest.

[0633] Example of a prompt:

[0634] Write code to retrieve data on historical events based on user input and display it on a VR device. The code should display a 3D model, text information, audio narration, and interactive elements for the event selected by the user. After the experience, collect user feedback and send it to the server.

[0635] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0636] Step 1:

[0637] The user launches the application on their device. The user interacts with the interface and selects a specific historical event. The input is the name of the historical event selected by the user, and the output is data related to that selection.

[0638] Step 2:

[0639] The terminal sends a data request to the server regarding the selected historical event. The input is the user's selection, and the output is the result of sending the data request. Specifically, the terminal issues an HTTP request to the server, which includes identification information for the selected historical event.

[0640] Step 3:

[0641] The server receives a data request and retrieves information related to the relevant historical event from its internal database. The input is the identification information of the selected historical event, and the output is a dataset of related information (text, images, videos, 3D models). Specifically, the server issues queries to the database and collects the necessary data.

[0642] Step 4:

[0643] The server processes the acquired information, colorizing, visualizing, and creating VR versions of monochrome images and videos. The input is the acquired dataset, and the output is the visualized dataset. Specifically, it uses a generative AI model to colorize images and videos and render 3D models.

[0644] Step 5:

[0645] The server integrates the processed data and generates a data package optimized for the user's VR experience. The input is a visualized dataset, and the output is a data package. Specifically, it packages various types of data and prepares them for streaming.

[0646] Step 6:

[0647] The server generates data packages and streams them to the terminal in real time. The input is the data package, and the output is the streaming data sent to the terminal. Specifically, the data is compressed and sent to the terminal over the network.

[0648] Step 7:

[0649] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. The input is streaming data, and the output is the experience scene displayed on the VR goggles. Specifically, the data is rendered using the VR goggles' API.

[0650] Step 8:

[0651] The user interacts with the experience, performing actions such as approaching specific tanks or soldiers. Input is the user's actions, and output is visual and auditory feedback based on those actions. Specifically, information is displayed in response to the user's viewpoint movements and clicks.

[0652] Step 9:

[0653] After the experience ends, users enter feedback through the application. This feedback consists of user impressions and suggestions for improvement, while the output is the feedback data saved on the device. Specifically, feedback is collected using an input form.

[0654] Step 10:

[0655] The device sends user feedback to the server. The input is the feedback data, and the output is the feedback sent to the server. Specifically, the feedback data is sent to the server via an HTTP request.

[0656] Step 11:

[0657] The server receives feedback, analyzes it, and extracts insights to improve the learning experience. The input is feedback data, and the output is improvement suggestions. Specifically, a feedback analysis algorithm is used to generate data for customizing the next experience.

[0658] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0659] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[0660] First, the user launches the application on their device and checks the home screen. The application then performs the necessary initial setup and prepares to run the emotion engine.

[0661] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0662] The terminal receives the selection information and sends a data request containing that information to the server.

[0663] The server receives a data request and searches its internal database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[0664] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[0665] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[0666] The server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, it displays detailed information.

[0667] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0668] The server streams the generated data package to the terminal in real time.

[0669] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0670] While users experience historical events in a VR environment and interact with interactive elements, the emotion engine collects and recognizes emotional data from the user's facial expressions and voice in real time.

[0671] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[0672] After the user finishes their experience, a screen is displayed for them to provide feedback, allowing them to enter their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[0673] The device sends user feedback and sentiment data to the server.

[0674] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. This includes customizing content based on the user's emotional state.

[0675] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0676] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their awareness of peace, and using an emotion engine to deliver a personalized learning experience.

[0677] The following describes the processing flow.

[0678] Step 1:

[0679] The user launches the application on their device and checks the home screen. The application performs the necessary initial setup and prepares the emotion engine.

[0680] Step 2:

[0681] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0682] Step 3:

[0683] The terminal receives the user's selection information and sends a data request containing that information to the server.

[0684] Step 4:

[0685] The server receives a data request. The request includes event identification information and the required data type (text information, images, videos, 3D models, etc.).

[0686] Step 5:

[0687] The server searches the database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[0688] Step 6:

[0689] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[0690] Step 7:

[0691] The server generates or optimizes 3D models and places the necessary objects (such as tanks and soldiers) for the historical scene.

[0692] Step 8:

[0693] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[0694] Step 9:

[0695] The server integrates all the data and generates a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0696] Step 10:

[0697] The server streams the generated data package to the terminal in real time.

[0698] Step 11:

[0699] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0700] Step 12:

[0701] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[0702] Step 13:

[0703] The emotion engine collects and recognizes emotional data in real time from the user's facial expressions and voice.

[0704] Step 14:

[0705] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[0706] Step 15:

[0707] After the user finishes their experience, a screen is displayed where they can provide feedback, including their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[0708] Step 16:

[0709] The device sends user feedback and sentiment data to the server.

[0710] Step 17:

[0711] The server analyzes user feedback and sentiment data to extract insights for improving the next learning experience. For example, it can customize content based on the user's emotional state.

[0712] Step 18:

[0713] The server updates the system based on the analysis results, improving the quality of the learning experience. Specific examples include adding new interactive elements or adjusting the pace of scenes.

[0714] This allows users to learn history visually and experientially, deepen their awareness of peace, and enables a personalized learning experience using an emotion engine.

[0715] (Example 2)

[0716] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0717] Traditional history learning systems have the challenge of making it difficult for users to learn intuitively and experientially. Furthermore, the lack of technology to collect emotional data in real time and feed it back into the learning experience prevents maximizing user learning effectiveness. Additionally, there is a lack of means to effectively utilize user feedback to customize subsequent learning experiences.

[0718] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotional state in real time and adjusting the display content, means for an emotion recognition engine to collect and analyze emotional data from the user's facial expressions and voice, and means for utilizing a generative AI model to generate colorized images and videos. This enables users to learn about historical events intuitively and experientially, and to provide a individually customized learning experience by utilizing emotional data.

[0719] An "interface" is a means for a user to select a specific historical event.

[0720] An "information processing device" is a computer system that receives data requests and retrieves and analyzes related information.

[0721] A "storage device" refers to a database system or storage device used to store related information.

[0722] "Colorization" is the process of adding color to monochrome images or videos.

[0723] "Visualization" is a means of displaying data visually.

[0724] "Virtualization" is a method of recreating elements of the real world and past events in a virtual reality environment.

[0725] A "display device" refers to a device that allows a user to experience a virtual reality environment, and more specifically, virtual reality goggles.

[0726] "Streaming" is the process of continuously transmitting data to a device in real time.

[0727] "An interactive means of manipulation" refers to a function that allows users to select and manipulate objects within a virtual reality environment.

[0728] "Feedback" refers to information such as impressions and suggestions for improvement that users provide after experiencing something.

[0729] An "emotion recognition engine" is software that collects and analyzes emotional data from a user's facial expressions and voice.

[0730] A "generative AI model" is an algorithm that uses artificial intelligence to colorize monochrome images or generate other data.

[0731] An "interaction log" is data that records the user's operation history.

[0732] Adjusting "display content" is the process of changing the pace of the experience and the content displayed to match the user's emotional state.

[0733] This invention is a system that enables users to learn about historical events intuitively and experientially. The system includes an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[0734] The system primarily consists of the following components: terminals, servers, an emotion recognition engine, and a communication network connecting them. Terminals are devices for user connection, while servers are central devices that perform data processing, storage, and analysis. The emotion recognition engine is software for collecting and analyzing emotion data.

[0735] Hardware and software specifications

[0736] Devices: For example, general tablet devices and personal computers (e.g., iPad Pro, Microsoft Surface), or virtual reality goggles (e.g., Oculus Rift).

[0737] Servers: For cloud servers, we will use Amazon Web Services (AWS) or other cloud service providers.

[0738] Emotion Recognition Engine: Uses NVIDIA's DeepStream SDK to analyze emotional data from the user's facial expressions and voice.

[0739] Generative AI Model: Uses OpenAI APIs and other AI models to colorize images and videos.

[0740] Database: Relational databases such as MySQL and PostgreSQL are used for storing and managing information.

[0741] 3D modeling software: Use Blender or Unity to generate 3D models related to historical events.

[0742] Program Processing Overview

[0743] Application startup and initial setup

[0744] When a user launches the application on their device, the home screen is displayed. The application internally starts its emotion recognition engine and waits for the user to be ready.

[0745] Selection of Historical Events

[0746] The user selects a specific historical event from the home screen. For example, they might select "World War II" and then "The Normandy Landings."

[0747] Sending a data request

[0748] The terminal generates a data request based on the user's selection information and sends it to the server.

[0749] Searching and retrieving data

[0750] The server receives a data request and retrieves the corresponding data (text information, images, videos, 3D models) from its internal database.

[0751] Colorization of data and generation of 3D models

[0752] The server processes the acquired data using a generation AI model to colorize monochrome images. It also optimizes 3D models and places objects such as tanks and soldiers.

[0753] Data integration and preparation for VR experiences

[0754] The server integrates text information, audio narration, and interactive elements to generate a data package for the VR experience.

[0755] Providing streaming and VR experiences

[0756] The server streams the data package to the terminal in real time, and the user puts on VR goggles to begin the experience.

[0757] Collection and analysis of emotional data

[0758] The emotion recognition engine collects emotional data in real time from the user's facial expressions and voice, and adapts it to the user's experience.

[0759] Providing feedback and improving the learning experience

[0760] After the experience ends, the user provides feedback, and the device sends that data to the server. The server analyzes the feedback and sentiment data to extract insights for improving the next learning experience.

[0761] Examples of specific cases and prompt statements

[0762] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. This data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion recognition engine analyzes the user's emotions in real time and adjusts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0763] Examples of prompts to input into a generative AI model:

[0764] "Please provide detailed information regarding the Normandy landings."

[0765] "Please analyze the emotional data of users when they selected the Normandy landings."

[0766] "Please integrate the data for the relevant historical event into the VR experience."

[0767] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0768] Step 1: Launching and initial setup of the application

[0769] The user launches the "HistoricalLearn" application on their device.

[0770] Input: Application launched by user action.

[0771] The device displays the application's home screen and initializes the emotion recognition engine.

[0772] Output: Home screen displayed, emotion recognition engine ready.

[0773] Specific operation: The application calls the Emotion Recognition API in the backend to complete the initial setup.

[0774] Step 2: Selecting Historical Events

[0775] The user selects a specific historical event, "World War II," from the application's menu, and then chooses "The Normandy Landings."

[0776] Input: User selection operation.

[0777] The device collects the selected information.

[0778] Output: Selected information "World War II" and "Normandy landings".

[0779] Specific operation: The application retrieves and prepares the selection information from the interface.

[0780] Step 3: Sending the Data Request

[0781] The device creates a data request based on the user's selection information and sends it to the server.

[0782] Input: User's selected information.

[0783] The terminal generates a data request and sends it to the server over the network.

[0784] Output: Data request sent to the server.

[0785] Specific operation: The data request includes selection information and is sent to the server via the API.

[0786] Step 4: Search and retrieve data

[0787] The server receives a data request and searches for and retrieves the relevant information from its internal database.

[0788] Input: Data request.

[0789] The server queries the database (MySQL) to retrieve relevant text information, images, videos, and 3D models.

[0790] Output: Acquired data (text information, images, videos, 3D models).

[0791] Specific operation: Use SQL queries to retrieve necessary information from the database and load it into memory.

[0792] Step 5: Colorizing the data and generating a 3D model

[0793] The server processes the acquired data using an AI model to colorize it and generate a 3D model.

[0794] Input: Acquired data (monochrome images, 3D models).

[0795] The server uses a generated AI model (OpenAI API) to colorize the image. It also uses 3D modeling software (Blender, Unity) to optimize the 3D model.

[0796] Output: Colorized image and optimized 3D model.

[0797] Specific operation: Colorize the image using an image processing algorithm, then position and adjust the model using a 3D modeling tool.

[0798] Step 6: Data Integration and Preparing for the VR Experience

[0799] The server integrates text information, audio narration, and interactive elements to generate a VR experience package.

[0800] Input: Colorized images, optimized 3D models, text information, audio narration, and interactive elements.

[0801] The server integrates the data and generates a data package for the VR experience.

[0802] Output: Data package for VR experience.

[0803] Specific action: Use data integration software to combine each element into one.

[0804] Step 7: Providing streaming and VR experiences

[0805] The server streams the generated data package to the terminal in real time.

[0806] Input: VR experience data package.

[0807] The server transmits data in real time using a streaming protocol.

[0808] The device processes the received data and displays it on the VR goggles.

[0809] Output: VR experience started.

[0810] Specific operation: Data is transmitted via a streaming server and displayed on the display device.

[0811] Step 8: Collecting and analyzing emotional data

[0812] While the user experiences historical events within the VR environment, the emotion engine collects the user's emotional data.

[0813] Input: User's facial expressions and voice.

[0814] The emotion recognition engine analyzes emotion data using facial expression recognition algorithms.

[0815] Output: Real-time sentiment data.

[0816] Specific operation: Emotional data is collected through the camera and microphone, and the emotion recognition engine analyzes the data.

[0817] Step 9: Providing feedback and improving the learning experience

[0818] Users provide feedback after completing the experience.

[0819] Input: User feedback information.

[0820] The device sends feedback information to the server.

[0821] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience.

[0822] Output: Analysis results and insights.

[0823] Specific actions: Analyze feedback data using natural language processing and data analysis tools to identify areas for improvement.

[0824] (Application Example 2)

[0825] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0826] In modern education, methods for learning historical events visually and experientially are limited. As a result, students and learners find it difficult to achieve the deep immersion necessary to understand history. Furthermore, traditional methods cannot individually customize the learning experience according to the learner's emotions and level of understanding. Therefore, new methods are needed to enhance the effectiveness of history learning.

[0827] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0828] This invention includes a server that recognizes the user's emotions in real time and adjusts the experience content, a server that analyzes the user's emotions using a generative AI model and dynamically adjusts the experience content using prompt sentences, and a server that analyzes feedback and emotion data to improve the learning experience. This enables learners to gain a deeply immersive, emotion-based experience while learning history in a individually customized way.

[0829] A "user" refers to a person who uses the system to experience and learn about historical events.

[0830] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[0831] A "server" refers to a computer system that receives data requests and retrieves, processes, and distributes information related to the relevant historical event.

[0832] A "data request" refers to a communication in which a user requests information about a historical event selected by the user from the server.

[0833] A "database" refers to data storage that accumulates data such as text information, images, videos, and 3D models related to historical events.

[0834] "Colorization" refers to the process of adding color to monochrome images or videos using artificial intelligence.

[0835] "Visualization" refers to the process of converting acquired data into a format that can be visually observed by the user.

[0836] "VR conversion" refers to the process of converting data into a format that can be applied to a virtual reality environment.

[0837] A "terminal" refers to a device used by a user to operate the system and experience VR.

[0838] "Streaming" refers to a method of continuously delivering data in real time.

[0839] "VR goggles" refers to a headset device used by users to experience virtual reality.

[0840] "Interactive" refers to the ability of users to interact with the system.

[0841] "Recognizing emotions in real time" refers to instantly reading emotions from the user's facial expressions, voice, and other cues.

[0842] "Adjusting the experience" refers to changing the content displayed and the progression of scenes according to the user's emotions and situation.

[0843] "Feedback" refers to the impressions and evaluations that users provide after completing an experience.

[0844] "Emotional data" refers to data about the user's emotional state collected during their experience.

[0845] A "generative AI model" refers to a model generated using artificial intelligence technology, and is primarily used for image colorization and sentiment analysis.

[0846] A "prompt" refers to a text instruction used to give instructions to a generative AI model.

[0847] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[0848] First, the user launches the application on their device (e.g., a smartphone or VR goggles) and checks the home screen. The application performs the necessary initial setup and prepares to activate the emotion engine. The user then selects a specific historical event through the interface.

[0849] The terminal receives the selection information and sends a data request containing that information to the server. The server receives the data request, searches its internal database for information related to the relevant historical event (text information, images, videos, 3D models), and retrieves it. The retrieved data is colorized using artificial intelligence (e.g., generative AI models). Monochrome footage is analyzed frame by frame, and color is added.

[0850] Next, the server generates or optimizes 3D models and places objects necessary for the historical scene (e.g., tanks and soldiers). Furthermore, the server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, detailed information is displayed. The server integrates all the data to generate a data package for the virtual reality experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements. The server streams the generated data package to the terminal in real time.

[0851] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience. During the experience, the emotion engine collects and analyzes emotional data from the user's facial expressions and voice in real time. The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the speed of the scene progression and the displayed content.

[0852] After the experience ends, users enter their thoughts and suggestions for improvement on a feedback screen. The user's feedback and emotional data are sent from the device to the server. The server analyzes the feedback and emotional data to extract insights for improving the next learning experience. This includes customizing the content to match the user's emotional state.

[0853] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[0854] Example of a prompt:

[0855] "Start a learning session that simulates the Normandy landings of World War II. Adjust the experience based on user emotional feedback."

[0856] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0857] Step 1:

[0858] The device launches the application. The user launches the application and checks the home screen. The application performs initial setup and prepares the emotion engine to operate. The input data is the user's login information and initial setup information, and the output data is the system's initialization status.

[0859] Step 2:

[0860] The user selects a specific historical event using an interface. The input data is the event selection information from the user, and the output data is the identification information of the selected event. The terminal receives this information and sends a data request to the server.

[0861] Step 3:

[0862] The server receives a data request and retrieves information related to the relevant historical event from the database. The input data is the identification information of the selected event, and the output data includes text information, images, videos, and 3D models related to the event. The server executes database queries to collect the necessary data.

[0863] Step 4:

[0864] The server processes the acquired information. First, monochrome images and videos are colorized using a generative AI model. The input data is monochrome images and videos, and the output data is colorized images and videos. The generative AI model is used to analyze each frame and add color.

[0865] Step 5:

[0866] The server generates or optimizes 3D models and places the necessary objects for the historical scene. The input data is the initial data of the acquired 3D model, and the output data is the optimized 3D model. The server uses dedicated modeling software to place objects and construct the scene.

[0867] Step 6:

[0868] The server integrates text information and audio narration, and adds interactive elements. Input data consists of text and audio data, while output data is interactive educational content. Detailed information displayed when the user selects a specific object is configured.

[0869] Step 7:

[0870] The server integrates all the data to generate a data package for the virtual reality experience. Input data includes colorized video, optimized 3D models, integrated text, and audio; output data is a unified data package for VR. This package is streamed to the user in real time.

[0871] Step 8:

[0872] The device processes the received streaming data and displays it on the VR goggles. The input data is a data package streamed from the server, and the output data is the video and interactive content displayed on the VR goggles. The user puts on the VR goggles and begins the experience.

[0873] Step 9:

[0874] During the experience, the emotion engine analyzes the user's facial expressions and voice in real time and collects emotional data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional data. The emotion engine recognizes the user's real-time emotional state.

[0875] Step 10:

[0876] The emotion engine adjusts the displayed content according to the user's emotional state. The input data is analyzed emotion data, and the output data is the adjusted content. For example, if the user feels excited or anxious, the speed of the scene progression and the displayed content will be changed.

[0877] Step 11:

[0878] After the experience ends, users enter their feedback and suggestions for improvement on a feedback screen. The input data is the user's feedback, and the output data is the feedback information. This information is sent from the device to the server.

[0879] Step 12:

[0880] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. Input data consists of user feedback and sentiment data, while output data provides insights into the improved learning experience. The server then uses this information to customize the learning content.

[0881] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0882] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0883] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0884] [Third Embodiment]

[0885] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0886] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0887] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0888] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0889] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0890] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0891] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0892] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0893] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0894] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0895] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0896] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0897] This invention provides a system that allows users to learn about historical events intuitively and experientially. The program for this system operates in the following steps.

[0898] First, the application is launched on the user's device. The application displays a screen through its user interface where the user can select a specific historical event. For example, the user might select "World War II" and then "The Normandy landings."

[0899] Next, the terminal receives the user's selection and sends that information to the server as a data request. The request includes event identification information and the required data type (e.g., text information, image, video, 3D model, etc.).

[0900] The server receives a request and retrieves information related to the relevant historical event from its internal database. The retrieved information includes text, images, videos, and 3D model data. The server also uses AI technology to colorize monochrome images and videos and convert them into a visually recognizable format.

[0901] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server streams the generated data package to the device in real time.

[0902] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. Users wear VR goggles and "present" on the beach where the Normandy landings took place, experiencing historical events in real time. Interactive elements allow users to explore different perspectives and details. For example, users can "approach" specific tanks or soldiers and view their details.

[0903] After the experience, users enter feedback within the application. This feedback includes their impressions of the experience, suggestions for improvement, and information they would like to see added.

[0904] The device sends user feedback to the server. The server receives this feedback, analyzes the data, and extracts insights to improve the next learning experience.

[0905] As a concrete example, consider the case where a user selects "the Normandy landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings by wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[0906] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0907] The following describes the processing flow.

[0908] Step 1:

[0909] The user launches the application on their device and checks the home screen. The application then performs the necessary initial setup.

[0910] Step 2:

[0911] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[0912] Step 3:

[0913] The terminal receives the user's selection information and sends a data request containing that information to the server.

[0914] Step 4:

[0915] The server receives a data request. The request includes event identification information and the required data type.

[0916] Step 5:

[0917] The server searches its internal database for and retrieves information related to the relevant historical event. This information includes text, images, videos, and 3D models.

[0918] Step 6:

[0919] The server uses AI technology to colorize images and videos it has acquired. Specifically, it analyzes monochrome video frame by frame and adds color to it.

[0920] Step 7:

[0921] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[0922] Step 8:

[0923] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[0924] Step 9:

[0925] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[0926] Step 10:

[0927] The server streams the generated data package to the terminal in real time.

[0928] Step 11:

[0929] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[0930] Step 12:

[0931] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[0932] Step 13:

[0933] After the user finishes their experience, a screen will appear where they can provide feedback, including their thoughts and suggestions for improvement.

[0934] Step 14:

[0935] The device receives user feedback and sends it to the server.

[0936] Step 15:

[0937] The server analyzes user feedback data to extract insights for improving the next learning experience.

[0938] Step 16:

[0939] The server updates the system based on the analysis results, improving the quality of the learning experience. For example, improvements such as adding new interactive elements may be made.

[0940] (Example 1)

[0941] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0942] Traditional history education systems rely solely on text information and static images, failing to provide users with sufficient opportunities to learn about past events visually and experientially. This limits learners' interest and understanding, making it particularly difficult for younger generations to deepen their appreciation for the importance of history and their awareness of peace. Furthermore, static data alone is insufficient to meet the diverse learning needs of users, preventing the provision of interactive learning experiences.

[0943] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0944] In this invention, the server includes means for providing an interface for a user to select a specific past event; means for sending a data request to the server regarding the selected past event; means for obtaining information related to the relevant past event from an information aggregate; means for processing, colorizing, visualizing, and virtualizing the obtained information; means for sequentially transmitting the processed data to an information processing device in real time; means for the user to experience and interact with the past event through a virtual reality device; means for sending user feedback to the server; and means for the server to analyze the feedback and improve the learning experience. As a result, the user can learn about past events visually and experientially, deepen their interest in and understanding of learning, and deepen their awareness of the importance of history and peace through an interactive learning experience.

[0945] "Specific past events" refer to historically significant events or occurrences that users can learn about or experience.

[0946] "Interface" refers to the means by which a user interacts with a system, and includes menus, buttons, and other input devices on the screen.

[0947] A "data request" refers to the process of a user requesting information about a past event from a server.

[0948] An "information repository" refers to a database or other information system that stores data related to past events.

[0949] "Colorization" refers to the process of adding color to monochrome images or videos, making them easier to recognize visually.

[0950] "Visualization" refers to the process of displaying acquired information in the form of charts, graphs, 3D models, and other similar formats.

[0951] "Virtualization" refers to the process of transforming information into a virtual reality environment, allowing users to experience it in an immersive way.

[0952] "Information processing device" refers to a terminal used for processing and displaying data, and includes computers, smartphones, tablets, etc.

[0953] "Sequential transmission" refers to the process of continuously transmitting data in real time.

[0954] "Virtual reality equipment" refers to devices used by users to experience a virtual reality environment, and includes VR goggles, headsets, and the like.

[0955] "Interactive manipulation" refers to users interacting with objects and scenes within a virtual environment.

[0956] "Feedback" refers to the opinions and impressions that users provide regarding their experiences or learning content.

[0957] "Analysis" refers to the process of compiling and evaluating collected feedback to extract insights for improving the next learning experience.

[0958] This invention is a system for allowing users to visually and experientially learn about important past events. This system is based on the interaction between a server, a terminal, and a user, and uses virtual reality technology to recreate historical events. The specific method for carrying out this invention is described below.

[0959] Hardware and software to be used

[0960] terminal

[0961] A terminal is a device that a user uses to launch and operate applications. Typically, a smartphone, tablet, or personal computer is used. Applications running on a terminal allow the user to select specific past events through a user interface.

[0962] server

[0963] A server is a central information processing unit that processes data requests from users and uses the acquired data to generate virtual reality experiences. Servers are equipped with high-performance processors and can process large amounts of data in real time.

[0964] Virtual reality device

[0965] Virtual reality devices are devices used by users to experience a virtual reality environment. Typical examples include VR goggles and headsets. This allows users to become visually immersed and engage in interactive activities.

[0966] Data processing and data calculation

[0967] Colorization and visualization

[0968] The server colorizes the acquired monochrome images and videos and converts them into a visually recognizable format. This is done using a generative AI model based on deep learning technology. For example, image processing libraries such as OpenCV or PIL can be used.

[0969] Virtual reality

[0970] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. High-performance game engines such as Unity or Unreal Engine are used to generate the 3D models.

[0971] Real-time streaming

[0972] The server streams the generated data package to the terminal in real time. This is achieved using streaming protocols such as WebRTC and RTMP.

[0973] Specific example

[0974] For example, if the user selects "The Normandy landings"

[0975] 1. The user launches the application and selects "World War II" and "Normandy landings".

[0976] 2. The server collects data about the selected event and generates colorized images, videos, and 3D models as needed.

[0977] 3. The server streams the generated data package to the terminal in real time.

[0978] 4. Users wear VR goggles and "be present" on the beach where the Normandy landings took place, experiencing the historical event in real time.

[0979] 5. Users provide feedback after completing the experience, and the server analyzes this information to improve the quality of the user experience.

[0980] Examples of prompt statements for a generative AI model are as follows:

[0981] Please provide detailed information about the Normandy landings during World War II. This should include text, colorized images, videos, and 3D models.

[0982] In this way, the present invention provides users with an opportunity to learn history intuitively and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[0983] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0984] Step 1:

[0985] The user launches the application on their device and selects a specific past event.

[0986] Specific operation: The user launches the app by tapping it from the home screen of their smartphone or computer, and selects "World War II" and "Normandy landings" from a dropdown menu or list.

[0987] Input: A past event selected by the user (for example, "The Normandy landings").

[0988] Output: The application retrieves identification information for the selected event.

[0989] Step 2:

[0990] The terminal sends a data request to the server regarding selected past events.

[0991] Specific operation: The application sends a request to the server containing the identification ID of the selected event and the required data type (text information, image, video, 3D model, etc.).

[0992] Input: Identification information for the event selected by the user.

[0993] Output: Data request sent to the server.

[0994] Step 3:

[0995] The server retrieves information related to the relevant past event from its internal database.

[0996] Specific operation: The server queries the database and retrieves relevant text, images, videos, and 3D models.

[0997] Input: Identification information for the event included in the data request.

[0998] Output: Acquired text, images, videos, and 3D models.

[0999] Step 4:

[1000] The server takes monochrome images and videos, colors them, and converts them into a visually identifiable format.

[1001] Specific operation: The server uses a generative AI model employing deep learning techniques (e.g., OpenCV or PIL) to colorize a monochrome image.

[1002] Input: Acquired monochrome images or videos.

[1003] Output: Colorized images or videos.

[1004] Step 5:

[1005] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience.

[1006] Specific operation: The server generates 3D models using game engines such as Unity or Unreal Engine, and integrates them with text information, voice narration, and interactive elements.

[1007] Input: Colorized images, 3D models, text information, audio data.

[1008] Output: Data package optimized for VR experience.

[1009] Step 6:

[1010] The server streams the generated data package to the terminal in real time.

[1011] Specific operation: Data packages are transmitted sequentially in real time using streaming protocols such as WebRTC and RTMP.

[1012] Input: A data package optimized for VR experiences.

[1013] Output: Data transmitted sequentially.

[1014] Step 7:

[1015] The device displays the received data package on the VR goggles, allowing the user to experience past events in a VR environment.

[1016] Specific operation: The device displays the received data on the VR goggles, and the user interacts with it while wearing the VR goggles.

[1017] Input: Data transmitted sequentially.

[1018] Output: User experience through a VR environment.

[1019] Step 8:

[1020] Users enter feedback within the application after completing the experience.

[1021] Specific actions: Enter your thoughts, suggestions for improvement, and information you'd like to see added into the application's feedback form.

[1022] Input: User feedback.

[1023] Output: Input feedback information.

[1024] Step 9:

[1025] The device sends user feedback to the server.

[1026] Specific action: Send the data entered in the feedback form to the server.

[1027] Input: User feedback information.

[1028] Output: Sent feedback data.

[1029] Step 10:

[1030] The server analyzes the feedback and extracts insights to improve the next learning experience.

[1031] Specific actions: Use analytical algorithms to evaluate feedback data and extract insights for improving the next learning experience.

[1032] Input: Submitted feedback data.

[1033] Output: Extracted insights.

[1034] (Application Example 1)

[1035] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1036] Traditional history education has primarily relied on providing visual and auditory information through textbooks and documentaries. However, these methods struggle to maintain user engagement and foster a visceral understanding of historical events. Furthermore, they lack the ability to customize the learning experience based on individual learners' comprehension levels and interests, resulting in limited learning effectiveness.

[1037] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1038] In this invention, the server includes means for providing an interface for a user to select a specific historical event; means for sending a data request to the server regarding the selected historical event; means for obtaining information related to the relevant historical event from a database; means for processing the obtained information, colorizing it, visualizing it, and creating a VR version of it; means for streaming the processed data to a terminal in real time; means for the user to experience and interact with the historical event through VR goggles; means for sending user feedback to the server; means for the server to analyze the feedback and improve the learning experience; means for recording and saving user experience information on the terminal in real time; and means for reflecting the recorded information in the next experience and providing a customized learning experience. This makes it possible for the user to experience a customized historical event in real time and enhance the learning effect.

[1039] A "user" is an entity that uses the system to experience historical events.

[1040] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[1041] A "server" is a central processing unit that receives user requests and provides the relevant data.

[1042] A "data request" is a request for information about a historical event selected by the user, which is sent to the server.

[1043] A "database" is a collection of information that stores historical events for a system to access.

[1044] "Colorization" refers to the process of converting monochrome images or videos into color using artificial intelligence technology.

[1045] "Visualization" refers to the process of displaying acquired data in a way that is easy for users to understand.

[1046] "VR conversion" is the process of transforming historical events into a format that allows users to experience them with a sense of presence, using virtual reality technology.

[1047] "Streaming" is a technology that transmits processed data to the user's device in real time and plays it back continuously.

[1048] "Device" refers to devices owned by the user, such as smartphones, smart glasses, and head-mounted displays.

[1049] "VR goggles" are virtual reality devices worn by users to experience historical events.

[1050] "Feedback" refers to information that users enter after experiencing a product or service, including their impressions and suggestions for improvement.

[1051] "Analysis" is the process by which the server analyzes feedback to improve the next learning experience.

[1052] "Experience information" refers to data such as operation logs and viewpoint information recorded when a user experiences a historical event.

[1053] "Customization" refers to adjusting the content of the next experience based on the individual user's interests and level of understanding.

[1054] This invention relates to a system that allows users to learn about historical events intuitively and experientially. The entire system operates through the coordinated efforts of the user's terminal, a server, and devices such as VR goggles.

[1055] First, the application is launched on the user's device. The application provides an interface for the user to select a specific historical event. Through this interface, the user selects, for example, "The Normandy Landings." Once this selection is made, the device sends a data request to the server.

[1056] The server receives this request, searches its internal database for information related to the relevant historical event, and retrieves the necessary data. This data includes text information, images, videos, and 3D models. The server also uses AI technology to colorize monochrome images and videos and prepare them in a visually identifiable format.

[1057] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server then streams the generated data package to the device in real time.

[1058] The device displays the received streaming data on VR goggles, allowing users to experience historical events in a VR environment. By wearing the VR goggles, users can "be present" in the Normandy landings through a realistic experience, experiencing historical events in real time. Interactive elements also allow users to explore different perspectives and detailed information. For example, users can "approach" specific tanks or soldiers to view their details.

[1059] Once the experience ends, users enter feedback within the application. This feedback includes their impressions of the experience, areas for improvement, and additional information they would like to see. The device sends this feedback to the server. The server analyzes the feedback and extracts insights to improve the learning experience. Based on the results of the feedback analysis, the next experience is customized, providing a learning experience tailored to each individual user.

[1060] As a concrete example, consider the case where the user selects "The Normandy Landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings while wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[1061] This application allows users to learn history in an engaging way, enhancing their learning effectiveness. Furthermore, because the experience is customized based on each user's individual history, it can foster a deeper understanding and greater interest.

[1062] Example of a prompt:

[1063] Write code to retrieve data on historical events based on user input and display it on a VR device. The code should display a 3D model, text information, audio narration, and interactive elements for the event selected by the user. After the experience, collect user feedback and send it to the server.

[1064] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1065] Step 1:

[1066] The user launches the application on their device. The user interacts with the interface and selects a specific historical event. The input is the name of the historical event selected by the user, and the output is data related to that selection.

[1067] Step 2:

[1068] The terminal sends a data request to the server regarding the selected historical event. The input is the user's selection, and the output is the result of sending the data request. Specifically, the terminal issues an HTTP request to the server, which includes identification information for the selected historical event.

[1069] Step 3:

[1070] The server receives a data request and retrieves information related to the relevant historical event from its internal database. The input is the identification information of the selected historical event, and the output is a dataset of related information (text, images, videos, 3D models). Specifically, the server issues queries to the database and collects the necessary data.

[1071] Step 4:

[1072] The server processes the acquired information, colorizing, visualizing, and creating VR versions of monochrome images and videos. The input is the acquired dataset, and the output is the visualized dataset. Specifically, it uses a generative AI model to colorize images and videos and render 3D models.

[1073] Step 5:

[1074] The server integrates the processed data and generates a data package optimized for the user's VR experience. The input is a visualized dataset, and the output is a data package. Specifically, it packages various types of data and prepares them for streaming.

[1075] Step 6:

[1076] The server generates data packages and streams them to the terminal in real time. The input is the data package, and the output is the streaming data sent to the terminal. Specifically, the data is compressed and sent to the terminal over the network.

[1077] Step 7:

[1078] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. The input is streaming data, and the output is the experience scene displayed on the VR goggles. Specifically, the data is rendered using the VR goggles' API.

[1079] Step 8:

[1080] The user interacts with the experience, performing actions such as approaching specific tanks or soldiers. Input is the user's actions, and output is visual and auditory feedback based on those actions. Specifically, information is displayed in response to the user's viewpoint movements and clicks.

[1081] Step 9:

[1082] After the experience ends, users enter feedback through the application. This feedback consists of user impressions and suggestions for improvement, while the output is the feedback data saved on the device. Specifically, feedback is collected using an input form.

[1083] Step 10:

[1084] The device sends user feedback to the server. The input is the feedback data, and the output is the feedback sent to the server. Specifically, the feedback data is sent to the server via an HTTP request.

[1085] Step 11:

[1086] The server receives feedback, analyzes it, and extracts insights to improve the learning experience. The input is feedback data, and the output is improvement suggestions. Specifically, a feedback analysis algorithm is used to generate data for customizing the next experience.

[1087] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1088] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1089] First, the user launches the application on their device and checks the home screen. The application then performs the necessary initial setup and prepares to run the emotion engine.

[1090] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[1091] The terminal receives the selection information and sends a data request containing that information to the server.

[1092] The server receives a data request and searches its internal database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[1093] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[1094] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[1095] The server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, it displays detailed information.

[1096] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[1097] The server streams the generated data package to the terminal in real time.

[1098] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[1099] While users experience historical events in a VR environment and interact with interactive elements, the emotion engine collects and recognizes emotional data from the user's facial expressions and voice in real time.

[1100] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[1101] After the user finishes their experience, a screen is displayed for them to provide feedback, allowing them to enter their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[1102] The device sends user feedback and sentiment data to the server.

[1103] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. This includes customizing content based on the user's emotional state.

[1104] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1105] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their awareness of peace, and using an emotion engine to deliver a personalized learning experience.

[1106] The following describes the processing flow.

[1107] Step 1:

[1108] The user launches the application on their device and checks the home screen. The application performs the necessary initial setup and prepares the emotion engine.

[1109] Step 2:

[1110] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[1111] Step 3:

[1112] The terminal receives the user's selection information and sends a data request containing that information to the server.

[1113] Step 4:

[1114] The server receives a data request. The request includes event identification information and the required data type (text information, images, videos, 3D models, etc.).

[1115] Step 5:

[1116] The server searches the database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[1117] Step 6:

[1118] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[1119] Step 7:

[1120] The server generates or optimizes 3D models and places the necessary objects (such as tanks and soldiers) for the historical scene.

[1121] Step 8:

[1122] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[1123] Step 9:

[1124] The server integrates all the data and generates a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[1125] Step 10:

[1126] The server streams the generated data package to the terminal in real time.

[1127] Step 11:

[1128] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[1129] Step 12:

[1130] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[1131] Step 13:

[1132] The emotion engine collects and recognizes emotional data in real time from the user's facial expressions and voice.

[1133] Step 14:

[1134] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[1135] Step 15:

[1136] After the user finishes their experience, a screen is displayed where they can provide feedback, including their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[1137] Step 16:

[1138] The device sends user feedback and sentiment data to the server.

[1139] Step 17:

[1140] The server analyzes user feedback and sentiment data to extract insights for improving the next learning experience. For example, it can customize content based on the user's emotional state.

[1141] Step 18:

[1142] The server updates the system based on the analysis results, improving the quality of the learning experience. Specific examples include adding new interactive elements or adjusting the pace of scenes.

[1143] This allows users to learn history visually and experientially, deepen their awareness of peace, and enables a personalized learning experience using an emotion engine.

[1144] (Example 2)

[1145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1146] Traditional history learning systems have the challenge of making it difficult for users to learn intuitively and experientially. Furthermore, the lack of technology to collect emotional data in real time and feed it back into the learning experience prevents maximizing user learning effectiveness. Additionally, there is a lack of means to effectively utilize user feedback to customize subsequent learning experiences.

[1147] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotional state in real time and adjusting the display content, means for an emotion recognition engine to collect and analyze emotional data from the user's facial expressions and voice, and means for utilizing a generative AI model to generate colorized images and videos. This enables users to learn about historical events intuitively and experientially, and to provide a individually customized learning experience by utilizing emotional data.

[1148] An "interface" is a means for a user to select a specific historical event.

[1149] An "information processing device" is a computer system that receives data requests and retrieves and analyzes related information.

[1150] A "storage device" refers to a database system or storage device used to store related information.

[1151] "Colorization" is the process of adding color to monochrome images or videos.

[1152] "Visualization" is a means of displaying data visually.

[1153] "Virtualization" is a method of recreating elements of the real world and past events in a virtual reality environment.

[1154] A "display device" refers to a device that allows a user to experience a virtual reality environment, and more specifically, virtual reality goggles.

[1155] "Streaming" is the process of continuously transmitting data to a device in real time.

[1156] "An interactive means of manipulation" refers to a function that allows users to select and manipulate objects within a virtual reality environment.

[1157] "Feedback" refers to information such as impressions and suggestions for improvement that users provide after experiencing something.

[1158] An "emotion recognition engine" is software that collects and analyzes emotional data from a user's facial expressions and voice.

[1159] A "generative AI model" is an algorithm that uses artificial intelligence to colorize monochrome images or generate other data.

[1160] An "interaction log" is data that records the user's operation history.

[1161] Adjusting "display content" is the process of changing the pace of the experience and the content displayed to match the user's emotional state.

[1162] This invention is a system that enables users to learn about historical events intuitively and experientially. The system includes an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1163] The system primarily consists of the following components: terminals, servers, an emotion recognition engine, and a communication network connecting them. Terminals are devices for user connection, while servers are central devices that perform data processing, storage, and analysis. The emotion recognition engine is software for collecting and analyzing emotion data.

[1164] Hardware and software specifications

[1165] Devices: For example, general tablet devices and personal computers (e.g., iPad Pro, Microsoft Surface), or virtual reality goggles (e.g., Oculus Rift).

[1166] Servers: For cloud servers, we will use Amazon Web Services (AWS) or other cloud service providers.

[1167] Emotion Recognition Engine: Uses NVIDIA's DeepStream SDK to analyze emotional data from the user's facial expressions and voice.

[1168] Generative AI Model: Uses OpenAI APIs and other AI models to colorize images and videos.

[1169] Database: Relational databases such as MySQL and PostgreSQL are used for storing and managing information.

[1170] 3D modeling software: Use Blender or Unity to generate 3D models related to historical events.

[1171] Program Processing Overview

[1172] Application startup and initial setup

[1173] When a user launches the application on their device, the home screen is displayed. The application internally starts its emotion recognition engine and waits for the user to be ready.

[1174] Selection of Historical Events

[1175] The user selects a specific historical event from the home screen. For example, they might select "World War II" and then "The Normandy Landings."

[1176] Sending a data request

[1177] The terminal generates a data request based on the user's selection information and sends it to the server.

[1178] Searching and retrieving data

[1179] The server receives a data request and retrieves the corresponding data (text information, images, videos, 3D models) from its internal database.

[1180] Colorization of data and generation of 3D models

[1181] The server processes the acquired data using a generation AI model to colorize monochrome images. It also optimizes 3D models and places objects such as tanks and soldiers.

[1182] Data integration and preparation for VR experiences

[1183] The server integrates text information, audio narration, and interactive elements to generate a data package for the VR experience.

[1184] Providing streaming and VR experiences

[1185] The server streams the data package to the terminal in real time, and the user puts on VR goggles to begin the experience.

[1186] Collection and analysis of emotional data

[1187] The emotion recognition engine collects emotional data in real time from the user's facial expressions and voice, and adapts it to the user's experience.

[1188] Providing feedback and improving the learning experience

[1189] After the experience ends, the user provides feedback, and the device sends that data to the server. The server analyzes the feedback and sentiment data to extract insights for improving the next learning experience.

[1190] Examples of specific cases and prompt statements

[1191] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. This data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion recognition engine analyzes the user's emotions in real time and adjusts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1192] Examples of prompts to input into a generative AI model:

[1193] "Please provide detailed information regarding the Normandy landings."

[1194] "Please analyze the emotional data of users when they selected the Normandy landings."

[1195] "Please integrate the data for the relevant historical event into the VR experience."

[1196] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1197] Step 1: Launching and initial setup of the application

[1198] The user launches the "HistoricalLearn" application on their device.

[1199] Input: Application launched by user action.

[1200] The device displays the application's home screen and initializes the emotion recognition engine.

[1201] Output: Home screen displayed, emotion recognition engine ready.

[1202] Specific operation: The application calls the Emotion Recognition API in the backend to complete the initial setup.

[1203] Step 2: Selecting Historical Events

[1204] The user selects a specific historical event, "World War II," from the application's menu, and then chooses "The Normandy Landings."

[1205] Input: User selection operation.

[1206] The device collects the selected information.

[1207] Output: Selected information "World War II" and "Normandy landings".

[1208] Specific operation: The application retrieves and prepares the selection information from the interface.

[1209] Step 3: Sending the Data Request

[1210] The device creates a data request based on the user's selection information and sends it to the server.

[1211] Input: User's selected information.

[1212] The terminal generates a data request and sends it to the server over the network.

[1213] Output: Data request sent to the server.

[1214] Specific operation: The data request includes selection information and is sent to the server via the API.

[1215] Step 4: Search and retrieve data

[1216] The server receives a data request and searches for and retrieves the relevant information from its internal database.

[1217] Input: Data request.

[1218] The server queries the database (MySQL) to retrieve relevant text information, images, videos, and 3D models.

[1219] Output: Acquired data (text information, images, videos, 3D models).

[1220] Specific operation: Use SQL queries to retrieve necessary information from the database and load it into memory.

[1221] Step 5: Colorizing the data and generating a 3D model

[1222] The server processes the acquired data using an AI model to colorize it and generate a 3D model.

[1223] Input: Acquired data (monochrome images, 3D models).

[1224] The server uses a generated AI model (OpenAI API) to colorize the image. It also uses 3D modeling software (Blender, Unity) to optimize the 3D model.

[1225] Output: Colorized image and optimized 3D model.

[1226] Specific operation: Colorize the image using an image processing algorithm, then position and adjust the model using a 3D modeling tool.

[1227] Step 6: Data Integration and Preparing for the VR Experience

[1228] The server integrates text information, audio narration, and interactive elements to generate a VR experience package.

[1229] Input: Colorized images, optimized 3D models, text information, audio narration, and interactive elements.

[1230] The server integrates the data and generates a data package for the VR experience.

[1231] Output: Data package for VR experience.

[1232] Specific action: Use data integration software to combine each element into one.

[1233] Step 7: Providing streaming and VR experiences

[1234] The server streams the generated data package to the terminal in real time.

[1235] Input: VR experience data package.

[1236] The server transmits data in real time using a streaming protocol.

[1237] The device processes the received data and displays it on the VR goggles.

[1238] Output: VR experience started.

[1239] Specific operation: Data is transmitted via a streaming server and displayed on the display device.

[1240] Step 8: Collecting and analyzing emotional data

[1241] While the user experiences historical events within the VR environment, the emotion engine collects the user's emotional data.

[1242] Input: User's facial expressions and voice.

[1243] The emotion recognition engine analyzes emotion data using facial expression recognition algorithms.

[1244] Output: Real-time sentiment data.

[1245] Specific operation: Emotional data is collected through the camera and microphone, and the emotion recognition engine analyzes the data.

[1246] Step 9: Providing feedback and improving the learning experience

[1247] Users provide feedback after completing the experience.

[1248] Input: User feedback information.

[1249] The device sends feedback information to the server.

[1250] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience.

[1251] Output: Analysis results and insights.

[1252] Specific actions: Analyze feedback data using natural language processing and data analysis tools to identify areas for improvement.

[1253] (Application Example 2)

[1254] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1255] In modern education, methods for learning historical events visually and experientially are limited. As a result, students and learners find it difficult to achieve the deep immersion necessary to understand history. Furthermore, traditional methods cannot individually customize the learning experience according to the learner's emotions and level of understanding. Therefore, new methods are needed to enhance the effectiveness of history learning.

[1256] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1257] This invention includes a server that recognizes the user's emotions in real time and adjusts the experience content, a server that analyzes the user's emotions using a generative AI model and dynamically adjusts the experience content using prompt sentences, and a server that analyzes feedback and emotion data to improve the learning experience. This enables learners to gain a deeply immersive, emotion-based experience while learning history in a individually customized way.

[1258] A "user" refers to a person who uses the system to experience and learn about historical events.

[1259] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[1260] A "server" refers to a computer system that receives data requests and retrieves, processes, and distributes information related to the relevant historical event.

[1261] A "data request" refers to a communication in which a user requests information about a historical event selected by the user from the server.

[1262] A "database" refers to data storage that accumulates data such as text information, images, videos, and 3D models related to historical events.

[1263] "Colorization" refers to the process of adding color to monochrome images or videos using artificial intelligence.

[1264] "Visualization" refers to the process of converting acquired data into a format that can be visually observed by the user.

[1265] "VR conversion" refers to the process of converting data into a format that can be applied to a virtual reality environment.

[1266] A "terminal" refers to a device used by a user to operate the system and experience VR.

[1267] "Streaming" refers to a method of continuously delivering data in real time.

[1268] "VR goggles" refers to a headset device used by users to experience virtual reality.

[1269] "Interactive" refers to the ability of users to interact with the system.

[1270] "Recognizing emotions in real time" refers to instantly reading emotions from the user's facial expressions, voice, and other cues.

[1271] "Adjusting the experience" refers to changing the content displayed and the progression of scenes according to the user's emotions and situation.

[1272] "Feedback" refers to the impressions and evaluations that users provide after completing an experience.

[1273] "Emotional data" refers to data about the user's emotional state collected during their experience.

[1274] A "generative AI model" refers to a model generated using artificial intelligence technology, and is primarily used for image colorization and sentiment analysis.

[1275] A "prompt" refers to a text instruction used to give instructions to a generative AI model.

[1276] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1277] First, the user launches the application on their device (e.g., a smartphone or VR goggles) and checks the home screen. The application performs the necessary initial setup and prepares to activate the emotion engine. The user then selects a specific historical event through the interface.

[1278] The terminal receives the selection information and sends a data request containing that information to the server. The server receives the data request, searches its internal database for information related to the relevant historical event (text information, images, videos, 3D models), and retrieves it. The retrieved data is colorized using artificial intelligence (e.g., generative AI models). Monochrome footage is analyzed frame by frame, and color is added.

[1279] Next, the server generates or optimizes 3D models and places objects necessary for the historical scene (e.g., tanks and soldiers). Furthermore, the server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, detailed information is displayed. The server integrates all the data to generate a data package for the virtual reality experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements. The server streams the generated data package to the terminal in real time.

[1280] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience. During the experience, the emotion engine collects and analyzes emotional data from the user's facial expressions and voice in real time. The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the speed of the scene progression and the displayed content.

[1281] After the experience ends, users enter their thoughts and suggestions for improvement on a feedback screen. The user's feedback and emotional data are sent from the device to the server. The server analyzes the feedback and emotional data to extract insights for improving the next learning experience. This includes customizing the content to match the user's emotional state.

[1282] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1283] Example of a prompt:

[1284] "Start a learning session that simulates the Normandy landings of World War II. Adjust the experience based on user emotional feedback."

[1285] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1286] Step 1:

[1287] The device launches the application. The user launches the application and checks the home screen. The application performs initial setup and prepares the emotion engine to operate. The input data is the user's login information and initial setup information, and the output data is the system's initialization status.

[1288] Step 2:

[1289] The user selects a specific historical event using an interface. The input data is the event selection information from the user, and the output data is the identification information of the selected event. The terminal receives this information and sends a data request to the server.

[1290] Step 3:

[1291] The server receives a data request and retrieves information related to the relevant historical event from the database. The input data is the identification information of the selected event, and the output data includes text information, images, videos, and 3D models related to the event. The server executes database queries to collect the necessary data.

[1292] Step 4:

[1293] The server processes the acquired information. First, monochrome images and videos are colorized using a generative AI model. The input data is monochrome images and videos, and the output data is colorized images and videos. The generative AI model is used to analyze each frame and add color.

[1294] Step 5:

[1295] The server generates or optimizes 3D models and places the necessary objects for the historical scene. The input data is the initial data of the acquired 3D model, and the output data is the optimized 3D model. The server uses dedicated modeling software to place objects and construct the scene.

[1296] Step 6:

[1297] The server integrates text information and audio narration, and adds interactive elements. Input data consists of text and audio data, while output data is interactive educational content. Detailed information displayed when the user selects a specific object is configured.

[1298] Step 7:

[1299] The server integrates all the data to generate a data package for the virtual reality experience. Input data includes colorized video, optimized 3D models, integrated text, and audio; output data is a unified data package for VR. This package is streamed to the user in real time.

[1300] Step 8:

[1301] The device processes the received streaming data and displays it on the VR goggles. The input data is a data package streamed from the server, and the output data is the video and interactive content displayed on the VR goggles. The user puts on the VR goggles and begins the experience.

[1302] Step 9:

[1303] During the experience, the emotion engine analyzes the user's facial expressions and voice in real time and collects emotional data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional data. The emotion engine recognizes the user's real-time emotional state.

[1304] Step 10:

[1305] The emotion engine adjusts the displayed content according to the user's emotional state. The input data is analyzed emotion data, and the output data is the adjusted content. For example, if the user feels excited or anxious, the speed of the scene progression and the displayed content will be changed.

[1306] Step 11:

[1307] After the experience ends, users enter their feedback and suggestions for improvement on a feedback screen. The input data is the user's feedback, and the output data is the feedback information. This information is sent from the device to the server.

[1308] Step 12:

[1309] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. Input data consists of user feedback and sentiment data, while output data provides insights into the improved learning experience. The server then uses this information to customize the learning content.

[1310] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1311] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1312] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1313] [Fourth Embodiment]

[1314] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1315] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1316] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1317] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1318] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1319] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1320] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1321] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1322] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1323] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1324] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1325] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1326] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1327] This invention provides a system that allows users to learn about historical events intuitively and experientially. The program for this system operates in the following steps.

[1328] First, the application is launched on the user's device. The application displays a screen through its user interface where the user can select a specific historical event. For example, the user might select "World War II" and then "The Normandy landings."

[1329] Next, the terminal receives the user's selection and sends that information to the server as a data request. The request includes event identification information and the required data type (e.g., text information, image, video, 3D model, etc.).

[1330] The server receives a request and retrieves information related to the relevant historical event from its internal database. The retrieved information includes text, images, videos, and 3D model data. The server also uses AI technology to colorize monochrome images and videos and convert them into a visually recognizable format.

[1331] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server streams the generated data package to the device in real time.

[1332] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. Users wear VR goggles and "present" on the beach where the Normandy landings took place, experiencing historical events in real time. Interactive elements allow users to explore different perspectives and details. For example, users can "approach" specific tanks or soldiers and view their details.

[1333] After the experience, users enter feedback within the application. This feedback includes their impressions of the experience, suggestions for improvement, and information they would like to see added.

[1334] The device sends user feedback to the server. The server receives this feedback, analyzes the data, and extracts insights to improve the next learning experience.

[1335] As a concrete example, consider the case where a user selects "the Normandy landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings by wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[1336] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[1337] The following describes the processing flow.

[1338] Step 1:

[1339] The user launches the application on their device and checks the home screen. The application then performs the necessary initial setup.

[1340] Step 2:

[1341] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[1342] Step 3:

[1343] The terminal receives the user's selection information and sends a data request containing that information to the server.

[1344] Step 4:

[1345] The server receives a data request. The request includes event identification information and the required data type.

[1346] Step 5:

[1347] The server searches its internal database for and retrieves information related to the relevant historical event. This information includes text, images, videos, and 3D models.

[1348] Step 6:

[1349] The server uses AI technology to colorize images and videos it has acquired. Specifically, it analyzes monochrome video frame by frame and adds color to it.

[1350] Step 7:

[1351] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[1352] Step 8:

[1353] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[1354] Step 9:

[1355] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[1356] Step 10:

[1357] The server streams the generated data package to the terminal in real time.

[1358] Step 11:

[1359] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[1360] Step 12:

[1361] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[1362] Step 13:

[1363] After the user finishes their experience, a screen will appear where they can provide feedback, including their thoughts and suggestions for improvement.

[1364] Step 14:

[1365] The device receives user feedback and sends it to the server.

[1366] Step 15:

[1367] The server analyzes user feedback data to extract insights for improving the next learning experience.

[1368] Step 16:

[1369] The server updates the system based on the analysis results, improving the quality of the learning experience. For example, improvements such as adding new interactive elements may be made.

[1370] (Example 1)

[1371] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1372] Traditional history education systems rely solely on text information and static images, failing to provide users with sufficient opportunities to learn about past events visually and experientially. This limits learners' interest and understanding, making it particularly difficult for younger generations to deepen their appreciation for the importance of history and their awareness of peace. Furthermore, static data alone is insufficient to meet the diverse learning needs of users, preventing the provision of interactive learning experiences.

[1373] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1374] In this invention, the server includes means for providing an interface for a user to select a specific past event; means for sending a data request to the server regarding the selected past event; means for obtaining information related to the relevant past event from an information aggregate; means for processing, colorizing, visualizing, and virtualizing the obtained information; means for sequentially transmitting the processed data to an information processing device in real time; means for the user to experience and interact with the past event through a virtual reality device; means for sending user feedback to the server; and means for the server to analyze the feedback and improve the learning experience. As a result, the user can learn about past events visually and experientially, deepen their interest in and understanding of learning, and deepen their awareness of the importance of history and peace through an interactive learning experience.

[1375] "Specific past events" refer to historically significant events or occurrences that users can learn about or experience.

[1376] "Interface" refers to the means by which a user interacts with a system, and includes menus, buttons, and other input devices on the screen.

[1377] A "data request" refers to the process of a user requesting information about a past event from a server.

[1378] An "information repository" refers to a database or other information system that stores data related to past events.

[1379] "Colorization" refers to the process of adding color to monochrome images or videos, making them easier to recognize visually.

[1380] "Visualization" refers to the process of displaying acquired information in the form of charts, graphs, 3D models, and other similar formats.

[1381] "Virtualization" refers to the process of transforming information into a virtual reality environment, allowing users to experience it in an immersive way.

[1382] "Information processing device" refers to a terminal used for processing and displaying data, and includes computers, smartphones, tablets, etc.

[1383] "Sequential transmission" refers to the process of continuously transmitting data in real time.

[1384] "Virtual reality equipment" refers to devices used by users to experience a virtual reality environment, and includes VR goggles, headsets, and the like.

[1385] "Interactive manipulation" refers to users interacting with objects and scenes within a virtual environment.

[1386] "Feedback" refers to the opinions and impressions that users provide regarding their experiences or learning content.

[1387] "Analysis" refers to the process of compiling and evaluating collected feedback to extract insights for improving the next learning experience.

[1388] This invention is a system for allowing users to visually and experientially learn about important past events. This system is based on the interaction between a server, a terminal, and a user, and uses virtual reality technology to recreate historical events. The specific method for carrying out this invention is described below.

[1389] Hardware and software to be used

[1390] terminal

[1391] A terminal is a device that a user uses to launch and operate applications. Typically, a smartphone, tablet, or personal computer is used. Applications running on a terminal allow the user to select specific past events through a user interface.

[1392] server

[1393] A server is a central information processing unit that processes data requests from users and uses the acquired data to generate virtual reality experiences. Servers are equipped with high-performance processors and can process large amounts of data in real time.

[1394] Virtual reality device

[1395] Virtual reality devices are devices used by users to experience a virtual reality environment. Typical examples include VR goggles and headsets. This allows users to become visually immersed and engage in interactive activities.

[1396] Data processing and data calculation

[1397] Colorization and visualization

[1398] The server colorizes the acquired monochrome images and videos and converts them into a visually recognizable format. This is done using a generative AI model based on deep learning technology. For example, image processing libraries such as OpenCV or PIL can be used.

[1399] Virtual reality

[1400] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. High-performance game engines such as Unity or Unreal Engine are used to generate the 3D models.

[1401] Real-time streaming

[1402] The server streams the generated data package to the terminal in real time. This is achieved using streaming protocols such as WebRTC and RTMP.

[1403] Specific example

[1404] For example, if the user selects "The Normandy landings"

[1405] 1. The user launches the application and selects "World War II" and "Normandy landings".

[1406] 2. The server collects data about the selected event and generates colorized images, videos, and 3D models as needed.

[1407] 3. The server streams the generated data package to the terminal in real time.

[1408] 4. Users wear VR goggles and "be present" on the beach where the Normandy landings took place, experiencing the historical event in real time.

[1409] 5. Users provide feedback after completing the experience, and the server analyzes this information to improve the quality of the user experience.

[1410] Examples of prompt statements for a generative AI model are as follows:

[1411] Please provide detailed information about the Normandy landings during World War II. This should include text, colorized images, videos, and 3D models.

[1412] In this way, the present invention provides users with an opportunity to learn history intuitively and experientially, helping to deepen their understanding of the importance of history and their awareness of peace.

[1413] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1414] Step 1:

[1415] The user launches the application on their device and selects a specific past event.

[1416] Specific operation: The user launches the app by tapping it from the home screen of their smartphone or computer, and selects "World War II" and "Normandy landings" from a dropdown menu or list.

[1417] Input: A past event selected by the user (for example, "The Normandy landings").

[1418] Output: The application retrieves identification information for the selected event.

[1419] Step 2:

[1420] The terminal sends a data request to the server regarding selected past events.

[1421] Specific operation: The application sends a request to the server containing the identification ID of the selected event and the required data type (text information, image, video, 3D model, etc.).

[1422] Input: Identification information for the event selected by the user.

[1423] Output: Data request sent to the server.

[1424] Step 3:

[1425] The server retrieves information related to the relevant past event from its internal database.

[1426] Specific operation: The server queries the database and retrieves relevant text, images, videos, and 3D models.

[1427] Input: Identification information for the event included in the data request.

[1428] Output: Acquired text, images, videos, and 3D models.

[1429] Step 4:

[1430] The server takes monochrome images and videos, colors them, and converts them into a visually identifiable format.

[1431] Specific operation: The server uses a generative AI model employing deep learning techniques (e.g., OpenCV or PIL) to colorize a monochrome image.

[1432] Input: Acquired monochrome images or videos.

[1433] Output: Colorized images or videos.

[1434] Step 5:

[1435] The server integrates the collected and processed data to generate a data package optimized for the user's VR experience.

[1436] Specific operation: The server generates 3D models using game engines such as Unity or Unreal Engine, and integrates them with text information, voice narration, and interactive elements.

[1437] Input: Colorized images, 3D models, text information, audio data.

[1438] Output: Data package optimized for VR experience.

[1439] Step 6:

[1440] The server streams the generated data package to the terminal in real time.

[1441] Specific operation: Data packages are transmitted sequentially in real time using streaming protocols such as WebRTC and RTMP.

[1442] Input: A data package optimized for VR experiences.

[1443] Output: Data transmitted sequentially.

[1444] Step 7:

[1445] The device displays the received data package on the VR goggles, allowing the user to experience past events in a VR environment.

[1446] Specific operation: The device displays the received data on the VR goggles, and the user interacts with it while wearing the VR goggles.

[1447] Input: Data transmitted sequentially.

[1448] Output: User experience through a VR environment.

[1449] Step 8:

[1450] Users enter feedback within the application after completing the experience.

[1451] Specific actions: Enter your thoughts, suggestions for improvement, and information you'd like to see added into the application's feedback form.

[1452] Input: User feedback.

[1453] Output: Input feedback information.

[1454] Step 9:

[1455] The device sends user feedback to the server.

[1456] Specific action: Send the data entered in the feedback form to the server.

[1457] Input: User feedback information.

[1458] Output: Sent feedback data.

[1459] Step 10:

[1460] The server analyzes the feedback and extracts insights to improve the next learning experience.

[1461] Specific actions: Use analytical algorithms to evaluate feedback data and extract insights for improving the next learning experience.

[1462] Input: Submitted feedback data.

[1463] Output: Extracted insights.

[1464] (Application Example 1)

[1465] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1466] Traditional history education has primarily relied on providing visual and auditory information through textbooks and documentaries. However, these methods struggle to maintain user engagement and foster a visceral understanding of historical events. Furthermore, they lack the ability to customize the learning experience based on individual learners' comprehension levels and interests, resulting in limited learning effectiveness.

[1467] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1468] In this invention, the server includes means for providing an interface for a user to select a specific historical event; means for sending a data request to the server regarding the selected historical event; means for obtaining information related to the relevant historical event from a database; means for processing the obtained information, colorizing it, visualizing it, and creating a VR version of it; means for streaming the processed data to a terminal in real time; means for the user to experience and interact with the historical event through VR goggles; means for sending user feedback to the server; means for the server to analyze the feedback and improve the learning experience; means for recording and saving user experience information on the terminal in real time; and means for reflecting the recorded information in the next experience and providing a customized learning experience. This makes it possible for the user to experience a customized historical event in real time and enhance the learning effect.

[1469] A "user" is an entity that uses the system to experience historical events.

[1470] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[1471] A "server" is a central processing unit that receives user requests and provides the relevant data.

[1472] A "data request" is a request for information about a historical event selected by the user, which is sent to the server.

[1473] A "database" is a collection of information that stores historical events for a system to access.

[1474] "Colorization" refers to the process of converting monochrome images or videos into color using artificial intelligence technology.

[1475] "Visualization" refers to the process of displaying acquired data in a way that is easy for users to understand.

[1476] "VR conversion" is the process of transforming historical events into a format that allows users to experience them with a sense of presence, using virtual reality technology.

[1477] "Streaming" is a technology that transmits processed data to the user's device in real time and plays it back continuously.

[1478] "Device" refers to devices owned by the user, such as smartphones, smart glasses, and head-mounted displays.

[1479] "VR goggles" are virtual reality devices worn by users to experience historical events.

[1480] "Feedback" refers to information that users enter after experiencing a product or service, including their impressions and suggestions for improvement.

[1481] "Analysis" is the process by which the server analyzes feedback to improve the next learning experience.

[1482] "Experience information" refers to data such as operation logs and viewpoint information recorded when a user experiences a historical event.

[1483] "Customization" refers to adjusting the content of the next experience based on the individual user's interests and level of understanding.

[1484] This invention relates to a system that allows users to learn about historical events intuitively and experientially. The entire system operates through the coordinated efforts of the user's terminal, a server, and devices such as VR goggles.

[1485] First, the application is launched on the user's device. The application provides an interface for the user to select a specific historical event. Through this interface, the user selects, for example, "The Normandy Landings." Once this selection is made, the device sends a data request to the server.

[1486] The server receives this request, searches its internal database for information related to the relevant historical event, and retrieves the necessary data. This data includes text information, images, videos, and 3D models. The server also uses AI technology to colorize monochrome images and videos and prepare them in a visually identifiable format.

[1487] Next, the server integrates the collected and processed data to generate a data package optimized for the user's VR experience. This data package includes 3D models of historical scenes, associated text information, audio narration, and interactive elements. The server then streams the generated data package to the device in real time.

[1488] The device displays the received streaming data on VR goggles, allowing users to experience historical events in a VR environment. By wearing the VR goggles, users can "be present" in the Normandy landings through a realistic experience, experiencing historical events in real time. Interactive elements also allow users to explore different perspectives and detailed information. For example, users can "approach" specific tanks or soldiers to view their details.

[1489] Once the experience ends, users enter feedback within the application. This feedback includes their impressions of the experience, areas for improvement, and additional information they would like to see. The device sends this feedback to the server. The server analyzes the feedback and extracts insights to improve the learning experience. Based on the results of the feedback analysis, the next experience is customized, providing a learning experience tailored to each individual user.

[1490] As a concrete example, consider the case where the user selects "The Normandy Landings." The user launches the application and selects a specific historical event. The server then collects relevant data and generates colorized images, videos, and 3D models. The data is streamed in real time, and the user experiences the landings while wearing VR goggles. After the experience, the user provides feedback, and the server analyzes this information to improve the quality of the user experience.

[1491] This application allows users to learn history in an engaging way, enhancing their learning effectiveness. Furthermore, because the experience is customized based on each user's individual history, it can foster a deeper understanding and greater interest.

[1492] Example of a prompt:

[1493] Write code to retrieve data on historical events based on user input and display it on a VR device. The code should display a 3D model, text information, audio narration, and interactive elements for the event selected by the user. After the experience, collect user feedback and send it to the server.

[1494] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1495] Step 1:

[1496] The user launches the application on their device. The user interacts with the interface and selects a specific historical event. The input is the name of the historical event selected by the user, and the output is data related to that selection.

[1497] Step 2:

[1498] The terminal sends a data request to the server regarding the selected historical event. The input is the user's selection, and the output is the result of sending the data request. Specifically, the terminal issues an HTTP request to the server, which includes identification information for the selected historical event.

[1499] Step 3:

[1500] The server receives a data request and retrieves information related to the relevant historical event from its internal database. The input is the identification information of the selected historical event, and the output is a dataset of related information (text, images, videos, 3D models). Specifically, the server issues queries to the database and collects the necessary data.

[1501] Step 4:

[1502] The server processes the acquired information, colorizing, visualizing, and creating VR versions of monochrome images and videos. The input is the acquired dataset, and the output is the visualized dataset. Specifically, it uses a generative AI model to colorize images and videos and render 3D models.

[1503] Step 5:

[1504] The server integrates the processed data and generates a data package optimized for the user's VR experience. The input is a visualized dataset, and the output is a data package. Specifically, it packages various types of data and prepares them for streaming.

[1505] Step 6:

[1506] The server generates data packages and streams them to the terminal in real time. The input is the data package, and the output is the streaming data sent to the terminal. Specifically, the data is compressed and sent to the terminal over the network.

[1507] Step 7:

[1508] The system displays streaming data received by the device on VR goggles, allowing users to experience historical events in a VR environment. The input is streaming data, and the output is the experience scene displayed on the VR goggles. Specifically, the data is rendered using the VR goggles' API.

[1509] Step 8:

[1510] The user interacts with the experience, performing actions such as approaching specific tanks or soldiers. Input is the user's actions, and output is visual and auditory feedback based on those actions. Specifically, information is displayed in response to the user's viewpoint movements and clicks.

[1511] Step 9:

[1512] After the experience ends, users enter feedback through the application. This feedback consists of user impressions and suggestions for improvement, while the output is the feedback data saved on the device. Specifically, feedback is collected using an input form.

[1513] Step 10:

[1514] The device sends user feedback to the server. The input is the feedback data, and the output is the feedback sent to the server. Specifically, the feedback data is sent to the server via an HTTP request.

[1515] Step 11:

[1516] The server receives feedback, analyzes it, and extracts insights to improve the learning experience. The input is feedback data, and the output is improvement suggestions. Specifically, a feedback analysis algorithm is used to generate data for customizing the next experience.

[1517] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1518] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1519] First, the user launches the application on their device and checks the home screen. The application then performs the necessary initial setup and prepares to run the emotion engine.

[1520] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[1521] The terminal receives the selection information and sends a data request containing that information to the server.

[1522] The server receives a data request and searches its internal database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[1523] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[1524] The server generates or optimizes 3D models and places the necessary objects (e.g., tanks and soldiers) for the historical scene.

[1525] The server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, it displays detailed information.

[1526] The server integrates all the data to generate a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[1527] The server streams the generated data package to the terminal in real time.

[1528] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[1529] While users experience historical events in a VR environment and interact with interactive elements, the emotion engine collects and recognizes emotional data from the user's facial expressions and voice in real time.

[1530] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[1531] After the user finishes their experience, a screen is displayed for them to provide feedback, allowing them to enter their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[1532] The device sends user feedback and sentiment data to the server.

[1533] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. This includes customizing content based on the user's emotional state.

[1534] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1535] In this way, the system provides users with opportunities to learn history visually and experientially, helping to deepen their awareness of peace, and using an emotion engine to deliver a personalized learning experience.

[1536] The following describes the processing flow.

[1537] Step 1:

[1538] The user launches the application on their device and checks the home screen. The application performs the necessary initial setup and prepares the emotion engine.

[1539] Step 2:

[1540] The user selects a specific historical event from a menu within the application. For example, they might select "World War II" and then "The Normandy Landings."

[1541] Step 3:

[1542] The terminal receives the user's selection information and sends a data request containing that information to the server.

[1543] Step 4:

[1544] The server receives a data request. The request includes event identification information and the required data type (text information, images, videos, 3D models, etc.).

[1545] Step 5:

[1546] The server searches the database for and retrieves information related to the relevant historical event. The retrieved information includes text, images, videos, and 3D models.

[1547] Step 6:

[1548] The server uses AI technology to colorize images and videos it has acquired. It analyzes monochrome video frame by frame and adds color to each frame.

[1549] Step 7:

[1550] The server generates or optimizes 3D models and places the necessary objects (such as tanks and soldiers) for the historical scene.

[1551] Step 8:

[1552] The server integrates text information and audio narration, and adds interactive elements. For example, it can display detailed information when a user selects a specific object.

[1553] Step 9:

[1554] The server integrates all the data and generates a data package for the VR experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements.

[1555] Step 10:

[1556] The server streams the generated data package to the terminal in real time.

[1557] Step 11:

[1558] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience.

[1559] Step 12:

[1560] Users experience historical events in a VR environment and interact with interactive elements. For example, users can approach tanks or observe the actions of soldiers.

[1561] Step 13:

[1562] The emotion engine collects and recognizes emotional data in real time from the user's facial expressions and voice.

[1563] Step 14:

[1564] The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the pace of the scene and the content displayed.

[1565] Step 15:

[1566] After the user finishes their experience, a screen is displayed where they can provide feedback, including their thoughts and suggestions for improvement. Emotional data collected by the emotion engine is also included in the feedback information.

[1567] Step 16:

[1568] The device sends user feedback and sentiment data to the server.

[1569] Step 17:

[1570] The server analyzes user feedback and sentiment data to extract insights for improving the next learning experience. For example, it can customize content based on the user's emotional state.

[1571] Step 18:

[1572] The server updates the system based on the analysis results, improving the quality of the learning experience. Specific examples include adding new interactive elements or adjusting the pace of scenes.

[1573] This allows users to learn history visually and experientially, deepen their awareness of peace, and enables a personalized learning experience using an emotion engine.

[1574] (Example 2)

[1575] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1576] Traditional history learning systems have the challenge of making it difficult for users to learn intuitively and experientially. Furthermore, the lack of technology to collect emotional data in real time and feed it back into the learning experience prevents maximizing user learning effectiveness. Additionally, there is a lack of means to effectively utilize user feedback to customize subsequent learning experiences.

[1577] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotional state in real time and adjusting the display content, means for an emotion recognition engine to collect and analyze emotional data from the user's facial expressions and voice, and means for utilizing a generative AI model to generate colorized images and videos. This enables users to learn about historical events intuitively and experientially, and to provide a individually customized learning experience by utilizing emotional data.

[1578] An "interface" is a means for a user to select a specific historical event.

[1579] An "information processing device" is a computer system that receives data requests and retrieves and analyzes related information.

[1580] A "storage device" refers to a database system or storage device used to store related information.

[1581] "Colorization" is the process of adding color to monochrome images or videos.

[1582] "Visualization" is a means of displaying data visually.

[1583] "Virtualization" is a method of recreating elements of the real world and past events in a virtual reality environment.

[1584] A "display device" refers to a device that allows a user to experience a virtual reality environment, and more specifically, virtual reality goggles.

[1585] "Streaming" is the process of continuously transmitting data to a device in real time.

[1586] "An interactive means of manipulation" refers to a function that allows users to select and manipulate objects within a virtual reality environment.

[1587] "Feedback" refers to information such as impressions and suggestions for improvement that users provide after experiencing something.

[1588] An "emotion recognition engine" is software that collects and analyzes emotional data from a user's facial expressions and voice.

[1589] A "generative AI model" is an algorithm that uses artificial intelligence to colorize monochrome images or generate other data.

[1590] An "interaction log" is data that records the user's operation history.

[1591] Adjusting "display content" is the process of changing the pace of the experience and the content displayed to match the user's emotional state.

[1592] This invention is a system that enables users to learn about historical events intuitively and experientially. The system includes an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1593] The system primarily consists of the following components: terminals, servers, an emotion recognition engine, and a communication network connecting them. Terminals are devices for user connection, while servers are central devices that perform data processing, storage, and analysis. The emotion recognition engine is software for collecting and analyzing emotion data.

[1594] Hardware and software specifications

[1595] Devices: For example, general tablet devices and personal computers (e.g., iPad Pro, Microsoft Surface), or virtual reality goggles (e.g., Oculus Rift).

[1596] Servers: For cloud servers, we will use Amazon Web Services (AWS) or other cloud service providers.

[1597] Emotion Recognition Engine: Uses NVIDIA's DeepStream SDK to analyze emotional data from the user's facial expressions and voice.

[1598] Generative AI Model: Uses OpenAI APIs and other AI models to colorize images and videos.

[1599] Database: Relational databases such as MySQL and PostgreSQL are used for storing and managing information.

[1600] 3D modeling software: Use Blender or Unity to generate 3D models related to historical events.

[1601] Program Processing Overview

[1602] Application startup and initial setup

[1603] When a user launches the application on their device, the home screen is displayed. The application internally starts its emotion recognition engine and waits for the user to be ready.

[1604] Selection of Historical Events

[1605] The user selects a specific historical event from the home screen. For example, they might select "World War II" and then "The Normandy Landings."

[1606] Sending a data request

[1607] The terminal generates a data request based on the user's selection information and sends it to the server.

[1608] Searching and retrieving data

[1609] The server receives a data request and retrieves the corresponding data (text information, images, videos, 3D models) from its internal database.

[1610] Colorization of data and generation of 3D models

[1611] The server processes the acquired data using a generation AI model to colorize monochrome images. It also optimizes 3D models and places objects such as tanks and soldiers.

[1612] Data integration and preparation for VR experiences

[1613] The server integrates text information, audio narration, and interactive elements to generate a data package for the VR experience.

[1614] Providing streaming and VR experiences

[1615] The server streams the data package to the terminal in real time, and the user puts on VR goggles to begin the experience.

[1616] Collection and analysis of emotional data

[1617] The emotion recognition engine collects emotional data in real time from the user's facial expressions and voice, and adapts it to the user's experience.

[1618] Providing feedback and improving the learning experience

[1619] After the experience ends, the user provides feedback, and the device sends that data to the server. The server analyzes the feedback and sentiment data to extract insights for improving the next learning experience.

[1620] Examples of specific cases and prompt statements

[1621] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. This data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion recognition engine analyzes the user's emotions in real time and adjusts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1622] Examples of prompts to input into a generative AI model:

[1623] "Please provide detailed information regarding the Normandy landings."

[1624] "Please analyze the emotional data of users when they selected the Normandy landings."

[1625] "Please integrate the data for the relevant historical event into the VR experience."

[1626] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1627] Step 1: Launching and initial setup of the application

[1628] The user launches the "HistoricalLearn" application on their device.

[1629] Input: Application launched by user action.

[1630] The device displays the application's home screen and initializes the emotion recognition engine.

[1631] Output: Home screen displayed, emotion recognition engine ready.

[1632] Specific operation: The application calls the Emotion Recognition API in the backend to complete the initial setup.

[1633] Step 2: Selecting Historical Events

[1634] The user selects a specific historical event, "World War II," from the application's menu, and then chooses "The Normandy Landings."

[1635] Input: User selection operation.

[1636] The device collects the selected information.

[1637] Output: Selected information "World War II" and "Normandy landings".

[1638] Specific operation: The application retrieves and prepares the selection information from the interface.

[1639] Step 3: Sending the Data Request

[1640] The device creates a data request based on the user's selection information and sends it to the server.

[1641] Input: User's selected information.

[1642] The terminal generates a data request and sends it to the server over the network.

[1643] Output: Data request sent to the server.

[1644] Specific operation: The data request includes selection information and is sent to the server via the API.

[1645] Step 4: Search and retrieve data

[1646] The server receives a data request and searches for and retrieves the relevant information from its internal database.

[1647] Input: Data request.

[1648] The server queries the database (MySQL) to retrieve relevant text information, images, videos, and 3D models.

[1649] Output: Acquired data (text information, images, videos, 3D models).

[1650] Specific operation: Use SQL queries to retrieve necessary information from the database and load it into memory.

[1651] Step 5: Colorizing the data and generating a 3D model

[1652] The server processes the acquired data using an AI model to colorize it and generate a 3D model.

[1653] Input: Acquired data (monochrome images, 3D models).

[1654] The server uses a generated AI model (OpenAI API) to colorize the image. It also uses 3D modeling software (Blender, Unity) to optimize the 3D model.

[1655] Output: Colorized image and optimized 3D model.

[1656] Specific operation: Colorize the image using an image processing algorithm, then position and adjust the model using a 3D modeling tool.

[1657] Step 6: Data Integration and Preparing for the VR Experience

[1658] The server integrates text information, audio narration, and interactive elements to generate a VR experience package.

[1659] Input: Colorized images, optimized 3D models, text information, audio narration, and interactive elements.

[1660] The server integrates the data and generates a data package for the VR experience.

[1661] Output: Data package for VR experience.

[1662] Specific action: Use data integration software to combine each element into one.

[1663] Step 7: Providing streaming and VR experiences

[1664] The server streams the generated data package to the terminal in real time.

[1665] Input: VR experience data package.

[1666] The server transmits data in real time using a streaming protocol.

[1667] The device processes the received data and displays it on the VR goggles.

[1668] Output: VR experience started.

[1669] Specific operation: Data is transmitted via a streaming server and displayed on the display device.

[1670] Step 8: Collecting and analyzing emotional data

[1671] While the user experiences historical events within the VR environment, the emotion engine collects the user's emotional data.

[1672] Input: User's facial expressions and voice.

[1673] The emotion recognition engine analyzes emotion data using facial expression recognition algorithms.

[1674] Output: Real-time sentiment data.

[1675] Specific operation: Emotional data is collected through the camera and microphone, and the emotion recognition engine analyzes the data.

[1676] Step 9: Providing feedback and improving the learning experience

[1677] Users provide feedback after completing the experience.

[1678] Input: User feedback information.

[1679] The device sends feedback information to the server.

[1680] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience.

[1681] Output: Analysis results and insights.

[1682] Specific actions: Analyze feedback data using natural language processing and data analysis tools to identify areas for improvement.

[1683] (Application Example 2)

[1684] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1685] In modern education, methods for learning historical events visually and experientially are limited. As a result, students and learners find it difficult to achieve the deep immersion necessary to understand history. Furthermore, traditional methods cannot individually customize the learning experience according to the learner's emotions and level of understanding. Therefore, new methods are needed to enhance the effectiveness of history learning.

[1686] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1687] This invention includes a server that recognizes the user's emotions in real time and adjusts the experience content, a server that analyzes the user's emotions using a generative AI model and dynamically adjusts the experience content using prompt sentences, and a server that analyzes feedback and emotion data to improve the learning experience. This enables learners to gain a deeply immersive, emotion-based experience while learning history in a individually customized way.

[1688] A "user" refers to a person who uses the system to experience and learn about historical events.

[1689] "Interface" refers to the screen or means of operation that allows a user to select a specific historical event.

[1690] A "server" refers to a computer system that receives data requests and retrieves, processes, and distributes information related to the relevant historical event.

[1691] A "data request" refers to a communication in which a user requests information about a historical event selected by the user from the server.

[1692] A "database" refers to data storage that accumulates data such as text information, images, videos, and 3D models related to historical events.

[1693] "Colorization" refers to the process of adding color to monochrome images or videos using artificial intelligence.

[1694] "Visualization" refers to the process of converting acquired data into a format that can be visually observed by the user.

[1695] "VR conversion" refers to the process of converting data into a format that can be applied to a virtual reality environment.

[1696] A "terminal" refers to a device used by a user to operate the system and experience VR.

[1697] "Streaming" refers to a method of continuously delivering data in real time.

[1698] "VR goggles" refers to a headset device used by users to experience virtual reality.

[1699] "Interactive" refers to the ability of users to interact with the system.

[1700] "Recognizing emotions in real time" refers to instantly reading emotions from the user's facial expressions, voice, and other cues.

[1701] "Adjusting the experience" refers to changing the content displayed and the progression of scenes according to the user's emotions and situation.

[1702] "Feedback" refers to the impressions and evaluations that users provide after completing an experience.

[1703] "Emotional data" refers to data about the user's emotional state collected during their experience.

[1704] A "generative AI model" refers to a model generated using artificial intelligence technology, and is primarily used for image colorization and sentiment analysis.

[1705] A "prompt" refers to a text instruction used to give instructions to a generative AI model.

[1706] This invention provides a system that allows users to learn about historical events intuitively and experientially. The system incorporates an emotion engine that recognizes the user's emotions in real time and provides feedback to the learning experience.

[1707] First, the user launches the application on their device (e.g., a smartphone or VR goggles) and checks the home screen. The application performs the necessary initial setup and prepares to activate the emotion engine. The user then selects a specific historical event through the interface.

[1708] The terminal receives the selection information and sends a data request containing that information to the server. The server receives the data request, searches its internal database for information related to the relevant historical event (text information, images, videos, 3D models), and retrieves it. The retrieved data is colorized using artificial intelligence (e.g., generative AI models). Monochrome footage is analyzed frame by frame, and color is added.

[1709] Next, the server generates or optimizes 3D models and places objects necessary for the historical scene (e.g., tanks and soldiers). Furthermore, the server integrates text information and audio narration, and adds interactive elements. When the user selects a specific object, detailed information is displayed. The server integrates all the data to generate a data package for the virtual reality experience. This package includes 3D models, colorized video, text information, audio narration, and interactive elements. The server streams the generated data package to the terminal in real time.

[1710] The device processes the received streaming data and displays it on the VR goggles. The user puts on the VR goggles and begins the experience. During the experience, the emotion engine collects and analyzes emotional data from the user's facial expressions and voice in real time. The emotion engine adjusts the displayed content according to the user's emotional state. For example, if the user feels excited or anxious, it changes the speed of the scene progression and the displayed content.

[1711] After the experience ends, users enter their thoughts and suggestions for improvement on a feedback screen. The user's feedback and emotional data are sent from the device to the server. The server analyzes the feedback and emotional data to extract insights for improving the next learning experience. This includes customizing the content to match the user's emotional state.

[1712] For example, if a user selects "The Normandy Landings," the server collects relevant data, colorizes it, and generates a 3D model. The data is streamed in real time, and the user puts on VR goggles to begin the experience. An emotion engine analyzes the user's emotions in real time and adapts the experience accordingly, providing a more immersive learning experience. After the experience ends, the user provides feedback, and this information is used to further improve the learning experience.

[1713] Example of a prompt:

[1714] "Start a learning session that simulates the Normandy landings of World War II. Adjust the experience based on user emotional feedback."

[1715] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1716] Step 1:

[1717] The device launches the application. The user launches the application and checks the home screen. The application performs initial setup and prepares the emotion engine to operate. The input data is the user's login information and initial setup information, and the output data is the system's initialization status.

[1718] Step 2:

[1719] The user selects a specific historical event using an interface. The input data is the event selection information from the user, and the output data is the identification information of the selected event. The terminal receives this information and sends a data request to the server.

[1720] Step 3:

[1721] The server receives a data request and retrieves information related to the relevant historical event from the database. The input data is the identification information of the selected event, and the output data includes text information, images, videos, and 3D models related to the event. The server executes database queries to collect the necessary data.

[1722] Step 4:

[1723] The server processes the acquired information. First, monochrome images and videos are colorized using a generative AI model. The input data is monochrome images and videos, and the output data is colorized images and videos. The generative AI model is used to analyze each frame and add color.

[1724] Step 5:

[1725] The server generates or optimizes 3D models and places the necessary objects for the historical scene. The input data is the initial data of the acquired 3D model, and the output data is the optimized 3D model. The server uses dedicated modeling software to place objects and construct the scene.

[1726] Step 6:

[1727] The server integrates text information and audio narration, and adds interactive elements. Input data consists of text and audio data, while output data is interactive educational content. Detailed information displayed when the user selects a specific object is configured.

[1728] Step 7:

[1729] The server integrates all the data to generate a data package for the virtual reality experience. Input data includes colorized video, optimized 3D models, integrated text, and audio; output data is a unified data package for VR. This package is streamed to the user in real time.

[1730] Step 8:

[1731] The device processes the received streaming data and displays it on the VR goggles. The input data is a data package streamed from the server, and the output data is the video and interactive content displayed on the VR goggles. The user puts on the VR goggles and begins the experience.

[1732] Step 9:

[1733] During the experience, the emotion engine analyzes the user's facial expressions and voice in real time and collects emotional data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional data. The emotion engine recognizes the user's real-time emotional state.

[1734] Step 10:

[1735] The emotion engine adjusts the displayed content according to the user's emotional state. The input data is analyzed emotion data, and the output data is the adjusted content. For example, if the user feels excited or anxious, the speed of the scene progression and the displayed content will be changed.

[1736] Step 11:

[1737] After the experience ends, users enter their feedback and suggestions for improvement on a feedback screen. The input data is the user's feedback, and the output data is the feedback information. This information is sent from the device to the server.

[1738] Step 12:

[1739] The server analyzes feedback and sentiment data to extract insights for improving the next learning experience. Input data consists of user feedback and sentiment data, while output data provides insights into the improved learning experience. The server then uses this information to customize the learning content.

[1740] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1741] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1742] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1743] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1744] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1745] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1746] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1747] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1748] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1749] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1750] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1751] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1752] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1753] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1754] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1755] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1756] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1757] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1758] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1759] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1760] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1761] The following is further disclosed regarding the embodiments described above.

[1762] (Claim 1)

[1763] A means of providing an interface for users to select specific historical events,

[1764] A means of sending a data request to a server regarding a selected historical event,

[1765] A means of obtaining information related to the relevant historical event from a database,

[1766] A means of processing acquired information, colorizing it, visualizing it, and creating a VR version,

[1767] A means of streaming processed data to a terminal in real time,

[1768] A means for users to experience and interact with historical events through VR goggles,

[1769] A means of sending user feedback to the server,

[1770] A system that includes a server to analyze feedback and implement measures to improve the learning experience.

[1771] (Claim 2)

[1772] The system according to claim 1, comprising means for using artificial intelligence to generate colorized images and videos.

[1773] (Claim 3)

[1774] The system according to claim 1, comprising means for recording user interaction logs and feedback and for customizing the next learning experience.

[1775] "Example 1"

[1776] (Claim 1)

[1777] A means of providing an interface for the user to select a specific past event,

[1778] A means of sending a data request to a server regarding selected past events,

[1779] A means of obtaining information related to a relevant past event from a collection of information,

[1780] A means of processing acquired information, colorizing it, visualizing it, and creating a virtual reality version of it,

[1781] A means for sequentially transmitting processed data to an information processing device in real time,

[1782] A means by which users can experience and interactively manipulate past events through a virtual reality device,

[1783] A means of sending user feedback to the server,

[1784] A system that includes a server to analyze feedback and implement measures to improve the learning experience.

[1785] (Claim 2)

[1786] The system according to claim 1, comprising means for utilizing an intelligent device to generate colorized images and videos.

[1787] (Claim 3)

[1788] The system according to claim 1, comprising means for recording user interactions and feedback and for customizing the next learning experience.

[1789] "Application Example 1"

[1790] (Claim 1)

[1791] A means of providing an interface for users to select specific historical events,

[1792] A means of sending a data request to a server regarding a selected historical event,

[1793] A means of obtaining information related to the relevant historical event from a database,

[1794] A means of processing acquired information, colorizing it, visualizing it, and creating a VR version,

[1795] A means of streaming processed data to a terminal in real time,

[1796] A means for users to experience and interact with historical events through VR goggles,

[1797] A means of sending user feedback to the server,

[1798] The server analyzes the feedback and provides means to improve the learning experience.

[1799] A means of recording and saving user experience information on the device in real time,

[1800] A system that includes means to reflect recorded information in subsequent experiences and provide a customized learning experience.

[1801] (Claim 2)

[1802] The system according to claim 1, comprising means for using artificial intelligence to generate colorized images and videos.

[1803] (Claim 3)

[1804] The system according to claim 1, comprising means for transferring acquired data to a VR device so that the user can experience it in an immersive way.

[1805] "Example 2 of combining an emotion engine"

[1806] (Claim 1)

[1807] A means of providing an interface for users to select specific historical events,

[1808] A means for sending a data request regarding a selected historical event to an information processing device,

[1809] A means of obtaining information related to the relevant historical event from a storage device,

[1810] A means of processing acquired information, colorizing it, visualizing it, and creating a virtual reality version of it,

[1811] A means for streaming processed data to a display device in real time,

[1812] A means by which users can experience and interact with historical events through a display device,

[1813] A means for transmitting user feedback to an information processing device,

[1814] Information processing devices analyze feedback and provide means to improve the learning experience.

[1815] A means of recognizing the user's emotional state in real time and adjusting the displayed content,

[1816] A system that includes means for an emotion recognition engine to collect and analyze emotion data from the user's facial expressions and voice.

[1817] (Claim 2)

[1818] The system according to claim 1, comprising means for utilizing a generative AI model to generate colorized images and videos.

[1819] (Claim 3)

[1820] The system according to claim 1, comprising means for recording user interaction logs and feedback and for customizing the next learning experience.

[1821] "Application example 2 when combining with an emotional engine"

[1822] (Claim 1)

[1823] A means of providing an interface for users to select specific historical events,

[1824] A means of sending a data request to a server regarding a selected historical event,

[1825] A means of obtaining information related to the relevant historical event from a database,

[1826] A means of processing acquired information, colorizing it, visualizing it, and creating a VR version,

[1827] A means of streaming processed data to a terminal in real time,

[1828] A means for users to experience and interact with historical events through VR goggles,

[1829] A means of recognizing user emotions in real time and adjusting the experience accordingly,

[1830] Means for sending user feedback and sentiment data to a server,

[1831] A system that includes means for servers to analyze feedback and sentiment data and improve the learning experience.

[1832] (Claim 2)

[1833] The system according to claim 1, comprising means for using artificial intelligence to generate colorized images and videos.

[1834] (Claim 3)

[1835] The system according to claim 1, comprising means for recording user interaction logs and feedback and for customizing the next learning experience.

[1836] (Claim 4)

[1837] The system according to claim 1, comprising means for analyzing user emotions using a generative AI model and dynamically adjusting the experience content using prompt sentences. [Explanation of symbols]

[1838] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of providing an interface for users to select specific historical events, A means of sending a data request to a server regarding a selected historical event, A means of obtaining information related to the relevant historical event from a database, A means of processing acquired information, colorizing it, visualizing it, and creating a VR version, A means of streaming processed data to a terminal in real time, A means for users to experience and interact with historical events through VR goggles, A means of sending user feedback to the server, A system that includes a server to analyze feedback and implement measures to improve the learning experience.

2. The system according to claim 1, comprising means for using artificial intelligence to generate colorized images and videos.

3. The system according to claim 1, comprising means for recording user interaction logs and feedback and for customizing the next learning experience.

Images