System

Abstract

A system is provided.SOLUTION: A system comprising: means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with teachers and other users in real-time; means for recreating historical events and cultural experiences using virtual reality technology to allow the user to experience them; and means for allowing the user to take lectures and answer questions in real-time with remote experts and instructors.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's educational environment, improving students' concentration and motivation is essential. However, traditional online learning lacks real-time interaction and a sense of presence, limiting learning effectiveness. Furthermore, geographical constraints make it difficult to provide diverse knowledge and cultural experiences. Given these circumstances, a more effective and engaging educational support system is needed. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following system.

[0006] a means for users to participate in a virtual classroom using a virtual reality headset and interact with the teacher and other users in real time;

[0007] A means of recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them;

[0008] A means to attend lectures and ask questions in real time with experts and lecturers in remote locations;

[0009] It is a system including:

[0010] Also, a means for transmitting user location information and interaction data to a server and synchronizing the latest status for all users;

[0011] Interactions with specific objects and characters in the virtual classroom are sent to the server, and events are triggered based on the results.

[0012] Further providing a means for synchronous distribution to other users improves the quality of the learning experience.

[0013] A "virtual reality headset" is a device that allows a user to visually and audibly experience a virtual environment.

[0014] A "virtual classroom" is a virtual educational space created using virtual reality technology.

[0015] "Real time" means happening or taking place simultaneously without a time lag.

[0016] A "user" is an individual such as a student or teacher who uses the system.

[0017] A "historical event" refers to an important event or phenomenon that actually occurred in the past.

[0018] A "cultural experience" is an experience of activities or environments associated with a particular culture.

[0019] A "remote location" refers to a location away from where the user currently resides.

[0020] An "expert" is someone who has advanced knowledge and experience in a particular field.

[0021] A "lecturer" is a person who teaches classes or lectures to conduct educational activities.

[0022] "Location information" refers to information about the user's current location within the virtual space.

[0023] "Interaction data" is information about the operations and actions performed by the user within the virtual space.

[0024] An "object" is a physical object that exists in a virtual space.

[0025] An "event" refers to a specific occurrence or action that occurs within a virtual space.

[0026] "Synchronous delivery" means delivering the same information to multiple users at the same time. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0028] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0029] First, the terms used in the following description will be explained.

[0030] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0031] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0032] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0033] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0034] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0035] [First embodiment]

[0036] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0037] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0038] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0039] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0040] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0041] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0042] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0043] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0044] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0045] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0046] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0047] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] This invention provides an educational support system that utilizes virtual reality (VR) technology to enable students to have a more immersive learning experience. Specifically, the system includes three main functions: virtual classroom, historical and cultural experience, and remote lecture participation.

[0049] Program processing

[0050] The main stakeholders are the server, the terminal, and the user.

[0051] 1. VR Virtual Classroom Experience

[0052] The server plays a central role, generating a 3D model of the virtual classroom and determining the initial positions and settings of each user. After the virtual classroom data is generated, the server transmits it to the terminals.

[0053] The device then renders the received 3D model data onto the VR headset, allowing the user to experience the visual and auditory sensations of being in the classroom. As the user uses the controller to move around the VR environment and interact with the teacher, the device sends positional and interaction data to the server.

[0054] The server receives this data and synchronizes it with all participants' devices, providing a real-time interactive environment.

[0055] 2. History and Culture Experience

[0056] The server generates a specific historical scenario and prepares the associated 3D model and simulation data, which are then sent to the device.

[0057] The device starts a simulation based on the received scenario data, providing the user with an experience that makes them feel as if they are actually there. The user explores and interacts within the simulation, and this interaction data is sent from the device to the server.

[0058] The server triggers specific events and synchronously distributes the results to other users, allowing users to enjoy a real-time interactive historical and cultural experience.

[0059] 3. Participating in lectures from remote locations

[0060] The server manages real-time communication with remote experts and lecturers. When a lecture starts, the server transmits the lecturer's video stream to the terminal.

[0061] The terminal displays the received video stream to the user. If the user wants to ask a question during the lecture, they can send the question by voice or text. This question data is sent from the terminal to the server.

[0062] The server relays questions to the instructor and sends the instructor's answers back to the user's device in real time, allowing users to participate in high-quality lectures and receive direct answers from experts regardless of location.

[0063] Specific examples

[0064] During a history class, students don VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. The data is then sent to the device, which renders the VR environment, allowing students to explore the ancient Egyptian landscape. When a student touches a specific object or interacts with a character, the data is sent from the device to the server and synchronized with the other students.

[0065] On another day, a lecture is given by a European expert. The server starts the lecturer's video stream and distributes it to the devices. Students can participate in the lecture in real time and ask questions. Questions are sent from the devices to the server, which relays them to the lecturer. The lecturer's answers are sent back to the students' devices via the server.

[0066] In this way, the system of the present invention provides three main functions: virtual classroom, historical and cultural experience, and remote lecture participation, which can dramatically improve students' concentration and learning effectiveness.

[0067] The processing flow will be explained below.

[0068] Processing steps for VR virtual classroom experience

[0069] Step 1:

[0070] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[0071] Step 2:

[0072] The server transmits the generated 3D model data and initial position information to the terminal.

[0073] Step 3:

[0074] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[0075] Step 4:

[0076] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[0077] Step 5:

[0078] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[0079] Step 6:

[0080] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[0081] History and Culture Experience Processing Steps

[0082] Step 1:

[0083] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[0084] Step 2:

[0085] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[0086] Step 3:

[0087] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[0088] Step 4:

[0089] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[0090] Step 5:

[0091] The terminal transmits the user's interaction data to the server in real time.

[0092] Step 6:

[0093] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[0094] Processing steps for remote lecture participation

[0095] Step 1:

[0096] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[0097] Step 2:

[0098] The server initiates the instructor's video stream and streams it to the terminal in real time.

[0099] Step 3:

[0100] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[0101] Step 4:

[0102] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[0103] Step 5:

[0104] The terminal transmits the user's question data to the server in real time.

[0105] Step 6:

[0106] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[0107] This allows users to have an interactive, high-quality learning experience without being restricted by geography.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] In conventional educational systems, it is difficult for students to participate in actual classrooms, to realistically recreate historical events or cultural experiences, or to receive lectures from experts or lecturers in remote locations in real time. As a result, there are problems that limit learning effectiveness and concentration. Furthermore, there is also the issue of incomplete synchronization of user interaction data and location information within the virtual environment, making it difficult to provide a smooth real-time interactive environment.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes: a means for allowing users to participate in a virtual learning environment using a virtual reality display device and interact with educators and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a central control unit for generating 3D model data and transmitting it to terminals; a terminal device for transmitting user position information and interaction data within the virtual environment to the central control unit and synchronizing the latest status for all users; and a means for transmitting interactions with specific objects and characters to the central control unit, triggering events, and synchronously distributing them to other users. This provides an environment in which multiple users can simultaneously learn while interacting in the same virtual space in real time, and also enables historical and cultural experiences to be realistically reproduced using virtual reality technology and lectures by lecturers in remote locations.

[0113] A "virtual reality display device" is a device that allows users to experience images and sounds in a virtual space in real time. Specifically, this includes head-mounted displays and VR goggles.

[0114] A "virtual learning environment" is a virtual space where users can learn or train using virtual reality technology. It is designed to mimic a real-world classroom or learning scenario.

[0115] An "educator" is a person who has a teaching role within a virtual learning environment, such as a teacher, lecturer, or instructor.

[0116] The "central control device" is the central computer of the system, and is a device that manages the generation of virtual environments, synchronization of user data, triggering of specific events, etc.

[0117] A "terminal device" is a device through which a user accesses a virtual reality environment, such as a computer, smartphone, or tablet.

[0118] "Interaction data" refers to data about the actions or behaviors a user performs within a virtual space, such as movements, clicks, and interactions with objects.

[0119] "3D model data" refers to data on digital objects that are represented in three-dimensional space. This includes classrooms, historical buildings, characters, and other virtual objects.

[0120] "Real-time synchronization of execution" is a technology that allows multiple users to simultaneously display and reflect status and events in a virtual space shared by multiple users.

[0121] An "event trigger" is a process that is automatically executed when a specific condition or action occurs. This occurs based on the scenario progression or user interaction within the virtual learning environment.

[0122] "Distance learning" is a format in which a lecturer in a physically distant location delivers a lecture in real time through a virtual learning environment.

[0123] This invention relates to an educational support system that allows users to experience a realistic learning environment using virtual reality (VR) technology. The system provides three main functions: virtual classroom experience, historical and cultural experience, and remote lecture participation.

[0124] Virtual Classroom Experience

[0125] Hardware and Software:

[0126] The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D model of the virtual classroom, and uses a communication protocol such as WebSocket to synchronize data in real time.

[0127] The device uses a VR headset such as Oculus or HTC Vive, and the device uses VR-specific software to render the received 3D model data in real time.

[0128] Users use a VR headset and controllers to move freely around the virtual classroom and interact with the educator.

[0129] Examples:

[0130] When a user puts on a VR headset and enters a virtual classroom, the server generates a 3D model and initializes the user's location information. The device instantly renders the classroom environment based on the data sent from the server to the device. When a user raises their hand to ask a question or moves around the classroom, this interaction data is sent via the device to the server and synchronized with the devices of other participants.

[0131] History and Cultural Experience

[0132] Hardware and Software:

[0133] The server generates historical scenarios using 3D modeling software such as Blender or Maya, and also uses AI techniques to set specific event triggers.

[0134] The device uses Unity or Unreal Engine to render the VR environment based on the received scenario data.

[0135] Users interact with and explore historical buildings and characters using VR controllers.

[0136] Examples:

[0137] When students take an "Ancient Egypt" class, the server generates 3D models of the pyramids, pharaoh's palaces, and other structures. The server sends data to the device, which then displays these models on the headset. Using the controller, users explore the camopets and murals inside the pyramids, and their interaction data is instantly shared with other students.

[0138] Participating in remote lectures

[0139] Hardware and Software:

[0140] The server retrieves the instructor's video stream using streaming technologies such as Zoom or WebRTC.

[0141] The device uses software to display the video stream in real time, allowing users to watch the lecture through a display or VR headset.

[0142] The user inputs a question using a microphone or keyboard and sends it to the server via a communication protocol.

[0143] Examples:

[0144] When a European expert gives a lecture, the server captures his video stream and sends it to the device. Students view the video on their VR headsets and, if they have questions, use their microphones to send them to the server in real time. The server then relays the questions to the lecturer, obtains the lecturer's answers, and immediately sends them back to the user. This process allows for seamless communication even with lecturers in remote locations.

[0145] Example prompt sentence:

[0146] "Write a program that generates a VR simulation of Ancient Egypt and allows users to explore inside the pyramids."

[0147] Through each function, this system can add new value to traditional teaching methods and dramatically improve the learning experience.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] System program processing

[0150] VR virtual classroom experience

[0151] Step 1:

[0152] The server generates a 3D model of the virtual classroom using a 3D engine such as Unity or Unreal Engine.

[0153] Input: 3D model template, user's initial position data.

[0154] Data processing: The 3D engine generates a 3D model based on the template data and sets the initial position for each user.

[0155] Output: The completed 3D model data.

[0156] Specific operation: The server constructs a virtual classroom using the template and initial position data.

[0157] Step 2:

[0158] The server sends the generated 3D model data to the terminal.

[0159] Input: Completed 3D model data.

[0160] Data processing: Converting the generated data into a format that can be sent.

[0161] Output: 3D model data for transmission.

[0162] Specific operation: The server encodes the data and sends it to the device.

[0163] Step 3:

[0164] The device renders the received 3D model data on the VR headset.

[0165] Input: 3D model data received from the server.

[0166] Data processing: Decode the data and render it as a VR environment.

[0167] Output: The rendered VR environment.

[0168] What it does: A virtual classroom is displayed in real time on the headset.

[0169] Step 4:

[0170] Users use VR controllers to move and interact within the virtual classroom.

[0171] Input: User operation data (movements, questions, etc.).

[0172] Data processing: Operation data is acquired in real time and converted into location information.

[0173] Output: User's current location and interaction data.

[0174] Specific actions: The user physically manipulates the controller and moves around in the virtual environment.

[0175] Step 5:

[0176] The device transmits the acquired user location information and interaction data to the server.

[0177] Input: User's current location and interaction data.

[0178] Data processing: Package the data and send it to the server.

[0179] Output: The packaged data.

[0180] Specific operation: Data is sent from the device to the server.

[0181] Step 6:

[0182] The server receives all user data and performs the synchronization process.

[0183] Input: Interaction data submitted by each user.

[0184] Data processing: Integrate data and update it to the latest information.

[0185] Output: The synchronized data.

[0186] Specific operation: The server sends data to all devices using WebSocket or similar and shares the latest status.

[0187] History and Cultural Experience

[0188] Step 1:

[0189] The server generates historical scenarios using 3D modeling software such as Blender or Maya.

[0190] Input: Historical scenario raw material file.

[0191] Data processing: Create and optimize scenarios using 3D modeling software.

[0192] Output: 3D scenario data.

[0193] What it does: The server builds and optimizes the scenario for the simulation.

[0194] Step 2:

[0195] The server transmits the generated historical scenario data to the terminal.

[0196] Input: Simulation data.

[0197] Data processing: Compression and conversion into a transmittable format.

[0198] Output: Compressed simulation data.

[0199] Specific operation: The server compresses the data and sends it to the terminal.

[0200] Step 3:

[0201] The device renders the received scenario data on the VR headset.

[0202] Input: Send data.

[0203] Data processing: Decode the data and render it into a VR environment.

[0204] Output: A rendered simulated environment.

[0205] Specific Behavior: Rendered to allow the user to begin exploring within the scenario.

[0206] Step 4:

[0207] The user interacts with specific objects and characters within the VR environment.

[0208] Input: User operation data.

[0209] Data processing: Convert the acquired operation data into interaction data.

[0210] Output: Interaction data.

[0211] Concrete behavior: The user explores and interacts with objects within the simulation.

[0212] Step 5:

[0213] The device sends the acquired interaction data to the server.

[0214] Input: Interaction data from the user.

[0215] Data processing: packaging and preparation for transmission.

[0216] Output: The packaged data.

[0217] Specific operation: Data is sent from the device to the server.

[0218] Step 6:

[0219] The server triggers specific events and synchronously distributes the results to other users.

[0220] Input: Interaction data.

[0221] Data processing: Executes event triggers and generates result data.

[0222] Output: Triggered event result data.

[0223] Specific operation: The event results are sent to other users' devices and synchronized.

[0224] Participating in remote lectures

[0225] Step 1:

[0226] The server retrieves the instructor's video stream using a streaming technology such as Zoom or WebRTC.

[0227] Input: Instructor video feed.

[0228] Data processing: Compression and optimization of video streams.

[0229] Output: Compressed video stream.

[0230] Specific operation: The instructor's video is uploaded to the server in real time.

[0231] Step 2:

[0232] The server transmits the video stream to the terminal.

[0233] Input: Compressed video stream.

[0234] Data processing: Format conversion as needed.

[0235] Output: Video stream for device.

[0236] Specific operation: The server delivers streaming data to the terminal.

[0237] Step 3:

[0238] The terminal displays the received video stream to the user.

[0239] Input: Video stream from the server.

[0240] Data processing: Decode and display the stream.

[0241] Output: Real-time video display.

[0242] What happens: The user watches the lecture on a display or VR headset.

[0243] Step 4:

[0244] The user inputs and submits a question by voice or text.

[0245] Input: Audio or text data.

[0246] Data processing: Converting audio to text based on data functions (in the case of audio).

[0247] Output: Question data.

[0248] What it does: The user creates and submits a question using a microphone or keyboard.

[0249] Step 5:

[0250] The terminal transmits the question data to the server.

[0251] Input: Question data.

[0252] Data processing: Packaging and preparing for transmission.

[0253] Output: Packaged question data.

[0254] Specific operation: Question data is sent from the terminal to the server.

[0255] Step 6:

[0256] The server relays the question to the instructor, obtains the instructor's answer, and returns it to the terminal.

[0257] Input: Question data.

[0258] Data processing: Converting data into a format suitable for instructors.

[0259] Output: Transformed question data and instructor answer data.

[0260] Specific operation: The server forwards the question to the instructor, obtains the answer, and then sends it back to the device.

[0261] In this way, the system can provide a complex learning environment and offer users a real-time interactive learning experience.

[0262] (Application example 1)

[0263] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0264] When providing educational support systems using virtual reality technology, the challenge is to ensure that users can enjoy a consistent, real-time learning experience while accessing high-quality educational content without geographical constraints. Conventional systems suffer from delays in real-time communication and insufficient information synchronization between users, making it difficult to maximize learning effectiveness.

[0265] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0266] In this invention, the server includes: means for allowing users to participate in a virtual classroom using a virtual reality headset and interact with teachers and other users in real time; means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; means for allowing users to attend lectures and engage in Q&A sessions with experts and lecturers in remote locations in real time; and means for providing a content distribution service and allowing users to enjoy a learning experience in virtual reality. This allows users to interact with teachers and other users in real time, enjoy an immersive learning experience about history and culture, and participate in lectures from high-quality experts without being restricted by geographical location.

[0267] - A "virtual reality headset" is a device that allows a user to immerse themselves in a computer-generated virtual environment through visual and auditory experiences.

[0268] "User" refers to a learner or participant who uses an educational support system using virtual reality technology.

[0269] A "virtual classroom" is a simulated educational space where teachers and students interact and learn in real time within a virtual reality environment.

[0270] "Real-time" refers to instantaneous communication and data synchronization with minimal delay.

[0271] "Interaction" means the act of users exchanging information through a communication means.

[0272] "Teacher" refers to a professional who conducts lessons within a virtual classroom and provides educational content to students.

[0273] "Other users" refers to participants in a simultaneous learning activity within a virtual classroom or other virtual environment.

[0274] "Historical events" refer to significant events or occurrences that actually occurred in the past and are re-enacted for educational purposes.

[0275] "Cultural Experience" refers to a virtual reality scenario that allows people to learn about the culture of a particular region or era through hands-on experience.

[0276] A "remote location" refers to a physically separate location that is accessed via a communication means such as the Internet.

[0277] "Expert" refers to a lecturer or advisor with advanced knowledge and skills in a particular field.

[0278] A "lecture" is an educational activity that involves a coherent explanation or instruction on a particular topic.

[0279] "Question and answer" is a process in which a user asks a question and an expert or lecturer provides an answer to the question.

[0280] "Content Delivery Service" refers to an online platform for providing educational content to users using virtual reality technology.

[0281] "Interaction data" refers to data related to user operations and behavior, which is processed and synchronized on the server.

[0282] "Server" refers to a central processing unit or system for providing virtual classrooms and related services.

[0283] The present invention is a system for enhancing educational experiences by utilizing virtual reality technology. Specific embodiments thereof are described below.

[0284] Hardware and Software Configuration

[0285] Hardware used:

[0286] VR headset (e.g. Oculus Rift, HTC Vive)

[0287] Server (standard server with high performance)

[0288] User device (PC or tablet)

[0289] Software used:

[0290] Unity 3D (3D model rendering engine)

[0291] WebRTC (protocol for real-time communication)

[0292] Node.js (provides real-time processing capabilities on the server side)

[0293] Amazon AWS (cloud storage and data distribution service)

[0294] Overview of program processing

[0295] Server Role:

[0296] 1. Creating a virtual classroom

[0297] The server generates a 3D model of the virtual classroom using Unity 3D, sets the initial position and environment for each user, and sends the generated 3D model data to the user's device.

[0298] 2. Real-time communication

[0299] The server uses WebRTC to manage real-time communication between users, supporting audio and video exchanges with teachers and other users, providing a smooth interactive environment.

[0300] 3. Data synchronization

[0301] It collects user location and interaction data and synchronizes it with other users in real time. The data is stored on Amazon AWS and distributed to other devices as needed.

[0302] Device role:

[0303] 1. Rendering the Data

[0304] The device receives 3D model data from the server and renders it on the VR headset using Unity 3D, allowing users to experience virtual classrooms and historical scenarios.

[0305] 2. Collecting interactions

[0306] It collects user actions and movements in the environment in real time and sends the data to a server, which keeps it synchronized with other users.

[0307] Examples:

[0308] Virtual Classroom Experience

[0309] A user puts on a VR headset and joins a virtual classroom. The server sets the user's initial position and sends 3D model data to the device. The device renders this data on the VR headset, allowing the user to move freely within the virtual classroom and interact with other users and the teacher in real time.

[0310] History and Cultural Experience

[0311] To learn about historical events, users put on a VR headset. For example, a server generates an ancient Egyptian scenario and transmits the data to the device. Users can explore ancient Egyptian streets and pyramids and interact with an interactive guide.

[0312] Example prompt sentence:

[0313] "Generate a VR experience scenario about Egyptian history, including detailed depictions of the inside of the pyramids and an interactive guide that users can interact with."

[0314] summary

[0315] The system of the present invention provides users with a comprehensive and immersive learning experience through the collaboration of the server and user terminals, enabling them to receive high-quality education without being restricted by geographical location, maximizing the effectiveness of their learning.

[0316] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0317] Step 1: The server generates a 3D model of the virtual classroom

[0318] Input: Virtual classroom design data, user initial location information

[0319] Processing: The server uses Unity 3D to generate a 3D model of the virtual classroom. It performs calculations to determine the user's initial position and environment settings.

[0320] Output: 3D model data of the completed virtual classroom, initial user position information

[0321] Step 2: The server sends the virtual classroom data to the device

[0322] Input: 3D model data of the virtual classroom, user's initial position information

[0323] Processing: The server sends the generated 3D model data and initial location information to each user's device.

[0324] Output: 3D model data sent to the device, initial position information

[0325] Step 3: The device renders the received data to the VR headset

[0326] Input: 3D model data received from the server, user's initial location information

[0327] Processing: The device uses Unity 3D to render the received 3D model data on the VR headset, allowing the user to experience the visual and auditory world of the virtual classroom.

[0328] Output: Visual and auditory data of the virtual classroom displayed on a VR headset

[0329] Step 4: User moves and interacts in the VR environment

[0330] Input: VR headset, controller

[0331] Processing: Using the headset and controllers, users move around the VR environment and interact with other users and objects, and their actions are collected as data.

[0332] Output: Collected user location and interaction data

[0333] Step 5: The device sends the interaction data to the server

[0334] Input: User location and interaction data

[0335] Processing: The device sends the collected data to the server in real time.

[0336] Output: User location and interaction data sent to the server

[0337] Step 6: The server synchronizes the data to other users

[0338] Input: User location and interaction data

[0339] Processing: The server analyzes the received location and interaction data and generates the data necessary to synchronize it with other users' devices.

[0340] Output: Location and interaction data synced to other users

[0341] Step 7: Trigger specific events in the scenario

[0342] Input: Interaction data stored on the server

[0343] Processing: When a certain condition is met, the server triggers an event in the scenario and notifies all users of that information.

[0344] Output: Event trigger information and related data sent to each device

[0345] Step 8: The server delivers the video stream of the remote lecture.

[0346] Input: Video stream from instructor

[0347] Processing: The server uses WebRTC to deliver the instructor's video and audio streams to each user's device in real time.

[0348] Output: Real-time video stream of lectures delivered to user devices

[0349] Step 9: Users attend the remote lecture and submit questions

[0350] Input: User voice or text data

[0351] Processing: While listening to the lecture, the user asks questions by voice or text, and this data is sent from the terminal to the server.

[0352] Output: Question data sent to the server

[0353] Step 10: The server relays the question to the instructor and delivers the answer

[0354] Input: User question data, instructor answer data

[0355] Processing: The server relays the user's questions to the instructor, collects the instructor's answers in real time, and delivers them to each user's device.

[0356] Output: Teacher's response data distributed to each user's device

[0357] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0358] This invention provides an educational support system that uses virtual reality (VR) technology to enable students to have a more immersive learning experience, and in particular, by combining it with an emotion engine, it realizes interactive education based on the user's emotional state. Specifically, this system includes a virtual classroom, historical and cultural experiences, remote lecture participation, and user emotion recognition.

[0359] Program processing

[0360] The main stakeholders are the server, the terminal, and the user.

[0361] 1. VR Virtual Classroom Experience

[0362] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device.

[0363] The device renders the received 3D model data on the VR headset, providing the user with a virtual classroom view. The user can explore the VR classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[0364] 2. History and Culture Experience

[0365] The server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[0366] 3. Participating in lectures from remote locations

[0367] The server prepares lecture information from remote experts and lecturers and sets access rights for users. When the lecture starts, the server starts the lecturer's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the lecturer and sends the lecturer's answer to the user's terminal in real time.

[0368] 4. Incorporating an Emotional Engine

[0369] The server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc., and recognizes them as emotion data.

[0370] The device collects the user's emotional data in real time and sends it to the server. The server then adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if they are dissatisfied or confused, the server will provide additional explanations or a different scenario.

[0371] Specific examples

[0372] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the teacher and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[0373] If one student shows interest in the pyramid while another struggles with a difficult problem, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[0374] Furthermore, when a lecture by a remote expert is being given, the server transmits the expert's video stream to the device. When a student has a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. Furthermore, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[0375] In this way, the system of the present invention integrates four main functions: virtual classroom, historical and cultural experience, remote lecture participation, and emotion recognition engine, thereby dramatically improving the user's learning experience.

[0376] The processing flow will be explained below.

[0377] Processing steps for VR virtual classroom experience

[0378] Step 1:

[0379] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[0380] Step 2:

[0381] The server transmits the generated 3D model data and initial position information to the terminal.

[0382] Step 3:

[0383] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[0384] Step 4:

[0385] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[0386] Step 5:

[0387] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[0388] Step 6:

[0389] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[0390] History and Culture Experience Processing Steps

[0391] Step 1:

[0392] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[0393] Step 2:

[0394] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[0395] Step 3:

[0396] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[0397] Step 4:

[0398] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[0399] Step 5:

[0400] The terminal transmits the user's interaction data to the server in real time.

[0401] Step 6:

[0402] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[0403] Processing steps for remote lecture participation

[0404] Step 1:

[0405] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[0406] Step 2:

[0407] The server initiates the instructor's video stream and streams it to the terminal in real time.

[0408] Step 3:

[0409] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[0410] Step 4:

[0411] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[0412] Step 5:

[0413] The terminal transmits the user's question data to the server in real time.

[0414] Step 6:

[0415] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[0416] Built-in processing steps for the emotion engine

[0417] Step 1:

[0418] The server uses an emotion engine to analyze the user's emotional state.

[0419] Step 2:

[0420] The device collects the user's facial expressions, voice tone, and behavioral patterns as emotional data in real time.

[0421] Step 3:

[0422] The device transmits the collected emotion data to a server in real time.

[0423] Step 4:

[0424] The server adjusts the educational content based on the received emotional data, for example, if the user expresses dissatisfaction, it provides additional explanations or new approaches.

[0425] Step 5:

[0426] The server changes the behavior of objects and characters in the virtual environment based on the emotional data to optimize the user's experience.

[0427] Step 6:

[0428] The device follows instructions from the server and provides appropriate feedback to the user, customizing the learning experience according to the user's emotional state.

[0429] Example 2

[0430] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Conventional educational systems have few opportunities for users to actively participate in the learning content, making it difficult to provide a realistic learning experience. Furthermore, they lack the technology to analyze the user's emotional state and dynamically adjust the educational content accordingly. As a result, there are issues with maintaining users' motivation to learn and making it difficult for them to progress effectively.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with an instructor and other users in real time, a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them, a means for allowing a user to attend lectures and engage in Q&A sessions with experts and instructors in remote locations in real time, and a means for analyzing the user's emotional state and dynamically adjusting educational content based on the results. This allows the user to increase their motivation to learn through a realistic learning experience and progress effectively in their studies.

[0433] A "virtual reality headset" is a device that allows a user to immerse themselves in a virtual reality environment, providing visual and audio information.

[0434] A "virtual classroom" is a virtual space that users can participate in through a virtual reality headset, recreating an environment for educational activities.

[0435] An "instructor" is a person or system that has the role of educating users, and includes teachers and lecturers.

[0436] "Historical events" refer to events or occurrences that actually occurred in the past and are used as educational material.

[0437] A "cultural experience" is an experience provided to users through a virtual environment based on a specific culture or historical background.

[0438] An "expert" is a person with advanced knowledge and skills in a particular field who is qualified to teach or lecture.

[0439] "Emotional state" refers to the internal psychological state that can be inferred from the user's facial expression, tone of voice, behavioral patterns, etc.

[0440] "Dynamic adjustment" refers to the process of changing educational content in real time based on the user's reactions and situation.

[0441] "Interaction data" is information about the actions and operations a user performs within a virtual environment that is used by the system to understand those actions and generate appropriate responses.

[0442] "Triggering an event" means that the system automatically initiates a specific action or scenario when certain conditions are met.

[0443] "Synchronous distribution" is the process of enabling multiple users to share the same information or state in real time.

[0444] This invention combines virtual reality (VR) technology with an emotion engine to provide users with a realistic learning experience. Below, we will explain in detail what hardware and software are used to build the system and how data processing and calculations are performed.

[0445] Hardware and software used

[0446] Servers are high-performance computers, and recommended specifications include a high-clock processor, large amounts of memory, and SSD storage. Software running on the server includes a data communication system using WebSocket or HTTP, and a game engine such as Unity or Unreal Engine.

[0447] A device is a user-interface device, which can include a VR headset with a high-resolution display (e.g., Oculus Rift or HTC Vive) and a connected PC or mobile device. The software running on the device includes Unity or Unreal Engine for rendering the VR environment and controller software for processing user input.

[0448] The user wears a VR headset and uses a controller to explore the virtual environment. To collect the user's emotional state, biometric devices such as a camera, microphone, and heart rate sensor are connected to the device.

[0449] System processing overview

[0450] The server generates a 3D model of the virtual classroom and sets the user's initial location. This data is sent from the server to the device using WebSocket or HTTP protocol. The device then renders the received 3D model data on the VR headset using Unity or Unreal Engine, providing the user with a view of the virtual classroom.

[0451] Users can explore the VR classroom using a controller and interact with instructors and other users in real time. During this time, the user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[0452] In a historical and cultural experience, the server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[0453] When participating in a remote lecture, the server prepares lecture information from a remote expert or instructor and sets access rights for the user. When the lecture starts, the server starts the instructor's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the instructor and sends the instructor's answer to the user's terminal in real time.

[0454] Regarding the incorporation of an emotion engine, the server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc. and recognizes them as emotional data. The device collects the user's emotional data in real time and sends it to the server. The server dynamically adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if the user is dissatisfied or confused, additional explanations or a different scenario will be provided.

[0455] Specific examples

[0456] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the instructor and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[0457] If one student shows interest in the pyramid while another struggles with a difficult problem during the scenario, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[0458] When a lecture is given by a remote expert, the server sends the expert's video stream to the device. When a student wants to ask a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. In addition, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[0459] Prompt Sentence Examples

[0460] "Describe a scenario in which students would use VR to learn about ancient Egypt. Also, detail how an emotion recognition engine would help."

[0461] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0462] Program processing flow

[0463] Step 1:

[0464] The server generates a 3D model of the virtual classroom and the user's initial position information.

[0465] Specifically, the server uses a game engine such as Unity or Unreal Engine to create a 3D model of the virtual classroom and sets the user's initial location information based on that. The input is the virtual classroom design and user ID, and the output is the generated 3D model data and initial location information, saved in JSON format.

[0466] Step 2:

[0467] The server sends the generated 3D model data and initial location information to the terminal.

[0468] The server sends this data to the device using WebSocket or HTTP protocol. The input includes the 3D model data and initial position information generated in step 1, and the output includes the data sent to the device.

[0469] Step 3:

[0470] The device renders the received 3D model data on the VR headset.

[0471] The device processes the received data using Unity or Unreal Engine and renders it in real time on the VR headset. The input includes 3D model data received from the server, and the output includes the virtual classroom view seen by the user.

[0472] Step 4:

[0473] Users explore and interact within the virtual classroom.

[0474] Using a VR headset and controllers, users can move freely around the virtual classroom and interact with other users and objects in real time. The inputs are the user's visual, auditory, and controller inputs, and the output is the interaction data generated.

[0475] Step 5:

[0476] The terminal transmits the user's interaction data to the server.

[0477] The device collects user location information and operation logs in real time and sends them to the server using WebSocket. The input includes user interaction data, and the output includes data sent to the server.

[0478] Step 6:

[0479] The server synchronizes the received interaction data with the terminals of other users.

[0480] The server synchronizes the received data to all other devices in real time. The input includes user interaction data, and the output includes data synchronized to other devices.

[0481] Step 7:

[0482] The server generates the data for a particular historical scenario.

[0483] The server generates 3D models and scenario scripts based on specific historical events and sends them to the terminal. The inputs include historical data and scenario design, and the outputs include the generated 3D models and scenario scripts.

[0484] Step 8:

[0485] The terminal starts the simulation, providing the user with a realistic historical experience.

[0486] The device starts a simulation using Unity or Unreal Engine based on the received scenario data. The input includes the received scenario data, and the output includes the virtual environment provided to the user.

[0487] Step 9:

[0488] The terminal transmits the interaction data to the server.

[0489] The device collects the interaction data of the user in the virtual environment in real time and sends it to the server.The input includes the user's interaction data, and the output includes the data sent to the server.

[0490] Step 10:

[0491] The server triggers a specific event and synchronizes the results to other devices.

[0492] The server triggers a specific event based on the received interaction data and synchronously distributes the result to all other terminals. The input includes the interaction data based on the trigger condition, and the output includes the synchronized event result.

[0493] Step 11:

[0494] The server prepares the lecture information for the remote location and transmits it to the terminal.

[0495] The server manages the contents and schedule of the lectures, sets access rights for participating users, and sends this information to the terminals. The input includes the contents and schedule of the lectures, and the output includes access rights information.

[0496] Step 12:

[0497] The terminal renders the instructor's video stream to the user.

[0498] The server starts the video stream at the beginning of the lecture, and the terminal renders it in real time.,The input includes the video stream data, and the output,includes the lecture video that is displayed to the user.

[0499] Step 13:

[0500] The terminal transmits the user's question data to the server.

[0501] When a user wants to ask a question, the question data is sent from the terminal to the server. The input includes the user's question data, and the output includes the data sent to the server.

[0502] Step 14:

[0503] The server relays questions to the instructor and delivers answers to the user.

[0504] The server relays questions to the instructor and sends the answers to the user's device in real time. The input includes the user's question data and the output includes the instructor's answer data.

[0505] Step 15:

[0506] The terminal transmits the user's emotion data to the server.

[0507] The device collects user emotion data using a camera and microphone and sends it to the server. The input includes the user emotion data, and the output includes the data sent to the server.

[0508] Step 16:

[0509] The server analyzes the received emotional data and dynamically adjusts the educational content.

[0510] The server uses an emotion analysis algorithm to analyze the user's emotional state and dynamically adjusts the educational content based on the results. The input is the user's emotional data, and the output is the adjusted educational content.

[0511] These steps enable the system of the present invention to provide an immersive learning experience for the user and to adapt educational content based on the user's emotional state in real time.

[0512] (Application example 2)

[0513] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0514] In modern educational and commercial environments, users have limited interactive options for enriching their experiences. Current virtual reality technologies lack the means to provide appropriate feedback and information based on the user's emotional state. This leads to a poor user experience and makes it difficult for users to learn or purchase effectively.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with a teacher and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a means for allowing a user to explore a virtual environment using smart glasses and present specific information or content based on the user's gaze or movements; and a means for recognizing the user's emotional state in real time and providing information or promotions according to that state. This enables an interactive experience according to the user's emotional state, dramatically improving the user experience in education and commerce.

[0516] A "virtual reality headset" is a device worn by a user to display a virtual reality environment, and is attached to the head.

[0517] A "virtual classroom" is a virtual classroom environment recreated using virtual reality technology that users can access through a virtual headset to create an immersive learning experience.

[0518] "Real-time" refers to the state in which data and information are processed and transmitted immediately, without delay, allowing users to interact with other users and systems instantly.

[0519] "Historical events" are important events or incidents that actually occurred in the past, which are recreated using virtual reality technology and can be experienced by users.

[0520] "Cultural experience" refers to the use of virtual reality technology to allow users to experience the cultural background, customs, and practices of a particular region or era.

[0521] The term "remote location" refers to a location away from the user's current location, where lectures and Q&A sessions can be held in real time with experts or lecturers in such locations.

[0522] "Experts and lecturers" are people who have advanced knowledge and skills in a particular field and who impart that knowledge to other users through lectures and education.

[0523] "Smart glasses" are eyeglass-type devices that can visually display information, allowing users to view information and content based on their line of sight and movements.

[0524] A "virtual environment" is a digital space created using virtual reality technology in which a user can have a virtual experience.

[0525] "Gaze" refers to the direction of the user's eyes, and is the direction of the user's gaze detected by a device such as smart glasses.

[0526] "Movement" refers to the movement of a user's hands or body, and is an action that conveys the user's intentions in an interactive system.

[0527] "Specific information and content" refers to data that is of interest to users and is provided through virtual reality or smart glasses, such as product information, promotions, and educational materials.

[0528] "Emotional state" refers to the user's current state of mind, and is recognized in real time by analyzing facial expressions, tone of voice, behavior, etc.

[0529] "Promotions" refer to advertising activities or special offers designed to increase purchasing interest in a particular product or service and are delivered based on a user's emotional state.

[0530] The present invention relates to an education and commerce system that uses smart glasses to provide customers with a rich experience in a virtual store, enabling interactive information provision based on the user's emotional state.

[0531] Using a virtual reality headset

[0532] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device. The device renders the received 3D model data on a virtual reality headset, providing the user with a view of the virtual classroom. The user can explore the virtual classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[0533] Using smart glasses in virtual stores

[0534] The server generates a 3D model of the virtual store and sets the user's initial location. The user puts on the smart glasses and freely explores the virtual store. Specific product information and promotions are automatically displayed on the smart glasses display based on the user's gaze and movements.

[0535] Real-time recognition of emotional states

[0536] The device is equipped with a camera and microphone to analyze the user's facial expressions and tone of voice. The server uses this data to recognize the user's emotional state in real time. For example, if the user is confused, a special promotion and support options will be displayed.

[0537] Hardware and software used

[0538] Hardware

[0539] Virtual reality headset: A device worn on the head that displays a virtual classroom or storefront.

[0540] Smart glasses: Eyeglasses-like devices that display information visually.

[0541] Camera and microphone: Input devices for analyzing the user's facial expressions and voice.

[0542] software

[0543] Emotion engine: A program that analyzes user emotions in real time.

[0544] VR interface: A program for generating and manipulating a virtual environment.

[0545] Specific examples

[0546] A customer puts on a virtual reality headset and explores a virtual store. When the customer looks at a particular product, detailed information about that product appears on the display. If the customer shows signs of confusion, the system displays a special promotion in real time and also offers the option to video chat with a staff member. In this way, the customer receives personalized information and support.

[0547] Prompt Sentence Examples

[0548] Create an example of a smart glasses application that uses an emotion engine to help customers explore products in a virtual store, including displaying more information when the customer looks at a product and showing special promotions if the customer's emotion is negative.

[0549] The above is a detailed description of the embodiments of the present invention, which can dramatically improve a user's educational and business transaction experience in a virtual environment.

[0550] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0551] Step 1:

[0552] The server generates a 3D model of the virtual classroom or virtual store and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the terminal. The input in this step is the design data of the virtual environment, and the output is the 3D model data and initial location information sent to the terminal.

[0553] Step 2:

[0554] The terminal renders the 3D model data received from the server onto a virtual reality headset or smart glasses, providing the user with a virtual classroom or virtual store view. The input in this step is the 3D model data sent from the server, and the output is the virtual environment displayed to the user.

[0555] Step 3:

[0556] The user explores the virtual classroom or virtual store using a controller or gaze. The user's location information and interaction data are sent from the device to the server. The input in this step is the user's movement and gaze information, and the output is the user's location information and interaction data sent to the server.

[0557] Step 4:

[0558] The server synchronizes the received user location information and interaction data with other users' devices, allowing all users to share the latest status. The input in this step is the user's location information and interaction data, and the output is synchronized data sent to other users' devices.

[0559] Step 5:

[0560] The camera and microphone installed on the device analyze the user's facial expressions and tone of voice in real time to obtain emotional data. The input in this step is the user's facial expression data and voice data, and the output is the analyzed emotional data.

[0561] Step 6:

[0562] The server analyzes the user's emotional state based on the emotional data sent from the terminal. For example, if the user is confused, it generates special promotions and support options. The input in this step is the emotional data, and the output is the generated promotions and support options.

[0563] Step 7:

[0564] The device presents the promotions and support options sent from the server to the user. This is done by displaying the information on the smart glasses display. The input in this step is the promotions and support options from the server, and the output is the information presented to the user.

[0565] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0566] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0567] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0568] [Second embodiment]

[0569] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0570] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0571] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0572] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0573] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0574] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0575] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0576] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0577] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0578] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0579] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0580] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0581] This invention provides an educational support system that utilizes virtual reality (VR) technology to enable students to have a more immersive learning experience. Specifically, the system includes three main functions: virtual classroom, historical and cultural experience, and remote lecture participation.

[0582] Program processing

[0583] The main stakeholders are the server, the terminal, and the user.

[0584] 1. VR Virtual Classroom Experience

[0585] The server plays a central role, generating a 3D model of the virtual classroom and determining the initial positions and settings of each user. After the virtual classroom data is generated, the server transmits it to the terminals.

[0586] The device then renders the received 3D model data onto the VR headset, allowing the user to experience the visual and auditory sensations of being in the classroom. As the user uses the controller to move around the VR environment and interact with the teacher, the device sends positional and interaction data to the server.

[0587] The server receives this data and synchronizes it with all participants' devices, providing a real-time interactive environment.

[0588] 2. History and Culture Experience

[0589] The server generates a specific historical scenario and prepares the associated 3D model and simulation data, which are then sent to the device.

[0590] The device starts a simulation based on the received scenario data, providing the user with an experience that makes them feel as if they are actually there. The user explores and interacts within the simulation, and this interaction data is sent from the device to the server.

[0591] The server triggers specific events and synchronously distributes the results to other users, allowing users to enjoy a real-time interactive historical and cultural experience.

[0592] 3. Participating in lectures from remote locations

[0593] The server manages real-time communication with remote experts and lecturers. When a lecture starts, the server transmits the lecturer's video stream to the terminal.

[0594] The terminal displays the received video stream to the user. If the user wants to ask a question during the lecture, they can send the question by voice or text. This question data is sent from the terminal to the server.

[0595] The server relays questions to the instructor and sends the instructor's answers back to the user's device in real time, allowing users to participate in high-quality lectures and receive direct answers from experts regardless of location.

[0596] Specific examples

[0597] During a history class, students don VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. The data is then sent to the device, which renders the VR environment, allowing students to explore the ancient Egyptian landscape. When a student touches a specific object or interacts with a character, the data is sent from the device to the server and synchronized with the other students.

[0598] On another day, a lecture is given by a European expert. The server starts the lecturer's video stream and distributes it to the devices. Students can participate in the lecture in real time and ask questions. Questions are sent from the devices to the server, which relays them to the lecturer. The lecturer's answers are sent back to the students' devices via the server.

[0599] In this way, the system of the present invention provides three main functions: virtual classroom, historical and cultural experience, and remote lecture participation, which can dramatically improve students' concentration and learning effectiveness.

[0600] The processing flow will be explained below.

[0601] Processing steps for VR virtual classroom experience

[0602] Step 1:

[0603] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[0604] Step 2:

[0605] The server transmits the generated 3D model data and initial position information to the terminal.

[0606] Step 3:

[0607] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[0608] Step 4:

[0609] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[0610] Step 5:

[0611] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[0612] Step 6:

[0613] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[0614] History and Culture Experience Processing Steps

[0615] Step 1:

[0616] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[0617] Step 2:

[0618] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[0619] Step 3:

[0620] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[0621] Step 4:

[0622] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[0623] Step 5:

[0624] The terminal transmits the user's interaction data to the server in real time.

[0625] Step 6:

[0626] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[0627] Processing steps for remote lecture participation

[0628] Step 1:

[0629] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[0630] Step 2:

[0631] The server initiates the instructor's video stream and streams it to the terminal in real time.

[0632] Step 3:

[0633] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[0634] Step 4:

[0635] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[0636] Step 5:

[0637] The terminal transmits the user's question data to the server in real time.

[0638] Step 6:

[0639] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[0640] This allows users to have an interactive, high-quality learning experience without being restricted by geography.

[0641] Example 1

[0642] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] In conventional educational systems, it is difficult for students to participate in actual classrooms, to realistically recreate historical events or cultural experiences, or to receive lectures from experts or lecturers in remote locations in real time. As a result, there are problems that limit learning effectiveness and concentration. Furthermore, there is also the issue of incomplete synchronization of user interaction data and location information within the virtual environment, making it difficult to provide a smooth real-time interactive environment.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0645] In this invention, the server includes: a means for allowing users to participate in a virtual learning environment using a virtual reality display device and interact with educators and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a central control unit for generating 3D model data and transmitting it to terminals; a terminal device for transmitting user position information and interaction data within the virtual environment to the central control unit and synchronizing the latest status for all users; and a means for transmitting interactions with specific objects and characters to the central control unit, triggering events, and synchronously distributing them to other users. This provides an environment in which multiple users can simultaneously learn while interacting in the same virtual space in real time, and also enables historical and cultural experiences to be realistically reproduced using virtual reality technology and lectures by lecturers in remote locations.

[0646] A "virtual reality display device" is a device that allows users to experience images and sounds in a virtual space in real time. Specifically, this includes head-mounted displays and VR goggles.

[0647] A "virtual learning environment" is a virtual space where users can learn or train using virtual reality technology. It is designed to mimic a real-world classroom or learning scenario.

[0648] An "educator" is a person who has a teaching role within a virtual learning environment, such as a teacher, lecturer, or instructor.

[0649] The "central control device" is the central computer of the system, and is a device that manages the generation of virtual environments, synchronization of user data, triggering of specific events, etc.

[0650] A "terminal device" is a device through which a user accesses a virtual reality environment, such as a computer, smartphone, or tablet.

[0651] "Interaction data" refers to data about the actions or behaviors a user performs within a virtual space, such as movements, clicks, and interactions with objects.

[0652] "3D model data" refers to data on digital objects that are represented in three-dimensional space. This includes classrooms, historical buildings, characters, and other virtual objects.

[0653] "Real-time synchronization of execution" is a technology that allows multiple users to simultaneously display and reflect status and events in a virtual space shared by multiple users.

[0654] An "event trigger" is a process that is automatically executed when a specific condition or action occurs. This occurs based on the scenario progression or user interaction within the virtual learning environment.

[0655] "Distance learning" is a format in which a lecturer in a physically distant location delivers a lecture in real time through a virtual learning environment.

[0656] This invention relates to an educational support system that allows users to experience a realistic learning environment using virtual reality (VR) technology. The system provides three main functions: virtual classroom experience, historical and cultural experience, and remote lecture participation.

[0657] Virtual Classroom Experience

[0658] Hardware and Software:

[0659] The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D model of the virtual classroom, and uses a communication protocol such as WebSocket to synchronize data in real time.

[0660] The device uses a VR headset such as Oculus or HTC Vive, and the device uses VR-specific software to render the received 3D model data in real time.

[0661] Users use a VR headset and controllers to move freely around the virtual classroom and interact with the educator.

[0662] Examples:

[0663] When a user puts on a VR headset and enters a virtual classroom, the server generates a 3D model and initializes the user's location information. The device instantly renders the classroom environment based on the data sent from the server to the device. When a user raises their hand to ask a question or moves around the classroom, this interaction data is sent via the device to the server and synchronized with the devices of other participants.

[0664] History and Cultural Experience

[0665] Hardware and Software:

[0666] The server generates historical scenarios using 3D modeling software such as Blender or Maya, and also uses AI techniques to set specific event triggers.

[0667] The device uses Unity or Unreal Engine to render the VR environment based on the received scenario data.

[0668] Users interact with and explore historical buildings and characters using VR controllers.

[0669] Examples:

[0670] When students take an "Ancient Egypt" class, the server generates 3D models of the pyramids, pharaoh's palaces, and other structures. The server sends data to the device, which then displays these models on the headset. Using the controller, users explore the camopets and murals inside the pyramids, and their interaction data is instantly shared with other students.

[0671] Participating in remote lectures

[0672] Hardware and Software:

[0673] The server retrieves the instructor's video stream using streaming technologies such as Zoom or WebRTC.

[0674] The device uses software to display the video stream in real time, allowing users to watch the lecture through a display or VR headset.

[0675] The user inputs a question using a microphone or keyboard and sends it to the server via a communication protocol.

[0676] Examples:

[0677] When a European expert gives a lecture, the server captures his video stream and sends it to the device. Students view the video on their VR headsets and, if they have questions, use their microphones to send them to the server in real time. The server then relays the questions to the lecturer, obtains the lecturer's answers, and immediately sends them back to the user. This process allows for seamless communication even with lecturers in remote locations.

[0678] Example prompt sentence:

[0679] "Write a program that generates a VR simulation of Ancient Egypt and allows users to explore inside the pyramids."

[0680] Through each function, this system can add new value to traditional teaching methods and dramatically improve the learning experience.

[0681] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0682] System program processing

[0683] VR virtual classroom experience

[0684] Step 1:

[0685] The server generates a 3D model of the virtual classroom using a 3D engine such as Unity or Unreal Engine.

[0686] Input: 3D model template, user's initial position data.

[0687] Data processing: The 3D engine generates a 3D model based on the template data and sets the initial position for each user.

[0688] Output: The completed 3D model data.

[0689] Specific operation: The server constructs a virtual classroom using the template and initial position data.

[0690] Step 2:

[0691] The server sends the generated 3D model data to the terminal.

[0692] Input: Completed 3D model data.

[0693] Data processing: Converting the generated data into a format that can be sent.

[0694] Output: 3D model data for transmission.

[0695] Specific operation: The server encodes the data and sends it to the device.

[0696] Step 3:

[0697] The device renders the received 3D model data on the VR headset.

[0698] Input: 3D model data received from the server.

[0699] Data processing: Decode the data and render it as a VR environment.

[0700] Output: The rendered VR environment.

[0701] What it does: A virtual classroom is displayed in real time on the headset.

[0702] Step 4:

[0703] Users use VR controllers to move and interact within the virtual classroom.

[0704] Input: User operation data (movements, questions, etc.).

[0705] Data processing: Operation data is acquired in real time and converted into location information.

[0706] Output: User's current location and interaction data.

[0707] Specific actions: The user physically manipulates the controller and moves around in the virtual environment.

[0708] Step 5:

[0709] The device transmits the acquired user location information and interaction data to the server.

[0710] Input: User's current location and interaction data.

[0711] Data processing: Package the data and send it to the server.

[0712] Output: The packaged data.

[0713] Specific operation: Data is sent from the device to the server.

[0714] Step 6:

[0715] The server receives all user data and performs the synchronization process.

[0716] Input: Interaction data submitted by each user.

[0717] Data processing: Integrate data and update it to the latest information.

[0718] Output: The synchronized data.

[0719] Specific operation: The server sends data to all devices using WebSocket or similar and shares the latest status.

[0720] History and Cultural Experience

[0721] Step 1:

[0722] The server generates historical scenarios using 3D modeling software such as Blender or Maya.

[0723] Input: Historical scenario raw material file.

[0724] Data processing: Create and optimize scenarios using 3D modeling software.

[0725] Output: 3D scenario data.

[0726] What it does: The server builds and optimizes the scenario for the simulation.

[0727] Step 2:

[0728] The server transmits the generated historical scenario data to the terminal.

[0729] Input: Simulation data.

[0730] Data processing: Compression and conversion into a transmittable format.

[0731] Output: Compressed simulation data.

[0732] Specific operation: The server compresses the data and sends it to the terminal.

[0733] Step 3:

[0734] The device renders the received scenario data on the VR headset.

[0735] Input: Send data.

[0736] Data processing: Decode the data and render it into a VR environment.

[0737] Output: A rendered simulated environment.

[0738] Specific Behavior: Rendered to allow the user to begin exploring within the scenario.

[0739] Step 4:

[0740] The user interacts with specific objects and characters within the VR environment.

[0741] Input: User operation data.

[0742] Data processing: Convert the acquired operation data into interaction data.

[0743] Output: Interaction data.

[0744] Concrete behavior: The user explores and interacts with objects within the simulation.

[0745] Step 5:

[0746] The device sends the acquired interaction data to the server.

[0747] Input: Interaction data from the user.

[0748] Data processing: packaging and preparation for transmission.

[0749] Output: The packaged data.

[0750] Specific operation: Data is sent from the device to the server.

[0751] Step 6:

[0752] The server triggers specific events and synchronously distributes the results to other users.

[0753] Input: Interaction data.

[0754] Data processing: Executes event triggers and generates result data.

[0755] Output: Triggered event result data.

[0756] Specific operation: The event results are sent to other users' devices and synchronized.

[0757] Participating in remote lectures

[0758] Step 1:

[0759] The server retrieves the instructor's video stream using a streaming technology such as Zoom or WebRTC.

[0760] Input: Instructor video feed.

[0761] Data processing: Compression and optimization of video streams.

[0762] Output: Compressed video stream.

[0763] Specific operation: The instructor's video is uploaded to the server in real time.

[0764] Step 2:

[0765] The server transmits the video stream to the terminal.

[0766] Input: Compressed video stream.

[0767] Data processing: Format conversion as needed.

[0768] Output: Video stream for device.

[0769] Specific operation: The server delivers streaming data to the terminal.

[0770] Step 3:

[0771] The terminal displays the received video stream to the user.

[0772] Input: Video stream from the server.

[0773] Data processing: Decode and display the stream.

[0774] Output: Real-time video display.

[0775] What happens: The user watches the lecture on a display or VR headset.

[0776] Step 4:

[0777] The user inputs and submits a question by voice or text.

[0778] Input: Audio or text data.

[0779] Data processing: Converting audio to text based on data functions (in the case of audio).

[0780] Output: Question data.

[0781] What it does: The user creates and submits a question using a microphone or keyboard.

[0782] Step 5:

[0783] The terminal transmits the question data to the server.

[0784] Input: Question data.

[0785] Data processing: Packaging and preparing for transmission.

[0786] Output: Packaged question data.

[0787] Specific operation: Question data is sent from the terminal to the server.

[0788] Step 6:

[0789] The server relays the question to the instructor, obtains the instructor's answer, and returns it to the terminal.

[0790] Input: Question data.

[0791] Data processing: Converting data into a format suitable for instructors.

[0792] Output: Transformed question data and instructor answer data.

[0793] Specific operation: The server forwards the question to the instructor, obtains the answer, and then sends it back to the device.

[0794] In this way, the system can provide a complex learning environment and offer users a real-time interactive learning experience.

[0795] (Application example 1)

[0796] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0797] When providing educational support systems using virtual reality technology, the challenge is to ensure that users can enjoy a consistent, real-time learning experience while accessing high-quality educational content without geographical constraints. Conventional systems suffer from delays in real-time communication and insufficient information synchronization between users, making it difficult to maximize learning effectiveness.

[0798] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0799] In this invention, the server includes: means for allowing users to participate in a virtual classroom using a virtual reality headset and interact with teachers and other users in real time; means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; means for allowing users to attend lectures and engage in Q&A sessions with experts and lecturers in remote locations in real time; and means for providing a content distribution service and allowing users to enjoy a learning experience in virtual reality. This allows users to interact with teachers and other users in real time, enjoy an immersive learning experience about history and culture, and participate in lectures from high-quality experts without being restricted by geographical location.

[0800] - A "virtual reality headset" is a device that allows a user to immerse themselves in a computer-generated virtual environment through visual and auditory experiences.

[0801] "User" refers to a learner or participant who uses an educational support system using virtual reality technology.

[0802] A "virtual classroom" is a simulated educational space where teachers and students interact and learn in real time within a virtual reality environment.

[0803] "Real-time" refers to instantaneous communication and data synchronization with minimal delay.

[0804] "Interaction" means the act of users exchanging information through a communication means.

[0805] "Teacher" refers to a professional who conducts lessons within a virtual classroom and provides educational content to students.

[0806] "Other users" refers to participants in a simultaneous learning activity within a virtual classroom or other virtual environment.

[0807] "Historical events" refer to significant events or occurrences that actually occurred in the past and are re-enacted for educational purposes.

[0808] "Cultural Experience" refers to a virtual reality scenario that allows people to learn about the culture of a particular region or era through hands-on experience.

[0809] A "remote location" refers to a physically separate location that is accessed via a communication means such as the Internet.

[0810] "Expert" refers to a lecturer or advisor with advanced knowledge and skills in a particular field.

[0811] A "lecture" is an educational activity that involves a coherent explanation or instruction on a particular topic.

[0812] "Question and answer" is a process in which a user asks a question and an expert or lecturer provides an answer to the question.

[0813] "Content Delivery Service" refers to an online platform for providing educational content to users using virtual reality technology.

[0814] "Interaction data" refers to data related to user operations and behavior, which is processed and synchronized on the server.

[0815] "Server" refers to a central processing unit or system for providing virtual classrooms and related services.

[0816] The present invention is a system for enhancing educational experiences by utilizing virtual reality technology. Specific embodiments thereof are described below.

[0817] Hardware and Software Configuration

[0818] Hardware used:

[0819] VR headset (e.g. Oculus Rift, HTC Vive)

[0820] Server (standard server with high performance)

[0821] User device (PC or tablet)

[0822] Software used:

[0823] Unity 3D (3D model rendering engine)

[0824] WebRTC (protocol for real-time communication)

[0825] Node.js (provides real-time processing capabilities on the server side)

[0826] Amazon AWS (cloud storage and data distribution service)

[0827] Overview of program processing

[0828] Server Role:

[0829] 1. Creating a virtual classroom

[0830] The server generates a 3D model of the virtual classroom using Unity 3D, sets the initial position and environment for each user, and sends the generated 3D model data to the user's device.

[0831] 2. Real-time communication

[0832] The server uses WebRTC to manage real-time communication between users, supporting audio and video exchanges with teachers and other users, providing a smooth interactive environment.

[0833] 3. Data synchronization

[0834] It collects user location and interaction data and synchronizes it with other users in real time. The data is stored on Amazon AWS and distributed to other devices as needed.

[0835] Device role:

[0836] 1. Rendering the Data

[0837] The device receives 3D model data from the server and renders it on the VR headset using Unity 3D, allowing users to experience virtual classrooms and historical scenarios.

[0838] 2. Collecting interactions

[0839] It collects user actions and movements in the environment in real time and sends the data to a server, which keeps it synchronized with other users.

[0840] Examples:

[0841] Virtual Classroom Experience

[0842] A user puts on a VR headset and joins a virtual classroom. The server sets the user's initial position and sends 3D model data to the device. The device renders this data on the VR headset, allowing the user to move freely within the virtual classroom and interact with other users and the teacher in real time.

[0843] History and Cultural Experience

[0844] To learn about historical events, users put on a VR headset. For example, a server generates an ancient Egyptian scenario and transmits the data to the device. Users can explore ancient Egyptian streets and pyramids and interact with an interactive guide.

[0845] Example prompt sentence:

[0846] "Generate a VR experience scenario about Egyptian history, including detailed depictions of the inside of the pyramids and an interactive guide that users can interact with."

[0847] summary

[0848] The system of the present invention provides users with a comprehensive and immersive learning experience through the collaboration of the server and user terminals, enabling them to receive high-quality education without being restricted by geographical location, maximizing the effectiveness of their learning.

[0849] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0850] Step 1: The server generates a 3D model of the virtual classroom

[0851] Input: Virtual classroom design data, user initial location information

[0852] Processing: The server uses Unity 3D to generate a 3D model of the virtual classroom. It performs calculations to determine the user's initial position and environment settings.

[0853] Output: 3D model data of the completed virtual classroom, initial user position information

[0854] Step 2: The server sends the virtual classroom data to the device

[0855] Input: 3D model data of the virtual classroom, user's initial position information

[0856] Processing: The server sends the generated 3D model data and initial location information to each user's device.

[0857] Output: 3D model data sent to the device, initial position information

[0858] Step 3: The device renders the received data to the VR headset

[0859] Input: 3D model data received from the server, user's initial location information

[0860] Processing: The device uses Unity 3D to render the received 3D model data on the VR headset, allowing the user to experience the visual and auditory world of the virtual classroom.

[0861] Output: Visual and auditory data of the virtual classroom displayed on a VR headset

[0862] Step 4: User moves and interacts in the VR environment

[0863] Input: VR headset, controller

[0864] Processing: Using the headset and controllers, users move around the VR environment and interact with other users and objects, and their actions are collected as data.

[0865] Output: Collected user location and interaction data

[0866] Step 5: The device sends the interaction data to the server

[0867] Input: User location and interaction data

[0868] Processing: The device sends the collected data to the server in real time.

[0869] Output: User location and interaction data sent to the server

[0870] Step 6: The server synchronizes the data to other users

[0871] Input: User location and interaction data

[0872] Processing: The server analyzes the received location and interaction data and generates the data necessary to synchronize it with other users' devices.

[0873] Output: Location and interaction data synced to other users

[0874] Step 7: Trigger specific events in the scenario

[0875] Input: Interaction data stored on the server

[0876] Processing: When a certain condition is met, the server triggers an event in the scenario and notifies all users of that information.

[0877] Output: Event trigger information and related data sent to each device

[0878] Step 8: The server delivers the video stream of the remote lecture.

[0879] Input: Video stream from instructor

[0880] Processing: The server uses WebRTC to deliver the instructor's video and audio streams to each user's device in real time.

[0881] Output: Real-time video stream of lectures delivered to user devices

[0882] Step 9: Users attend the remote lecture and submit questions

[0883] Input: User voice or text data

[0884] Processing: While listening to the lecture, the user asks questions by voice or text, and this data is sent from the terminal to the server.

[0885] Output: Question data sent to the server

[0886] Step 10: The server relays the question to the instructor and delivers the answer

[0887] Input: User question data, instructor answer data

[0888] Processing: The server relays the user's questions to the instructor, collects the instructor's answers in real time, and delivers them to each user's device.

[0889] Output: Teacher's response data distributed to each user's device

[0890] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0891] This invention provides an educational support system that uses virtual reality (VR) technology to enable students to have a more immersive learning experience, and in particular, by combining it with an emotion engine, it realizes interactive education based on the user's emotional state. Specifically, this system includes a virtual classroom, historical and cultural experiences, remote lecture participation, and user emotion recognition.

[0892] Program processing

[0893] The main stakeholders are the server, the terminal, and the user.

[0894] 1. VR Virtual Classroom Experience

[0895] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device.

[0896] The device renders the received 3D model data on the VR headset, providing the user with a virtual classroom view. The user can explore the VR classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[0897] 2. History and Culture Experience

[0898] The server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[0899] 3. Participating in lectures from remote locations

[0900] The server prepares lecture information from remote experts and lecturers and sets access rights for users. When the lecture starts, the server starts the lecturer's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the lecturer and sends the lecturer's answer to the user's terminal in real time.

[0901] 4. Incorporating an Emotional Engine

[0902] The server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc., and recognizes them as emotion data.

[0903] The device collects the user's emotional data in real time and sends it to the server. The server then adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if they are dissatisfied or confused, the server will provide additional explanations or a different scenario.

[0904] Specific examples

[0905] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the teacher and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[0906] If one student shows interest in the pyramid while another struggles with a difficult problem, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[0907] Furthermore, when a lecture by a remote expert is being given, the server transmits the expert's video stream to the device. When a student has a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. Furthermore, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[0908] In this way, the system of the present invention integrates four main functions: virtual classroom, historical and cultural experience, remote lecture participation, and emotion recognition engine, thereby dramatically improving the user's learning experience.

[0909] The processing flow will be explained below.

[0910] Processing steps for VR virtual classroom experience

[0911] Step 1:

[0912] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[0913] Step 2:

[0914] The server transmits the generated 3D model data and initial position information to the terminal.

[0915] Step 3:

[0916] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[0917] Step 4:

[0918] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[0919] Step 5:

[0920] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[0921] Step 6:

[0922] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[0923] History and Culture Experience Processing Steps

[0924] Step 1:

[0925] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[0926] Step 2:

[0927] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[0928] Step 3:

[0929] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[0930] Step 4:

[0931] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[0932] Step 5:

[0933] The terminal transmits the user's interaction data to the server in real time.

[0934] Step 6:

[0935] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[0936] Processing steps for remote lecture participation

[0937] Step 1:

[0938] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[0939] Step 2:

[0940] The server initiates the instructor's video stream and streams it to the terminal in real time.

[0941] Step 3:

[0942] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[0943] Step 4:

[0944] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[0945] Step 5:

[0946] The terminal transmits the user's question data to the server in real time.

[0947] Step 6:

[0948] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[0949] Built-in processing steps for the emotion engine

[0950] Step 1:

[0951] The server uses an emotion engine to analyze the user's emotional state.

[0952] Step 2:

[0953] The device collects the user's facial expressions, voice tone, and behavioral patterns as emotional data in real time.

[0954] Step 3:

[0955] The device transmits the collected emotion data to a server in real time.

[0956] Step 4:

[0957] The server adjusts the educational content based on the received emotional data, for example, if the user expresses dissatisfaction, it provides additional explanations or new approaches.

[0958] Step 5:

[0959] The server changes the behavior of objects and characters in the virtual environment based on the emotional data to optimize the user's experience.

[0960] Step 6:

[0961] The device follows instructions from the server and provides appropriate feedback to the user, customizing the learning experience according to the user's emotional state.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] Conventional educational systems have few opportunities for users to actively participate in the learning content, making it difficult to provide a realistic learning experience. Furthermore, they lack the technology to analyze the user's emotional state and dynamically adjust the educational content accordingly. As a result, there are issues with maintaining users' motivation to learn and making it difficult for them to progress effectively.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with an instructor and other users in real time, a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them, a means for allowing a user to attend lectures and engage in Q&A sessions with experts and instructors in remote locations in real time, and a means for analyzing the user's emotional state and dynamically adjusting educational content based on the results. This allows the user to increase their motivation to learn through a realistic learning experience and progress effectively in their studies.

[0966] A "virtual reality headset" is a device that allows a user to immerse themselves in a virtual reality environment, providing visual and audio information.

[0967] A "virtual classroom" is a virtual space that users can participate in through a virtual reality headset, recreating an environment for educational activities.

[0968] An "instructor" is a person or system that has the role of educating users, and includes teachers and lecturers.

[0969] "Historical events" refer to events or occurrences that actually occurred in the past and are used as educational material.

[0970] A "cultural experience" is an experience provided to users through a virtual environment based on a specific culture or historical background.

[0971] An "expert" is a person with advanced knowledge and skills in a particular field who is qualified to teach or lecture.

[0972] "Emotional state" refers to the internal psychological state that can be inferred from the user's facial expression, tone of voice, behavioral patterns, etc.

[0973] "Dynamic adjustment" refers to the process of changing educational content in real time based on the user's reactions and situation.

[0974] "Interaction data" is information about the actions and operations a user performs within a virtual environment that is used by the system to understand those actions and generate appropriate responses.

[0975] "Triggering an event" means that the system automatically initiates a specific action or scenario when certain conditions are met.

[0976] "Synchronous distribution" is the process of enabling multiple users to share the same information or state in real time.

[0977] This invention combines virtual reality (VR) technology with an emotion engine to provide users with a realistic learning experience. Below, we will explain in detail what hardware and software are used to build the system and how data processing and calculations are performed.

[0978] Hardware and software used

[0979] Servers are high-performance computers, and recommended specifications include a high-clock processor, large amounts of memory, and SSD storage. Software running on the server includes a data communication system using WebSocket or HTTP, and a game engine such as Unity or Unreal Engine.

[0980] A device is a user-interface device, which can include a VR headset with a high-resolution display (e.g., Oculus Rift or HTC Vive) and a connected PC or mobile device. The software running on the device includes Unity or Unreal Engine for rendering the VR environment and controller software for processing user input.

[0981] The user wears a VR headset and uses a controller to explore the virtual environment. To collect the user's emotional state, biometric devices such as a camera, microphone, and heart rate sensor are connected to the device.

[0982] System processing overview

[0983] The server generates a 3D model of the virtual classroom and sets the user's initial location. This data is sent from the server to the device using WebSocket or HTTP protocol. The device then renders the received 3D model data on the VR headset using Unity or Unreal Engine, providing the user with a view of the virtual classroom.

[0984] Users can explore the VR classroom using a controller and interact with instructors and other users in real time. During this time, the user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[0985] In a historical and cultural experience, the server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[0986] When participating in a remote lecture, the server prepares lecture information from a remote expert or instructor and sets access rights for the user. When the lecture starts, the server starts the instructor's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the instructor and sends the instructor's answer to the user's terminal in real time.

[0987] Regarding the incorporation of an emotion engine, the server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc. and recognizes them as emotional data. The device collects the user's emotional data in real time and sends it to the server. The server dynamically adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if the user is dissatisfied or confused, additional explanations or a different scenario will be provided.

[0988] Specific examples

[0989] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the instructor and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[0990] If one student shows interest in the pyramid while another struggles with a difficult problem during the scenario, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[0991] When a lecture is given by a remote expert, the server sends the expert's video stream to the device. When a student wants to ask a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. In addition, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[0992] Prompt Sentence Examples

[0993] "Describe a scenario in which students would use VR to learn about ancient Egypt. Also, detail how an emotion recognition engine would help."

[0994] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0995] Program processing flow

[0996] Step 1:

[0997] The server generates a 3D model of the virtual classroom and the user's initial position information.

[0998] Specifically, the server uses a game engine such as Unity or Unreal Engine to create a 3D model of the virtual classroom and sets the user's initial location information based on that. The input is the virtual classroom design and user ID, and the output is the generated 3D model data and initial location information, saved in JSON format.

[0999] Step 2:

[1000] The server sends the generated 3D model data and initial location information to the terminal.

[1001] The server sends this data to the device using WebSocket or HTTP protocol. The input includes the 3D model data and initial position information generated in step 1, and the output includes the data sent to the device.

[1002] Step 3:

[1003] The device renders the received 3D model data on the VR headset.

[1004] The device processes the received data using Unity or Unreal Engine and renders it in real time on the VR headset. The input includes 3D model data received from the server, and the output includes the virtual classroom view seen by the user.

[1005] Step 4:

[1006] Users explore and interact within the virtual classroom.

[1007] Using a VR headset and controllers, users can move freely around the virtual classroom and interact with other users and objects in real time. The inputs are the user's visual, auditory, and controller inputs, and the output is the interaction data generated.

[1008] Step 5:

[1009] The terminal transmits the user's interaction data to the server.

[1010] The device collects user location information and operation logs in real time and sends them to the server using WebSocket. The input includes user interaction data, and the output includes data sent to the server.

[1011] Step 6:

[1012] The server synchronizes the received interaction data with the terminals of other users.

[1013] The server synchronizes the received data to all other devices in real time. The input includes user interaction data, and the output includes data synchronized to other devices.

[1014] Step 7:

[1015] The server generates the data for a particular historical scenario.

[1016] The server generates 3D models and scenario scripts based on specific historical events and sends them to the terminal. The inputs include historical data and scenario design, and the outputs include the generated 3D models and scenario scripts.

[1017] Step 8:

[1018] The terminal starts the simulation, providing the user with a realistic historical experience.

[1019] The device starts a simulation using Unity or Unreal Engine based on the received scenario data. The input includes the received scenario data, and the output includes the virtual environment provided to the user.

[1020] Step 9:

[1021] The terminal transmits the interaction data to the server.

[1022] The device collects the interaction data of the user in the virtual environment in real time and sends it to the server.The input includes the user's interaction data, and the output includes the data sent to the server.

[1023] Step 10:

[1024] The server triggers a specific event and synchronizes the results to other devices.

[1025] The server triggers a specific event based on the received interaction data and synchronously distributes the result to all other terminals. The input includes the interaction data based on the trigger condition, and the output includes the synchronized event result.

[1026] Step 11:

[1027] The server prepares the lecture information for the remote location and transmits it to the terminal.

[1028] The server manages the contents and schedule of the lectures, sets access rights for participating users, and sends this information to the terminals. The input includes the contents and schedule of the lectures, and the output includes access rights information.

[1029] Step 12:

[1030] The terminal renders the instructor's video stream to the user.

[1031] The server starts the video stream at the beginning of the lecture, and the terminal renders it in real time.,The input includes the video stream data, and the output,includes the lecture video that is displayed to the user.

[1032] Step 13:

[1033] The terminal transmits the user's question data to the server.

[1034] When a user wants to ask a question, the question data is sent from the terminal to the server. The input includes the user's question data, and the output includes the data sent to the server.

[1035] Step 14:

[1036] The server relays questions to the instructor and delivers answers to the user.

[1037] The server relays questions to the instructor and sends the answers to the user's device in real time. The input includes the user's question data and the output includes the instructor's answer data.

[1038] Step 15:

[1039] The terminal transmits the user's emotion data to the server.

[1040] The device collects user emotion data using a camera and microphone and sends it to the server. The input includes the user emotion data, and the output includes the data sent to the server.

[1041] Step 16:

[1042] The server analyzes the received emotional data and dynamically adjusts the educational content.

[1043] The server uses an emotion analysis algorithm to analyze the user's emotional state and dynamically adjusts the educational content based on the results. The input is the user's emotional data, and the output is the adjusted educational content.

[1044] These steps enable the system of the present invention to provide an immersive learning experience for the user and to adapt educational content based on the user's emotional state in real time.

[1045] (Application example 2)

[1046] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1047] In modern educational and commercial environments, users have limited interactive options for enriching their experiences. Current virtual reality technologies lack the means to provide appropriate feedback and information based on the user's emotional state. This leads to a poor user experience and makes it difficult for users to learn or purchase effectively.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with a teacher and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a means for allowing a user to explore a virtual environment using smart glasses and present specific information or content based on the user's gaze or movements; and a means for recognizing the user's emotional state in real time and providing information or promotions according to that state. This enables an interactive experience according to the user's emotional state, dramatically improving the user experience in education and commerce.

[1049] A "virtual reality headset" is a device worn by a user to display a virtual reality environment, and is attached to the head.

[1050] A "virtual classroom" is a virtual classroom environment recreated using virtual reality technology that users can access through a virtual headset to create an immersive learning experience.

[1051] "Real-time" refers to the state in which data and information are processed and transmitted immediately, without delay, allowing users to interact with other users and systems instantly.

[1052] "Historical events" are important events or incidents that actually occurred in the past, which are recreated using virtual reality technology and can be experienced by users.

[1053] "Cultural experience" refers to the use of virtual reality technology to allow users to experience the cultural background, customs, and practices of a particular region or era.

[1054] The term "remote location" refers to a location away from the user's current location, where lectures and Q&A sessions can be held in real time with experts or lecturers in such locations.

[1055] "Experts and lecturers" are people who have advanced knowledge and skills in a particular field and who impart that knowledge to other users through lectures and education.

[1056] "Smart glasses" are eyeglass-type devices that can visually display information, allowing users to view information and content based on their line of sight and movements.

[1057] A "virtual environment" is a digital space created using virtual reality technology in which a user can have a virtual experience.

[1058] "Gaze" refers to the direction of the user's eyes, and is the direction of the user's gaze detected by a device such as smart glasses.

[1059] "Movement" refers to the movement of a user's hands or body, and is an action that conveys the user's intentions in an interactive system.

[1060] "Specific information and content" refers to data that is of interest to users and is provided through virtual reality or smart glasses, such as product information, promotions, and educational materials.

[1061] "Emotional state" refers to the user's current state of mind, and is recognized in real time by analyzing facial expressions, tone of voice, behavior, etc.

[1062] "Promotions" refer to advertising activities or special offers designed to increase purchasing interest in a particular product or service and are delivered based on a user's emotional state.

[1063] The present invention relates to an education and commerce system that uses smart glasses to provide customers with a rich experience in a virtual store, enabling interactive information provision based on the user's emotional state.

[1064] Using a virtual reality headset

[1065] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device. The device renders the received 3D model data on a virtual reality headset, providing the user with a view of the virtual classroom. The user can explore the virtual classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[1066] Using smart glasses in virtual stores

[1067] The server generates a 3D model of the virtual store and sets the user's initial location. The user puts on the smart glasses and freely explores the virtual store. Specific product information and promotions are automatically displayed on the smart glasses display based on the user's gaze and movements.

[1068] Real-time recognition of emotional states

[1069] The device is equipped with a camera and microphone to analyze the user's facial expressions and tone of voice. The server uses this data to recognize the user's emotional state in real time. For example, if the user is confused, a special promotion and support options will be displayed.

[1070] Hardware and software used

[1071] Hardware

[1072] Virtual reality headset: A device worn on the head that displays a virtual classroom or storefront.

[1073] Smart glasses: Eyeglasses-like devices that display information visually.

[1074] Camera and microphone: Input devices for analyzing the user's facial expressions and voice.

[1075] software

[1076] Emotion engine: A program that analyzes user emotions in real time.

[1077] VR interface: A program for generating and manipulating a virtual environment.

[1078] Specific examples

[1079] A customer puts on a virtual reality headset and explores a virtual store. When the customer looks at a particular product, detailed information about that product appears on the display. If the customer shows signs of confusion, the system displays a special promotion in real time and also offers the option to video chat with a staff member. In this way, the customer receives personalized information and support.

[1080] Prompt Sentence Examples

[1081] Create an example of a smart glasses application that uses an emotion engine to help customers explore products in a virtual store, including displaying more information when the customer looks at a product and showing special promotions if the customer's emotion is negative.

[1082] The above is a detailed description of the embodiments of the present invention, which can dramatically improve a user's educational and business transaction experience in a virtual environment.

[1083] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1084] Step 1:

[1085] The server generates a 3D model of the virtual classroom or virtual store and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the terminal. The input in this step is the design data of the virtual environment, and the output is the 3D model data and initial location information sent to the terminal.

[1086] Step 2:

[1087] The terminal renders the 3D model data received from the server onto a virtual reality headset or smart glasses, providing the user with a virtual classroom or virtual store view. The input in this step is the 3D model data sent from the server, and the output is the virtual environment displayed to the user.

[1088] Step 3:

[1089] The user explores the virtual classroom or virtual store using a controller or gaze. The user's location information and interaction data are sent from the device to the server. The input in this step is the user's movement and gaze information, and the output is the user's location information and interaction data sent to the server.

[1090] Step 4:

[1091] The server synchronizes the received user location information and interaction data with other users' devices, allowing all users to share the latest status. The input in this step is the user's location information and interaction data, and the output is synchronized data sent to other users' devices.

[1092] Step 5:

[1093] The camera and microphone installed on the device analyze the user's facial expressions and tone of voice in real time to obtain emotional data. The input in this step is the user's facial expression data and voice data, and the output is the analyzed emotional data.

[1094] Step 6:

[1095] The server analyzes the user's emotional state based on the emotional data sent from the terminal. For example, if the user is confused, it generates special promotions and support options. The input in this step is the emotional data, and the output is the generated promotions and support options.

[1096] Step 7:

[1097] The device presents the promotions and support options sent from the server to the user. This is done by displaying the information on the smart glasses display. The input in this step is the promotions and support options from the server, and the output is the information presented to the user.

[1098] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1099] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1100] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1101] [Third embodiment]

[1102] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1103] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1104] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1105] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1106] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1107] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1108] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1109] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1110] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1111] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1112] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1113] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1114] This invention provides an educational support system that utilizes virtual reality (VR) technology to enable students to have a more immersive learning experience. Specifically, the system includes three main functions: virtual classroom, historical and cultural experience, and remote lecture participation.

[1115] Program processing

[1116] The main stakeholders are the server, the terminal, and the user.

[1117] 1. VR Virtual Classroom Experience

[1118] The server plays a central role, generating a 3D model of the virtual classroom and determining the initial positions and settings of each user. After the virtual classroom data is generated, the server transmits it to the terminals.

[1119] The device then renders the received 3D model data onto the VR headset, allowing the user to experience the visual and auditory sensations of being in the classroom. As the user uses the controller to move around the VR environment and interact with the teacher, the device sends positional and interaction data to the server.

[1120] The server receives this data and synchronizes it with all participants' devices, providing a real-time interactive environment.

[1121] 2. History and Culture Experience

[1122] The server generates a specific historical scenario and prepares the associated 3D model and simulation data, which are then sent to the device.

[1123] The device starts a simulation based on the received scenario data, providing the user with an experience that makes them feel as if they are actually there. The user explores and interacts within the simulation, and this interaction data is sent from the device to the server.

[1124] The server triggers specific events and synchronously distributes the results to other users, allowing users to enjoy a real-time interactive historical and cultural experience.

[1125] 3. Participating in lectures from remote locations

[1126] The server manages real-time communication with remote experts and lecturers. When a lecture starts, the server transmits the lecturer's video stream to the terminal.

[1127] The terminal displays the received video stream to the user. If the user wants to ask a question during the lecture, they can send the question by voice or text. This question data is sent from the terminal to the server.

[1128] The server relays questions to the instructor and sends the instructor's answers back to the user's device in real time, allowing users to participate in high-quality lectures and receive direct answers from experts regardless of location.

[1129] Specific examples

[1130] During a history class, students don VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. The data is then sent to the device, which renders the VR environment, allowing students to explore the ancient Egyptian landscape. When a student touches a specific object or interacts with a character, the data is sent from the device to the server and synchronized with the other students.

[1131] On another day, a lecture is given by a European expert. The server starts the lecturer's video stream and distributes it to the devices. Students can participate in the lecture in real time and ask questions. Questions are sent from the devices to the server, which relays them to the lecturer. The lecturer's answers are sent back to the students' devices via the server.

[1132] In this way, the system of the present invention provides three main functions: virtual classroom, historical and cultural experience, and remote lecture participation, which can dramatically improve students' concentration and learning effectiveness.

[1133] The processing flow will be explained below.

[1134] Processing steps for VR virtual classroom experience

[1135] Step 1:

[1136] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[1137] Step 2:

[1138] The server transmits the generated 3D model data and initial position information to the terminal.

[1139] Step 3:

[1140] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[1141] Step 4:

[1142] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[1143] Step 5:

[1144] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[1145] Step 6:

[1146] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[1147] History and Culture Experience Processing Steps

[1148] Step 1:

[1149] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[1150] Step 2:

[1151] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[1152] Step 3:

[1153] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[1154] Step 4:

[1155] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[1156] Step 5:

[1157] The terminal transmits the user's interaction data to the server in real time.

[1158] Step 6:

[1159] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[1160] Processing steps for remote lecture participation

[1161] Step 1:

[1162] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[1163] Step 2:

[1164] The server initiates the instructor's video stream and streams it to the terminal in real time.

[1165] Step 3:

[1166] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[1167] Step 4:

[1168] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[1169] Step 5:

[1170] The terminal transmits the user's question data to the server in real time.

[1171] Step 6:

[1172] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[1173] This allows users to have an interactive, high-quality learning experience without being restricted by geography.

[1174] Example 1

[1175] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] In conventional educational systems, it is difficult for students to participate in actual classrooms, to realistically recreate historical events or cultural experiences, or to receive lectures from experts or lecturers in remote locations in real time. As a result, there are problems that limit learning effectiveness and concentration. Furthermore, there is also the issue of incomplete synchronization of user interaction data and location information within the virtual environment, making it difficult to provide a smooth real-time interactive environment.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1178] In this invention, the server includes: a means for allowing users to participate in a virtual learning environment using a virtual reality display device and interact with educators and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a central control unit for generating 3D model data and transmitting it to terminals; a terminal device for transmitting user position information and interaction data within the virtual environment to the central control unit and synchronizing the latest status for all users; and a means for transmitting interactions with specific objects and characters to the central control unit, triggering events, and synchronously distributing them to other users. This provides an environment in which multiple users can simultaneously learn while interacting in the same virtual space in real time, and also enables historical and cultural experiences to be realistically reproduced using virtual reality technology and lectures by lecturers in remote locations.

[1179] A "virtual reality display device" is a device that allows users to experience images and sounds in a virtual space in real time. Specifically, this includes head-mounted displays and VR goggles.

[1180] A "virtual learning environment" is a virtual space where users can learn or train using virtual reality technology. It is designed to mimic a real-world classroom or learning scenario.

[1181] An "educator" is a person who has a teaching role within a virtual learning environment, such as a teacher, lecturer, or instructor.

[1182] The "central control device" is the central computer of the system, and is a device that manages the generation of virtual environments, synchronization of user data, triggering of specific events, etc.

[1183] A "terminal device" is a device through which a user accesses a virtual reality environment, such as a computer, smartphone, or tablet.

[1184] "Interaction data" refers to data about the actions or behaviors a user performs within a virtual space, such as movements, clicks, and interactions with objects.

[1185] "3D model data" refers to data on digital objects that are represented in three-dimensional space. This includes classrooms, historical buildings, characters, and other virtual objects.

[1186] "Real-time synchronization of execution" is a technology that allows multiple users to simultaneously display and reflect status and events in a virtual space shared by multiple users.

[1187] An "event trigger" is a process that is automatically executed when a specific condition or action occurs. This occurs based on the scenario progression or user interaction within the virtual learning environment.

[1188] "Distance learning" is a format in which a lecturer in a physically distant location delivers a lecture in real time through a virtual learning environment.

[1189] This invention relates to an educational support system that allows users to experience a realistic learning environment using virtual reality (VR) technology. The system provides three main functions: virtual classroom experience, historical and cultural experience, and remote lecture participation.

[1190] Virtual Classroom Experience

[1191] Hardware and Software:

[1192] The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D model of the virtual classroom, and uses a communication protocol such as WebSocket to synchronize data in real time.

[1193] The device uses a VR headset such as Oculus or HTC Vive, and the device uses VR-specific software to render the received 3D model data in real time.

[1194] Users use a VR headset and controllers to move freely around the virtual classroom and interact with the educator.

[1195] Examples:

[1196] When a user puts on a VR headset and enters a virtual classroom, the server generates a 3D model and initializes the user's location information. The device instantly renders the classroom environment based on the data sent from the server to the device. When a user raises their hand to ask a question or moves around the classroom, this interaction data is sent via the device to the server and synchronized with the devices of other participants.

[1197] History and Cultural Experience

[1198] Hardware and Software:

[1199] The server generates historical scenarios using 3D modeling software such as Blender or Maya, and also uses AI techniques to set specific event triggers.

[1200] The device uses Unity or Unreal Engine to render the VR environment based on the received scenario data.

[1201] Users interact with and explore historical buildings and characters using VR controllers.

[1202] Examples:

[1203] When students take an "Ancient Egypt" class, the server generates 3D models of the pyramids, pharaoh's palaces, and other structures. The server sends data to the device, which then displays these models on the headset. Using the controller, users explore the camopets and murals inside the pyramids, and their interaction data is instantly shared with other students.

[1204] Participating in remote lectures

[1205] Hardware and Software:

[1206] The server retrieves the instructor's video stream using streaming technologies such as Zoom or WebRTC.

[1207] The device uses software to display the video stream in real time, allowing users to watch the lecture through a display or VR headset.

[1208] The user inputs a question using a microphone or keyboard and sends it to the server via a communication protocol.

[1209] Examples:

[1210] When a European expert gives a lecture, the server captures his video stream and sends it to the device. Students view the video on their VR headsets and, if they have questions, use their microphones to send them to the server in real time. The server then relays the questions to the lecturer, obtains the lecturer's answers, and immediately sends them back to the user. This process allows for seamless communication even with lecturers in remote locations.

[1211] Example prompt sentence:

[1212] "Write a program that generates a VR simulation of Ancient Egypt and allows users to explore inside the pyramids."

[1213] Through each function, this system can add new value to traditional teaching methods and dramatically improve the learning experience.

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] System program processing

[1216] VR virtual classroom experience

[1217] Step 1:

[1218] The server generates a 3D model of the virtual classroom using a 3D engine such as Unity or Unreal Engine.

[1219] Input: 3D model template, user's initial position data.

[1220] Data processing: The 3D engine generates a 3D model based on the template data and sets the initial position for each user.

[1221] Output: The completed 3D model data.

[1222] Specific operation: The server constructs a virtual classroom using the template and initial position data.

[1223] Step 2:

[1224] The server sends the generated 3D model data to the terminal.

[1225] Input: Completed 3D model data.

[1226] Data processing: Converting the generated data into a format that can be sent.

[1227] Output: 3D model data for transmission.

[1228] Specific operation: The server encodes the data and sends it to the device.

[1229] Step 3:

[1230] The device renders the received 3D model data on the VR headset.

[1231] Input: 3D model data received from the server.

[1232] Data processing: Decode the data and render it as a VR environment.

[1233] Output: The rendered VR environment.

[1234] What it does: A virtual classroom is displayed in real time on the headset.

[1235] Step 4:

[1236] Users use VR controllers to move and interact within the virtual classroom.

[1237] Input: User operation data (movements, questions, etc.).

[1238] Data processing: Operation data is acquired in real time and converted into location information.

[1239] Output: User's current location and interaction data.

[1240] Specific actions: The user physically manipulates the controller and moves around in the virtual environment.

[1241] Step 5:

[1242] The device transmits the acquired user location information and interaction data to the server.

[1243] Input: User's current location and interaction data.

[1244] Data processing: Package the data and send it to the server.

[1245] Output: The packaged data.

[1246] Specific operation: Data is sent from the device to the server.

[1247] Step 6:

[1248] The server receives all user data and performs the synchronization process.

[1249] Input: Interaction data submitted by each user.

[1250] Data processing: Integrate data and update it to the latest information.

[1251] Output: The synchronized data.

[1252] Specific operation: The server sends data to all devices using WebSocket or similar and shares the latest status.

[1253] History and Cultural Experience

[1254] Step 1:

[1255] The server generates historical scenarios using 3D modeling software such as Blender or Maya.

[1256] Input: Historical scenario raw material file.

[1257] Data processing: Create and optimize scenarios using 3D modeling software.

[1258] Output: 3D scenario data.

[1259] What it does: The server builds and optimizes the scenario for the simulation.

[1260] Step 2:

[1261] The server transmits the generated historical scenario data to the terminal.

[1262] Input: Simulation data.

[1263] Data processing: Compression and conversion into a transmittable format.

[1264] Output: Compressed simulation data.

[1265] Specific operation: The server compresses the data and sends it to the terminal.

[1266] Step 3:

[1267] The device renders the received scenario data on the VR headset.

[1268] Input: Send data.

[1269] Data processing: Decode the data and render it into a VR environment.

[1270] Output: A rendered simulated environment.

[1271] Specific Behavior: Rendered to allow the user to begin exploring within the scenario.

[1272] Step 4:

[1273] The user interacts with specific objects and characters within the VR environment.

[1274] Input: User operation data.

[1275] Data processing: Convert the acquired operation data into interaction data.

[1276] Output: Interaction data.

[1277] Concrete behavior: The user explores and interacts with objects within the simulation.

[1278] Step 5:

[1279] The device sends the acquired interaction data to the server.

[1280] Input: Interaction data from the user.

[1281] Data processing: packaging and preparation for transmission.

[1282] Output: The packaged data.

[1283] Specific operation: Data is sent from the device to the server.

[1284] Step 6:

[1285] The server triggers specific events and synchronously distributes the results to other users.

[1286] Input: Interaction data.

[1287] Data processing: Executes event triggers and generates result data.

[1288] Output: Triggered event result data.

[1289] Specific operation: The event results are sent to other users' devices and synchronized.

[1290] Participating in remote lectures

[1291] Step 1:

[1292] The server retrieves the instructor's video stream using a streaming technology such as Zoom or WebRTC.

[1293] Input: Instructor video feed.

[1294] Data processing: Compression and optimization of video streams.

[1295] Output: Compressed video stream.

[1296] Specific operation: The instructor's video is uploaded to the server in real time.

[1297] Step 2:

[1298] The server transmits the video stream to the terminal.

[1299] Input: Compressed video stream.

[1300] Data processing: Format conversion as needed.

[1301] Output: Video stream for device.

[1302] Specific operation: The server delivers streaming data to the terminal.

[1303] Step 3:

[1304] The terminal displays the received video stream to the user.

[1305] Input: Video stream from the server.

[1306] Data processing: Decode and display the stream.

[1307] Output: Real-time video display.

[1308] What happens: The user watches the lecture on a display or VR headset.

[1309] Step 4:

[1310] The user inputs and submits a question by voice or text.

[1311] Input: Audio or text data.

[1312] Data processing: Converting audio to text based on data functions (in the case of audio).

[1313] Output: Question data.

[1314] What it does: The user creates and submits a question using a microphone or keyboard.

[1315] Step 5:

[1316] The terminal transmits the question data to the server.

[1317] Input: Question data.

[1318] Data processing: Packaging and preparing for transmission.

[1319] Output: Packaged question data.

[1320] Specific operation: Question data is sent from the terminal to the server.

[1321] Step 6:

[1322] The server relays the question to the instructor, obtains the instructor's answer, and returns it to the terminal.

[1323] Input: Question data.

[1324] Data processing: Converting data into a format suitable for instructors.

[1325] Output: Transformed question data and instructor answer data.

[1326] Specific operation: The server forwards the question to the instructor, obtains the answer, and then sends it back to the device.

[1327] In this way, the system can provide a complex learning environment and offer users a real-time interactive learning experience.

[1328] (Application example 1)

[1329] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1330] When providing educational support systems using virtual reality technology, the challenge is to ensure that users can enjoy a consistent, real-time learning experience while accessing high-quality educational content without geographical constraints. Conventional systems suffer from delays in real-time communication and insufficient information synchronization between users, making it difficult to maximize learning effectiveness.

[1331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1332] In this invention, the server includes: means for allowing users to participate in a virtual classroom using a virtual reality headset and interact with teachers and other users in real time; means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; means for allowing users to attend lectures and engage in Q&A sessions with experts and lecturers in remote locations in real time; and means for providing a content distribution service and allowing users to enjoy a learning experience in virtual reality. This allows users to interact with teachers and other users in real time, enjoy an immersive learning experience about history and culture, and participate in lectures from high-quality experts without being restricted by geographical location.

[1333] - A "virtual reality headset" is a device that allows a user to immerse themselves in a computer-generated virtual environment through visual and auditory experiences.

[1334] "User" refers to a learner or participant who uses an educational support system using virtual reality technology.

[1335] A "virtual classroom" is a simulated educational space where teachers and students interact and learn in real time within a virtual reality environment.

[1336] "Real-time" refers to instantaneous communication and data synchronization with minimal delay.

[1337] "Interaction" means the act of users exchanging information through a communication means.

[1338] "Teacher" refers to a professional who conducts lessons within a virtual classroom and provides educational content to students.

[1339] "Other users" refers to participants in a simultaneous learning activity within a virtual classroom or other virtual environment.

[1340] "Historical events" refer to significant events or occurrences that actually occurred in the past and are re-enacted for educational purposes.

[1341] "Cultural Experience" refers to a virtual reality scenario that allows people to learn about the culture of a particular region or era through hands-on experience.

[1342] A "remote location" refers to a physically separate location that is accessed via a communication means such as the Internet.

[1343] "Expert" refers to a lecturer or advisor with advanced knowledge and skills in a particular field.

[1344] A "lecture" is an educational activity that involves a coherent explanation or instruction on a particular topic.

[1345] "Question and answer" is a process in which a user asks a question and an expert or lecturer provides an answer to the question.

[1346] "Content Delivery Service" refers to an online platform for providing educational content to users using virtual reality technology.

[1347] "Interaction data" refers to data related to user operations and behavior, which is processed and synchronized on the server.

[1348] "Server" refers to a central processing unit or system for providing virtual classrooms and related services.

[1349] The present invention is a system for enhancing educational experiences by utilizing virtual reality technology. Specific embodiments thereof are described below.

[1350] Hardware and Software Configuration

[1351] Hardware used:

[1352] VR headset (e.g. Oculus Rift, HTC Vive)

[1353] Server (standard server with high performance)

[1354] User device (PC or tablet)

[1355] Software used:

[1356] Unity 3D (3D model rendering engine)

[1357] WebRTC (protocol for real-time communication)

[1358] Node.js (provides real-time processing capabilities on the server side)

[1359] Amazon AWS (cloud storage and data distribution service)

[1360] Overview of program processing

[1361] Server Role:

[1362] 1. Creating a virtual classroom

[1363] The server generates a 3D model of the virtual classroom using Unity 3D, sets the initial position and environment for each user, and sends the generated 3D model data to the user's device.

[1364] 2. Real-time communication

[1365] The server uses WebRTC to manage real-time communication between users, supporting audio and video exchanges with teachers and other users, providing a smooth interactive environment.

[1366] 3. Data synchronization

[1367] It collects user location and interaction data and synchronizes it with other users in real time. The data is stored on Amazon AWS and distributed to other devices as needed.

[1368] Device role:

[1369] 1. Rendering the Data

[1370] The device receives 3D model data from the server and renders it on the VR headset using Unity 3D, allowing users to experience virtual classrooms and historical scenarios.

[1371] 2. Collecting interactions

[1372] It collects user actions and movements in the environment in real time and sends the data to a server, which keeps it synchronized with other users.

[1373] Examples:

[1374] Virtual Classroom Experience

[1375] A user puts on a VR headset and joins a virtual classroom. The server sets the user's initial position and sends 3D model data to the device. The device renders this data on the VR headset, allowing the user to move freely within the virtual classroom and interact with other users and the teacher in real time.

[1376] History and Cultural Experience

[1377] To learn about historical events, users put on a VR headset. For example, a server generates an ancient Egyptian scenario and transmits the data to the device. Users can explore ancient Egyptian streets and pyramids and interact with an interactive guide.

[1378] Example prompt sentence:

[1379] "Generate a VR experience scenario about Egyptian history, including detailed depictions of the inside of the pyramids and an interactive guide that users can interact with."

[1380] summary

[1381] The system of the present invention provides users with a comprehensive and immersive learning experience through the collaboration of the server and user terminals, enabling them to receive high-quality education without being restricted by geographical location, maximizing the effectiveness of their learning.

[1382] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1383] Step 1: The server generates a 3D model of the virtual classroom

[1384] Input: Virtual classroom design data, user initial location information

[1385] Processing: The server uses Unity 3D to generate a 3D model of the virtual classroom. It performs calculations to determine the user's initial position and environment settings.

[1386] Output: 3D model data of the completed virtual classroom, initial user position information

[1387] Step 2: The server sends the virtual classroom data to the device

[1388] Input: 3D model data of the virtual classroom, user's initial position information

[1389] Processing: The server sends the generated 3D model data and initial location information to each user's device.

[1390] Output: 3D model data sent to the device, initial position information

[1391] Step 3: The device renders the received data to the VR headset

[1392] Input: 3D model data received from the server, user's initial location information

[1393] Processing: The device uses Unity 3D to render the received 3D model data on the VR headset, allowing the user to experience the visual and auditory world of the virtual classroom.

[1394] Output: Visual and auditory data of the virtual classroom displayed on a VR headset

[1395] Step 4: User moves and interacts in the VR environment

[1396] Input: VR headset, controller

[1397] Processing: Using the headset and controllers, users move around the VR environment and interact with other users and objects, and their actions are collected as data.

[1398] Output: Collected user location and interaction data

[1399] Step 5: The device sends the interaction data to the server

[1400] Input: User location and interaction data

[1401] Processing: The device sends the collected data to the server in real time.

[1402] Output: User location and interaction data sent to the server

[1403] Step 6: The server synchronizes the data to other users

[1404] Input: User location and interaction data

[1405] Processing: The server analyzes the received location and interaction data and generates the data necessary to synchronize it with other users' devices.

[1406] Output: Location and interaction data synced to other users

[1407] Step 7: Trigger specific events in the scenario

[1408] Input: Interaction data stored on the server

[1409] Processing: When a certain condition is met, the server triggers an event in the scenario and notifies all users of that information.

[1410] Output: Event trigger information and related data sent to each device

[1411] Step 8: The server delivers the video stream of the remote lecture.

[1412] Input: Video stream from instructor

[1413] Processing: The server uses WebRTC to deliver the instructor's video and audio streams to each user's device in real time.

[1414] Output: Real-time video stream of lectures delivered to user devices

[1415] Step 9: Users attend the remote lecture and submit questions

[1416] Input: User voice or text data

[1417] Processing: While listening to the lecture, the user asks questions by voice or text, and this data is sent from the terminal to the server.

[1418] Output: Question data sent to the server

[1419] Step 10: The server relays the question to the instructor and delivers the answer

[1420] Input: User question data, instructor answer data

[1421] Processing: The server relays the user's questions to the instructor, collects the instructor's answers in real time, and delivers them to each user's device.

[1422] Output: Teacher's response data distributed to each user's device

[1423] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1424] This invention provides an educational support system that uses virtual reality (VR) technology to enable students to have a more immersive learning experience, and in particular, by combining it with an emotion engine, it realizes interactive education based on the user's emotional state. Specifically, this system includes a virtual classroom, historical and cultural experiences, remote lecture participation, and user emotion recognition.

[1425] Program processing

[1426] The main stakeholders are the server, the terminal, and the user.

[1427] 1. VR Virtual Classroom Experience

[1428] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device.

[1429] The device renders the received 3D model data on the VR headset, providing the user with a virtual classroom view. The user can explore the VR classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[1430] 2. History and Culture Experience

[1431] The server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[1432] 3. Participating in lectures from remote locations

[1433] The server prepares lecture information from remote experts and lecturers and sets access rights for users. When the lecture starts, the server starts the lecturer's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the lecturer and sends the lecturer's answer to the user's terminal in real time.

[1434] 4. Incorporating an Emotional Engine

[1435] The server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc., and recognizes them as emotion data.

[1436] The device collects the user's emotional data in real time and sends it to the server. The server then adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if they are dissatisfied or confused, the server will provide additional explanations or a different scenario.

[1437] Specific examples

[1438] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the teacher and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[1439] If one student shows interest in the pyramid while another struggles with a difficult problem, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[1440] Furthermore, when a lecture by a remote expert is being given, the server transmits the expert's video stream to the device. When a student has a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. Furthermore, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[1441] In this way, the system of the present invention integrates four main functions: virtual classroom, historical and cultural experience, remote lecture participation, and emotion recognition engine, thereby dramatically improving the user's learning experience.

[1442] The processing flow will be explained below.

[1443] Processing steps for VR virtual classroom experience

[1444] Step 1:

[1445] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[1446] Step 2:

[1447] The server transmits the generated 3D model data and initial position information to the terminal.

[1448] Step 3:

[1449] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[1450] Step 4:

[1451] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[1452] Step 5:

[1453] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[1454] Step 6:

[1455] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[1456] History and Culture Experience Processing Steps

[1457] Step 1:

[1458] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[1459] Step 2:

[1460] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[1461] Step 3:

[1462] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[1463] Step 4:

[1464] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[1465] Step 5:

[1466] The terminal transmits the user's interaction data to the server in real time.

[1467] Step 6:

[1468] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[1469] Processing steps for remote lecture participation

[1470] Step 1:

[1471] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[1472] Step 2:

[1473] The server initiates the instructor's video stream and streams it to the terminal in real time.

[1474] Step 3:

[1475] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[1476] Step 4:

[1477] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[1478] Step 5:

[1479] The terminal transmits the user's question data to the server in real time.

[1480] Step 6:

[1481] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[1482] Built-in processing steps for the emotion engine

[1483] Step 1:

[1484] The server uses an emotion engine to analyze the user's emotional state.

[1485] Step 2:

[1486] The device collects the user's facial expressions, voice tone, and behavioral patterns as emotional data in real time.

[1487] Step 3:

[1488] The device transmits the collected emotion data to a server in real time.

[1489] Step 4:

[1490] The server adjusts the educational content based on the received emotional data, for example, if the user expresses dissatisfaction, it provides additional explanations or new approaches.

[1491] Step 5:

[1492] The server changes the behavior of objects and characters in the virtual environment based on the emotional data to optimize the user's experience.

[1493] Step 6:

[1494] The device follows instructions from the server and provides appropriate feedback to the user, customizing the learning experience according to the user's emotional state.

[1495] Example 2

[1496] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1497] Conventional educational systems have few opportunities for users to actively participate in the learning content, making it difficult to provide a realistic learning experience. Furthermore, they lack the technology to analyze the user's emotional state and dynamically adjust the educational content accordingly. As a result, there are issues with maintaining users' motivation to learn and making it difficult for them to progress effectively.

[1498] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with an instructor and other users in real time, a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them, a means for allowing a user to attend lectures and engage in Q&A sessions with experts and instructors in remote locations in real time, and a means for analyzing the user's emotional state and dynamically adjusting educational content based on the results. This allows the user to increase their motivation to learn through a realistic learning experience and progress effectively in their studies.

[1499] A "virtual reality headset" is a device that allows a user to immerse themselves in a virtual reality environment, providing visual and audio information.

[1500] A "virtual classroom" is a virtual space that users can participate in through a virtual reality headset, recreating an environment for educational activities.

[1501] An "instructor" is a person or system that has the role of educating users, and includes teachers and lecturers.

[1502] "Historical events" refer to events or occurrences that actually occurred in the past and are used as educational material.

[1503] A "cultural experience" is an experience provided to users through a virtual environment based on a specific culture or historical background.

[1504] An "expert" is a person with advanced knowledge and skills in a particular field who is qualified to teach or lecture.

[1505] "Emotional state" refers to the internal psychological state that can be inferred from the user's facial expression, tone of voice, behavioral patterns, etc.

[1506] "Dynamic adjustment" refers to the process of changing educational content in real time based on the user's reactions and situation.

[1507] "Interaction data" is information about the actions and operations a user performs within a virtual environment that is used by the system to understand those actions and generate appropriate responses.

[1508] "Triggering an event" means that the system automatically initiates a specific action or scenario when certain conditions are met.

[1509] "Synchronous distribution" is the process of enabling multiple users to share the same information or state in real time.

[1510] This invention combines virtual reality (VR) technology with an emotion engine to provide users with a realistic learning experience. Below, we will explain in detail what hardware and software are used to build the system and how data processing and calculations are performed.

[1511] Hardware and software used

[1512] Servers are high-performance computers, and recommended specifications include a high-clock processor, large amounts of memory, and SSD storage. Software running on the server includes a data communication system using WebSocket or HTTP, and a game engine such as Unity or Unreal Engine.

[1513] A device is a user-interface device, which can include a VR headset with a high-resolution display (e.g., Oculus Rift or HTC Vive) and a connected PC or mobile device. The software running on the device includes Unity or Unreal Engine for rendering the VR environment and controller software for processing user input.

[1514] The user wears a VR headset and uses a controller to explore the virtual environment. To collect the user's emotional state, biometric devices such as a camera, microphone, and heart rate sensor are connected to the device.

[1515] System processing overview

[1516] The server generates a 3D model of the virtual classroom and sets the user's initial location. This data is sent from the server to the device using WebSocket or HTTP protocol. The device then renders the received 3D model data on the VR headset using Unity or Unreal Engine, providing the user with a view of the virtual classroom.

[1517] Users can explore the VR classroom using a controller and interact with instructors and other users in real time. During this time, the user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[1518] In a historical and cultural experience, the server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[1519] When participating in a remote lecture, the server prepares lecture information from a remote expert or instructor and sets access rights for the user. When the lecture starts, the server starts the instructor's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the instructor and sends the instructor's answer to the user's terminal in real time.

[1520] Regarding the incorporation of an emotion engine, the server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc. and recognizes them as emotional data. The device collects the user's emotional data in real time and sends it to the server. The server dynamically adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if the user is dissatisfied or confused, additional explanations or a different scenario will be provided.

[1521] Specific examples

[1522] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the instructor and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[1523] If one student shows interest in the pyramid while another struggles with a difficult problem during the scenario, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[1524] When a lecture is given by a remote expert, the server sends the expert's video stream to the device. When a student wants to ask a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. In addition, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[1525] Prompt Sentence Examples

[1526] "Describe a scenario in which students would use VR to learn about ancient Egypt. Also, detail how an emotion recognition engine would help."

[1527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1528] Program processing flow

[1529] Step 1:

[1530] The server generates a 3D model of the virtual classroom and the user's initial position information.

[1531] Specifically, the server uses a game engine such as Unity or Unreal Engine to create a 3D model of the virtual classroom and sets the user's initial location information based on that. The input is the virtual classroom design and user ID, and the output is the generated 3D model data and initial location information, saved in JSON format.

[1532] Step 2:

[1533] The server sends the generated 3D model data and initial location information to the terminal.

[1534] The server sends this data to the device using WebSocket or HTTP protocol. The input includes the 3D model data and initial position information generated in step 1, and the output includes the data sent to the device.

[1535] Step 3:

[1536] The device renders the received 3D model data on the VR headset.

[1537] The device processes the received data using Unity or Unreal Engine and renders it in real time on the VR headset. The input includes 3D model data received from the server, and the output includes the virtual classroom view seen by the user.

[1538] Step 4:

[1539] Users explore and interact within the virtual classroom.

[1540] Using a VR headset and controllers, users can move freely around the virtual classroom and interact with other users and objects in real time. The inputs are the user's visual, auditory, and controller inputs, and the output is the interaction data generated.

[1541] Step 5:

[1542] The terminal transmits the user's interaction data to the server.

[1543] The device collects user location information and operation logs in real time and sends them to the server using WebSocket. The input includes user interaction data, and the output includes data sent to the server.

[1544] Step 6:

[1545] The server synchronizes the received interaction data with the terminals of other users.

[1546] The server synchronizes the received data to all other devices in real time. The input includes user interaction data, and the output includes data synchronized to other devices.

[1547] Step 7:

[1548] The server generates the data for a particular historical scenario.

[1549] The server generates 3D models and scenario scripts based on specific historical events and sends them to the terminal. The inputs include historical data and scenario design, and the outputs include the generated 3D models and scenario scripts.

[1550] Step 8:

[1551] The terminal starts the simulation, providing the user with a realistic historical experience.

[1552] The device starts a simulation using Unity or Unreal Engine based on the received scenario data. The input includes the received scenario data, and the output includes the virtual environment provided to the user.

[1553] Step 9:

[1554] The terminal transmits the interaction data to the server.

[1555] The device collects the interaction data of the user in the virtual environment in real time and sends it to the server.The input includes the user's interaction data, and the output includes the data sent to the server.

[1556] Step 10:

[1557] The server triggers a specific event and synchronizes the results to other devices.

[1558] The server triggers a specific event based on the received interaction data and synchronously distributes the result to all other terminals. The input includes the interaction data based on the trigger condition, and the output includes the synchronized event result.

[1559] Step 11:

[1560] The server prepares the lecture information for the remote location and transmits it to the terminal.

[1561] The server manages the contents and schedule of the lectures, sets access rights for participating users, and sends this information to the terminals. The input includes the contents and schedule of the lectures, and the output includes access rights information.

[1562] Step 12:

[1563] The terminal renders the instructor's video stream to the user.

[1564] The server starts the video stream at the beginning of the lecture, and the terminal renders it in real time.,The input includes the video stream data, and the output,includes the lecture video that is displayed to the user.

[1565] Step 13:

[1566] The terminal transmits the user's question data to the server.

[1567] When a user wants to ask a question, the question data is sent from the terminal to the server. The input includes the user's question data, and the output includes the data sent to the server.

[1568] Step 14:

[1569] The server relays questions to the instructor and delivers answers to the user.

[1570] The server relays questions to the instructor and sends the answers to the user's device in real time. The input includes the user's question data and the output includes the instructor's answer data.

[1571] Step 15:

[1572] The terminal transmits the user's emotion data to the server.

[1573] The device collects user emotion data using a camera and microphone and sends it to the server. The input includes the user emotion data, and the output includes the data sent to the server.

[1574] Step 16:

[1575] The server analyzes the received emotional data and dynamically adjusts the educational content.

[1576] The server uses an emotion analysis algorithm to analyze the user's emotional state and dynamically adjusts the educational content based on the results. The input is the user's emotional data, and the output is the adjusted educational content.

[1577] These steps enable the system of the present invention to provide an immersive learning experience for the user and to adapt educational content based on the user's emotional state in real time.

[1578] (Application example 2)

[1579] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1580] In modern educational and commercial environments, users have limited interactive options for enriching their experiences. Current virtual reality technologies lack the means to provide appropriate feedback and information based on the user's emotional state. This leads to a poor user experience and makes it difficult for users to learn or purchase effectively.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with a teacher and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a means for allowing a user to explore a virtual environment using smart glasses and present specific information or content based on the user's gaze or movements; and a means for recognizing the user's emotional state in real time and providing information or promotions according to that state. This enables an interactive experience according to the user's emotional state, dramatically improving the user experience in education and commerce.

[1582] A "virtual reality headset" is a device worn by a user to display a virtual reality environment, and is attached to the head.

[1583] A "virtual classroom" is a virtual classroom environment recreated using virtual reality technology that users can access through a virtual headset to create an immersive learning experience.

[1584] "Real-time" refers to the state in which data and information are processed and transmitted immediately, without delay, allowing users to interact with other users and systems instantly.

[1585] "Historical events" are important events or incidents that actually occurred in the past, which are recreated using virtual reality technology and can be experienced by users.

[1586] "Cultural experience" refers to the use of virtual reality technology to allow users to experience the cultural background, customs, and practices of a particular region or era.

[1587] The term "remote location" refers to a location away from the user's current location, where lectures and Q&A sessions can be held in real time with experts or lecturers in such locations.

[1588] "Experts and lecturers" are people who have advanced knowledge and skills in a particular field and who impart that knowledge to other users through lectures and education.

[1589] "Smart glasses" are eyeglass-type devices that can visually display information, allowing users to view information and content based on their line of sight and movements.

[1590] A "virtual environment" is a digital space created using virtual reality technology in which a user can have a virtual experience.

[1591] "Gaze" refers to the direction of the user's eyes, and is the direction of the user's gaze detected by a device such as smart glasses.

[1592] "Movement" refers to the movement of a user's hands or body, and is an action that conveys the user's intentions in an interactive system.

[1593] "Specific information and content" refers to data that is of interest to users and is provided through virtual reality or smart glasses, such as product information, promotions, and educational materials.

[1594] "Emotional state" refers to the user's current state of mind, and is recognized in real time by analyzing facial expressions, tone of voice, behavior, etc.

[1595] "Promotions" refer to advertising activities or special offers designed to increase purchasing interest in a particular product or service and are delivered based on a user's emotional state.

[1596] The present invention relates to an education and commerce system that uses smart glasses to provide customers with a rich experience in a virtual store, enabling interactive information provision based on the user's emotional state.

[1597] Using a virtual reality headset

[1598] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device. The device renders the received 3D model data on a virtual reality headset, providing the user with a view of the virtual classroom. The user can explore the virtual classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[1599] Using smart glasses in virtual stores

[1600] The server generates a 3D model of the virtual store and sets the user's initial location. The user puts on the smart glasses and freely explores the virtual store. Specific product information and promotions are automatically displayed on the smart glasses display based on the user's gaze and movements.

[1601] Real-time recognition of emotional states

[1602] The device is equipped with a camera and microphone to analyze the user's facial expressions and tone of voice. The server uses this data to recognize the user's emotional state in real time. For example, if the user is confused, a special promotion and support options will be displayed.

[1603] Hardware and software used

[1604] Hardware

[1605] Virtual reality headset: A device worn on the head that displays a virtual classroom or storefront.

[1606] Smart glasses: Eyeglasses-like devices that display information visually.

[1607] Camera and microphone: Input devices for analyzing the user's facial expressions and voice.

[1608] software

[1609] Emotion engine: A program that analyzes user emotions in real time.

[1610] VR interface: A program for generating and manipulating a virtual environment.

[1611] Specific examples

[1612] A customer puts on a virtual reality headset and explores a virtual store. When the customer looks at a particular product, detailed information about that product appears on the display. If the customer shows signs of confusion, the system displays a special promotion in real time and also offers the option to video chat with a staff member. In this way, the customer receives personalized information and support.

[1613] Prompt Sentence Examples

[1614] Create an example of a smart glasses application that uses an emotion engine to help customers explore products in a virtual store, including displaying more information when the customer looks at a product and showing special promotions if the customer's emotion is negative.

[1615] The above is a detailed description of the embodiments of the present invention, which can dramatically improve a user's educational and business transaction experience in a virtual environment.

[1616] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1617] Step 1:

[1618] The server generates a 3D model of the virtual classroom or virtual store and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the terminal. The input in this step is the design data of the virtual environment, and the output is the 3D model data and initial location information sent to the terminal.

[1619] Step 2:

[1620] The terminal renders the 3D model data received from the server onto a virtual reality headset or smart glasses, providing the user with a virtual classroom or virtual store view. The input in this step is the 3D model data sent from the server, and the output is the virtual environment displayed to the user.

[1621] Step 3:

[1622] The user explores the virtual classroom or virtual store using a controller or gaze. The user's location information and interaction data are sent from the device to the server. The input in this step is the user's movement and gaze information, and the output is the user's location information and interaction data sent to the server.

[1623] Step 4:

[1624] The server synchronizes the received user location information and interaction data with other users' devices, allowing all users to share the latest status. The input in this step is the user's location information and interaction data, and the output is synchronized data sent to other users' devices.

[1625] Step 5:

[1626] The camera and microphone installed on the device analyze the user's facial expressions and tone of voice in real time to obtain emotional data. The input in this step is the user's facial expression data and voice data, and the output is the analyzed emotional data.

[1627] Step 6:

[1628] The server analyzes the user's emotional state based on the emotional data sent from the terminal. For example, if the user is confused, it generates special promotions and support options. The input in this step is the emotional data, and the output is the generated promotions and support options.

[1629] Step 7:

[1630] The device presents the promotions and support options sent from the server to the user. This is done by displaying the information on the smart glasses display. The input in this step is the promotions and support options from the server, and the output is the information presented to the user.

[1631] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1632] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1633] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1634] [Fourth embodiment]

[1635] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1636] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1637] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1638] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1639] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1640] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1641] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1642] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1643] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1644] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1645] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1646] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1647] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1648] This invention provides an educational support system that utilizes virtual reality (VR) technology to enable students to have a more immersive learning experience. Specifically, the system includes three main functions: virtual classroom, historical and cultural experience, and remote lecture participation.

[1649] Program processing

[1650] The main stakeholders are the server, the terminal, and the user.

[1651] 1. VR Virtual Classroom Experience

[1652] The server plays a central role, generating a 3D model of the virtual classroom and determining the initial positions and settings of each user. After the virtual classroom data is generated, the server transmits it to the terminals.

[1653] The device then renders the received 3D model data onto the VR headset, allowing the user to experience the visual and auditory sensations of being in the classroom. As the user uses the controller to move around the VR environment and interact with the teacher, the device sends positional and interaction data to the server.

[1654] The server receives this data and synchronizes it with all participants' devices, providing a real-time interactive environment.

[1655] 2. History and Culture Experience

[1656] The server generates a specific historical scenario and prepares the associated 3D model and simulation data, which are then sent to the device.

[1657] The device starts a simulation based on the received scenario data, providing the user with an experience that makes them feel as if they are actually there. The user explores and interacts within the simulation, and this interaction data is sent from the device to the server.

[1658] The server triggers specific events and synchronously distributes the results to other users, allowing users to enjoy a real-time interactive historical and cultural experience.

[1659] 3. Participating in lectures from remote locations

[1660] The server manages real-time communication with remote experts and lecturers. When a lecture starts, the server transmits the lecturer's video stream to the terminal.

[1661] The terminal displays the received video stream to the user. If the user wants to ask a question during the lecture, they can send the question by voice or text. This question data is sent from the terminal to the server.

[1662] The server relays questions to the instructor and sends the instructor's answers back to the user's device in real time, allowing users to participate in high-quality lectures and receive direct answers from experts regardless of location.

[1663] Specific examples

[1664] During a history class, students don VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. The data is then sent to the device, which renders the VR environment, allowing students to explore the ancient Egyptian landscape. When a student touches a specific object or interacts with a character, the data is sent from the device to the server and synchronized with the other students.

[1665] On another day, a lecture is given by a European expert. The server starts the lecturer's video stream and distributes it to the devices. Students can participate in the lecture in real time and ask questions. Questions are sent from the devices to the server, which relays them to the lecturer. The lecturer's answers are sent back to the students' devices via the server.

[1666] In this way, the system of the present invention provides three main functions: virtual classroom, historical and cultural experience, and remote lecture participation, which can dramatically improve students' concentration and learning effectiveness.

[1667] The processing flow will be explained below.

[1668] Processing steps for VR virtual classroom experience

[1669] Step 1:

[1670] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[1671] Step 2:

[1672] The server transmits the generated 3D model data and initial position information to the terminal.

[1673] Step 3:

[1674] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[1675] Step 4:

[1676] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[1677] Step 5:

[1678] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[1679] Step 6:

[1680] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[1681] History and Culture Experience Processing Steps

[1682] Step 1:

[1683] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[1684] Step 2:

[1685] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[1686] Step 3:

[1687] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[1688] Step 4:

[1689] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[1690] Step 5:

[1691] The terminal transmits the user's interaction data to the server in real time.

[1692] Step 6:

[1693] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[1694] Processing steps for remote lecture participation

[1695] Step 1:

[1696] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[1697] Step 2:

[1698] The server initiates the instructor's video stream and streams it to the terminal in real time.

[1699] Step 3:

[1700] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[1701] Step 4:

[1702] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[1703] Step 5:

[1704] The terminal transmits the user's question data to the server in real time.

[1705] Step 6:

[1706] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[1707] This allows users to have an interactive, high-quality learning experience without being restricted by geography.

[1708] Example 1

[1709] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1710] In conventional educational systems, it is difficult for students to participate in actual classrooms, to realistically recreate historical events or cultural experiences, or to receive lectures from experts or lecturers in remote locations in real time. As a result, there are problems that limit learning effectiveness and concentration. Furthermore, there is also the issue of incomplete synchronization of user interaction data and location information within the virtual environment, making it difficult to provide a smooth real-time interactive environment.

[1711] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1712] In this invention, the server includes: a means for allowing users to participate in a virtual learning environment using a virtual reality display device and interact with educators and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a central control unit for generating 3D model data and transmitting it to terminals; a terminal device for transmitting user position information and interaction data within the virtual environment to the central control unit and synchronizing the latest status for all users; and a means for transmitting interactions with specific objects and characters to the central control unit, triggering events, and synchronously distributing them to other users. This provides an environment in which multiple users can simultaneously learn while interacting in the same virtual space in real time, and also enables historical and cultural experiences to be realistically reproduced using virtual reality technology and lectures by lecturers in remote locations.

[1713] A "virtual reality display device" is a device that allows users to experience images and sounds in a virtual space in real time. Specifically, this includes head-mounted displays and VR goggles.

[1714] A "virtual learning environment" is a virtual space where users can learn or train using virtual reality technology. It is designed to mimic a real-world classroom or learning scenario.

[1715] An "educator" is a person who has a teaching role within a virtual learning environment, such as a teacher, lecturer, or instructor.

[1716] The "central control device" is the central computer of the system, and is a device that manages the generation of virtual environments, synchronization of user data, triggering of specific events, etc.

[1717] A "terminal device" is a device through which a user accesses a virtual reality environment, such as a computer, smartphone, or tablet.

[1718] "Interaction data" refers to data about the actions or behaviors a user performs within a virtual space, such as movements, clicks, and interactions with objects.

[1719] "3D model data" refers to data on digital objects that are represented in three-dimensional space. This includes classrooms, historical buildings, characters, and other virtual objects.

[1720] "Real-time synchronization of execution" is a technology that allows multiple users to simultaneously display and reflect status and events in a virtual space shared by multiple users.

[1721] An "event trigger" is a process that is automatically executed when a specific condition or action occurs. This occurs based on the scenario progression or user interaction within the virtual learning environment.

[1722] "Distance learning" is a format in which a lecturer in a physically distant location delivers a lecture in real time through a virtual learning environment.

[1723] This invention relates to an educational support system that allows users to experience a realistic learning environment using virtual reality (VR) technology. The system provides three main functions: virtual classroom experience, historical and cultural experience, and remote lecture participation.

[1724] Virtual Classroom Experience

[1725] Hardware and Software:

[1726] The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D model of the virtual classroom, and uses a communication protocol such as WebSocket to synchronize data in real time.

[1727] The device uses a VR headset such as Oculus or HTC Vive, and the device uses VR-specific software to render the received 3D model data in real time.

[1728] Users use a VR headset and controllers to move freely around the virtual classroom and interact with the educator.

[1729] Examples:

[1730] When a user puts on a VR headset and enters a virtual classroom, the server generates a 3D model and initializes the user's location information. The device instantly renders the classroom environment based on the data sent from the server to the device. When a user raises their hand to ask a question or moves around the classroom, this interaction data is sent via the device to the server and synchronized with the devices of other participants.

[1731] History and Cultural Experience

[1732] Hardware and Software:

[1733] The server generates historical scenarios using 3D modeling software such as Blender or Maya, and also uses AI techniques to set specific event triggers.

[1734] The device uses Unity or Unreal Engine to render the VR environment based on the received scenario data.

[1735] Users interact with and explore historical buildings and characters using VR controllers.

[1736] Examples:

[1737] When students take an "Ancient Egypt" class, the server generates 3D models of the pyramids, pharaoh's palaces, and other structures. The server sends data to the device, which then displays these models on the headset. Using the controller, users explore the camopets and murals inside the pyramids, and their interaction data is instantly shared with other students.

[1738] Participating in remote lectures

[1739] Hardware and Software:

[1740] The server retrieves the instructor's video stream using streaming technologies such as Zoom or WebRTC.

[1741] The device uses software to display the video stream in real time, allowing users to watch the lecture through a display or VR headset.

[1742] The user inputs a question using a microphone or keyboard and sends it to the server via a communication protocol.

[1743] Examples:

[1744] When a European expert gives a lecture, the server captures his video stream and sends it to the device. Students view the video on their VR headsets and, if they have questions, use their microphones to send them to the server in real time. The server then relays the questions to the lecturer, obtains the lecturer's answers, and immediately sends them back to the user. This process allows for seamless communication even with lecturers in remote locations.

[1745] Example prompt sentence:

[1746] "Write a program that generates a VR simulation of Ancient Egypt and allows users to explore inside the pyramids."

[1747] Through each function, this system can add new value to traditional teaching methods and dramatically improve the learning experience.

[1748] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1749] System program processing

[1750] VR virtual classroom experience

[1751] Step 1:

[1752] The server generates a 3D model of the virtual classroom using a 3D engine such as Unity or Unreal Engine.

[1753] Input: 3D model template, user's initial position data.

[1754] Data processing: The 3D engine generates a 3D model based on the template data and sets the initial position for each user.

[1755] Output: The completed 3D model data.

[1756] Specific operation: The server constructs a virtual classroom using the template and initial position data.

[1757] Step 2:

[1758] The server sends the generated 3D model data to the terminal.

[1759] Input: Completed 3D model data.

[1760] Data processing: Converting the generated data into a format that can be sent.

[1761] Output: 3D model data for transmission.

[1762] Specific operation: The server encodes the data and sends it to the device.

[1763] Step 3:

[1764] The device renders the received 3D model data on the VR headset.

[1765] Input: 3D model data received from the server.

[1766] Data processing: Decode the data and render it as a VR environment.

[1767] Output: The rendered VR environment.

[1768] What it does: A virtual classroom is displayed in real time on the headset.

[1769] Step 4:

[1770] Users use VR controllers to move and interact within the virtual classroom.

[1771] Input: User operation data (movements, questions, etc.).

[1772] Data processing: Operation data is acquired in real time and converted into location information.

[1773] Output: User's current location and interaction data.

[1774] Specific actions: The user physically manipulates the controller and moves around in the virtual environment.

[1775] Step 5:

[1776] The device transmits the acquired user location information and interaction data to the server.

[1777] Input: User's current location and interaction data.

[1778] Data processing: Package the data and send it to the server.

[1779] Output: The packaged data.

[1780] Specific operation: Data is sent from the device to the server.

[1781] Step 6:

[1782] The server receives all user data and performs the synchronization process.

[1783] Input: Interaction data submitted by each user.

[1784] Data processing: Integrate data and update it to the latest information.

[1785] Output: The synchronized data.

[1786] Specific operation: The server sends data to all devices using WebSocket or similar and shares the latest status.

[1787] History and Cultural Experience

[1788] Step 1:

[1789] The server generates historical scenarios using 3D modeling software such as Blender or Maya.

[1790] Input: Historical scenario raw material file.

[1791] Data processing: Create and optimize scenarios using 3D modeling software.

[1792] Output: 3D scenario data.

[1793] What it does: The server builds and optimizes the scenario for the simulation.

[1794] Step 2:

[1795] The server transmits the generated historical scenario data to the terminal.

[1796] Input: Simulation data.

[1797] Data processing: Compression and conversion into a transmittable format.

[1798] Output: Compressed simulation data.

[1799] Specific operation: The server compresses the data and sends it to the terminal.

[1800] Step 3:

[1801] The device renders the received scenario data on the VR headset.

[1802] Input: Send data.

[1803] Data processing: Decode the data and render it into a VR environment.

[1804] Output: A rendered simulated environment.

[1805] Specific Behavior: Rendered to allow the user to begin exploring within the scenario.

[1806] Step 4:

[1807] The user interacts with specific objects and characters within the VR environment.

[1808] Input: User operation data.

[1809] Data processing: Convert the acquired operation data into interaction data.

[1810] Output: Interaction data.

[1811] Concrete behavior: The user explores and interacts with objects within the simulation.

[1812] Step 5:

[1813] The device sends the acquired interaction data to the server.

[1814] Input: Interaction data from the user.

[1815] Data processing: packaging and preparation for transmission.

[1816] Output: The packaged data.

[1817] Specific operation: Data is sent from the device to the server.

[1818] Step 6:

[1819] The server triggers specific events and synchronously distributes the results to other users.

[1820] Input: Interaction data.

[1821] Data processing: Executes event triggers and generates result data.

[1822] Output: Triggered event result data.

[1823] Specific operation: The event results are sent to other users' devices and synchronized.

[1824] Participating in remote lectures

[1825] Step 1:

[1826] The server retrieves the instructor's video stream using a streaming technology such as Zoom or WebRTC.

[1827] Input: Instructor video feed.

[1828] Data processing: Compression and optimization of video streams.

[1829] Output: Compressed video stream.

[1830] Specific operation: The instructor's video is uploaded to the server in real time.

[1831] Step 2:

[1832] The server transmits the video stream to the terminal.

[1833] Input: Compressed video stream.

[1834] Data processing: Format conversion as needed.

[1835] Output: Video stream for device.

[1836] Specific operation: The server delivers streaming data to the terminal.

[1837] Step 3:

[1838] The terminal displays the received video stream to the user.

[1839] Input: Video stream from the server.

[1840] Data processing: Decode and display the stream.

[1841] Output: Real-time video display.

[1842] What happens: The user watches the lecture on a display or VR headset.

[1843] Step 4:

[1844] The user inputs and submits a question by voice or text.

[1845] Input: Audio or text data.

[1846] Data processing: Converting audio to text based on data functions (in the case of audio).

[1847] Output: Question data.

[1848] What it does: The user creates and submits a question using a microphone or keyboard.

[1849] Step 5:

[1850] The terminal transmits the question data to the server.

[1851] Input: Question data.

[1852] Data processing: Packaging and preparing for transmission.

[1853] Output: Packaged question data.

[1854] Specific operation: Question data is sent from the terminal to the server.

[1855] Step 6:

[1856] The server relays the question to the instructor, obtains the instructor's answer, and returns it to the terminal.

[1857] Input: Question data.

[1858] Data processing: Converting data into a format suitable for instructors.

[1859] Output: Transformed question data and instructor answer data.

[1860] Specific operation: The server forwards the question to the instructor, obtains the answer, and then sends it back to the device.

[1861] In this way, the system can provide a complex learning environment and offer users a real-time interactive learning experience.

[1862] (Application example 1)

[1863] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1864] When providing educational support systems using virtual reality technology, the challenge is to ensure that users can enjoy a consistent, real-time learning experience while accessing high-quality educational content without geographical constraints. Conventional systems suffer from delays in real-time communication and insufficient information synchronization between users, making it difficult to maximize learning effectiveness.

[1865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1866] In this invention, the server includes: means for allowing users to participate in a virtual classroom using a virtual reality headset and interact with teachers and other users in real time; means for recreating historical events and cultural experiences using virtual reality technology and allowing users to experience them; means for allowing users to attend lectures and engage in Q&A sessions with experts and lecturers in remote locations in real time; and means for providing a content distribution service and allowing users to enjoy a learning experience in virtual reality. This allows users to interact with teachers and other users in real time, enjoy an immersive learning experience about history and culture, and participate in lectures from high-quality experts without being restricted by geographical location.

[1867] - A "virtual reality headset" is a device that allows a user to immerse themselves in a computer-generated virtual environment through visual and auditory experiences.

[1868] "User" refers to a learner or participant who uses an educational support system using virtual reality technology.

[1869] A "virtual classroom" is a simulated educational space where teachers and students interact and learn in real time within a virtual reality environment.

[1870] "Real-time" refers to instantaneous communication and data synchronization with minimal delay.

[1871] "Interaction" means the act of users exchanging information through a communication means.

[1872] "Teacher" refers to a professional who conducts lessons within a virtual classroom and provides educational content to students.

[1873] "Other users" refers to participants in a simultaneous learning activity within a virtual classroom or other virtual environment.

[1874] "Historical events" refer to significant events or occurrences that actually occurred in the past and are re-enacted for educational purposes.

[1875] "Cultural Experience" refers to a virtual reality scenario that allows people to learn about the culture of a particular region or era through hands-on experience.

[1876] A "remote location" refers to a physically separate location that is accessed via a communication means such as the Internet.

[1877] "Expert" refers to a lecturer or advisor with advanced knowledge and skills in a particular field.

[1878] A "lecture" is an educational activity that involves a coherent explanation or instruction on a particular topic.

[1879] "Question and answer" is a process in which a user asks a question and an expert or lecturer provides an answer to the question.

[1880] "Content Delivery Service" refers to an online platform for providing educational content to users using virtual reality technology.

[1881] "Interaction data" refers to data related to user operations and behavior, which is processed and synchronized on the server.

[1882] "Server" refers to a central processing unit or system for providing virtual classrooms and related services.

[1883] The present invention is a system for enhancing educational experiences by utilizing virtual reality technology. Specific embodiments thereof are described below.

[1884] Hardware and Software Configuration

[1885] Hardware used:

[1886] VR headset (e.g. Oculus Rift, HTC Vive)

[1887] Server (standard server with high performance)

[1888] User device (PC or tablet)

[1889] Software used:

[1890] Unity 3D (3D model rendering engine)

[1891] WebRTC (protocol for real-time communication)

[1892] Node.js (provides real-time processing capabilities on the server side)

[1893] Amazon AWS (cloud storage and data distribution service)

[1894] Overview of program processing

[1895] Server Role:

[1896] 1. Creating a virtual classroom

[1897] The server generates a 3D model of the virtual classroom using Unity 3D, sets the initial position and environment for each user, and sends the generated 3D model data to the user's device.

[1898] 2. Real-time communication

[1899] The server uses WebRTC to manage real-time communication between users, supporting audio and video exchanges with teachers and other users, providing a smooth interactive environment.

[1900] 3. Data synchronization

[1901] It collects user location and interaction data and synchronizes it with other users in real time. The data is stored on Amazon AWS and distributed to other devices as needed.

[1902] Device role:

[1903] 1. Rendering the Data

[1904] The device receives 3D model data from the server and renders it on the VR headset using Unity 3D, allowing users to experience virtual classrooms and historical scenarios.

[1905] 2. Collecting interactions

[1906] It collects user actions and movements in the environment in real time and sends the data to a server, which keeps it synchronized with other users.

[1907] Examples:

[1908] Virtual Classroom Experience

[1909] A user puts on a VR headset and joins a virtual classroom. The server sets the user's initial position and sends 3D model data to the device. The device renders this data on the VR headset, allowing the user to move freely within the virtual classroom and interact with other users and the teacher in real time.

[1910] History and Cultural Experience

[1911] To learn about historical events, users put on a VR headset. For example, a server generates an ancient Egyptian scenario and transmits the data to the device. Users can explore ancient Egyptian streets and pyramids and interact with an interactive guide.

[1912] Example prompt sentence:

[1913] "Generate a VR experience scenario about Egyptian history, including detailed depictions of the inside of the pyramids and an interactive guide that users can interact with."

[1914] summary

[1915] The system of the present invention provides users with a comprehensive and immersive learning experience through the collaboration of the server and user terminals, enabling them to receive high-quality education without being restricted by geographical location, maximizing the effectiveness of their learning.

[1916] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1917] Step 1: The server generates a 3D model of the virtual classroom

[1918] Input: Virtual classroom design data, user initial location information

[1919] Processing: The server uses Unity 3D to generate a 3D model of the virtual classroom. It performs calculations to determine the user's initial position and environment settings.

[1920] Output: 3D model data of the completed virtual classroom, initial user position information

[1921] Step 2: The server sends the virtual classroom data to the device

[1922] Input: 3D model data of the virtual classroom, user's initial position information

[1923] Processing: The server sends the generated 3D model data and initial location information to each user's device.

[1924] Output: 3D model data sent to the device, initial position information

[1925] Step 3: The device renders the received data to the VR headset

[1926] Input: 3D model data received from the server, user's initial location information

[1927] Processing: The device uses Unity 3D to render the received 3D model data on the VR headset, allowing the user to experience the visual and auditory world of the virtual classroom.

[1928] Output: Visual and auditory data of the virtual classroom displayed on a VR headset

[1929] Step 4: User moves and interacts in the VR environment

[1930] Input: VR headset, controller

[1931] Processing: Using the headset and controllers, users move around the VR environment and interact with other users and objects, and their actions are collected as data.

[1932] Output: Collected user location and interaction data

[1933] Step 5: The device sends the interaction data to the server

[1934] Input: User location and interaction data

[1935] Processing: The device sends the collected data to the server in real time.

[1936] Output: User location and interaction data sent to the server

[1937] Step 6: The server synchronizes the data to other users

[1938] Input: User location and interaction data

[1939] Processing: The server analyzes the received location and interaction data and generates the data necessary to synchronize it with other users' devices.

[1940] Output: Location and interaction data synced to other users

[1941] Step 7: Trigger specific events in the scenario

[1942] Input: Interaction data stored on the server

[1943] Processing: When a certain condition is met, the server triggers an event in the scenario and notifies all users of that information.

[1944] Output: Event trigger information and related data sent to each device

[1945] Step 8: The server delivers the video stream of the remote lecture.

[1946] Input: Video stream from instructor

[1947] Processing: The server uses WebRTC to deliver the instructor's video and audio streams to each user's device in real time.

[1948] Output: Real-time video stream of lectures delivered to user devices

[1949] Step 9: Users attend the remote lecture and submit questions

[1950] Input: User voice or text data

[1951] Processing: While listening to the lecture, the user asks questions by voice or text, and this data is sent from the terminal to the server.

[1952] Output: Question data sent to the server

[1953] Step 10: The server relays the question to the instructor and delivers the answer

[1954] Input: User question data, instructor answer data

[1955] Processing: The server relays the user's questions to the instructor, collects the instructor's answers in real time, and delivers them to each user's device.

[1956] Output: Teacher's response data distributed to each user's device

[1957] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1958] This invention provides an educational support system that uses virtual reality (VR) technology to enable students to have a more immersive learning experience, and in particular, by combining it with an emotion engine, it realizes interactive education based on the user's emotional state. Specifically, this system includes a virtual classroom, historical and cultural experiences, remote lecture participation, and user emotion recognition.

[1959] Program processing

[1960] The main stakeholders are the server, the terminal, and the user.

[1961] 1. VR Virtual Classroom Experience

[1962] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device.

[1963] The device renders the received 3D model data on the VR headset, providing the user with a virtual classroom view. The user can explore the VR classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[1964] 2. History and Culture Experience

[1965] The server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[1966] 3. Participating in lectures from remote locations

[1967] The server prepares lecture information from remote experts and lecturers and sets access rights for users. When the lecture starts, the server starts the lecturer's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the lecturer and sends the lecturer's answer to the user's terminal in real time.

[1968] 4. Incorporating an Emotional Engine

[1969] The server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc., and recognizes them as emotion data.

[1970] The device collects the user's emotional data in real time and sends it to the server. The server then adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if they are dissatisfied or confused, the server will provide additional explanations or a different scenario.

[1971] Specific examples

[1972] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the teacher and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[1973] If one student shows interest in the pyramid while another struggles with a difficult problem, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[1974] Furthermore, when a lecture by a remote expert is being given, the server transmits the expert's video stream to the device. When a student has a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. Furthermore, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[1975] In this way, the system of the present invention integrates four main functions: virtual classroom, historical and cultural experience, remote lecture participation, and emotion recognition engine, thereby dramatically improving the user's learning experience.

[1976] The processing flow will be explained below.

[1977] Processing steps for VR virtual classroom experience

[1978] Step 1:

[1979] The server generates a 3D model of the virtual classroom and sets the initial location information for users to join.

[1980] Step 2:

[1981] The server transmits the generated 3D model data and initial position information to the terminal.

[1982] Step 3:

[1983] The device receives the 3D model data and location information sent from the server and renders the virtual classroom scene to the user through the VR headset.

[1984] Step 4:

[1985] Users use a VR headset and controllers to explore the virtual classroom and interact with the teacher and other users.

[1986] Step 5:

[1987] The device transmits the user's location information and interaction data (e.g., utterances and movements) to the server in real time.

[1988] Step 6:

[1989] The server synchronizes the received data with the other users' terminals, so that the latest status is reflected in all participants.

[1990] History and Culture Experience Processing Steps

[1991] Step 1:

[1992] The server generates a specific historical scenario and prepares the associated 3D model and simulation data.

[1993] Step 2:

[1994] The server transmits scenario data to the terminal and sets the user's initial position and interaction points.

[1995] Step 3:

[1996] The device starts a simulation based on the received scenario data, providing the user with an immersive historical experience through a VR headset.

[1997] Step 4:

[1998] The user explores the virtual environment and interacts with specific objects and characters within the simulation.

[1999] Step 5:

[2000] The terminal transmits the user's interaction data to the server in real time.

[2001] Step 6:

[2002] The server triggers an event based on a specific interaction and synchronously distributes the results to other users' terminals.

[2003] Processing steps for remote lecture participation

[2004] Step 1:

[2005] The server prepares lecture information from experts and lecturers in remote locations and sets access rights for users.

[2006] Step 2:

[2007] The server initiates the instructor's video stream and streams it to the terminal in real time.

[2008] Step 3:

[2009] The terminal receives the video stream sent from the server and displays the video and audio of the lecture to the user through a VR headset.

[2010] Step 4:

[2011] If a user wishes to ask a question during a lecture, the user can input the question by voice or text.

[2012] Step 5:

[2013] The terminal transmits the user's question data to the server in real time.

[2014] Step 6:

[2015] The server relays the questions to the instructor's terminal and transmits the instructor's answers to the user's terminal in real time.

[2016] Built-in processing steps for the emotion engine

[2017] Step 1:

[2018] The server uses an emotion engine to analyze the user's emotional state.

[2019] Step 2:

[2020] The device collects the user's facial expressions, voice tone, and behavioral patterns as emotional data in real time.

[2021] Step 3:

[2022] The device transmits the collected emotion data to a server in real time.

[2023] Step 4:

[2024] The server adjusts the educational content based on the received emotional data, for example, if the user expresses dissatisfaction, it provides additional explanations or new approaches.

[2025] Step 5:

[2026] The server changes the behavior of objects and characters in the virtual environment based on the emotional data to optimize the user's experience.

[2027] Step 6:

[2028] The device follows instructions from the server and provides appropriate feedback to the user, customizing the learning experience according to the user's emotional state.

[2029] Example 2

[2030] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2031] Conventional educational systems have few opportunities for users to actively participate in the learning content, making it difficult to provide a realistic learning experience. Furthermore, they lack the technology to analyze the user's emotional state and dynamically adjust the educational content accordingly. As a result, there are issues with maintaining users' motivation to learn and making it difficult for them to progress effectively.

[2032] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with an instructor and other users in real time, a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them, a means for allowing a user to attend lectures and engage in Q&A sessions with experts and instructors in remote locations in real time, and a means for analyzing the user's emotional state and dynamically adjusting educational content based on the results. This allows the user to increase their motivation to learn through a realistic learning experience and progress effectively in their studies.

[2033] A "virtual reality headset" is a device that allows a user to immerse themselves in a virtual reality environment, providing visual and audio information.

[2034] A "virtual classroom" is a virtual space that users can participate in through a virtual reality headset, recreating an environment for educational activities.

[2035] An "instructor" is a person or system that has the role of educating users, and includes teachers and lecturers.

[2036] "Historical events" refer to events or occurrences that actually occurred in the past and are used as educational material.

[2037] A "cultural experience" is an experience provided to users through a virtual environment based on a specific culture or historical background.

[2038] An "expert" is a person with advanced knowledge and skills in a particular field who is qualified to teach or lecture.

[2039] "Emotional state" refers to the internal psychological state that can be inferred from the user's facial expression, tone of voice, behavioral patterns, etc.

[2040] "Dynamic adjustment" refers to the process of changing educational content in real time based on the user's reactions and situation.

[2041] "Interaction data" is information about the actions and operations a user performs within a virtual environment that is used by the system to understand those actions and generate appropriate responses.

[2042] "Triggering an event" means that the system automatically initiates a specific action or scenario when certain conditions are met.

[2043] "Synchronous distribution" is the process of enabling multiple users to share the same information or state in real time.

[2044] This invention combines virtual reality (VR) technology with an emotion engine to provide users with a realistic learning experience. Below, we will explain in detail what hardware and software are used to build the system and how data processing and calculations are performed.

[2045] Hardware and software used

[2046] Servers are high-performance computers, and recommended specifications include a high-clock processor, large amounts of memory, and SSD storage. Software running on the server includes a data communication system using WebSocket or HTTP, and a game engine such as Unity or Unreal Engine.

[2047] A device is a user-interface device, which can include a VR headset with a high-resolution display (e.g., Oculus Rift or HTC Vive) and a connected PC or mobile device. The software running on the device includes Unity or Unreal Engine for rendering the VR environment and controller software for processing user input.

[2048] The user wears a VR headset and uses a controller to explore the virtual environment. To collect the user's emotional state, biometric devices such as a camera, microphone, and heart rate sensor are connected to the device.

[2049] System processing overview

[2050] The server generates a 3D model of the virtual classroom and sets the user's initial location. This data is sent from the server to the device using WebSocket or HTTP protocol. The device then renders the received 3D model data on the VR headset using Unity or Unreal Engine, providing the user with a view of the virtual classroom.

[2051] Users can explore the VR classroom using a controller and interact with instructors and other users in real time. During this time, the user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[2052] In a historical and cultural experience, the server generates a specific historical scenario and prepares the associated 3D models and simulation data. This data is sent from the server to the device, which then starts the simulation, providing the user with an immersive historical experience. The user explores the virtual environment and interacts with specific objects and characters. This interaction data is sent from the device to the server, which triggers specific events and synchronously distributes the results to other users' devices.

[2053] When participating in a remote lecture, the server prepares lecture information from a remote expert or instructor and sets access rights for the user. When the lecture starts, the server starts the instructor's video stream and sends it to the terminal. The terminal receives the video stream and provides the user with video and audio of the lecture in real time. When the user asks a question, the question data is sent from the terminal to the server, and the server relays the question to the instructor and sends the instructor's answer to the user's terminal in real time.

[2054] Regarding the incorporation of an emotion engine, the server is equipped with an emotion engine for analyzing the user's emotions. This engine analyzes the user's facial expressions, tone of voice, behavioral patterns, etc. and recognizes them as emotional data. The device collects the user's emotional data in real time and sends it to the server. The server dynamically adjusts the educational content based on the received emotional data. For example, if the user is enjoying themselves, the current scenario will continue, but if the user is dissatisfied or confused, additional explanations or a different scenario will be provided.

[2055] Specific examples

[2056] One day in a history class, students put on VR headsets to learn about "Ancient Egypt." The server generates a 3D model of the virtual classroom and determines each student's initial position. When the data is sent to the device, the device renders the VR environment, and the student explores the ancient Egyptian landscape. They interact with the instructor and other students in the classroom, and their location and interaction data are sent from the device to the server and synchronized to all users' devices.

[2057] If one student shows interest in the pyramid while another struggles with a difficult problem during the scenario, the device sends that emotional data to the server, which then provides additional information to the interested student and additional explanations and hints to the struggling student.

[2058] When a lecture is given by a remote expert, the server sends the expert's video stream to the device. When a student wants to ask a question, the question data is sent from the device to the server, which relays it to the instructor and provides the instructor's answer remotely to the student. In addition, student emotion data is analyzed during the lecture, and if dissatisfaction or a decline in interest is detected, the instructor can take appropriate action according to the situation.

[2059] Prompt Sentence Examples

[2060] "Describe a scenario in which students would use VR to learn about ancient Egypt. Also, detail how an emotion recognition engine would help."

[2061] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2062] Program processing flow

[2063] Step 1:

[2064] The server generates a 3D model of the virtual classroom and the user's initial position information.

[2065] Specifically, the server uses a game engine such as Unity or Unreal Engine to create a 3D model of the virtual classroom and sets the user's initial location information based on that. The input is the virtual classroom design and user ID, and the output is the generated 3D model data and initial location information, saved in JSON format.

[2066] Step 2:

[2067] The server sends the generated 3D model data and initial location information to the terminal.

[2068] The server sends this data to the device using WebSocket or HTTP protocol. The input includes the 3D model data and initial position information generated in step 1, and the output includes the data sent to the device.

[2069] Step 3:

[2070] The device renders the received 3D model data on the VR headset.

[2071] The device processes the received data using Unity or Unreal Engine and renders it in real time on the VR headset. The input includes 3D model data received from the server, and the output includes the virtual classroom view seen by the user.

[2072] Step 4:

[2073] Users explore and interact within the virtual classroom.

[2074] Using a VR headset and controllers, users can move freely around the virtual classroom and interact with other users and objects in real time. The inputs are the user's visual, auditory, and controller inputs, and the output is the interaction data generated.

[2075] Step 5:

[2076] The terminal transmits the user's interaction data to the server.

[2077] The device collects user location information and operation logs in real time and sends them to the server using WebSocket. The input includes user interaction data, and the output includes data sent to the server.

[2078] Step 6:

[2079] The server synchronizes the received interaction data with the terminals of other users.

[2080] The server synchronizes the received data to all other devices in real time. The input includes user interaction data, and the output includes data synchronized to other devices.

[2081] Step 7:

[2082] The server generates the data for a particular historical scenario.

[2083] The server generates 3D models and scenario scripts based on specific historical events and sends them to the terminal. The inputs include historical data and scenario design, and the outputs include the generated 3D models and scenario scripts.

[2084] Step 8:

[2085] The terminal starts the simulation, providing the user with a realistic historical experience.

[2086] The device starts a simulation using Unity or Unreal Engine based on the received scenario data. The input includes the received scenario data, and the output includes the virtual environment provided to the user.

[2087] Step 9:

[2088] The terminal transmits the interaction data to the server.

[2089] The device collects the interaction data of the user in the virtual environment in real time and sends it to the server.The input includes the user's interaction data, and the output includes the data sent to the server.

[2090] Step 10:

[2091] The server triggers a specific event and synchronizes the results to other devices.

[2092] The server triggers a specific event based on the received interaction data and synchronously distributes the result to all other terminals. The input includes the interaction data based on the trigger condition, and the output includes the synchronized event result.

[2093] Step 11:

[2094] The server prepares the lecture information for the remote location and transmits it to the terminal.

[2095] The server manages the contents and schedule of the lectures, sets access rights for participating users, and sends this information to the terminals. The input includes the contents and schedule of the lectures, and the output includes access rights information.

[2096] Step 12:

[2097] The terminal renders the instructor's video stream to the user.

[2098] The server starts the video stream at the beginning of the lecture, and the terminal renders it in real time.,The input includes the video stream data, and the output,includes the lecture video that is displayed to the user.

[2099] Step 13:

[2100] The terminal transmits the user's question data to the server.

[2101] When a user wants to ask a question, the question data is sent from the terminal to the server. The input includes the user's question data, and the output includes the data sent to the server.

[2102] Step 14:

[2103] The server relays questions to the instructor and delivers answers to the user.

[2104] The server relays questions to the instructor and sends the answers to the user's device in real time. The input includes the user's question data and the output includes the instructor's answer data.

[2105] Step 15:

[2106] The terminal transmits the user's emotion data to the server.

[2107] The device collects user emotion data using a camera and microphone and sends it to the server. The input includes the user emotion data, and the output includes the data sent to the server.

[2108] Step 16:

[2109] The server analyzes the received emotional data and dynamically adjusts the educational content.

[2110] The server uses an emotion analysis algorithm to analyze the user's emotional state and dynamically adjusts the educational content based on the results. The input is the user's emotional data, and the output is the adjusted educational content.

[2111] These steps enable the system of the present invention to provide an immersive learning experience for the user and to adapt educational content based on the user's emotional state in real time.

[2112] (Application example 2)

[2113] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2114] In modern educational and commercial environments, users have limited interactive options for enriching their experiences. Current virtual reality technologies lack the means to provide appropriate feedback and information based on the user's emotional state. This leads to a poor user experience and makes it difficult for users to learn or purchase effectively.

[2115] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with a teacher and other users in real time; a means for recreating historical events and cultural experiences using virtual reality technology and allowing the user to experience them; a means for attending lectures and engaging in Q&A sessions with experts and lecturers in remote locations in real time; a means for allowing a user to explore a virtual environment using smart glasses and present specific information or content based on the user's gaze or movements; and a means for recognizing the user's emotional state in real time and providing information or promotions according to that state. This enables an interactive experience according to the user's emotional state, dramatically improving the user experience in education and commerce.

[2116] A "virtual reality headset" is a device worn by a user to display a virtual reality environment, and is attached to the head.

[2117] A "virtual classroom" is a virtual classroom environment recreated using virtual reality technology that users can access through a virtual headset to create an immersive learning experience.

[2118] "Real-time" refers to the state in which data and information are processed and transmitted immediately, without delay, allowing users to interact with other users and systems instantly.

[2119] "Historical events" are important events or incidents that actually occurred in the past, which are recreated using virtual reality technology and can be experienced by users.

[2120] "Cultural experience" refers to the use of virtual reality technology to allow users to experience the cultural background, customs, and practices of a particular region or era.

[2121] The term "remote location" refers to a location away from the user's current location, where lectures and Q&A sessions can be held in real time with experts or lecturers in such locations.

[2122] "Experts and lecturers" are people who have advanced knowledge and skills in a particular field and who impart that knowledge to other users through lectures and education.

[2123] "Smart glasses" are eyeglass-type devices that can visually display information, allowing users to view information and content based on their line of sight and movements.

[2124] A "virtual environment" is a digital space created using virtual reality technology in which a user can have a virtual experience.

[2125] "Gaze" refers to the direction of the user's eyes, and is the direction of the user's gaze detected by a device such as smart glasses.

[2126] "Movement" refers to the movement of a user's hands or body, and is an action that conveys the user's intentions in an interactive system.

[2127] "Specific information and content" refers to data that is of interest to users and is provided through virtual reality or smart glasses, such as product information, promotions, and educational materials.

[2128] "Emotional state" refers to the user's current state of mind, and is recognized in real time by analyzing facial expressions, tone of voice, behavior, etc.

[2129] "Promotions" refer to advertising activities or special offers designed to increase purchasing interest in a particular product or service and are delivered based on a user's emotional state.

[2130] The present invention relates to an education and commerce system that uses smart glasses to provide customers with a rich experience in a virtual store, enabling interactive information provision based on the user's emotional state.

[2131] Using a virtual reality headset

[2132] The server generates a 3D model of the virtual classroom and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the device. The device renders the received 3D model data on a virtual reality headset, providing the user with a view of the virtual classroom. The user can explore the virtual classroom using a controller and interact with the teacher and other users in real time. The user's location information and interaction data are sent from the device to the server, and the server synchronizes it with the devices of other users.

[2133] Using smart glasses in virtual stores

[2134] The server generates a 3D model of the virtual store and sets the user's initial location. The user puts on the smart glasses and freely explores the virtual store. Specific product information and promotions are automatically displayed on the smart glasses display based on the user's gaze and movements.

[2135] Real-time recognition of emotional states

[2136] The device is equipped with a camera and microphone to analyze the user's facial expressions and tone of voice. The server uses this data to recognize the user's emotional state in real time. For example, if the user is confused, a special promotion and support options will be displayed.

[2137] Hardware and software used

[2138] Hardware

[2139] Virtual reality headset: A device worn on the head that displays a virtual classroom or storefront.

[2140] Smart glasses: Eyeglasses-like devices that display information visually.

[2141] Camera and microphone: Input devices for analyzing the user's facial expressions and voice.

[2142] software

[2143] Emotion engine: A program that analyzes user emotions in real time.

[2144] VR interface: A program for generating and manipulating a virtual environment.

[2145] Specific examples

[2146] A customer puts on a virtual reality headset and explores a virtual store. When the customer looks at a particular product, detailed information about that product appears on the display. If the customer shows signs of confusion, the system displays a special promotion in real time and also offers the option to video chat with a staff member. In this way, the customer receives personalized information and support.

[2147] Prompt Sentence Examples

[2148] Create an example of a smart glasses application that uses an emotion engine to help customers explore products in a virtual store, including displaying more information when the customer looks at a product and showing special promotions if the customer's emotion is negative.

[2149] The above is a detailed description of the embodiments of the present invention, which can dramatically improve a user's educational and business transaction experience in a virtual environment.

[2150] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2151] Step 1:

[2152] The server generates a 3D model of the virtual classroom or virtual store and sets the user's initial location information. The generated 3D model and initial location information are sent from the server to the terminal. The input in this step is the design data of the virtual environment, and the output is the 3D model data and initial location information sent to the terminal.

[2153] Step 2:

[2154] The terminal renders the 3D model data received from the server onto a virtual reality headset or smart glasses, providing the user with a virtual classroom or virtual store view. The input in this step is the 3D model data sent from the server, and the output is the virtual environment displayed to the user.

[2155] Step 3:

[2156] The user explores the virtual classroom or virtual store using a controller or gaze. The user's location information and interaction data are sent from the device to the server. The input in this step is the user's movement and gaze information, and the output is the user's location information and interaction data sent to the server.

[2157] Step 4:

[2158] The server synchronizes the received user location information and interaction data with other users' devices, allowing all users to share the latest status. The input in this step is the user's location information and interaction data, and the output is sync...

Claims

1. a means for allowing a user to participate in a virtual classroom using a virtual reality headset and interact with the teacher and other users in real time; A means to recreate historical events and cultural experiences using virtual reality technology and allow users to experience them; A means to attend lectures and ask questions in real time with experts and lecturers in remote locations, A system including:

2. The system of claim 1 , further comprising means for transmitting user location information and interaction data to a server and synchronizing the latest status for all users.

3. The system according to claim 1, further comprising means for transmitting interactions with specific objects or characters in the virtual classroom to a server, triggering an event based on the results, and synchronously distributing the event to other users.

Images