system

Abstract

A system is provided.SOLUTION: A system comprising: means for storing digital narrative; means for identifying a character selected by a user; means for generating a 360 degree VR experience based on a viewpoint of the selected character; means for displaying the generated VR experience to the user; and means for collecting and analyzing user behavior in a AI to optimize upcoming experiences.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional educational toys and tools, users can only experience a story from one perspective, making it difficult to develop a deep understanding or empathy for the characters. Furthermore, conventional VR educational content does not offer experiences from multiple perspectives, making it difficult to provide an optimal experience to maximize the user's learning effect. Furthermore, it is not possible to analyze user behavior data and reflect it in the next experience, making it difficult to provide an individually optimized learning experience. [Means for solving the problem]

[0005] This invention provides a system that includes a means for storing digital data of a story, a means for identifying a character selected by the user, a means for generating 360-degree VR experience data based on the selected character's perspective, a means for displaying the generated VR experience data to the user, and a means for collecting user behavior data and analyzing it with AI to optimize the next experience. This system allows users to gain a deeper understanding and empathy from multiple perspectives of the story. Furthermore, the system includes a means for users to select characters using a digital pen and a means for users to experience the story in 360 degrees from the character's perspective using a VR headset, further enhancing the educational effect. This solves existing problems and makes it possible to provide individually optimized learning experiences.

[0006] "Digital story data" refers to data that electronically records the content of a story, and includes images, text, audio, etc.

[0007] A "user" is someone who uses the system to experience the story.

[0008] A "character to be selected" is a character whose perspective the user switches within the story.

[0009] "Means for identifying" refers to the technical means for detecting a user's selection or operation and transmitting that information to the system.

[0010] "Means for generating viewpoint-based 360-degree VR experience data" refers to technology for creating virtual reality data of a story from the viewpoint of a selected character.

[0011] "VR experience data" is data generated so that the content of a story can be visually experienced from all directions in 360 degrees.

[0012] The "means for displaying the generated VR experience data to the user" refers to a device such as a display or headset for visually providing the generated VR experience data to the user.

[0013] "Behavioral data" is information related to the user's actions during the VR experience, such as user operations and viewpoint movements.

[0014] "Means of analysis using AI" refers to a technique that uses artificial intelligence technology to analyze behavioral data and evaluate the user's interest and level of understanding.

[0015] The "means for optimizing the next experience" is a technology that makes adjustments based on the analysis results to make the next user experience more effective.

[0016] A "digital pen" is an electronic input device that a user uses to select characters in a story.

[0017] A "VR headset" is a display device worn by a user to experience a virtual reality environment. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that allows you to experience a picture book story in a 360-degree VR environment. Specific embodiments of this system will be described below.

[0040] System configuration

[0041] 1. Server

[0042] The server stores the digital data of the story.

[0043] The data is processed to identify the characters selected by the user.

[0044] Generates 360-degree VR experience data based on the perspective of the selected character.

[0045] The generated VR experience data is sent to the device.

[0046] Collect user behavior data and analyze it with AI to optimize the next experience.

[0047] 2. Terminal

[0048] An interface is provided whereby the user can select a character using a digital pen.

[0049] Receives VR experience data sent from the server and displays it to the user.

[0050] The user's behavioral data and viewpoint movements are tracked in real time and sent to the server.

[0051] 3. Users

[0052] Users use the AI ​​pen to select characters for the story.

[0053] Users put on a VR headset and experience the story from the perspective of a selected character.

[0054] Users can freely explore the VR environment and provide behavioral data during the experience.

[0055] Program processing explanation

[0056] Preserving digital data of picture books

[0057] The server receives the digital data of the picture book and stores it in separate files, including images, text, and audio data for each page, making it easy for users to access specific pages or scenes later.

[0058] Identify the characters the user selects

[0059] When a user selects a character in a picture book using the AI ​​pen, the device identifies that information and sends it to the server, which then obtains the viewpoint data of the selected character based on the information received.

[0060] Generate 360-degree VR experience data based on the perspective of a selected character

[0061] The server uses a VR engine to generate VR experience data from the perspective of a selected character, for example, creating 360-degree video and simulation data to observe Momotaro's actions from the perspective of the ogre.

[0062] Display the generated VR experience data to the user

[0063] The terminal receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story in virtual reality by wearing the headset.

[0064] Collect user behavior data and analyze it with AI

[0065] Data on the user's viewpoint movements and interactions during the experience is collected. This behavioral data is periodically sent to a server and analyzed by AI. This allows the next experience to be optimized based on the user's interests and level of understanding.

[0066] Specific examples

[0067] Take the story of Momotaro for example. The user uses the AI ​​pen to select the "demon." The server generates a VR scene of Kagoshima from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to those actions and the demon's feelings. After the experience, data on the user's viewpoint and behavior is collected, and the AI ​​optimizes the next experience.

[0068] This system allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved from a single conventional perspective, providing a groundbreaking educational tool that can enhance empathy and understanding.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The server receives the digital data of the picture book. The data includes images, text, and audio. The server analyzes the data, divides it into pages, and stores them.

[0072] Step 2:

[0073] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[0074] Step 3:

[0075] The device analyzes the received location data to identify which character was selected and sends this information to the server.

[0076] Step 4:

[0077] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[0078] Step 5:

[0079] The generated VR experience data is sent from the server to the device, which then displays the received data on the user's VR headset.

[0080] Step 6:

[0081] Users wear a VR headset and experience the story world from the perspective of a selected character. Users can freely move their viewpoint and interact with the character.

[0082] Step 7:

[0083] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data, which is then sent to a server.

[0084] Step 8:

[0085] The server uses AI to analyze the received user behavior data and, based on the analysis results, sets parameters to optimize the user's next experience.

[0086] Step 9:

[0087] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In modern education and entertainment, experiences that allow users to actively participate are important. However, traditional visual experiences do not allow users to experience the story from the perspective of the characters, making it difficult to enhance users' empathy and understanding. Furthermore, they lack a mechanism for collecting user behavioral data and applying it to the next experience. Therefore, there is a demand for more interactive and personalized experiences.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for storing digital data of the story, means for identifying a character selected by the user, means for generating virtual reality experience data based on the viewpoint of the selected character, means for displaying the generated virtual reality experience data to the user, and means for collecting user behavior data, analyzing it with artificial intelligence, and optimizing the next experience. This allows the user to experience the story from the viewpoint of the character, and the user's behavior and viewpoint are analyzed as data and reflected in the next experience, enabling a more effective and personalized experience.

[0093] "Digital story data" refers to data formats such as images, text, and audio that are used to electronically store and manage the content of a story.

[0094] A "user-selected character" is a character in the story that the user selects through the interface at the start of the experience.

[0095] "Virtual reality experience data" is data that includes 360-degree video and audio generated based on the perspective of a selected character.

[0096] "Generated virtual reality experience data" refers to specific data for a virtual reality experience that is generated on a server and sent to a user's terminal.

[0097] "User behavior data" refers to information about interactions such as viewpoint movement and operations performed by the user during the virtual reality experience.

[0098] "Analysis with artificial intelligence" is the process of using collected user behavior data to analyze it using machine learning models and AI algorithms, and then adjusting the content and difficulty of the next experience.

[0099] "Means to optimize the next experience" is a function that individually adjusts the content and difficulty of the experience based on AI analysis of user behavior data.

[0100] An "input device" is a device such as a digital pen or touch screen that a user uses to select characters.

[0101] A "display device" is a device, such as a VR headset or computer screen, that a user uses to visually enjoy a virtual reality experience.

[0102] This invention relates to a system for experiencing a story in a virtual reality environment. This system allows users to experience a 360-degree virtual reality experience from the perspective of a character in the story, and provides a mechanism for optimizing the next experience by collecting and analyzing user behavior data.

[0103] System configuration

[0104] 1. Server

[0105] How to digitally preserve your stories:

[0106] The server receives the digital data of the storybook and stores it in separate files, including images, text, and audio data for each page, using a specific database management system (e.g., MySQL or PostgreSQL).

[0107] A way to identify the character the user selects:

[0108] The server receives data about the character selected by the user and identifies that character's information. This process uses software (e.g., device drivers and APIs) to interpret data from input devices such as digital pens and touchscreens.

[0109] Means for generating virtual reality experience data:

[0110] A VR engine (e.g., Unity, Unreal Engine) is used to generate 360-degree VR experience data based on the perspective of the selected character, creating an environment in which the user can experience the story from the perspective of the selected character.

[0111] Collecting user behavior data and analyzing it with artificial intelligence:

[0112] Data on the user's viewpoint movements and interactions during the VR experience is collected by the server, analyzed using an artificial intelligence model (e.g., TensorFlow, PyTorch), and the results are used to optimize the next experience.

[0113] 2. Terminal

[0114] A means for displaying the generated virtual reality experience data to the user:

[0115] It receives the VR experience data sent from the server and displays it to the user through the VR headset (e.g., Oculus Rift, HTC Vive). The device converts this data into an appropriate format so that the user can enjoy an immersive experience.

[0116] An interface that identifies the character the user can select:

[0117] It provides an interface for users to select characters using a digital pen or touch screen, and this interface plays a role in transmitting information about the selected characters to the server.

[0118] 3. Users

[0119] Story Experience:

[0120] Users select a character in the story using a digital pen and then wear a VR headset to experience the story in virtual reality. For example, if a user selects the "demon," they can observe Momotaro's actions from the demon's perspective.

[0121] Behavioral data provided:

[0122] The user's viewpoint movements and interactions during the experience are collected in real time and used to optimize the next experience.

[0123] Specific examples

[0124] For example, in the story of Momotaro, the user selects the "demon" using the AI ​​pen. The server generates a VR scene of Okayama from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to his actions and the demon's feelings. After the experience, the user's viewpoint and behavior data are collected, and the AI ​​optimizes the next experience.

[0125] Examples of prompt statements

[0126] "This is the story of Momotaro. The user selects the ogre with the AI ​​pen. The server should generate VR experience data from the ogre's point of view and send it to the device. The device should display this data on the user's VR headset, collect user behavior data, and send it back to the server."

[0127] This invention allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved through a traditional single perspective, making it a groundbreaking educational tool that can enhance empathy and understanding.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Program processing flow

[0130] Step 1:

[0131] The server receives the digital data of picture books uploaded by the administrator. The digital data is divided into image files (JPEG, PNG) for each page, text files (TXT, XML), and audio files (MP3, WAV) and saved. Specifically, it stores the data in a database using a database management system (e.g., MySQL or PostgreSQL). The input is the picture book data uploaded by the administrator, and the output is the individual digital data saved in the database.

[0132] Step 2:

[0133] The user selects a character in a picture book using a digital pen. The device receives position input from the digital pen and identifies the character selected by the user. Specifically, the device reads the pen's position data and executes an identification algorithm to identify the corresponding character. The input is the position data from the digital pen, and the output is the identified character information.

[0134] Step 3:

[0135] The terminal sends character information to the server. The server receives this and obtains the viewpoint data of the selected character. Specifically, the server searches for and obtains the corresponding viewpoint data from a database based on the character information it receives. The input is character information, and the output is viewpoint data.

[0136] Step 4:

[0137] The server uses a VR engine (e.g., Unity, Unreal Engine) to generate VR experience data from the viewpoint of the selected character. Specifically, it runs a VR engine program to generate 360-degree simulation images and scenes in real time based on the viewpoint data. The input is viewpoint data, and the output is the generated VR experience data.

[0138] Step 5:

[0139] The device receives the VR experience data sent from the server and displays it on the user's VR headset (e.g., Oculus Rift, HTC Vive). Specifically, the device decodes the VR experience data and converts it into a format that can be displayed on the headset. The input is the VR experience data from the server, and the output is the image displayed on the VR headset.

[0140] Step 6:

[0141] The user wears a VR headset and experiences the story from the perspective of a selected character. Specific actions include moving freely within the VR environment, changing viewpoints, and interacting with the character. The input is the visual and audio experience data obtained through the VR headset, and the output is the user's experience and behavioral data.

[0142] Step 7:

[0143] The device collects user behavior data (eye movement and operations) in real time and sends it to a server. Specifically, it runs a program that collects sensor data to track user behavior and sends it to the server for analysis. The input is the user behavior data, and the output is the raw data sent to the server.

[0144] Step 8:

[0145] The server inputs the received user behavior data into an AI analysis model (e.g., TensorFlow, PyTorch) for analysis. Specifically, the AI ​​analysis model analyzes the user's behavioral patterns and executes algorithms to evaluate their interests and level of understanding. The input is the user's behavioral data, and the output is the analysis results.

[0146] Step 9:

[0147] The server adjusts the content and difficulty of the next experience based on the analysis results. Specifically, it runs a program to update the VR experience scenario based on the analysis results and prepares to generate new VR experience data. The input is the analysis results, and the output is optimized experience data for the next experience.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Today, there is a demand for interactive content as a way for users to gain a deeper understanding of stories and learn from them. However, conventional methods only allow users to enjoy stories as a visual experience, making it difficult to provide diverse experiences. Furthermore, there is a lack of mechanisms for effectively utilizing user behavior data to optimize the next experience. For this reason, it is necessary to develop an interactive system that allows users to learn from stories from multiple angles.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes: [means for storing digital data of the story;] [means for identifying characters selected by the user;] [means for generating 360-degree VR experience data based on the viewpoint of the selected character;] [means for displaying the generated VR experience data to the user;] [means for collecting user behavior data, analyzing it with AI, and optimizing the next experience; and] [means for the user to freely move their viewpoint within the story and provide interactions with specific objects and characters. This allows the user to experience the story from multiple perspectives and gain deeper learning.

[0153] "Digital story data" refers to data in digital format that includes the text, images, audio, etc. of a story.

[0154] "User-selected character" refers to the character the user wants to experience in the story.

[0155] "360-degree VR experience data" is data that users can experience from a 360-degree perspective in a virtual reality space.

[0156] "Generated VR experience data" refers to data of a 360-degree VR environment generated by the server based on the user's selection.

[0157] "User behavior data" refers to data such as viewpoint movements and interactions that users make during a VR experience.

[0158] "Analyzing with AI and optimizing the next experience" refers to the process of analyzing collected user behavior data using AI technology and adjusting the content of the next experience based on the user's interests and behavior.

[0159] "Users can freely move their viewpoint within the story" refers to a function that allows users to move their viewpoint in any direction within the VR environment.

[0160] "Providing interaction with specific objects and characters" refers to a feature that allows the user to actually interact with specific items and characters within the story.

[0161] A "pen-type device" is a device that a user uses to select characters and items in a story.

[0162] "Display device" means a device that provides the user with visual information for a VR experience, including a VR headset and a smartphone.

[0163] This invention provides an interactive system that allows users to experience stories in 360-degree VR. This system consists of three elements: a server, a terminal, and a user. Each element and its processing will be explained below.

[0164] server

[0165] The server has the following features:

[0166] 1. Digital Data Storage:

[0167] The server stores the digital data of the stories (text, images, audio), using database software such as MySQL.

[0168] 2. Character Identification:

[0169] The system receives information about the character selected by the user through the terminal and identifies them. The software used is Python and Flask.

[0170] 3. Generating 360-degree VR experience data:

[0171] The server generates 360-degree VR experience data from the perspective of the selected character using Unity and Google VR SDK. For example, if the user selects "Oni," it generates a scene from the story of Momotaro from the perspective of the ogre.

[0172] 4. View generated data:

[0173] The generated VR experience data is sent to the device, allowing the user to visually experience the story through a VR headset.

[0174] 5. Behavioral data collection and AI analysis:

[0175] User behavior data (eye movement and interactions) is collected and analyzed using AI. TensorFlow is used for the AI ​​analysis. The next experience content is optimized based on the results of this analysis.

[0176] Terminal

[0177] The terminal has the following features:

[0178] 1. Provide an interface:

[0179] It provides an interface for users to select characters using a pen-like device, using Android Studio and a corresponding UI framework.

[0180] 2. Receiving and displaying VR experience data:

[0181] The VR experience data sent from the server is received and displayed to the user. The device uses a VR headset (e.g., a smartphone + VR headset) to provide a 360-degree experience.

[0182] 3. Transmission of behavioral data:

[0183] The user's behavior data is sent to the server in real time using a network communication module.

[0184] User

[0185] The user uses the system as follows:

[0186] 1. Character Selection:

[0187] Use the pen-shaped device to select a character in the story. For example, you can choose the "demon" in the story of Momotaro.

[0188] 2. VR Experience:

[0189] Put on a VR headset and experience the story in 360 degrees from the perspective of a selected character, with the ability to move freely between viewpoints and interact with specific objects and characters.

[0190] 3. Providing behavioral data:

[0191] Data on viewpoint movements and interactions during the experience is collected, allowing the next experience to be optimized for each individual user.

[0192] Examples and prompts

[0193] To illustrate, consider the following scenario:

[0194] When a user selects the story of Momotaro, they choose the "Oni's perspective." This information is sent to the server.

[0195] Prompt: "The user selected the story of Momotaro."

[0196] The server generates 360-degree VR data from the demon's perspective and sends it to the device.

[0197] Prompt: "The user selected the demon's perspective."

[0198] The user wears a VR headset and observes Momotaro's actions as the demon. Interactions also occur when touching specific objects.

[0199] Prompt: "The user observed Momotaro from the demon's perspective and touched a specific object."

[0200] User behavior data is sent to the server to optimize the experience next time.

[0201] Prompt: "AI identified the scenes that you found most interesting in your previous experience and optimized your next experience."

[0202] This system allows users to enjoy an interactive and multifaceted experience that goes beyond traditional storytelling methods.

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] The user selects a story and characters using a pen-like device.

[0206] Specific operation: The user selects one story from the list displayed on the device (smartphone) interface. Then, they use the pen-shaped device to select the character they want to experience from the characters appearing in the story.

[0207] Input: User-selected story and character information.

[0208] Output: Selected stories and character identities.

[0209] Data Processing: Processes input from the pen-type device and generates an ID for the selected story and character.

[0210] Step 2:

[0211] The terminal transmits information about the selected character to the server.

[0212] Specific operation: The device sends the selected character ID and story ID to the server via the network.

[0213] Input: Story ID and Character ID.

[0214] Output: The request to the server.

[0215] Data processing: Converting the data to be sent into the appropriate format (e.g., JSON or XML).

[0216] Step 3:

[0217] The server generates 360-degree VR experience data based on the viewpoint of the selected character.

[0218] What it does: The server uses Unity and the Google VR SDK to render a story scene from the perspective of a selected character as a 360-degree VR video.

[0219] Input: Story ID and Character ID.

[0220] Output: 360 degree VR data.

[0221] Data calculation: Generates a virtual space seen from the viewpoint of the selected character and renders the scene data.

[0222] Step 4:

[0223] The server sends the generated 360-degree VR data to the device.

[0224] Specific operation: The server transfers the generated VR data to the terminal via the network.

[0225] Input: 360-degree VR data.

[0226] Output: Sending VR data to the device.

[0227] Data processing: Compress large video data and convert it into an efficient streaming format.

[0228] Step 5:

[0229] The 360-degree VR data received by the device is displayed on the VR headset.

[0230] Specific operation: The terminal decompresses the received VR data and displays it on the VR headset (smartphone and VR headset).

[0231] Input: 360-degree VR data.

[0232] Output: The user's VR experience.

[0233] Data Computation: Rendering data into a format suitable for the display of a VR headset.

[0234] Step 6:

[0235] The device records the user's viewpoint movements and interactions during the VR experience.

[0236] Specific actions: The user uses the VR headset to explore the story and interact with specific objects and characters. The device records these actions in real time.

[0237] Input: User gaze movement and interaction data.

[0238] Output: Behavioral data.

[0239] Data collection: Data such as gaze position, gaze duration, and type of interaction are collected as logs.

[0240] Step 7:

[0241] The device transmits the collected behavioral data to a server.

[0242] Specific operation: The device compresses the behavioral data periodically or at the end of the experience and sends it to a server via the Internet.

[0243] Input: Behavioral data.

[0244] Output: Sending behavioral data to the server.

[0245] Data processing: Encoding raw behavioral data into a transmittable format.

[0246] Step 8:

[0247] The server uses AI to analyze behavioral data and optimize the next experience.

[0248] How it works: The server uses AI tools like TensorFlow to analyze the collected behavioral data, identify user interests and behavioral patterns, and use this information to fine-tune the next experience.

[0249] Input: Behavioral data.

[0250] Output: Analysis results and optimized experience data.

[0251] Data Computing: Analyzing behavioral data, identifying user interests and behavioral patterns, and applying optimization algorithms.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention is a system that allows users to experience a picture book story in a 360-degree VR environment and optimizes the next experience by recognizing and analyzing the user's emotions. This system stores the digital data of the story, generates a VR experience based on character selection and their viewpoint, and collects and analyzes the user's behavioral and emotional data.

[0254] System configuration

[0255] 1. Server

[0256] The server stores the digital data of the story, which includes images, text, and audio.

[0257] It identifies the selected character and generates 360-degree VR experience data based on their perspective.

[0258] The generated VR experience data is sent to the terminal, and the user's behavioral and emotional data is collected.

[0259] The collected data is analyzed using AI to optimize the next experience.

[0260] 2. Terminal

[0261] An interface is provided whereby the user can select a character using a digital pen.

[0262] It receives VR experience data sent from the server and provides it to the user via a VR headset for display.

[0263] Emotion data is collected in real time using user behavior data and an emotion engine and sent to a server.

[0264] 3. Users

[0265] The user selects a character in the picture book using a digital pen.

[0266] Put on a VR headset and experience the story from the perspective of a selected character.

[0267] It provides the system with information about viewpoint changes, interactions, and emotional changes during the experience.

[0268] Program processing explanation

[0269] Preserving digital data of picture books

[0270] The server receives the digital data of the picture book and stores it in separate parts: images, text, and audio, making it easier for users to access specific pages or scenes.

[0271] Identify the characters the user selects

[0272] When a user selects a character in a picture book using a digital pen, the device identifies that information and sends it to the server, which then acquires the character's viewpoint data based on the information received.

[0273] Generate 360-degree VR experience data based on the perspective of a selected character

[0274] The server uses a VR engine to generate 360-degree VR experience data from the perspective of a selected character, for example, creating video and simulation data for observing Momotaro's actions from the perspective of the ogre.

[0275] Display the generated VR experience data to the user

[0276] The device receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story world from the character's perspective.

[0277] Collect and analyze user behavioral and emotional data

[0278] The device collects data on the user's viewpoint, interactions, and emotional data collected through the emotion engine during the experience. This data is sent to a server and analyzed by AI. Based on the analysis results, the next experience is optimized based on the user's interests and emotions.

[0279] Specific examples

[0280] Take the story of Momotaro for example. The user selects the "demon" using a digital pen, and the server generates VR data from the demon's perspective and sends it to the device. The user then wears a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes (such as surprise or excitement) in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[0281] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[0282] The processing flow will be explained below.

[0283] Step 1:

[0284] The server receives the digital data of the picture book, which includes images, text, and audio. The server analyzes the data, splits it into pages, and stores them, making it easier for users to access specific pages or scenes.

[0285] Step 2:

[0286] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[0287] Step 3:

[0288] The device analyzes the received location data, identifies which character has been selected, and sends the identification result to the server.

[0289] Step 4:

[0290] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[0291] Step 5:

[0292] The server sends the generated VR experience data to the device, which then forwards the received data to the VR headset.

[0293] Step 6:

[0294] Users wear a VR headset and explore the story world from the perspective of a selected character, with the ability to move the viewpoint and interact with the character.

[0295] Step 7:

[0296] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data. It also uses an emotion engine to analyze the user's facial expressions and voice to collect emotional data. The collected data is periodically sent to a server.

[0297] Step 8:

[0298] The server uses AI to analyze the received user behavioral and emotional data, and based on the analysis results, it makes settings to individually optimize the user's next experience.

[0299] Step 9:

[0300] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[0301] Example 2

[0302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] In today's education and entertainment fields, new methods are needed to engage users and deepen their understanding. In particular, there is a lack of technology to individually optimize story experiences to achieve deep empathy and learning effects. Furthermore, there has been no system that can collect user emotional and behavioral data in real time and use it to optimize the next experience.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, and means for collecting user behavioral data and emotional data, analyzing them using AI, and optimizing the next experience. This allows the user to deeply understand the story and empathize with it through an individually optimized VR experience. Furthermore, real-time emotional data analysis allows the next experience to be optimized based on the user's interests and emotions, thereby providing more effective education and entertainment.

[0305] "Storing digital story data" means receiving picture books and other story content in digital format and storing it in separate images, text, and audio data.

[0306] "Identifying a character selected by a user" means recognizing and accurately identifying the selected information when a user selects a character in a picture book using a digital pen or other interface.

[0307] "Generating 360-degree VR experience data based on the viewpoint of a selected character" means generating data that allows the user to experience a virtual reality environment from the viewpoint of a character selected by the user in a 360-degree view.

[0308] "Displaying the generated VR experience data to the user" means sending the VR experience data generated by the server to the terminal and presenting it to the user through a display device such as a VR headset.

[0309] "Collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience" means collecting the user's viewpoint movements, interactions, and emotional changes during the VR experience in real time, analyzing this data using AI, and using the results to improve the next VR experience based on the user's interests and emotions.

[0310] "Means for selecting a character using a digital pen" refers to a digital input device that a user uses to select a character on a picture book, and to transmit that selection information to the system.

[0311] A "VR headset that allows you to experience stories in 360 degrees" is a head-mounted display device that allows users to experience a virtual reality environment in all directions.

[0312] MODE FOR CARRYING OUT THE INVENTION

[0313] System Configuration

[0314] This invention is a system that stores digital data of a story, generates a 360-degree VR experience based on the viewpoint of a character selected by the user, and optimizes the next experience by analyzing the user's emotions. Below is a detailed explanation of how to implement this system.

[0315] Hardware and software used

[0316] 1. Server

[0317] Servers use high performance data storage and computation hardware (e.g., data center servers).

[0318] The software used includes database management systems (e.g., MySQL, MongoDB) for managing digital data, VR engines (e.g., Unity, Unreal Engine) for generating VR experiences, and AI models (e.g., TensorFlow, PyTorch) for analyzing emotional data.

[0319] 2. Terminal

[0320] A terminal is a device for user interaction, typically a computer with graphics processing capabilities (e.g., a gaming PC), a digital pen, or a VR headset (e.g., Oculus Rift or HTC Vive).

[0321] The software used includes applications to provide a user interface (e.g., custom VR apps, interactive UI software), and emotion engines (e.g., Affectiva, Realeyes) to collect emotion data in real time.

[0322] 3. Users

[0323] Users use a digital pen to select characters from the picture book and then put on a VR headset to immerse themselves in the story.

[0324] The user moves their viewpoint and interacts with the system, providing emotional data to the system.

[0325] Natural language explanation of program processing

[0326] The server receives the digital data (images, text, and audio) from the picture book, divides it, and stores it in a database. When the user selects a character in the picture book using the digital pen, the device captures the selection information and sends it to the server. The server uses a VR engine to generate 360-degree VR experience data based on the viewpoint of the selected character and sends this data to the device. The device displays this data to the user via the VR headset. Behavioral and emotional data from the user's experience is collected in real time through the emotion engine and sent to the server. The server uses an AI model to analyze this data and optimize the next experience based on the user's interests and emotions.

[0327] Examples of concrete examples and prompts

[0328] As a concrete example, when using the story of Momotaro, if the user selects "Oni" with the digital pen, the server generates VR experience data from the Oni's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the Oni's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by an AI model to further optimize the next experience.

[0329] Example prompt sentence:

[0330] "Create a VR experience from the perspective of the ogre in the story of Momotaro, and generate a program that will analyze the user's emotional data and optimize the next experience."

[0331] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1:

[0334] The server receives the digital data of the story (images, text, audio). The input is the digital data of the story, and the output is the stored data. The server splits the received data into individual files and stores them in the appropriate directories using a database management system (e.g., MySQL, MongoDB). This allows users to efficiently access specific pages or scenes.

[0335] Step 2:

[0336] The user selects a character in a picture book using a digital pen. The input is the information of the character selected by the user, and the output is the selected character information. The device captures the digital pen input and sends the information to the server in JSON format.

[0337] Step 3:

[0338] The server receives information about the character selected by the user. The input is the selected character information, and the output is the character's viewpoint data. Based on the received information, the server obtains the character's viewpoint data and generates 360-degree VR experience data using a VR engine (e.g., Unity, Unreal Engine). Specifically, it generates simulation data and exports it as a 360-degree video.

[0339] Step 4:

[0340] The server sends the generated VR experience data to the device. The input is 360-degree VR experience data, and the output is data sent to the device. The device streams the received VR experience data to a VR headset (e.g., Oculus Rift, HTC Vive) to provide the experience to the user. The user can realistically experience the world of the story from the character's perspective.

[0341] Step 5:

[0342] The device collects in real time the user's viewpoint movements and interactions during the experience, as well as emotional data collected using an emotion engine (e.g., Affectiva, Realeyes). The input is the user's behavioral data and emotional data, and the output is the collected data. These data are sent to the server.

[0343] Step 6:

[0344] The server receives the user's behavioral and emotional data sent from the device. The input is the collected data, and the output is the analysis results. The server analyzes the data using an AI model (e.g., TensorFlow, PyTorch) and stores the results in a database. Based on the analysis results, the next VR experience is optimized based on the user's interests and emotions.

[0345] (Application example 2)

[0346] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] Traditional picture book and story experiences are static and one-way, making it difficult for users to immerse themselves in the story and receive an individually optimized experience. Furthermore, there is a lack of means to personalize the next experience based on emotional and behavioral data, making it difficult to improve the user's learning effect and entertainment value.

[0348] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, means for collecting behavioral data and emotional data of the user, analyzing the data using AI, and optimizing the next experience, and means for individually optimizing the next VR experience based on the user's emotional data. This allows the user to feel highly immersed in the world of the story while enjoying a personalized experience based on the emotional and behavioral data.

[0349] "Digital narrative data" refers to data that stores the visual and auditory content (images, text, and audio) of a narrative in digital form.

[0350] A "user-selected character" is a character within a story from which the user can choose a perspective within the experience.

[0351] "360-degree VR experience data" is virtual reality data that allows you to visually experience a story from any direction from the perspective of a selected character.

[0352] "Displaying to the user" means providing the generated VR experience data visually and audibly through a device (such as a headset) used by the user.

[0353] "Behavioral data" refers to data about the user's movements during a VR experience, such as their viewpoint, interactions, and gestures.

[0354] "Emotional data" refers to data about the user's emotional state obtained from facial expressions, tone of voice, heart rate, etc. during the VR experience.

[0355] "Analysis with AI" means analyzing and interpreting collected behavioral and emotional data using artificial intelligence technology.

[0356] "Optimizing the next experience" means adjusting the next VR experience to be more appropriate and interesting for the user based on the user's previous behavioral and emotional data.

[0357] A "digital input device" is a device (e.g., a stylus, touchpad) that a user uses to select characters within a story.

[0358] A "viewing device" is a device (e.g., a VR headset, smart glasses) used by a user to view VR experience data.

[0359] MODE FOR CARRYING OUT THE INVENTION

[0360] This invention is a system that allows users to immerse themselves in a story world and provide an individually optimized experience. This system includes storing digital data of the story, generating a 360-degree VR experience based on character selection, collecting and analyzing user behavioral and emotional data, and optimizing the next experience.

[0361] System configuration

[0362] server

[0363] Preserving the story digitally

[0364] The server stores the visual and auditory content of the story (images, text, audio) in digital form, making it easy to access specific pages or scenes.

[0365] VR experience generation based on character selection

[0366] The system identifies the character selected by the user and generates 360-degree VR experience data based on that character's perspective, allowing the user to experience the story from the perspective of the selected character.

[0367] Collecting and analyzing behavioral and emotional data

[0368] The server collects user behavioral and emotional data and analyzes it using AI technology, optimizing the content of the next experience based on the user's interests and emotions.

[0369] Terminal

[0370] Character selection interface

[0371] The user selects a character in the story using a digital input device (e.g., a touch pen or a touchpad), and the terminal identifies this selection and transmits it to the server.

[0372] Displaying VR experience data

[0373] The device receives the VR experience data sent from the server and displays it to the user using a viewing device such as a VR headset or smart glasses.

[0374] Collecting behavioral and emotional data

[0375] The device collects behavioral data such as the user's gaze movement, interactions, and facial expressions. It also uses an emotion engine to collect the user's emotional data in real time and transmits it to the server.

[0376] User

[0377] Character Selection

[0378] The user selects a character in the picture book using a digital input device.

[0379] VR experience

[0380] Users put on a VR headset and experience the story from the perspective of a character of their choice.

[0381] Specific examples

[0382] Take the story of Momotaro for example. The user selects the "demon" using a digital input device, and the server generates VR data from the demon's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[0383] Prompt Sentence Examples

[0384] You experience a story in a VR environment. First, choose a character. Then, enjoy the story from the perspective of the character you selected. Based on emotional and behavioral data from the experience, your next experience will be optimized to make it even more enjoyable.

[0385] If you experience any of the following emotions during the experience, you will enjoy the next optimized scene more.

[0386] surprise

[0387] joy

[0388] sorrow

[0389] anger

[0390] This system improves the user's immersion and learning effectiveness by individually optimizing the next VR experience based on the user's emotional and behavioral data.

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] The server stores the digital data of the story. This data includes images, text, and audio. The input is the digital file of the story, and the output is the stored data.

[0394] Step 2:

[0395] The terminal identifies the character selected by the user using a digital input device, with the input being the user's selection action and the output being the identified character information.

[0396] Step 3:

[0397] The server generates 360-degree VR experience data from the viewpoint of the selected character based on the identified character information. The input is character information, and the output is 360-degree VR data. For data processing, a VR engine is used to generate visual and audio data.

[0398] Step 4:

[0399] The device receives the VR experience data sent from the server and displays it to the user. The input is VR data, and the output is the display to the user through the VR headset. Specifically, the device uses VR display software.

[0400] Step 5:

[0401] The device collects user behavior data (eye movement and interactions). The input is the user's actions, and the output is the collected behavior data. The device uses the built-in gyro sensor and accelerometer to process the data.

[0402] Step 6:

[0403] It uses an emotion engine to collect user emotion data in real time. The input is the user's facial expressions and voice, and the output is the collected emotion data. Specific operations include analyzing data from the camera and microphone.

[0404] Step 7:

[0405] The terminal sends the collected behavioral data and emotional data to the server. The input is behavioral data and emotional data, and the output is data sent to the server.

[0406] Step 8:

[0407] The server analyzes the collected data using AI. The input is behavioral data and emotional data, and the output is the analysis results. Data is analyzed using a generative AI model.

[0408] Step 9:

[0409] The server optimizes the next VR experience based on the analysis results. The input is the analysis results, and the output is the optimized VR experience data. Specifically, it uses an algorithm to personalize the content of the next experience.

[0410] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0412] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0413] [Second embodiment]

[0414] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0415] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0417] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0421] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0422] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0423] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0424] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0425] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0426] This invention is a system that allows you to experience a picture book story in a 360-degree VR environment. Specific embodiments of this system will be described below.

[0427] System configuration

[0428] 1. Server

[0429] The server stores the digital data of the story.

[0430] The data is processed to identify the characters selected by the user.

[0431] Generates 360-degree VR experience data based on the perspective of the selected character.

[0432] The generated VR experience data is sent to the device.

[0433] Collect user behavior data and analyze it with AI to optimize the next experience.

[0434] 2. Terminal

[0435] An interface is provided whereby the user can select a character using a digital pen.

[0436] Receives VR experience data sent from the server and displays it to the user.

[0437] The user's behavioral data and viewpoint movements are tracked in real time and sent to the server.

[0438] 3. Users

[0439] Users use the AI ​​pen to select characters for the story.

[0440] Users put on a VR headset and experience the story from the perspective of a selected character.

[0441] Users can freely explore the VR environment and provide behavioral data during the experience.

[0442] Program processing explanation

[0443] Preserving digital data of picture books

[0444] The server receives the digital data of the picture book and stores it in separate files, including images, text, and audio data for each page, making it easy for users to access specific pages or scenes later.

[0445] Identify the characters the user selects

[0446] When a user selects a character in a picture book using the AI ​​pen, the device identifies that information and sends it to the server, which then obtains the viewpoint data of the selected character based on the information received.

[0447] Generate 360-degree VR experience data based on the perspective of a selected character

[0448] The server uses a VR engine to generate VR experience data from the perspective of a selected character, for example, creating 360-degree video and simulation data to observe Momotaro's actions from the perspective of the ogre.

[0449] Display the generated VR experience data to the user

[0450] The terminal receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story in virtual reality by wearing the headset.

[0451] Collect user behavior data and analyze it with AI

[0452] Data on the user's viewpoint movements and interactions during the experience is collected. This behavioral data is periodically sent to a server and analyzed by AI. This allows the next experience to be optimized based on the user's interests and level of understanding.

[0453] Specific examples

[0454] Take the story of Momotaro for example. The user uses the AI ​​pen to select the "demon." The server generates a VR scene of Kagoshima from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to those actions and the demon's feelings. After the experience, data on the user's viewpoint and behavior is collected, and the AI ​​optimizes the next experience.

[0455] This system allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved from a single conventional perspective, providing a groundbreaking educational tool that can enhance empathy and understanding.

[0456] The processing flow will be explained below.

[0457] Step 1:

[0458] The server receives the digital data of the picture book. The data includes images, text, and audio. The server analyzes the data, divides it into pages, and stores them.

[0459] Step 2:

[0460] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[0461] Step 3:

[0462] The device analyzes the received location data to identify which character was selected and sends this information to the server.

[0463] Step 4:

[0464] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[0465] Step 5:

[0466] The generated VR experience data is sent from the server to the device, which then displays the received data on the user's VR headset.

[0467] Step 6:

[0468] Users wear a VR headset and experience the story world from the perspective of a selected character. Users can freely move their viewpoint and interact with the character.

[0469] Step 7:

[0470] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data, which is then sent to a server.

[0471] Step 8:

[0472] The server uses AI to analyze the received user behavior data and, based on the analysis results, sets parameters to optimize the user's next experience.

[0473] Step 9:

[0474] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[0475] Example 1

[0476] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0477] In modern education and entertainment, experiences that allow users to actively participate are important. However, traditional visual experiences do not allow users to experience the story from the perspective of the characters, making it difficult to enhance users' empathy and understanding. Furthermore, they lack a mechanism for collecting user behavioral data and applying it to the next experience. Therefore, there is a demand for more interactive and personalized experiences.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0479] In this invention, the server includes means for storing digital data of the story, means for identifying a character selected by the user, means for generating virtual reality experience data based on the viewpoint of the selected character, means for displaying the generated virtual reality experience data to the user, and means for collecting user behavior data, analyzing it with artificial intelligence, and optimizing the next experience. This allows the user to experience the story from the viewpoint of the character, and the user's behavior and viewpoint are analyzed as data and reflected in the next experience, enabling a more effective and personalized experience.

[0480] "Digital story data" refers to data formats such as images, text, and audio that are used to electronically store and manage the content of a story.

[0481] A "user-selected character" is a character in the story that the user selects through the interface at the start of the experience.

[0482] "Virtual reality experience data" is data that includes 360-degree video and audio generated based on the perspective of a selected character.

[0483] "Generated virtual reality experience data" refers to specific data for a virtual reality experience that is generated on a server and sent to a user's terminal.

[0484] "User behavior data" refers to information about interactions such as viewpoint movement and operations performed by the user during the virtual reality experience.

[0485] "Analysis with artificial intelligence" is the process of using collected user behavior data to analyze it using machine learning models and AI algorithms, and then adjusting the content and difficulty of the next experience.

[0486] "Means to optimize the next experience" is a function that individually adjusts the content and difficulty of the experience based on AI analysis of user behavior data.

[0487] An "input device" is a device such as a digital pen or touch screen that a user uses to select characters.

[0488] A "display device" is a device, such as a VR headset or computer screen, that a user uses to visually enjoy a virtual reality experience.

[0489] This invention relates to a system for experiencing a story in a virtual reality environment. This system allows users to experience a 360-degree virtual reality experience from the perspective of a character in the story, and provides a mechanism for optimizing the next experience by collecting and analyzing user behavior data.

[0490] System configuration

[0491] 1. Server

[0492] How to digitally preserve your stories:

[0493] The server receives the digital data of the storybook and stores it in separate files, including images, text, and audio data for each page, using a specific database management system (e.g., MySQL or PostgreSQL).

[0494] A way to identify the character the user selects:

[0495] The server receives data about the character selected by the user and identifies that character's information. This process uses software (e.g., device drivers and APIs) to interpret data from input devices such as digital pens and touchscreens.

[0496] Means for generating virtual reality experience data:

[0497] A VR engine (e.g., Unity, Unreal Engine) is used to generate 360-degree VR experience data based on the perspective of the selected character, creating an environment in which the user can experience the story from the perspective of the selected character.

[0498] Collecting user behavior data and analyzing it with artificial intelligence:

[0499] Data on the user's viewpoint movements and interactions during the VR experience is collected by the server, analyzed using an artificial intelligence model (e.g., TensorFlow, PyTorch), and the results are used to optimize the next experience.

[0500] 2. Terminal

[0501] A means for displaying the generated virtual reality experience data to the user:

[0502] It receives the VR experience data sent from the server and displays it to the user through the VR headset (e.g., Oculus Rift, HTC Vive). The device converts this data into an appropriate format so that the user can enjoy an immersive experience.

[0503] An interface that identifies the character the user can select:

[0504] It provides an interface for users to select characters using a digital pen or touch screen, and this interface plays a role in transmitting information about the selected characters to the server.

[0505] 3. Users

[0506] Story Experience:

[0507] Users select a character in the story using a digital pen and then wear a VR headset to experience the story in virtual reality. For example, if a user selects the "demon," they can observe Momotaro's actions from the demon's perspective.

[0508] Behavioral data provided:

[0509] The user's viewpoint movements and interactions during the experience are collected in real time and used to optimize the next experience.

[0510] Specific examples

[0511] For example, in the story of Momotaro, the user selects the "demon" using the AI ​​pen. The server generates a VR scene of Okayama from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to his actions and the demon's feelings. After the experience, the user's viewpoint and behavior data are collected, and the AI ​​optimizes the next experience.

[0512] Examples of prompt statements

[0513] "This is the story of Momotaro. The user selects the ogre with the AI ​​pen. The server should generate VR experience data from the ogre's point of view and send it to the device. The device should display this data on the user's VR headset, collect user behavior data, and send it back to the server."

[0514] This invention allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved through a traditional single perspective, making it a groundbreaking educational tool that can enhance empathy and understanding.

[0515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0516] Program processing flow

[0517] Step 1:

[0518] The server receives the digital data of picture books uploaded by the administrator. The digital data is divided into image files (JPEG, PNG) for each page, text files (TXT, XML), and audio files (MP3, WAV) and saved. Specifically, it stores the data in a database using a database management system (e.g., MySQL or PostgreSQL). The input is the picture book data uploaded by the administrator, and the output is the individual digital data saved in the database.

[0519] Step 2:

[0520] The user selects a character in a picture book using a digital pen. The device receives position input from the digital pen and identifies the character selected by the user. Specifically, the device reads the pen's position data and executes an identification algorithm to identify the corresponding character. The input is the position data from the digital pen, and the output is the identified character information.

[0521] Step 3:

[0522] The terminal sends character information to the server. The server receives this and obtains the viewpoint data of the selected character. Specifically, the server searches for and obtains the corresponding viewpoint data from a database based on the character information it receives. The input is character information, and the output is viewpoint data.

[0523] Step 4:

[0524] The server uses a VR engine (e.g., Unity, Unreal Engine) to generate VR experience data from the viewpoint of the selected character. Specifically, it runs a VR engine program to generate 360-degree simulation images and scenes in real time based on the viewpoint data. The input is viewpoint data, and the output is the generated VR experience data.

[0525] Step 5:

[0526] The device receives the VR experience data sent from the server and displays it on the user's VR headset (e.g., Oculus Rift, HTC Vive). Specifically, the device decodes the VR experience data and converts it into a format that can be displayed on the headset. The input is the VR experience data from the server, and the output is the image displayed on the VR headset.

[0527] Step 6:

[0528] The user wears a VR headset and experiences the story from the perspective of a selected character. Specific actions include moving freely within the VR environment, changing viewpoints, and interacting with the character. The input is the visual and audio experience data obtained through the VR headset, and the output is the user's experience and behavioral data.

[0529] Step 7:

[0530] The device collects user behavior data (eye movement and operations) in real time and sends it to a server. Specifically, it runs a program that collects sensor data to track user behavior and sends it to the server for analysis. The input is the user behavior data, and the output is the raw data sent to the server.

[0531] Step 8:

[0532] The server inputs the received user behavior data into an AI analysis model (e.g., TensorFlow, PyTorch) for analysis. Specifically, the AI ​​analysis model analyzes the user's behavioral patterns and executes algorithms to evaluate their interests and level of understanding. The input is the user's behavioral data, and the output is the analysis results.

[0533] Step 9:

[0534] The server adjusts the content and difficulty of the next experience based on the analysis results. Specifically, it runs a program to update the VR experience scenario based on the analysis results and prepares to generate new VR experience data. The input is the analysis results, and the output is optimized experience data for the next experience.

[0535] (Application example 1)

[0536] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0537] Today, there is a demand for interactive content as a way for users to gain a deeper understanding of stories and learn from them. However, conventional methods only allow users to enjoy stories as a visual experience, making it difficult to provide diverse experiences. Furthermore, there is a lack of mechanisms for effectively utilizing user behavior data to optimize the next experience. For this reason, it is necessary to develop an interactive system that allows users to learn from stories from multiple angles.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0539] In this invention, the server includes: [means for storing digital data of the story;] [means for identifying characters selected by the user;] [means for generating 360-degree VR experience data based on the viewpoint of the selected character;] [means for displaying the generated VR experience data to the user;] [means for collecting user behavior data, analyzing it with AI, and optimizing the next experience; and] [means for the user to freely move their viewpoint within the story and provide interactions with specific objects and characters. This allows the user to experience the story from multiple perspectives and gain deeper learning.

[0540] "Digital story data" refers to data in digital format that includes the text, images, audio, etc. of a story.

[0541] "User-selected character" refers to the character the user wants to experience in the story.

[0542] "360-degree VR experience data" is data that users can experience from a 360-degree perspective in a virtual reality space.

[0543] "Generated VR experience data" refers to data of a 360-degree VR environment generated by the server based on the user's selection.

[0544] "User behavior data" refers to data such as viewpoint movements and interactions that users make during a VR experience.

[0545] "Analyzing with AI and optimizing the next experience" refers to the process of analyzing collected user behavior data using AI technology and adjusting the content of the next experience based on the user's interests and behavior.

[0546] "Users can freely move their viewpoint within the story" refers to a function that allows users to move their viewpoint in any direction within the VR environment.

[0547] "Providing interaction with specific objects and characters" refers to a feature that allows the user to actually interact with specific items and characters within the story.

[0548] A "pen-type device" is a device that a user uses to select characters and items in a story.

[0549] "Display device" means a device that provides the user with visual information for a VR experience, including a VR headset and a smartphone.

[0550] This invention provides an interactive system that allows users to experience stories in 360-degree VR. This system consists of three elements: a server, a terminal, and a user. Each element and its processing will be explained below.

[0551] server

[0552] The server has the following features:

[0553] 1. Digital Data Storage:

[0554] The server stores the digital data of the stories (text, images, audio), using database software such as MySQL.

[0555] 2. Character Identification:

[0556] The system receives information about the character selected by the user through the terminal and identifies them. The software used is Python and Flask.

[0557] 3. Generating 360-degree VR experience data:

[0558] The server generates 360-degree VR experience data from the perspective of the selected character using Unity and Google VR SDK. For example, if the user selects "Oni," it generates a scene from the story of Momotaro from the perspective of the ogre.

[0559] 4. View generated data:

[0560] The generated VR experience data is sent to the device, allowing the user to visually experience the story through a VR headset.

[0561] 5. Behavioral data collection and AI analysis:

[0562] User behavior data (eye movement and interactions) is collected and analyzed using AI. TensorFlow is used for the AI ​​analysis. The next experience content is optimized based on the results of this analysis.

[0563] Terminal

[0564] The terminal has the following features:

[0565] 1. Provide an interface:

[0566] It provides an interface for users to select characters using a pen-like device, using Android Studio and a corresponding UI framework.

[0567] 2. Receiving and displaying VR experience data:

[0568] The VR experience data sent from the server is received and displayed to the user. The device uses a VR headset (e.g., a smartphone + VR headset) to provide a 360-degree experience.

[0569] 3. Transmission of behavioral data:

[0570] The user's behavior data is sent to the server in real time using a network communication module.

[0571] User

[0572] The user uses the system as follows:

[0573] 1. Character Selection:

[0574] Use the pen-shaped device to select a character in the story. For example, you can choose the "demon" in the story of Momotaro.

[0575] 2. VR Experience:

[0576] Put on a VR headset and experience the story in 360 degrees from the perspective of a selected character, with the ability to move freely between viewpoints and interact with specific objects and characters.

[0577] 3. Providing behavioral data:

[0578] Data on viewpoint movements and interactions during the experience is collected, allowing the next experience to be optimized for each individual user.

[0579] Examples and prompts

[0580] To illustrate, consider the following scenario:

[0581] When a user selects the story of Momotaro, they choose the "Oni's perspective." This information is sent to the server.

[0582] Prompt: "The user selected the story of Momotaro."

[0583] The server generates 360-degree VR data from the demon's perspective and sends it to the device.

[0584] Prompt: "The user selected the demon's perspective."

[0585] The user wears a VR headset and observes Momotaro's actions as the demon. Interactions also occur when touching specific objects.

[0586] Prompt: "The user observed Momotaro from the demon's perspective and touched a specific object."

[0587] User behavior data is sent to the server to optimize the experience next time.

[0588] Prompt: "AI identified the scenes that you found most interesting in your previous experience and optimized your next experience."

[0589] This system allows users to enjoy an interactive and multifaceted experience that goes beyond traditional storytelling methods.

[0590] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0591] Step 1:

[0592] The user selects a story and characters using a pen-like device.

[0593] Specific operation: The user selects one story from the list displayed on the device (smartphone) interface. Then, they use the pen-shaped device to select the character they want to experience from the characters appearing in the story.

[0594] Input: User-selected story and character information.

[0595] Output: Selected stories and character identities.

[0596] Data Processing: Processes input from the pen-type device and generates an ID for the selected story and character.

[0597] Step 2:

[0598] The terminal transmits information about the selected character to the server.

[0599] Specific operation: The device sends the selected character ID and story ID to the server via the network.

[0600] Input: Story ID and Character ID.

[0601] Output: The request to the server.

[0602] Data processing: Converting the data to be sent into the appropriate format (e.g., JSON or XML).

[0603] Step 3:

[0604] The server generates 360-degree VR experience data based on the viewpoint of the selected character.

[0605] What it does: The server uses Unity and the Google VR SDK to render a story scene from the perspective of a selected character as a 360-degree VR video.

[0606] Input: Story ID and Character ID.

[0607] Output: 360 degree VR data.

[0608] Data calculation: Generates a virtual space seen from the viewpoint of the selected character and renders the scene data.

[0609] Step 4:

[0610] The server sends the generated 360-degree VR data to the device.

[0611] Specific operation: The server transfers the generated VR data to the terminal via the network.

[0612] Input: 360-degree VR data.

[0613] Output: Sending VR data to the device.

[0614] Data processing: Compress large video data and convert it into an efficient streaming format.

[0615] Step 5:

[0616] The 360-degree VR data received by the device is displayed on the VR headset.

[0617] Specific operation: The terminal decompresses the received VR data and displays it on the VR headset (smartphone and VR headset).

[0618] Input: 360-degree VR data.

[0619] Output: The user's VR experience.

[0620] Data Computation: Rendering data into a format suitable for the display of a VR headset.

[0621] Step 6:

[0622] The device records the user's viewpoint movements and interactions during the VR experience.

[0623] Specific actions: The user uses the VR headset to explore the story and interact with specific objects and characters. The device records these actions in real time.

[0624] Input: User gaze movement and interaction data.

[0625] Output: Behavioral data.

[0626] Data collection: Data such as gaze position, gaze duration, and type of interaction are collected as logs.

[0627] Step 7:

[0628] The device transmits the collected behavioral data to a server.

[0629] Specific operation: The device compresses the behavioral data periodically or at the end of the experience and sends it to a server via the Internet.

[0630] Input: Behavioral data.

[0631] Output: Sending behavioral data to the server.

[0632] Data processing: Encoding raw behavioral data into a transmittable format.

[0633] Step 8:

[0634] The server uses AI to analyze behavioral data and optimize the next experience.

[0635] How it works: The server uses AI tools like TensorFlow to analyze the collected behavioral data, identify user interests and behavioral patterns, and use this information to fine-tune the next experience.

[0636] Input: Behavioral data.

[0637] Output: Analysis results and optimized experience data.

[0638] Data Computing: Analyzing behavioral data, identifying user interests and behavioral patterns, and applying optimization algorithms.

[0639] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0640] This invention is a system that allows users to experience a picture book story in a 360-degree VR environment and optimizes the next experience by recognizing and analyzing the user's emotions. This system stores the digital data of the story, generates a VR experience based on character selection and their viewpoint, and collects and analyzes the user's behavioral and emotional data.

[0641] System configuration

[0642] 1. Server

[0643] The server stores the digital data of the story, which includes images, text, and audio.

[0644] It identifies the selected character and generates 360-degree VR experience data based on their perspective.

[0645] The generated VR experience data is sent to the terminal, and the user's behavioral and emotional data is collected.

[0646] The collected data is analyzed using AI to optimize the next experience.

[0647] 2. Terminal

[0648] An interface is provided whereby the user can select a character using a digital pen.

[0649] It receives VR experience data sent from the server and provides it to the user via a VR headset for display.

[0650] Emotion data is collected in real time using user behavior data and an emotion engine and sent to a server.

[0651] 3. Users

[0652] The user selects a character in the picture book using a digital pen.

[0653] Put on a VR headset and experience the story from the perspective of a selected character.

[0654] It provides the system with information about viewpoint changes, interactions, and emotional changes during the experience.

[0655] Program processing explanation

[0656] Preserving digital data of picture books

[0657] The server receives the digital data of the picture book and stores it in separate parts: images, text, and audio, making it easier for users to access specific pages or scenes.

[0658] Identify the characters the user selects

[0659] When a user selects a character in a picture book using a digital pen, the device identifies that information and sends it to the server, which then acquires the character's viewpoint data based on the information received.

[0660] Generate 360-degree VR experience data based on the perspective of a selected character

[0661] The server uses a VR engine to generate 360-degree VR experience data from the perspective of a selected character, for example, creating video and simulation data for observing Momotaro's actions from the perspective of the ogre.

[0662] Display the generated VR experience data to the user

[0663] The device receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story world from the character's perspective.

[0664] Collect and analyze user behavioral and emotional data

[0665] The device collects data on the user's viewpoint, interactions, and emotional data collected through the emotion engine during the experience. This data is sent to a server and analyzed by AI. Based on the analysis results, the next experience is optimized based on the user's interests and emotions.

[0666] Specific examples

[0667] Take the story of Momotaro for example. The user selects the "demon" using a digital pen, and the server generates VR data from the demon's perspective and sends it to the device. The user then wears a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes (such as surprise or excitement) in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[0668] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[0669] The processing flow will be explained below.

[0670] Step 1:

[0671] The server receives the digital data of the picture book, which includes images, text, and audio. The server analyzes the data, splits it into pages, and stores them, making it easier for users to access specific pages or scenes.

[0672] Step 2:

[0673] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[0674] Step 3:

[0675] The device analyzes the received location data, identifies which character has been selected, and sends the identification result to the server.

[0676] Step 4:

[0677] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[0678] Step 5:

[0679] The server sends the generated VR experience data to the device, which then forwards the received data to the VR headset.

[0680] Step 6:

[0681] Users wear a VR headset and explore the story world from the perspective of a selected character, with the ability to move the viewpoint and interact with the character.

[0682] Step 7:

[0683] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data. It also uses an emotion engine to analyze the user's facial expressions and voice to collect emotional data. The collected data is periodically sent to a server.

[0684] Step 8:

[0685] The server uses AI to analyze the received user behavioral and emotional data, and based on the analysis results, it makes settings to individually optimize the user's next experience.

[0686] Step 9:

[0687] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[0688] Example 2

[0689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In today's education and entertainment fields, new methods are needed to engage users and deepen their understanding. In particular, there is a lack of technology to individually optimize story experiences to achieve deep empathy and learning effects. Furthermore, there has been no system that can collect user emotional and behavioral data in real time and use it to optimize the next experience.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, and means for collecting user behavioral data and emotional data, analyzing them using AI, and optimizing the next experience. This allows the user to deeply understand the story and empathize with it through an individually optimized VR experience. Furthermore, real-time emotional data analysis allows the next experience to be optimized based on the user's interests and emotions, thereby providing more effective education and entertainment.

[0692] "Storing digital story data" means receiving picture books and other story content in digital format and storing it in separate images, text, and audio data.

[0693] "Identifying a character selected by a user" means recognizing and accurately identifying the selected information when a user selects a character in a picture book using a digital pen or other interface.

[0694] "Generating 360-degree VR experience data based on the viewpoint of a selected character" means generating data that allows the user to experience a virtual reality environment from the viewpoint of a character selected by the user in a 360-degree view.

[0695] "Displaying the generated VR experience data to the user" means sending the VR experience data generated by the server to the terminal and presenting it to the user through a display device such as a VR headset.

[0696] "Collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience" means collecting the user's viewpoint movements, interactions, and emotional changes during the VR experience in real time, analyzing this data using AI, and using the results to improve the next VR experience based on the user's interests and emotions.

[0697] "Means for selecting a character using a digital pen" refers to a digital input device that a user uses to select a character on a picture book, and to transmit that selection information to the system.

[0698] A "VR headset that allows you to experience stories in 360 degrees" is a head-mounted display device that allows users to experience a virtual reality environment in all directions.

[0699] MODE FOR CARRYING OUT THE INVENTION

[0700] System Configuration

[0701] This invention is a system that stores digital data of a story, generates a 360-degree VR experience based on the viewpoint of a character selected by the user, and optimizes the next experience by analyzing the user's emotions. Below is a detailed explanation of how to implement this system.

[0702] Hardware and software used

[0703] 1. Server

[0704] Servers use high performance data storage and computation hardware (e.g., data center servers).

[0705] The software used includes database management systems (e.g., MySQL, MongoDB) for managing digital data, VR engines (e.g., Unity, Unreal Engine) for generating VR experiences, and AI models (e.g., TensorFlow, PyTorch) for analyzing emotional data.

[0706] 2. Terminal

[0707] A terminal is a device for user interaction, typically a computer with graphics processing capabilities (e.g., a gaming PC), a digital pen, or a VR headset (e.g., Oculus Rift or HTC Vive).

[0708] The software used includes applications to provide a user interface (e.g., custom VR apps, interactive UI software), and emotion engines (e.g., Affectiva, Realeyes) to collect emotion data in real time.

[0709] 3. Users

[0710] Users use a digital pen to select characters from the picture book and then put on a VR headset to immerse themselves in the story.

[0711] The user moves their viewpoint and interacts with the system, providing emotional data to the system.

[0712] Natural language explanation of program processing

[0713] The server receives the digital data (images, text, and audio) from the picture book, divides it, and stores it in a database. When the user selects a character in the picture book using the digital pen, the device captures the selection information and sends it to the server. The server uses a VR engine to generate 360-degree VR experience data based on the viewpoint of the selected character and sends this data to the device. The device displays this data to the user via the VR headset. Behavioral and emotional data from the user's experience is collected in real time through the emotion engine and sent to the server. The server uses an AI model to analyze this data and optimize the next experience based on the user's interests and emotions.

[0714] Examples of concrete examples and prompts

[0715] As a concrete example, when using the story of Momotaro, if the user selects "Oni" with the digital pen, the server generates VR experience data from the Oni's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the Oni's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by an AI model to further optimize the next experience.

[0716] Example prompt sentence:

[0717] "Create a VR experience from the perspective of the ogre in the story of Momotaro, and generate a program that will analyze the user's emotional data and optimize the next experience."

[0718] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[0719] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0720] Step 1:

[0721] The server receives the digital data of the story (images, text, audio). The input is the digital data of the story, and the output is the stored data. The server splits the received data into individual files and stores them in the appropriate directories using a database management system (e.g., MySQL, MongoDB). This allows users to efficiently access specific pages or scenes.

[0722] Step 2:

[0723] The user selects a character in a picture book using a digital pen. The input is the information of the character selected by the user, and the output is the selected character information. The device captures the digital pen input and sends the information to the server in JSON format.

[0724] Step 3:

[0725] The server receives information about the character selected by the user. The input is the selected character information, and the output is the character's viewpoint data. Based on the received information, the server obtains the character's viewpoint data and generates 360-degree VR experience data using a VR engine (e.g., Unity, Unreal Engine). Specifically, it generates simulation data and exports it as a 360-degree video.

[0726] Step 4:

[0727] The server sends the generated VR experience data to the device. The input is 360-degree VR experience data, and the output is data sent to the device. The device streams the received VR experience data to a VR headset (e.g., Oculus Rift, HTC Vive) to provide the experience to the user. The user can realistically experience the world of the story from the character's perspective.

[0728] Step 5:

[0729] The device collects in real time the user's viewpoint movements and interactions during the experience, as well as emotional data collected using an emotion engine (e.g., Affectiva, Realeyes). The input is the user's behavioral data and emotional data, and the output is the collected data. These data are sent to the server.

[0730] Step 6:

[0731] The server receives the user's behavioral and emotional data sent from the device. The input is the collected data, and the output is the analysis results. The server analyzes the data using an AI model (e.g., TensorFlow, PyTorch) and stores the results in a database. Based on the analysis results, the next VR experience is optimized based on the user's interests and emotions.

[0732] (Application example 2)

[0733] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0734] Traditional picture book and story experiences are static and one-way, making it difficult for users to immerse themselves in the story and receive an individually optimized experience. Furthermore, there is a lack of means to personalize the next experience based on emotional and behavioral data, making it difficult to improve the user's learning effect and entertainment value.

[0735] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, means for collecting behavioral data and emotional data of the user, analyzing the data using AI, and optimizing the next experience, and means for individually optimizing the next VR experience based on the user's emotional data. This allows the user to feel highly immersed in the world of the story while enjoying a personalized experience based on the emotional and behavioral data.

[0736] "Digital narrative data" refers to data that stores the visual and auditory content (images, text, and audio) of a narrative in digital form.

[0737] A "user-selected character" is a character within a story from which the user can choose a perspective within the experience.

[0738] "360-degree VR experience data" is virtual reality data that allows you to visually experience a story from any direction from the perspective of a selected character.

[0739] "Displaying to the user" means providing the generated VR experience data visually and audibly through a device (such as a headset) used by the user.

[0740] "Behavioral data" refers to data about the user's movements during a VR experience, such as their viewpoint, interactions, and gestures.

[0741] "Emotional data" refers to data about the user's emotional state obtained from facial expressions, tone of voice, heart rate, etc. during the VR experience.

[0742] "Analysis with AI" means analyzing and interpreting collected behavioral and emotional data using artificial intelligence technology.

[0743] "Optimizing the next experience" means adjusting the next VR experience to be more appropriate and interesting for the user based on the user's previous behavioral and emotional data.

[0744] A "digital input device" is a device (e.g., a stylus, touchpad) that a user uses to select characters within a story.

[0745] A "viewing device" is a device (e.g., a VR headset, smart glasses) used by a user to view VR experience data.

[0746] MODE FOR CARRYING OUT THE INVENTION

[0747] This invention is a system that allows users to immerse themselves in a story world and provide an individually optimized experience. This system includes storing digital data of the story, generating a 360-degree VR experience based on character selection, collecting and analyzing user behavioral and emotional data, and optimizing the next experience.

[0748] System configuration

[0749] server

[0750] Preserving the story digitally

[0751] The server stores the visual and auditory content of the story (images, text, audio) in digital form, making it easy to access specific pages or scenes.

[0752] VR experience generation based on character selection

[0753] The system identifies the character selected by the user and generates 360-degree VR experience data based on that character's perspective, allowing the user to experience the story from the perspective of the selected character.

[0754] Collecting and analyzing behavioral and emotional data

[0755] The server collects user behavioral and emotional data and analyzes it using AI technology, optimizing the content of the next experience based on the user's interests and emotions.

[0756] Terminal

[0757] Character selection interface

[0758] The user selects a character in the story using a digital input device (e.g., a touch pen or a touchpad), and the terminal identifies this selection and transmits it to the server.

[0759] Displaying VR experience data

[0760] The device receives the VR experience data sent from the server and displays it to the user using a viewing device such as a VR headset or smart glasses.

[0761] Collecting behavioral and emotional data

[0762] The device collects behavioral data such as the user's gaze movement, interactions, and facial expressions. It also uses an emotion engine to collect the user's emotional data in real time and transmits it to the server.

[0763] User

[0764] Character Selection

[0765] The user selects a character in the picture book using a digital input device.

[0766] VR experience

[0767] Users put on a VR headset and experience the story from the perspective of a character of their choice.

[0768] Specific examples

[0769] Take the story of Momotaro for example. The user selects the "demon" using a digital input device, and the server generates VR data from the demon's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[0770] Prompt Sentence Examples

[0771] You experience a story in a VR environment. First, choose a character. Then, enjoy the story from the perspective of the character you selected. Based on emotional and behavioral data from the experience, your next experience will be optimized to make it even more enjoyable.

[0772] If you experience any of the following emotions during the experience, you will enjoy the next optimized scene more.

[0773] surprise

[0774] joy

[0775] sorrow

[0776] anger

[0777] This system improves the user's immersion and learning effectiveness by individually optimizing the next VR experience based on the user's emotional and behavioral data.

[0778] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0779] Step 1:

[0780] The server stores the digital data of the story. This data includes images, text, and audio. The input is the digital file of the story, and the output is the stored data.

[0781] Step 2:

[0782] The terminal identifies the character selected by the user using a digital input device, with the input being the user's selection action and the output being the identified character information.

[0783] Step 3:

[0784] The server generates 360-degree VR experience data from the viewpoint of the selected character based on the identified character information. The input is character information, and the output is 360-degree VR data. For data processing, a VR engine is used to generate visual and audio data.

[0785] Step 4:

[0786] The device receives the VR experience data sent from the server and displays it to the user. The input is VR data, and the output is the display to the user through the VR headset. Specifically, the device uses VR display software.

[0787] Step 5:

[0788] The device collects user behavior data (eye movement and interactions). The input is the user's actions, and the output is the collected behavior data. The device uses the built-in gyro sensor and accelerometer to process the data.

[0789] Step 6:

[0790] It uses an emotion engine to collect user emotion data in real time. The input is the user's facial expressions and voice, and the output is the collected emotion data. Specific operations include analyzing data from the camera and microphone.

[0791] Step 7:

[0792] The terminal sends the collected behavioral data and emotional data to the server. The input is behavioral data and emotional data, and the output is data sent to the server.

[0793] Step 8:

[0794] The server analyzes the collected data using AI. The input is behavioral data and emotional data, and the output is the analysis results. Data is analyzed using a generative AI model.

[0795] Step 9:

[0796] The server optimizes the next VR experience based on the analysis results. The input is the analysis results, and the output is the optimized VR experience data. Specifically, it uses an algorithm to personalize the content of the next experience.

[0797] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0798] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0799] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0800] [Third embodiment]

[0801] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0802] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0803] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0804] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0805] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0806] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0807] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0808] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0809] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0810] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0811] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0812] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0813] This invention is a system that allows you to experience a picture book story in a 360-degree VR environment. Specific embodiments of this system will be described below.

[0814] System configuration

[0815] 1. Server

[0816] The server stores the digital data of the story.

[0817] The data is processed to identify the characters selected by the user.

[0818] Generates 360-degree VR experience data based on the perspective of the selected character.

[0819] The generated VR experience data is sent to the device.

[0820] Collect user behavior data and analyze it with AI to optimize the next experience.

[0821] 2. Terminal

[0822] An interface is provided whereby the user can select a character using a digital pen.

[0823] Receives VR experience data sent from the server and displays it to the user.

[0824] The user's behavioral data and viewpoint movements are tracked in real time and sent to the server.

[0825] 3. Users

[0826] Users use the AI ​​pen to select characters for the story.

[0827] Users put on a VR headset and experience the story from the perspective of a selected character.

[0828] Users can freely explore the VR environment and provide behavioral data during the experience.

[0829] Program processing explanation

[0830] Preserving digital data of picture books

[0831] The server receives the digital data of the picture book and stores it in separate files, including images, text, and audio data for each page, making it easy for users to access specific pages or scenes later.

[0832] Identify the characters the user selects

[0833] When a user selects a character in a picture book using the AI ​​pen, the device identifies that information and sends it to the server, which then obtains the viewpoint data of the selected character based on the information received.

[0834] Generate 360-degree VR experience data based on the perspective of a selected character

[0835] The server uses a VR engine to generate VR experience data from the perspective of a selected character, for example, creating 360-degree video and simulation data to observe Momotaro's actions from the perspective of the ogre.

[0836] Display the generated VR experience data to the user

[0837] The terminal receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story in virtual reality by wearing the headset.

[0838] Collect user behavior data and analyze it with AI

[0839] Data on the user's viewpoint movements and interactions during the experience is collected. This behavioral data is periodically sent to a server and analyzed by AI. This allows the next experience to be optimized based on the user's interests and level of understanding.

[0840] Specific examples

[0841] Take the story of Momotaro for example. The user uses the AI ​​pen to select the "demon." The server generates a VR scene of Kagoshima from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to those actions and the demon's feelings. After the experience, data on the user's viewpoint and behavior is collected, and the AI ​​optimizes the next experience.

[0842] This system allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved from a single conventional perspective, providing a groundbreaking educational tool that can enhance empathy and understanding.

[0843] The processing flow will be explained below.

[0844] Step 1:

[0845] The server receives the digital data of the picture book. The data includes images, text, and audio. The server analyzes the data, divides it into pages, and stores them.

[0846] Step 2:

[0847] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[0848] Step 3:

[0849] The device analyzes the received location data to identify which character was selected and sends this information to the server.

[0850] Step 4:

[0851] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[0852] Step 5:

[0853] The generated VR experience data is sent from the server to the device, which then displays the received data on the user's VR headset.

[0854] Step 6:

[0855] Users wear a VR headset and experience the story world from the perspective of a selected character. Users can freely move their viewpoint and interact with the character.

[0856] Step 7:

[0857] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data, which is then sent to a server.

[0858] Step 8:

[0859] The server uses AI to analyze the received user behavior data and, based on the analysis results, sets parameters to optimize the user's next experience.

[0860] Step 9:

[0861] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[0862] Example 1

[0863] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0864] In modern education and entertainment, experiences that allow users to actively participate are important. However, traditional visual experiences do not allow users to experience the story from the perspective of the characters, making it difficult to enhance users' empathy and understanding. Furthermore, they lack a mechanism for collecting user behavioral data and applying it to the next experience. Therefore, there is a demand for more interactive and personalized experiences.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0866] In this invention, the server includes means for storing digital data of the story, means for identifying a character selected by the user, means for generating virtual reality experience data based on the viewpoint of the selected character, means for displaying the generated virtual reality experience data to the user, and means for collecting user behavior data, analyzing it with artificial intelligence, and optimizing the next experience. This allows the user to experience the story from the viewpoint of the character, and the user's behavior and viewpoint are analyzed as data and reflected in the next experience, enabling a more effective and personalized experience.

[0867] "Digital story data" refers to data formats such as images, text, and audio that are used to electronically store and manage the content of a story.

[0868] A "user-selected character" is a character in the story that the user selects through the interface at the start of the experience.

[0869] "Virtual reality experience data" is data that includes 360-degree video and audio generated based on the perspective of a selected character.

[0870] "Generated virtual reality experience data" refers to specific data for a virtual reality experience that is generated on a server and sent to a user's terminal.

[0871] "User behavior data" refers to information about interactions such as viewpoint movement and operations performed by the user during the virtual reality experience.

[0872] "Analysis with artificial intelligence" is the process of using collected user behavior data to analyze it using machine learning models and AI algorithms, and then adjusting the content and difficulty of the next experience.

[0873] "Means to optimize the next experience" is a function that individually adjusts the content and difficulty of the experience based on AI analysis of user behavior data.

[0874] An "input device" is a device such as a digital pen or touch screen that a user uses to select characters.

[0875] A "display device" is a device, such as a VR headset or computer screen, that a user uses to visually enjoy a virtual reality experience.

[0876] This invention relates to a system for experiencing a story in a virtual reality environment. This system allows users to experience a 360-degree virtual reality experience from the perspective of a character in the story, and provides a mechanism for optimizing the next experience by collecting and analyzing user behavior data.

[0877] System configuration

[0878] 1. Server

[0879] How to digitally preserve your stories:

[0880] The server receives the digital data of the storybook and stores it in separate files, including images, text, and audio data for each page, using a specific database management system (e.g., MySQL or PostgreSQL).

[0881] A way to identify the character the user selects:

[0882] The server receives data about the character selected by the user and identifies that character's information. This process uses software (e.g., device drivers and APIs) to interpret data from input devices such as digital pens and touchscreens.

[0883] Means for generating virtual reality experience data:

[0884] A VR engine (e.g., Unity, Unreal Engine) is used to generate 360-degree VR experience data based on the perspective of the selected character, creating an environment in which the user can experience the story from the perspective of the selected character.

[0885] Collecting user behavior data and analyzing it with artificial intelligence:

[0886] Data on the user's viewpoint movements and interactions during the VR experience is collected by the server, analyzed using an artificial intelligence model (e.g., TensorFlow, PyTorch), and the results are used to optimize the next experience.

[0887] 2. Terminal

[0888] A means for displaying the generated virtual reality experience data to the user:

[0889] It receives the VR experience data sent from the server and displays it to the user through the VR headset (e.g., Oculus Rift, HTC Vive). The device converts this data into an appropriate format so that the user can enjoy an immersive experience.

[0890] An interface that identifies the character the user can select:

[0891] It provides an interface for users to select characters using a digital pen or touch screen, and this interface plays a role in transmitting information about the selected characters to the server.

[0892] 3. Users

[0893] Story Experience:

[0894] Users select a character in the story using a digital pen and then wear a VR headset to experience the story in virtual reality. For example, if a user selects the "demon," they can observe Momotaro's actions from the demon's perspective.

[0895] Behavioral data provided:

[0896] The user's viewpoint movements and interactions during the experience are collected in real time and used to optimize the next experience.

[0897] Specific examples

[0898] For example, in the story of Momotaro, the user selects the "demon" using the AI ​​pen. The server generates a VR scene of Okayama from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to his actions and the demon's feelings. After the experience, the user's viewpoint and behavior data are collected, and the AI ​​optimizes the next experience.

[0899] Examples of prompt statements

[0900] "This is the story of Momotaro. The user selects the ogre with the AI ​​pen. The server should generate VR experience data from the ogre's point of view and send it to the device. The device should display this data on the user's VR headset, collect user behavior data, and send it back to the server."

[0901] This invention allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved through a traditional single perspective, making it a groundbreaking educational tool that can enhance empathy and understanding.

[0902] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0903] Program processing flow

[0904] Step 1:

[0905] The server receives the digital data of picture books uploaded by the administrator. The digital data is divided into image files (JPEG, PNG) for each page, text files (TXT, XML), and audio files (MP3, WAV) and saved. Specifically, it stores the data in a database using a database management system (e.g., MySQL or PostgreSQL). The input is the picture book data uploaded by the administrator, and the output is the individual digital data saved in the database.

[0906] Step 2:

[0907] The user selects a character in a picture book using a digital pen. The device receives position input from the digital pen and identifies the character selected by the user. Specifically, the device reads the pen's position data and executes an identification algorithm to identify the corresponding character. The input is the position data from the digital pen, and the output is the identified character information.

[0908] Step 3:

[0909] The terminal sends character information to the server. The server receives this and obtains the viewpoint data of the selected character. Specifically, the server searches for and obtains the corresponding viewpoint data from a database based on the character information it receives. The input is character information, and the output is viewpoint data.

[0910] Step 4:

[0911] The server uses a VR engine (e.g., Unity, Unreal Engine) to generate VR experience data from the viewpoint of the selected character. Specifically, it runs a VR engine program to generate 360-degree simulation images and scenes in real time based on the viewpoint data. The input is viewpoint data, and the output is the generated VR experience data.

[0912] Step 5:

[0913] The device receives the VR experience data sent from the server and displays it on the user's VR headset (e.g., Oculus Rift, HTC Vive). Specifically, the device decodes the VR experience data and converts it into a format that can be displayed on the headset. The input is the VR experience data from the server, and the output is the image displayed on the VR headset.

[0914] Step 6:

[0915] The user wears a VR headset and experiences the story from the perspective of a selected character. Specific actions include moving freely within the VR environment, changing viewpoints, and interacting with the character. The input is the visual and audio experience data obtained through the VR headset, and the output is the user's experience and behavioral data.

[0916] Step 7:

[0917] The device collects user behavior data (eye movement and operations) in real time and sends it to a server. Specifically, it runs a program that collects sensor data to track user behavior and sends it to the server for analysis. The input is the user behavior data, and the output is the raw data sent to the server.

[0918] Step 8:

[0919] The server inputs the received user behavior data into an AI analysis model (e.g., TensorFlow, PyTorch) for analysis. Specifically, the AI ​​analysis model analyzes the user's behavioral patterns and executes algorithms to evaluate their interests and level of understanding. The input is the user's behavioral data, and the output is the analysis results.

[0920] Step 9:

[0921] The server adjusts the content and difficulty of the next experience based on the analysis results. Specifically, it runs a program to update the VR experience scenario based on the analysis results and prepares to generate new VR experience data. The input is the analysis results, and the output is optimized experience data for the next experience.

[0922] (Application example 1)

[0923] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0924] Today, there is a demand for interactive content as a way for users to gain a deeper understanding of stories and learn from them. However, conventional methods only allow users to enjoy stories as a visual experience, making it difficult to provide diverse experiences. Furthermore, there is a lack of mechanisms for effectively utilizing user behavior data to optimize the next experience. For this reason, it is necessary to develop an interactive system that allows users to learn from stories from multiple angles.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0926] In this invention, the server includes: [means for storing digital data of the story;] [means for identifying characters selected by the user;] [means for generating 360-degree VR experience data based on the viewpoint of the selected character;] [means for displaying the generated VR experience data to the user;] [means for collecting user behavior data, analyzing it with AI, and optimizing the next experience; and] [means for the user to freely move their viewpoint within the story and provide interactions with specific objects and characters. This allows the user to experience the story from multiple perspectives and gain deeper learning.

[0927] "Digital story data" refers to data in digital format that includes the text, images, audio, etc. of a story.

[0928] "User-selected character" refers to the character the user wants to experience in the story.

[0929] "360-degree VR experience data" is data that users can experience from a 360-degree perspective in a virtual reality space.

[0930] "Generated VR experience data" refers to data of a 360-degree VR environment generated by the server based on the user's selection.

[0931] "User behavior data" refers to data such as viewpoint movements and interactions that users make during a VR experience.

[0932] "Analyzing with AI and optimizing the next experience" refers to the process of analyzing collected user behavior data using AI technology and adjusting the content of the next experience based on the user's interests and behavior.

[0933] "Users can freely move their viewpoint within the story" refers to a function that allows users to move their viewpoint in any direction within the VR environment.

[0934] "Providing interaction with specific objects and characters" refers to a feature that allows the user to actually interact with specific items and characters within the story.

[0935] A "pen-type device" is a device that a user uses to select characters and items in a story.

[0936] "Display device" means a device that provides the user with visual information for a VR experience, including a VR headset and a smartphone.

[0937] This invention provides an interactive system that allows users to experience stories in 360-degree VR. This system consists of three elements: a server, a terminal, and a user. Each element and its processing will be explained below.

[0938] server

[0939] The server has the following features:

[0940] 1. Digital Data Storage:

[0941] The server stores the digital data of the stories (text, images, audio), using database software such as MySQL.

[0942] 2. Character Identification:

[0943] The system receives information about the character selected by the user through the terminal and identifies them. The software used is Python and Flask.

[0944] 3. Generating 360-degree VR experience data:

[0945] The server generates 360-degree VR experience data from the perspective of the selected character using Unity and Google VR SDK. For example, if the user selects "Oni," it generates a scene from the story of Momotaro from the perspective of the ogre.

[0946] 4. View generated data:

[0947] The generated VR experience data is sent to the device, allowing the user to visually experience the story through a VR headset.

[0948] 5. Behavioral data collection and AI analysis:

[0949] User behavior data (eye movement and interactions) is collected and analyzed using AI. TensorFlow is used for the AI ​​analysis. The next experience content is optimized based on the results of this analysis.

[0950] Terminal

[0951] The terminal has the following features:

[0952] 1. Provide an interface:

[0953] It provides an interface for users to select characters using a pen-like device, using Android Studio and a corresponding UI framework.

[0954] 2. Receiving and displaying VR experience data:

[0955] The VR experience data sent from the server is received and displayed to the user. The device uses a VR headset (e.g., a smartphone + VR headset) to provide a 360-degree experience.

[0956] 3. Transmission of behavioral data:

[0957] The user's behavior data is sent to the server in real time using a network communication module.

[0958] User

[0959] The user uses the system as follows:

[0960] 1. Character Selection:

[0961] Use the pen-shaped device to select a character in the story. For example, you can choose the "demon" in the story of Momotaro.

[0962] 2. VR Experience:

[0963] Put on a VR headset and experience the story in 360 degrees from the perspective of a selected character, with the ability to move freely between viewpoints and interact with specific objects and characters.

[0964] 3. Providing behavioral data:

[0965] Data on viewpoint movements and interactions during the experience is collected, allowing the next experience to be optimized for each individual user.

[0966] Examples and prompts

[0967] To illustrate, consider the following scenario:

[0968] When a user selects the story of Momotaro, they choose the "Oni's perspective." This information is sent to the server.

[0969] Prompt: "The user selected the story of Momotaro."

[0970] The server generates 360-degree VR data from the demon's perspective and sends it to the device.

[0971] Prompt: "The user selected the demon's perspective."

[0972] The user wears a VR headset and observes Momotaro's actions as the demon. Interactions also occur when touching specific objects.

[0973] Prompt: "The user observed Momotaro from the demon's perspective and touched a specific object."

[0974] User behavior data is sent to the server to optimize the experience next time.

[0975] Prompt: "AI identified the scenes that you found most interesting in your previous experience and optimized your next experience."

[0976] This system allows users to enjoy an interactive and multifaceted experience that goes beyond traditional storytelling methods.

[0977] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0978] Step 1:

[0979] The user selects a story and characters using a pen-like device.

[0980] Specific operation: The user selects one story from the list displayed on the device (smartphone) interface. Then, they use the pen-shaped device to select the character they want to experience from the characters appearing in the story.

[0981] Input: User-selected story and character information.

[0982] Output: Selected stories and character identities.

[0983] Data Processing: Processes input from the pen-type device and generates an ID for the selected story and character.

[0984] Step 2:

[0985] The terminal transmits information about the selected character to the server.

[0986] Specific operation: The device sends the selected character ID and story ID to the server via the network.

[0987] Input: Story ID and Character ID.

[0988] Output: The request to the server.

[0989] Data processing: Converting the data to be sent into the appropriate format (e.g., JSON or XML).

[0990] Step 3:

[0991] The server generates 360-degree VR experience data based on the viewpoint of the selected character.

[0992] What it does: The server uses Unity and the Google VR SDK to render a story scene from the perspective of a selected character as a 360-degree VR video.

[0993] Input: Story ID and Character ID.

[0994] Output: 360 degree VR data.

[0995] Data calculation: Generates a virtual space seen from the viewpoint of the selected character and renders the scene data.

[0996] Step 4:

[0997] The server sends the generated 360-degree VR data to the device.

[0998] Specific operation: The server transfers the generated VR data to the terminal via the network.

[0999] Input: 360-degree VR data.

[1000] Output: Sending VR data to the device.

[1001] Data processing: Compress large video data and convert it into an efficient streaming format.

[1002] Step 5:

[1003] The 360-degree VR data received by the device is displayed on the VR headset.

[1004] Specific operation: The terminal decompresses the received VR data and displays it on the VR headset (smartphone and VR headset).

[1005] Input: 360-degree VR data.

[1006] Output: The user's VR experience.

[1007] Data Computation: Rendering data into a format suitable for the display of a VR headset.

[1008] Step 6:

[1009] The device records the user's viewpoint movements and interactions during the VR experience.

[1010] Specific actions: The user uses the VR headset to explore the story and interact with specific objects and characters. The device records these actions in real time.

[1011] Input: User gaze movement and interaction data.

[1012] Output: Behavioral data.

[1013] Data collection: Data such as gaze position, gaze duration, and type of interaction are collected as logs.

[1014] Step 7:

[1015] The device transmits the collected behavioral data to a server.

[1016] Specific operation: The device compresses the behavioral data periodically or at the end of the experience and sends it to a server via the Internet.

[1017] Input: Behavioral data.

[1018] Output: Sending behavioral data to the server.

[1019] Data processing: Encoding raw behavioral data into a transmittable format.

[1020] Step 8:

[1021] The server uses AI to analyze behavioral data and optimize the next experience.

[1022] How it works: The server uses AI tools like TensorFlow to analyze the collected behavioral data, identify user interests and behavioral patterns, and use this information to fine-tune the next experience.

[1023] Input: Behavioral data.

[1024] Output: Analysis results and optimized experience data.

[1025] Data Computing: Analyzing behavioral data, identifying user interests and behavioral patterns, and applying optimization algorithms.

[1026] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1027] This invention is a system that allows users to experience a picture book story in a 360-degree VR environment and optimizes the next experience by recognizing and analyzing the user's emotions. This system stores the digital data of the story, generates a VR experience based on character selection and their viewpoint, and collects and analyzes the user's behavioral and emotional data.

[1028] System configuration

[1029] 1. Server

[1030] The server stores the digital data of the story, which includes images, text, and audio.

[1031] It identifies the selected character and generates 360-degree VR experience data based on their perspective.

[1032] The generated VR experience data is sent to the terminal, and the user's behavioral and emotional data is collected.

[1033] The collected data is analyzed using AI to optimize the next experience.

[1034] 2. Terminal

[1035] An interface is provided whereby the user can select a character using a digital pen.

[1036] It receives VR experience data sent from the server and provides it to the user via a VR headset for display.

[1037] Emotion data is collected in real time using user behavior data and an emotion engine and sent to a server.

[1038] 3. Users

[1039] The user selects a character in the picture book using a digital pen.

[1040] Put on a VR headset and experience the story from the perspective of a selected character.

[1041] It provides the system with information about viewpoint changes, interactions, and emotional changes during the experience.

[1042] Program processing explanation

[1043] Preserving digital data of picture books

[1044] The server receives the digital data of the picture book and stores it in separate parts: images, text, and audio, making it easier for users to access specific pages or scenes.

[1045] Identify the characters the user selects

[1046] When a user selects a character in a picture book using a digital pen, the device identifies that information and sends it to the server, which then acquires the character's viewpoint data based on the information received.

[1047] Generate 360-degree VR experience data based on the perspective of a selected character

[1048] The server uses a VR engine to generate 360-degree VR experience data from the perspective of a selected character, for example, creating video and simulation data for observing Momotaro's actions from the perspective of the ogre.

[1049] Display the generated VR experience data to the user

[1050] The device receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story world from the character's perspective.

[1051] Collect and analyze user behavioral and emotional data

[1052] The device collects data on the user's viewpoint, interactions, and emotional data collected through the emotion engine during the experience. This data is sent to a server and analyzed by AI. Based on the analysis results, the next experience is optimized based on the user's interests and emotions.

[1053] Specific examples

[1054] Take the story of Momotaro for example. The user selects the "demon" using a digital pen, and the server generates VR data from the demon's perspective and sends it to the device. The user then wears a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes (such as surprise or excitement) in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[1055] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[1056] The processing flow will be explained below.

[1057] Step 1:

[1058] The server receives the digital data of the picture book, which includes images, text, and audio. The server analyzes the data, splits it into pages, and stores them, making it easier for users to access specific pages or scenes.

[1059] Step 2:

[1060] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[1061] Step 3:

[1062] The device analyzes the received location data, identifies which character has been selected, and sends the identification result to the server.

[1063] Step 4:

[1064] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[1065] Step 5:

[1066] The server sends the generated VR experience data to the device, which then forwards the received data to the VR headset.

[1067] Step 6:

[1068] Users wear a VR headset and explore the story world from the perspective of a selected character, with the ability to move the viewpoint and interact with the character.

[1069] Step 7:

[1070] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data. It also uses an emotion engine to analyze the user's facial expressions and voice to collect emotional data. The collected data is periodically sent to a server.

[1071] Step 8:

[1072] The server uses AI to analyze the received user behavioral and emotional data, and based on the analysis results, it makes settings to individually optimize the user's next experience.

[1073] Step 9:

[1074] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[1075] Example 2

[1076] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] In today's education and entertainment fields, new methods are needed to engage users and deepen their understanding. In particular, there is a lack of technology to individually optimize story experiences to achieve deep empathy and learning effects. Furthermore, there has been no system that can collect user emotional and behavioral data in real time and use it to optimize the next experience.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, and means for collecting user behavioral data and emotional data, analyzing them using AI, and optimizing the next experience. This allows the user to deeply understand the story and empathize with it through an individually optimized VR experience. Furthermore, real-time emotional data analysis allows the next experience to be optimized based on the user's interests and emotions, thereby providing more effective education and entertainment.

[1079] "Storing digital story data" means receiving picture books and other story content in digital format and storing it in separate images, text, and audio data.

[1080] "Identifying a character selected by a user" means recognizing and accurately identifying the selected information when a user selects a character in a picture book using a digital pen or other interface.

[1081] "Generating 360-degree VR experience data based on the viewpoint of a selected character" means generating data that allows the user to experience a virtual reality environment from the viewpoint of a character selected by the user in a 360-degree view.

[1082] "Displaying the generated VR experience data to the user" means sending the VR experience data generated by the server to the terminal and presenting it to the user through a display device such as a VR headset.

[1083] "Collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience" means collecting the user's viewpoint movements, interactions, and emotional changes during the VR experience in real time, analyzing this data using AI, and using the results to improve the next VR experience based on the user's interests and emotions.

[1084] "Means for selecting a character using a digital pen" refers to a digital input device that a user uses to select a character on a picture book, and to transmit that selection information to the system.

[1085] A "VR headset that allows you to experience stories in 360 degrees" is a head-mounted display device that allows users to experience a virtual reality environment in all directions.

[1086] MODE FOR CARRYING OUT THE INVENTION

[1087] System Configuration

[1088] This invention is a system that stores digital data of a story, generates a 360-degree VR experience based on the viewpoint of a character selected by the user, and optimizes the next experience by analyzing the user's emotions. Below is a detailed explanation of how to implement this system.

[1089] Hardware and software used

[1090] 1. Server

[1091] Servers use high performance data storage and computation hardware (e.g., data center servers).

[1092] The software used includes database management systems (e.g., MySQL, MongoDB) for managing digital data, VR engines (e.g., Unity, Unreal Engine) for generating VR experiences, and AI models (e.g., TensorFlow, PyTorch) for analyzing emotional data.

[1093] 2. Terminal

[1094] A terminal is a device for user interaction, typically a computer with graphics processing capabilities (e.g., a gaming PC), a digital pen, or a VR headset (e.g., Oculus Rift or HTC Vive).

[1095] The software used includes applications to provide a user interface (e.g., custom VR apps, interactive UI software), and emotion engines (e.g., Affectiva, Realeyes) to collect emotion data in real time.

[1096] 3. Users

[1097] Users use a digital pen to select characters from the picture book and then put on a VR headset to immerse themselves in the story.

[1098] The user moves their viewpoint and interacts with the system, providing emotional data to the system.

[1099] Natural language explanation of program processing

[1100] The server receives the digital data (images, text, and audio) from the picture book, divides it, and stores it in a database. When the user selects a character in the picture book using the digital pen, the device captures the selection information and sends it to the server. The server uses a VR engine to generate 360-degree VR experience data based on the viewpoint of the selected character and sends this data to the device. The device displays this data to the user via the VR headset. Behavioral and emotional data from the user's experience is collected in real time through the emotion engine and sent to the server. The server uses an AI model to analyze this data and optimize the next experience based on the user's interests and emotions.

[1101] Examples of concrete examples and prompts

[1102] As a concrete example, when using the story of Momotaro, if the user selects "Oni" with the digital pen, the server generates VR experience data from the Oni's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the Oni's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by an AI model to further optimize the next experience.

[1103] Example prompt sentence:

[1104] "Create a VR experience from the perspective of the ogre in the story of Momotaro, and generate a program that will analyze the user's emotional data and optimize the next experience."

[1105] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[1106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] The server receives the digital data of the story (images, text, audio). The input is the digital data of the story, and the output is the stored data. The server splits the received data into individual files and stores them in the appropriate directories using a database management system (e.g., MySQL, MongoDB). This allows users to efficiently access specific pages or scenes.

[1109] Step 2:

[1110] The user selects a character in a picture book using a digital pen. The input is the information of the character selected by the user, and the output is the selected character information. The device captures the digital pen input and sends the information to the server in JSON format.

[1111] Step 3:

[1112] The server receives information about the character selected by the user. The input is the selected character information, and the output is the character's viewpoint data. Based on the received information, the server obtains the character's viewpoint data and generates 360-degree VR experience data using a VR engine (e.g., Unity, Unreal Engine). Specifically, it generates simulation data and exports it as a 360-degree video.

[1113] Step 4:

[1114] The server sends the generated VR experience data to the device. The input is 360-degree VR experience data, and the output is data sent to the device. The device streams the received VR experience data to a VR headset (e.g., Oculus Rift, HTC Vive) to provide the experience to the user. The user can realistically experience the world of the story from the character's perspective.

[1115] Step 5:

[1116] The device collects in real time the user's viewpoint movements and interactions during the experience, as well as emotional data collected using an emotion engine (e.g., Affectiva, Realeyes). The input is the user's behavioral data and emotional data, and the output is the collected data. These data are sent to the server.

[1117] Step 6:

[1118] The server receives the user's behavioral and emotional data sent from the device. The input is the collected data, and the output is the analysis results. The server analyzes the data using an AI model (e.g., TensorFlow, PyTorch) and stores the results in a database. Based on the analysis results, the next VR experience is optimized based on the user's interests and emotions.

[1119] (Application example 2)

[1120] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] Traditional picture book and story experiences are static and one-way, making it difficult for users to immerse themselves in the story and receive an individually optimized experience. Furthermore, there is a lack of means to personalize the next experience based on emotional and behavioral data, making it difficult to improve the user's learning effect and entertainment value.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, means for collecting behavioral data and emotional data of the user, analyzing the data using AI, and optimizing the next experience, and means for individually optimizing the next VR experience based on the user's emotional data. This allows the user to feel highly immersed in the world of the story while enjoying a personalized experience based on the emotional and behavioral data.

[1123] "Digital narrative data" refers to data that stores the visual and auditory content (images, text, and audio) of a narrative in digital form.

[1124] A "user-selected character" is a character within a story from which the user can choose a perspective within the experience.

[1125] "360-degree VR experience data" is virtual reality data that allows you to visually experience a story from any direction from the perspective of a selected character.

[1126] "Displaying to the user" means providing the generated VR experience data visually and audibly through a device (such as a headset) used by the user.

[1127] "Behavioral data" refers to data about the user's movements during a VR experience, such as their viewpoint, interactions, and gestures.

[1128] "Emotional data" refers to data about the user's emotional state obtained from facial expressions, tone of voice, heart rate, etc. during the VR experience.

[1129] "Analysis with AI" means analyzing and interpreting collected behavioral and emotional data using artificial intelligence technology.

[1130] "Optimizing the next experience" means adjusting the next VR experience to be more appropriate and interesting for the user based on the user's previous behavioral and emotional data.

[1131] A "digital input device" is a device (e.g., a stylus, touchpad) that a user uses to select characters within a story.

[1132] A "viewing device" is a device (e.g., a VR headset, smart glasses) used by a user to view VR experience data.

[1133] MODE FOR CARRYING OUT THE INVENTION

[1134] This invention is a system that allows users to immerse themselves in a story world and provide an individually optimized experience. This system includes storing digital data of the story, generating a 360-degree VR experience based on character selection, collecting and analyzing user behavioral and emotional data, and optimizing the next experience.

[1135] System configuration

[1136] server

[1137] Preserving the story digitally

[1138] The server stores the visual and auditory content of the story (images, text, audio) in digital form, making it easy to access specific pages or scenes.

[1139] VR experience generation based on character selection

[1140] The system identifies the character selected by the user and generates 360-degree VR experience data based on that character's perspective, allowing the user to experience the story from the perspective of the selected character.

[1141] Collecting and analyzing behavioral and emotional data

[1142] The server collects user behavioral and emotional data and analyzes it using AI technology, optimizing the content of the next experience based on the user's interests and emotions.

[1143] Terminal

[1144] Character selection interface

[1145] The user selects a character in the story using a digital input device (e.g., a touch pen or a touchpad), and the terminal identifies this selection and transmits it to the server.

[1146] Displaying VR experience data

[1147] The device receives the VR experience data sent from the server and displays it to the user using a viewing device such as a VR headset or smart glasses.

[1148] Collecting behavioral and emotional data

[1149] The device collects behavioral data such as the user's gaze movement, interactions, and facial expressions. It also uses an emotion engine to collect the user's emotional data in real time and transmits it to the server.

[1150] User

[1151] Character Selection

[1152] The user selects a character in the picture book using a digital input device.

[1153] VR experience

[1154] Users put on a VR headset and experience the story from the perspective of a character of their choice.

[1155] Specific examples

[1156] Take the story of Momotaro for example. The user selects the "demon" using a digital input device, and the server generates VR data from the demon's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[1157] Prompt Sentence Examples

[1158] You experience a story in a VR environment. First, choose a character. Then, enjoy the story from the perspective of the character you selected. Based on emotional and behavioral data from the experience, your next experience will be optimized to make it even more enjoyable.

[1159] If you experience any of the following emotions during the experience, you will enjoy the next optimized scene more.

[1160] surprise

[1161] joy

[1162] sorrow

[1163] anger

[1164] This system improves the user's immersion and learning effectiveness by individually optimizing the next VR experience based on the user's emotional and behavioral data.

[1165] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1166] Step 1:

[1167] The server stores the digital data of the story. This data includes images, text, and audio. The input is the digital file of the story, and the output is the stored data.

[1168] Step 2:

[1169] The terminal identifies the character selected by the user using a digital input device, with the input being the user's selection action and the output being the identified character information.

[1170] Step 3:

[1171] The server generates 360-degree VR experience data from the viewpoint of the selected character based on the identified character information. The input is character information, and the output is 360-degree VR data. For data processing, a VR engine is used to generate visual and audio data.

[1172] Step 4:

[1173] The device receives the VR experience data sent from the server and displays it to the user. The input is VR data, and the output is the display to the user through the VR headset. Specifically, the device uses VR display software.

[1174] Step 5:

[1175] The device collects user behavior data (eye movement and interactions). The input is the user's actions, and the output is the collected behavior data. The device uses the built-in gyro sensor and accelerometer to process the data.

[1176] Step 6:

[1177] It uses an emotion engine to collect user emotion data in real time. The input is the user's facial expressions and voice, and the output is the collected emotion data. Specific operations include analyzing data from the camera and microphone.

[1178] Step 7:

[1179] The terminal sends the collected behavioral data and emotional data to the server. The input is behavioral data and emotional data, and the output is data sent to the server.

[1180] Step 8:

[1181] The server analyzes the collected data using AI. The input is behavioral data and emotional data, and the output is the analysis results. Data is analyzed using a generative AI model.

[1182] Step 9:

[1183] The server optimizes the next VR experience based on the analysis results. The input is the analysis results, and the output is the optimized VR experience data. Specifically, it uses an algorithm to personalize the content of the next experience.

[1184] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1185] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1186] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1187] [Fourth embodiment]

[1188] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1189] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1190] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1191] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1192] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1193] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1194] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1195] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1196] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1197] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1198] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1199] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1200] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1201] This invention is a system that allows you to experience a picture book story in a 360-degree VR environment. Specific embodiments of this system will be described below.

[1202] System configuration

[1203] 1. Server

[1204] The server stores the digital data of the story.

[1205] The data is processed to identify the characters selected by the user.

[1206] Generates 360-degree VR experience data based on the perspective of the selected character.

[1207] The generated VR experience data is sent to the device.

[1208] Collect user behavior data and analyze it with AI to optimize the next experience.

[1209] 2. Terminal

[1210] An interface is provided whereby the user can select a character using a digital pen.

[1211] Receives VR experience data sent from the server and displays it to the user.

[1212] The user's behavioral data and viewpoint movements are tracked in real time and sent to the server.

[1213] 3. Users

[1214] Users use the AI ​​pen to select characters for the story.

[1215] Users put on a VR headset and experience the story from the perspective of a selected character.

[1216] Users can freely explore the VR environment and provide behavioral data during the experience.

[1217] Program processing explanation

[1218] Preserving digital data of picture books

[1219] The server receives the digital data of the picture book and stores it in separate files, including images, text, and audio data for each page, making it easy for users to access specific pages or scenes later.

[1220] Identify the characters the user selects

[1221] When a user selects a character in a picture book using the AI ​​pen, the device identifies that information and sends it to the server, which then obtains the viewpoint data of the selected character based on the information received.

[1222] Generate 360-degree VR experience data based on the perspective of a selected character

[1223] The server uses a VR engine to generate VR experience data from the perspective of a selected character, for example, creating 360-degree video and simulation data to observe Momotaro's actions from the perspective of the ogre.

[1224] Display the generated VR experience data to the user

[1225] The terminal receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story in virtual reality by wearing the headset.

[1226] Collect user behavior data and analyze it with AI

[1227] Data on the user's viewpoint movements and interactions during the experience is collected. This behavioral data is periodically sent to a server and analyzed by AI. This allows the next experience to be optimized based on the user's interests and level of understanding.

[1228] Specific examples

[1229] Take the story of Momotaro for example. The user uses the AI ​​pen to select the "demon." The server generates a VR scene of Kagoshima from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to those actions and the demon's feelings. After the experience, data on the user's viewpoint and behavior is collected, and the AI ​​optimizes the next experience.

[1230] This system allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved from a single conventional perspective, providing a groundbreaking educational tool that can enhance empathy and understanding.

[1231] The processing flow will be explained below.

[1232] Step 1:

[1233] The server receives the digital data of the picture book. The data includes images, text, and audio. The server analyzes the data, divides it into pages, and stores them.

[1234] Step 2:

[1235] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[1236] Step 3:

[1237] The device analyzes the received location data to identify which character was selected and sends this information to the server.

[1238] Step 4:

[1239] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[1240] Step 5:

[1241] The generated VR experience data is sent from the server to the device, which then displays the received data on the user's VR headset.

[1242] Step 6:

[1243] Users wear a VR headset and experience the story world from the perspective of a selected character. Users can freely move their viewpoint and interact with the character.

[1244] Step 7:

[1245] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data, which is then sent to a server.

[1246] Step 8:

[1247] The server uses AI to analyze the received user behavior data and, based on the analysis results, sets parameters to optimize the user's next experience.

[1248] Step 9:

[1249] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[1250] Example 1

[1251] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1252] In modern education and entertainment, experiences that allow users to actively participate are important. However, traditional visual experiences do not allow users to experience the story from the perspective of the characters, making it difficult to enhance users' empathy and understanding. Furthermore, they lack a mechanism for collecting user behavioral data and applying it to the next experience. Therefore, there is a demand for more interactive and personalized experiences.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1254] In this invention, the server includes means for storing digital data of the story, means for identifying a character selected by the user, means for generating virtual reality experience data based on the viewpoint of the selected character, means for displaying the generated virtual reality experience data to the user, and means for collecting user behavior data, analyzing it with artificial intelligence, and optimizing the next experience. This allows the user to experience the story from the viewpoint of the character, and the user's behavior and viewpoint are analyzed as data and reflected in the next experience, enabling a more effective and personalized experience.

[1255] "Digital story data" refers to data formats such as images, text, and audio that are used to electronically store and manage the content of a story.

[1256] A "user-selected character" is a character in the story that the user selects through the interface at the start of the experience.

[1257] "Virtual reality experience data" is data that includes 360-degree video and audio generated based on the perspective of a selected character.

[1258] "Generated virtual reality experience data" refers to specific data for a virtual reality experience that is generated on a server and sent to a user's terminal.

[1259] "User behavior data" refers to information about interactions such as viewpoint movement and operations performed by the user during the virtual reality experience.

[1260] "Analysis with artificial intelligence" is the process of using collected user behavior data to analyze it using machine learning models and AI algorithms, and then adjusting the content and difficulty of the next experience.

[1261] "Means to optimize the next experience" is a function that individually adjusts the content and difficulty of the experience based on AI analysis of user behavior data.

[1262] An "input device" is a device such as a digital pen or touch screen that a user uses to select characters.

[1263] A "display device" is a device, such as a VR headset or computer screen, that a user uses to visually enjoy a virtual reality experience.

[1264] This invention relates to a system for experiencing a story in a virtual reality environment. This system allows users to experience a 360-degree virtual reality experience from the perspective of a character in the story, and provides a mechanism for optimizing the next experience by collecting and analyzing user behavior data.

[1265] System configuration

[1266] 1. Server

[1267] How to digitally preserve your stories:

[1268] The server receives the digital data of the storybook and stores it in separate files, including images, text, and audio data for each page, using a specific database management system (e.g., MySQL or PostgreSQL).

[1269] A way to identify the character the user selects:

[1270] The server receives data about the character selected by the user and identifies that character's information. This process uses software (e.g., device drivers and APIs) to interpret data from input devices such as digital pens and touchscreens.

[1271] Means for generating virtual reality experience data:

[1272] A VR engine (e.g., Unity, Unreal Engine) is used to generate 360-degree VR experience data based on the perspective of the selected character, creating an environment in which the user can experience the story from the perspective of the selected character.

[1273] Collecting user behavior data and analyzing it with artificial intelligence:

[1274] Data on the user's viewpoint movements and interactions during the VR experience is collected by the server, analyzed using an artificial intelligence model (e.g., TensorFlow, PyTorch), and the results are used to optimize the next experience.

[1275] 2. Terminal

[1276] A means for displaying the generated virtual reality experience data to the user:

[1277] It receives the VR experience data sent from the server and displays it to the user through the VR headset (e.g., Oculus Rift, HTC Vive). The device converts this data into an appropriate format so that the user can enjoy an immersive experience.

[1278] An interface that identifies the character the user can select:

[1279] It provides an interface for users to select characters using a digital pen or touch screen, and this interface plays a role in transmitting information about the selected characters to the server.

[1280] 3. Users

[1281] Story Experience:

[1282] Users select a character in the story using a digital pen and then wear a VR headset to experience the story in virtual reality. For example, if a user selects the "demon," they can observe Momotaro's actions from the demon's perspective.

[1283] Behavioral data provided:

[1284] The user's viewpoint movements and interactions during the experience are collected in real time and used to optimize the next experience.

[1285] Specific examples

[1286] For example, in the story of Momotaro, the user selects the "demon" using the AI ​​pen. The server generates a VR scene of Okayama from the demon's point of view and displays it to the user via the device. The user observes Momotaro's actions from the demon's point of view and understands the background to his actions and the demon's feelings. After the experience, the user's viewpoint and behavior data are collected, and the AI ​​optimizes the next experience.

[1287] Examples of prompt statements

[1288] "This is the story of Momotaro. The user selects the ogre with the AI ​​pen. The server should generate VR experience data from the ogre's point of view and send it to the device. The device should display this data on the user's VR headset, collect user behavior data, and send it back to the server."

[1289] This invention allows users to enjoy a multifaceted and in-depth learning experience that cannot be achieved through a traditional single perspective, making it a groundbreaking educational tool that can enhance empathy and understanding.

[1290] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1291] Program processing flow

[1292] Step 1:

[1293] The server receives the digital data of picture books uploaded by the administrator. The digital data is divided into image files (JPEG, PNG) for each page, text files (TXT, XML), and audio files (MP3, WAV) and saved. Specifically, it stores the data in a database using a database management system (e.g., MySQL or PostgreSQL). The input is the picture book data uploaded by the administrator, and the output is the individual digital data saved in the database.

[1294] Step 2:

[1295] The user selects a character in a picture book using a digital pen. The device receives position input from the digital pen and identifies the character selected by the user. Specifically, the device reads the pen's position data and executes an identification algorithm to identify the corresponding character. The input is the position data from the digital pen, and the output is the identified character information.

[1296] Step 3:

[1297] The terminal sends character information to the server. The server receives this and obtains the viewpoint data of the selected character. Specifically, the server searches for and obtains the corresponding viewpoint data from a database based on the character information it receives. The input is character information, and the output is viewpoint data.

[1298] Step 4:

[1299] The server uses a VR engine (e.g., Unity, Unreal Engine) to generate VR experience data from the viewpoint of the selected character. Specifically, it runs a VR engine program to generate 360-degree simulation images and scenes in real time based on the viewpoint data. The input is viewpoint data, and the output is the generated VR experience data.

[1300] Step 5:

[1301] The device receives the VR experience data sent from the server and displays it on the user's VR headset (e.g., Oculus Rift, HTC Vive). Specifically, the device decodes the VR experience data and converts it into a format that can be displayed on the headset. The input is the VR experience data from the server, and the output is the image displayed on the VR headset.

[1302] Step 6:

[1303] The user wears a VR headset and experiences the story from the perspective of a selected character. Specific actions include moving freely within the VR environment, changing viewpoints, and interacting with the character. The input is the visual and audio experience data obtained through the VR headset, and the output is the user's experience and behavioral data.

[1304] Step 7:

[1305] The device collects user behavior data (eye movement and operations) in real time and sends it to a server. Specifically, it runs a program that collects sensor data to track user behavior and sends it to the server for analysis. The input is the user behavior data, and the output is the raw data sent to the server.

[1306] Step 8:

[1307] The server inputs the received user behavior data into an AI analysis model (e.g., TensorFlow, PyTorch) for analysis. Specifically, the AI ​​analysis model analyzes the user's behavioral patterns and executes algorithms to evaluate their interests and level of understanding. The input is the user's behavioral data, and the output is the analysis results.

[1308] Step 9:

[1309] The server adjusts the content and difficulty of the next experience based on the analysis results. Specifically, it runs a program to update the VR experience scenario based on the analysis results and prepares to generate new VR experience data. The input is the analysis results, and the output is optimized experience data for the next experience.

[1310] (Application example 1)

[1311] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1312] Today, there is a demand for interactive content as a way for users to gain a deeper understanding of stories and learn from them. However, conventional methods only allow users to enjoy stories as a visual experience, making it difficult to provide diverse experiences. Furthermore, there is a lack of mechanisms for effectively utilizing user behavior data to optimize the next experience. For this reason, it is necessary to develop an interactive system that allows users to learn from stories from multiple angles.

[1313] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1314] In this invention, the server includes: [means for storing digital data of the story;] [means for identifying characters selected by the user;] [means for generating 360-degree VR experience data based on the viewpoint of the selected character;] [means for displaying the generated VR experience data to the user;] [means for collecting user behavior data, analyzing it with AI, and optimizing the next experience; and] [means for the user to freely move their viewpoint within the story and provide interactions with specific objects and characters. This allows the user to experience the story from multiple perspectives and gain deeper learning.

[1315] "Digital story data" refers to data in digital format that includes the text, images, audio, etc. of a story.

[1316] "User-selected character" refers to the character the user wants to experience in the story.

[1317] "360-degree VR experience data" is data that users can experience from a 360-degree perspective in a virtual reality space.

[1318] "Generated VR experience data" refers to data of a 360-degree VR environment generated by the server based on the user's selection.

[1319] "User behavior data" refers to data such as viewpoint movements and interactions that users make during a VR experience.

[1320] "Analyzing with AI and optimizing the next experience" refers to the process of analyzing collected user behavior data using AI technology and adjusting the content of the next experience based on the user's interests and behavior.

[1321] "Users can freely move their viewpoint within the story" refers to a function that allows users to move their viewpoint in any direction within the VR environment.

[1322] "Providing interaction with specific objects and characters" refers to a feature that allows the user to actually interact with specific items and characters within the story.

[1323] A "pen-type device" is a device that a user uses to select characters and items in a story.

[1324] "Display device" means a device that provides the user with visual information for a VR experience, including a VR headset and a smartphone.

[1325] This invention provides an interactive system that allows users to experience stories in 360-degree VR. This system consists of three elements: a server, a terminal, and a user. Each element and its processing will be explained below.

[1326] server

[1327] The server has the following features:

[1328] 1. Digital Data Storage:

[1329] The server stores the digital data of the stories (text, images, audio), using database software such as MySQL.

[1330] 2. Character Identification:

[1331] The system receives information about the character selected by the user through the terminal and identifies them. The software used is Python and Flask.

[1332] 3. Generating 360-degree VR experience data:

[1333] The server generates 360-degree VR experience data from the perspective of the selected character using Unity and Google VR SDK. For example, if the user selects "Oni," it generates a scene from the story of Momotaro from the perspective of the ogre.

[1334] 4. View generated data:

[1335] The generated VR experience data is sent to the device, allowing the user to visually experience the story through a VR headset.

[1336] 5. Behavioral data collection and AI analysis:

[1337] User behavior data (eye movement and interactions) is collected and analyzed using AI. TensorFlow is used for the AI ​​analysis. The next experience content is optimized based on the results of this analysis.

[1338] Terminal

[1339] The terminal has the following features:

[1340] 1. Provide an interface:

[1341] It provides an interface for users to select characters using a pen-like device, using Android Studio and a corresponding UI framework.

[1342] 2. Receiving and displaying VR experience data:

[1343] The VR experience data sent from the server is received and displayed to the user. The device uses a VR headset (e.g., a smartphone + VR headset) to provide a 360-degree experience.

[1344] 3. Transmission of behavioral data:

[1345] The user's behavior data is sent to the server in real time using a network communication module.

[1346] User

[1347] The user uses the system as follows:

[1348] 1. Character Selection:

[1349] Use the pen-shaped device to select a character in the story. For example, you can choose the "demon" in the story of Momotaro.

[1350] 2. VR Experience:

[1351] Put on a VR headset and experience the story in 360 degrees from the perspective of a selected character, with the ability to move freely between viewpoints and interact with specific objects and characters.

[1352] 3. Providing behavioral data:

[1353] Data on viewpoint movements and interactions during the experience is collected, allowing the next experience to be optimized for each individual user.

[1354] Examples and prompts

[1355] To illustrate, consider the following scenario:

[1356] When a user selects the story of Momotaro, they choose the "Oni's perspective." This information is sent to the server.

[1357] Prompt: "The user selected the story of Momotaro."

[1358] The server generates 360-degree VR data from the demon's perspective and sends it to the device.

[1359] Prompt: "The user selected the demon's perspective."

[1360] The user wears a VR headset and observes Momotaro's actions as the demon. Interactions also occur when touching specific objects.

[1361] Prompt: "The user observed Momotaro from the demon's perspective and touched a specific object."

[1362] User behavior data is sent to the server to optimize the experience next time.

[1363] Prompt: "AI identified the scenes that you found most interesting in your previous experience and optimized your next experience."

[1364] This system allows users to enjoy an interactive and multifaceted experience that goes beyond traditional storytelling methods.

[1365] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1366] Step 1:

[1367] The user selects a story and characters using a pen-like device.

[1368] Specific operation: The user selects one story from the list displayed on the device (smartphone) interface. Then, they use the pen-shaped device to select the character they want to experience from the characters appearing in the story.

[1369] Input: User-selected story and character information.

[1370] Output: Selected stories and character identities.

[1371] Data Processing: Processes input from the pen-type device and generates an ID for the selected story and character.

[1372] Step 2:

[1373] The terminal transmits information about the selected character to the server.

[1374] Specific operation: The device sends the selected character ID and story ID to the server via the network.

[1375] Input: Story ID and Character ID.

[1376] Output: The request to the server.

[1377] Data processing: Converting the data to be sent into the appropriate format (e.g., JSON or XML).

[1378] Step 3:

[1379] The server generates 360-degree VR experience data based on the viewpoint of the selected character.

[1380] What it does: The server uses Unity and the Google VR SDK to render a story scene from the perspective of a selected character as a 360-degree VR video.

[1381] Input: Story ID and Character ID.

[1382] Output: 360 degree VR data.

[1383] Data calculation: Generates a virtual space seen from the viewpoint of the selected character and renders the scene data.

[1384] Step 4:

[1385] The server sends the generated 360-degree VR data to the device.

[1386] Specific operation: The server transfers the generated VR data to the terminal via the network.

[1387] Input: 360-degree VR data.

[1388] Output: Sending VR data to the device.

[1389] Data processing: Compress large video data and convert it into an efficient streaming format.

[1390] Step 5:

[1391] The 360-degree VR data received by the device is displayed on the VR headset.

[1392] Specific operation: The terminal decompresses the received VR data and displays it on the VR headset (smartphone and VR headset).

[1393] Input: 360-degree VR data.

[1394] Output: The user's VR experience.

[1395] Data Computation: Rendering data into a format suitable for the display of a VR headset.

[1396] Step 6:

[1397] The device records the user's viewpoint movements and interactions during the VR experience.

[1398] Specific actions: The user uses the VR headset to explore the story and interact with specific objects and characters. The device records these actions in real time.

[1399] Input: User gaze movement and interaction data.

[1400] Output: Behavioral data.

[1401] Data collection: Data such as gaze position, gaze duration, and type of interaction are collected as logs.

[1402] Step 7:

[1403] The device transmits the collected behavioral data to a server.

[1404] Specific operation: The device compresses the behavioral data periodically or at the end of the experience and sends it to a server via the Internet.

[1405] Input: Behavioral data.

[1406] Output: Sending behavioral data to the server.

[1407] Data processing: Encoding raw behavioral data into a transmittable format.

[1408] Step 8:

[1409] The server uses AI to analyze behavioral data and optimize the next experience.

[1410] How it works: The server uses AI tools like TensorFlow to analyze the collected behavioral data, identify user interests and behavioral patterns, and use this information to fine-tune the next experience.

[1411] Input: Behavioral data.

[1412] Output: Analysis results and optimized experience data.

[1413] Data Computing: Analyzing behavioral data, identifying user interests and behavioral patterns, and applying optimization algorithms.

[1414] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1415] This invention is a system that allows users to experience a picture book story in a 360-degree VR environment and optimizes the next experience by recognizing and analyzing the user's emotions. This system stores the digital data of the story, generates a VR experience based on character selection and their viewpoint, and collects and analyzes the user's behavioral and emotional data.

[1416] System configuration

[1417] 1. Server

[1418] The server stores the digital data of the story, which includes images, text, and audio.

[1419] It identifies the selected character and generates 360-degree VR experience data based on their perspective.

[1420] The generated VR experience data is sent to the terminal, and the user's behavioral and emotional data is collected.

[1421] The collected data is analyzed using AI to optimize the next experience.

[1422] 2. Terminal

[1423] An interface is provided whereby the user can select a character using a digital pen.

[1424] It receives VR experience data sent from the server and provides it to the user via a VR headset for display.

[1425] Emotion data is collected in real time using user behavior data and an emotion engine and sent to a server.

[1426] 3. Users

[1427] The user selects a character in the picture book using a digital pen.

[1428] Put on a VR headset and experience the story from the perspective of a selected character.

[1429] It provides the system with information about viewpoint changes, interactions, and emotional changes during the experience.

[1430] Program processing explanation

[1431] Preserving digital data of picture books

[1432] The server receives the digital data of the picture book and stores it in separate parts: images, text, and audio, making it easier for users to access specific pages or scenes.

[1433] Identify the characters the user selects

[1434] When a user selects a character in a picture book using a digital pen, the device identifies that information and sends it to the server, which then acquires the character's viewpoint data based on the information received.

[1435] Generate 360-degree VR experience data based on the perspective of a selected character

[1436] The server uses a VR engine to generate 360-degree VR experience data from the perspective of a selected character, for example, creating video and simulation data for observing Momotaro's actions from the perspective of the ogre.

[1437] Display the generated VR experience data to the user

[1438] The device receives the VR experience data sent from the server and displays it to the user through the VR headset, allowing the user to experience the story world from the character's perspective.

[1439] Collect and analyze user behavioral and emotional data

[1440] The device collects data on the user's viewpoint, interactions, and emotional data collected through the emotion engine during the experience. This data is sent to a server and analyzed by AI. Based on the analysis results, the next experience is optimized based on the user's interests and emotions.

[1441] Specific examples

[1442] Take the story of Momotaro for example. The user selects the "demon" using a digital pen, and the server generates VR data from the demon's perspective and sends it to the device. The user then wears a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes (such as surprise or excitement) in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[1443] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[1444] The processing flow will be explained below.

[1445] Step 1:

[1446] The server receives the digital data of the picture book, which includes images, text, and audio. The server analyzes the data, splits it into pages, and stores them, making it easier for users to access specific pages or scenes.

[1447] Step 2:

[1448] The user touches the characters in the picture book using the AI ​​pen, which detects the touched position and sends that information to the device.

[1449] Step 3:

[1450] The device analyzes the received location data, identifies which character has been selected, and sends the identification result to the server.

[1451] Step 4:

[1452] The server acquires the character's viewpoint data based on the received character information, and then generates 360-degree VR experience data from the selected character's viewpoint.

[1453] Step 5:

[1454] The server sends the generated VR experience data to the device, which then forwards the received data to the VR headset.

[1455] Step 6:

[1456] Users wear a VR headset and explore the story world from the perspective of a selected character, with the ability to move the viewpoint and interact with the character.

[1457] Step 7:

[1458] The device tracks the user's gaze movements and interactions in real time and collects this behavioral data. It also uses an emotion engine to analyze the user's facial expressions and voice to collect emotional data. The collected data is periodically sent to a server.

[1459] Step 8:

[1460] The server uses AI to analyze the received user behavioral and emotional data, and based on the analysis results, it makes settings to individually optimize the user's next experience.

[1461] Step 9:

[1462] The next time the user accesses the system, the server generates and serves optimized experience data to the user, allowing the user to build on what they learned from their previous experience.

[1463] Example 2

[1464] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1465] In today's education and entertainment fields, new methods are needed to engage users and deepen their understanding. In particular, there is a lack of technology to individually optimize story experiences to achieve deep empathy and learning effects. Furthermore, there has been no system that can collect user emotional and behavioral data in real time and use it to optimize the next experience.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, and means for collecting user behavioral data and emotional data, analyzing them using AI, and optimizing the next experience. This allows the user to deeply understand the story and empathize with it through an individually optimized VR experience. Furthermore, real-time emotional data analysis allows the next experience to be optimized based on the user's interests and emotions, thereby providing more effective education and entertainment.

[1467] "Storing digital story data" means receiving picture books and other story content in digital format and storing it in separate images, text, and audio data.

[1468] "Identifying a character selected by a user" means recognizing and accurately identifying the selected information when a user selects a character in a picture book using a digital pen or other interface.

[1469] "Generating 360-degree VR experience data based on the viewpoint of a selected character" means generating data that allows the user to experience a virtual reality environment from the viewpoint of a character selected by the user in a 360-degree view.

[1470] "Displaying the generated VR experience data to the user" means sending the VR experience data generated by the server to the terminal and presenting it to the user through a display device such as a VR headset.

[1471] "Collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience" means collecting the user's viewpoint movements, interactions, and emotional changes during the VR experience in real time, analyzing this data using AI, and using the results to improve the next VR experience based on the user's interests and emotions.

[1472] "Means for selecting a character using a digital pen" refers to a digital input device that a user uses to select a character on a picture book, and to transmit that selection information to the system.

[1473] A "VR headset that allows you to experience stories in 360 degrees" is a head-mounted display device that allows users to experience a virtual reality environment in all directions.

[1474] MODE FOR CARRYING OUT THE INVENTION

[1475] System Configuration

[1476] This invention is a system that stores digital data of a story, generates a 360-degree VR experience based on the viewpoint of a character selected by the user, and optimizes the next experience by analyzing the user's emotions. Below is a detailed explanation of how to implement this system.

[1477] Hardware and software used

[1478] 1. Server

[1479] Servers use high performance data storage and computation hardware (e.g., data center servers).

[1480] The software used includes database management systems (e.g., MySQL, MongoDB) for managing digital data, VR engines (e.g., Unity, Unreal Engine) for generating VR experiences, and AI models (e.g., TensorFlow, PyTorch) for analyzing emotional data.

[1481] 2. Terminal

[1482] A terminal is a device for user interaction, typically a computer with graphics processing capabilities (e.g., a gaming PC), a digital pen, or a VR headset (e.g., Oculus Rift or HTC Vive).

[1483] The software used includes applications to provide a user interface (e.g., custom VR apps, interactive UI software), and emotion engines (e.g., Affectiva, Realeyes) to collect emotion data in real time.

[1484] 3. Users

[1485] Users use a digital pen to select characters from the picture book and then put on a VR headset to immerse themselves in the story.

[1486] The user moves their viewpoint and interacts with the system, providing emotional data to the system.

[1487] Natural language explanation of program processing

[1488] The server receives the digital data (images, text, and audio) from the picture book, divides it, and stores it in a database. When the user selects a character in the picture book using the digital pen, the device captures the selection information and sends it to the server. The server uses a VR engine to generate 360-degree VR experience data based on the viewpoint of the selected character and sends this data to the device. The device displays this data to the user via the VR headset. Behavioral and emotional data from the user's experience is collected in real time through the emotion engine and sent to the server. The server uses an AI model to analyze this data and optimize the next experience based on the user's interests and emotions.

[1489] Examples of concrete examples and prompts

[1490] As a concrete example, when using the story of Momotaro, if the user selects "Oni" with the digital pen, the server generates VR experience data from the Oni's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the Oni's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by an AI model to further optimize the next experience.

[1491] Example prompt sentence:

[1492] "Create a VR experience from the perspective of the ogre in the story of Momotaro, and generate a program that will analyze the user's emotional data and optimize the next experience."

[1493] This system allows users to gain deep understanding and empathy through individually optimized learning experiences, rather than a one-sided experience. The introduction of an emotion engine realizes an educational experience that is attuned to the user's emotions.

[1494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1495] Step 1:

[1496] The server receives the digital data of the story (images, text, audio). The input is the digital data of the story, and the output is the stored data. The server splits the received data into individual files and stores them in the appropriate directories using a database management system (e.g., MySQL, MongoDB). This allows users to efficiently access specific pages or scenes.

[1497] Step 2:

[1498] The user selects a character in a picture book using a digital pen. The input is the information of the character selected by the user, and the output is the selected character information. The device captures the digital pen input and sends the information to the server in JSON format.

[1499] Step 3:

[1500] The server receives information about the character selected by the user. The input is the selected character information, and the output is the character's viewpoint data. Based on the received information, the server obtains the character's viewpoint data and generates 360-degree VR experience data using a VR engine (e.g., Unity, Unreal Engine). Specifically, it generates simulation data and exports it as a 360-degree video.

[1501] Step 4:

[1502] The server sends the generated VR experience data to the device. The input is 360-degree VR experience data, and the output is data sent to the device. The device streams the received VR experience data to a VR headset (e.g., Oculus Rift, HTC Vive) to provide the experience to the user. The user can realistically experience the world of the story from the character's perspective.

[1503] Step 5:

[1504] The device collects in real time the user's viewpoint movements and interactions during the experience, as well as emotional data collected using an emotion engine (e.g., Affectiva, Realeyes). The input is the user's behavioral data and emotional data, and the output is the collected data. These data are sent to the server.

[1505] Step 6:

[1506] The server receives the user's behavioral and emotional data sent from the device. The input is the collected data, and the output is the analysis results. The server analyzes the data using an AI model (e.g., TensorFlow, PyTorch) and stores the results in a database. Based on the analysis results, the next VR experience is optimized based on the user's interests and emotions.

[1507] (Application example 2)

[1508] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Traditional picture book and story experiences are static and one-way, making it difficult for users to immerse themselves in the story and receive an individually optimized experience. Furthermore, there is a lack of means to personalize the next experience based on emotional and behavioral data, making it difficult to improve the user's learning effect and entertainment value.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing digital data of a story, means for identifying a character selected by a user, means for generating 360-degree VR experience data based on the viewpoint of the selected character, means for displaying the generated VR experience data to the user, means for collecting behavioral data and emotional data of the user, analyzing the data using AI, and optimizing the next experience, and means for individually optimizing the next VR experience based on the user's emotional data. This allows the user to feel highly immersed in the world of the story while enjoying a personalized experience based on the emotional and behavioral data.

[1511] "Digital narrative data" refers to data that stores the visual and auditory content (images, text, and audio) of a narrative in digital form.

[1512] A "user-selected character" is a character within a story from which the user can choose a perspective within the experience.

[1513] "360-degree VR experience data" is virtual reality data that allows you to visually experience a story from any direction from the perspective of a selected character.

[1514] "Displaying to the user" means providing the generated VR experience data visually and audibly through a device (such as a headset) used by the user.

[1515] "Behavioral data" refers to data about the user's movements during a VR experience, such as their viewpoint, interactions, and gestures.

[1516] "Emotional data" refers to data about the user's emotional state obtained from facial expressions, tone of voice, heart rate, etc. during the VR experience.

[1517] "Analysis with AI" means analyzing and interpreting collected behavioral and emotional data using artificial intelligence technology.

[1518] "Optimizing the next experience" means adjusting the next VR experience to be more appropriate and interesting for the user based on the user's previous behavioral and emotional data.

[1519] A "digital input device" is a device (e.g., a stylus, touchpad) that a user uses to select characters within a story.

[1520] A "viewing device" is a device (e.g., a VR headset, smart glasses) used by a user to view VR experience data.

[1521] MODE FOR CARRYING OUT THE INVENTION

[1522] This invention is a system that allows users to immerse themselves in a story world and provide an individually optimized experience. This system includes storing digital data of the story, generating a 360-degree VR experience based on character selection, collecting and analyzing user behavioral and emotional data, and optimizing the next experience.

[1523] System configuration

[1524] server

[1525] Preserving the story digitally

[1526] The server stores the visual and auditory content of the story (images, text, audio) in digital form, making it easy to access specific pages or scenes.

[1527] VR experience generation based on character selection

[1528] The system identifies the character selected by the user and generates 360-degree VR experience data based on that character's perspective, allowing the user to experience the story from the perspective of the selected character.

[1529] Collecting and analyzing behavioral and emotional data

[1530] The server collects user behavioral and emotional data and analyzes it using AI technology, optimizing the content of the next experience based on the user's interests and emotions.

[1531] Terminal

[1532] Character selection interface

[1533] The user selects a character in the story using a digital input device (e.g., a touch pen or a touchpad), and the terminal identifies this selection and transmits it to the server.

[1534] Displaying VR experience data

[1535] The device receives the VR experience data sent from the server and displays it to the user using a viewing device such as a VR headset or smart glasses.

[1536] Collecting behavioral and emotional data

[1537] The device collects behavioral data such as the user's gaze movement, interactions, and facial expressions. It also uses an emotion engine to collect the user's emotional data in real time and transmits it to the server.

[1538] User

[1539] Character Selection

[1540] The user selects a character in the picture book using a digital input device.

[1541] VR experience

[1542] Users put on a VR headset and experience the story from the perspective of a character of their choice.

[1543] Specific examples

[1544] Take the story of Momotaro for example. The user selects the "demon" using a digital input device, and the server generates VR data from the demon's perspective and sends it to the device. The user puts on a VR headset and experiences the story from the demon's perspective. During this time, the emotion engine detects and collects the user's viewpoint movements, interactions, and emotional changes in real time. The collected data is sent to the server and analyzed by AI. The next time the user uses the system, a more optimized experience is provided based on the emotional data, improving the user's learning effect.

[1545] Prompt Sentence Examples

[1546] You experience a story in a VR environment. First, choose a character. Then, enjoy the story from the perspective of the character you selected. Based on emotional and behavioral data from the experience, your next experience will be optimized to make it even more enjoyable.

[1547] If you experience any of the following emotions during the experience, you will enjoy the next optimized scene more.

[1548] surprise

[1549] joy

[1550] sorrow

[1551] anger

[1552] This system improves the user's immersion and learning effectiveness by individually optimizing the next VR experience based on the user's emotional and behavioral data.

[1553] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1554] Step 1:

[1555] The server stores the digital data of the story. This data includes images, text, and audio. The input is the digital file of the story, and the output is the stored data.

[1556] Step 2:

[1557] The terminal identifies the character selected by the user using a digital input device, with the input being the user's selection action and the output being the identified character information.

[1558] Step 3:

[1559] The server generates 360-degree VR experience data from the viewpoint of the selected character based on the identified character information. The input is character information, and the output is 360-degree VR data. For data processing, a VR engine is used to generate visual and audio data.

[1560] Step 4:

[1561] The device receives the VR experience data sent from the server and displays it to the user. The input is VR data, and the output is the display to the user through the VR headset. Specifically, the device uses VR display software.

[1562] Step 5:

[1563] The device collects user behavior data (eye movement and interactions). The input is the user's actions, and the output is the collected behavior data. The device uses the built-in gyro sensor and accelerometer to process the data.

[1564] Step 6:

[1565] It uses an emotion engine to collect user emotion data in real time. The input is the user's facial expressions and voice, and the output is the collected emotion data. Specific operations include analyzing data from the camera and microphone.

[1566] Step 7:

[1567] The terminal sends the collected behavioral data and emotional data to the server. The input is behavioral data and emotional data, and the output is data sent to the server.

[1568] Step 8:

[1569] The server analyzes the collected data using AI. The input is behavioral data and emotional data, and the output is the analysis results. Data is analyzed using a generative AI model.

[1570] Step 9:

[1571] The server optimizes the next VR experience based on the analysis results. The input is the analysis results, and the output is the optimized VR experience data. Specifically, it uses an algorithm to personalize the content of the next experience.

[1572] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1573] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1574] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1575] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1576] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1577] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1578] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1579] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1580] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1581] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1582] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1583] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1584] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1585] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1586] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1587] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1588] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1589] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1590] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1591] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1592] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1593] The following is further disclosed regarding the above embodiment.

[1594] (Claim 1)

[1595] [Means for storing digital data of stories,

[1596] [means for identifying a character selected by a user;

[1597] [Means for generating 360-degree VR experience data based on the viewpoint of a selected character;

[1598] [Means for displaying the generated VR experience data to the user;

[1599] [Means of collecting user behavior data, analyzing it with AI, and optimizing the next experience,

[1600] A system including:

[1601] (Claim 2)

[1602] The system of claim 1, further comprising means for a user to select a character using a digital pen.

[1603] (Claim 3)

[1604] [The system of claim 1, further comprising means including a VR headset for experiencing the story in 360 degrees from the perspective of the characters.

[1605] "Example 1"

[1606] (Claim 1)

[1607] [Means for storing digital data of stories,

[1608] [means for identifying a character selected by a user;

[1609] [means for generating virtual reality experience data based on the viewpoint of a selected character;

[1610] [means for displaying the generated virtual reality experience data to a user;

[1611] [Means of collecting user behavior data, analyzing it with artificial intelligence, and optimizing the next experience,

[1612] A system including:

[1613] (Claim 2)

[1614] The system of claim 1, further comprising means for a user to select a character using an input device.

[1615] (Claim 3)

[1616] [The system of claim 1, further comprising means for including a display device that allows the story to be experienced from the perspective of a character.

[1617] "Application Example 1"

[1618] (Claim 1)

[1619] [Means for storing digital data of stories,

[1620] [means for identifying a character selected by a user;

[1621] [Means for generating 360-degree VR experience data based on the viewpoint of a selected character;

[1622] [Means for displaying the generated VR experience data to the user;

[1623] [Means of collecting user behavior data, analyzing it with AI, and optimizing the next experience,

[1624] [A means for users to freely move their viewpoint within the story and interact with specific objects and characters,

[1625] A system including:

[1626] (Claim 2)

[1627] The system of claim 1, further comprising means for a user to select a character using a pen-type device.

[1628] (Claim 3)

[1629] [The system of claim 1, further comprising means for including a display device that allows a 360-degree experience of the story from the perspective of the characters.

[1630] "Example 2: Combining Emotion Engines"

[1631] (Claim 1)

[1632] [Means for storing digital data of stories,

[1633] [means for identifying a character selected by a user;

[1634] [Means for generating 360-degree VR experience data based on the viewpoint of a selected character;

[1635] [Means for displaying the generated VR experience data to the user;

[1636] [Means of collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience,

[1637] A system including:

[1638] (Claim 2)

[1639] The system of claim 1, further comprising means for a user to select a character using a digital pen.

[1640] (Claim 3)

[1641] [The system of claim 1, further comprising means for including a device for allowing a story to be experienced in 360 degrees from the perspective of a character.

[1642] "Application example 2 when combining emotion engines"

[1643] (Claim 1)

[1644] [Means for storing digital data of stories,

[1645] [means for identifying a character selected by a user;

[1646] [Means for generating 360-degree VR experience data based on the viewpoint of a selected character;

[1647] [Means for displaying the generated VR experience data to the user;

[1648] [Means of collecting user behavioral and emotional data, analyzing it with AI, and optimizing the next experience,

[1649] [Means to individually optimize the next VR experience based on the user's emotional data,

[1650] A system including:

[1651] (Claim 2)

[1652] The system of claim 1, further comprising: means for a user to select a character using a digital input device.

[1653] (Claim 3)

[1654] [The system of claim 1, further comprising means for including a viewing device that allows a 360-degree experience of the story from the perspective of a character. [Explanation of symbols]

[1655] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of storing digital data of the story, means for identifying a character selected by a user; means for generating 360-degree VR experience data based on the viewpoint of a selected character; A means for displaying the generated VR experience data to a user; Collecting user behavior data, analyzing it with AI, and optimizing the next experience. A system including:

2. The system of claim 1 further comprising means for a user to select a character using a digital pen.

3. 10. The system of claim 1, further comprising means for including a VR headset that allows a 360-degree experience of the story from the perspective of a character.

Images