system

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting past data and storing the collected data in a database; means for developing a future prediction model based on the collected data; means for creating a VR space based on a generated future scenario; means for dynamically generating an event in the VR space according to a user's operation and branching a story; and means for allowing the user to freely act in the VR space and have a conversation with another character.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While it is practically impossible to travel to the future in a time machine, people have a strong desire to visually confirm future events. However, current technology has not yet fully realized a method for experiencing the future in a virtual reality space using past data and predictive models. Furthermore, there is no system that allows users to act freely in a VR space and experience future predictions while enjoying conversations with characters from the future. There is a need to solve these issues and provide an environment where users can realistically experience future scenery and events and freely interact with them. [Means for solving the problem]

[0005] The present invention is a system that includes a means for collecting past data and storing it in a database, a means for developing a future prediction model based on the collected data, a means for creating a VR space based on the generated future scenario, a means for dynamically generating events in the VR space and branching the story in response to user operations, and a means for the user to freely act within the VR space and converse with other characters. This combines past data and a future prediction model to create a realistic VR space under specific future conditions, allowing the user to act freely within the virtual reality space and experience future events according to the scenario. Furthermore, by including a means for cleansing necessary information from past data and storing it in a database, and a means for using machine learning and deep learning technologies in developing the future prediction model, it is possible to provide more reliable future predictions and realistic virtual reality spaces.

[0006] "Historical data" refers to information accumulated over time, including information obtained from various sources such as historical events, academic research, economic indicators, and social statistics.

[0007] A "database" is a system for storing and managing data in an organized manner, and refers to a relational database or NoSQL database for storing past data and predictive information.

[0008] "Future prediction model" refers to a mathematical or statistical model that predicts future events or situations based on past data, and includes models constructed using machine learning and deep learning technologies.

[0009] "VR space" refers to a three-dimensional virtual environment created using virtual reality technology, in which users can interact through sight, hearing, and touch.

[0010] "User operations" refers to actions and inputs made by the user within the VR space, including choices that affect the branching of the scenario or the occurrence of events.

[0011] An "event" refers to a specific occurrence or action that occurs within a VR space, including those that are dynamically generated in response to user operations.

[0012] "Branching of the story" refers to the scenario unfolding in different directions depending on the user's choices and actions, and implies the existence of multiple scenarios.

[0013] "Character" refers to a virtual person or agent that interacts with the user in a VR space, including those controlled by AI.

[0014] "Cleansing" refers to the process of filling in missing values, removing outliers, and standardizing data formats to ensure collected data is accurate and consistent.

[0015] "Machine learning" refers to an artificial intelligence technology that learns patterns from data and makes predictions and classifications, and is particularly used to build future prediction models based on past data.

[0016] "Deep learning" refers to an advanced machine learning technology based on neural networks, which is used to learn complex patterns from large amounts of data and build highly accurate predictive models. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] Understood. The following is a description of the "Mode for carrying out the invention."

[0039] The present invention is a system for recreating future events in a virtual reality space (VR space), and is realized by the following program. This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, and means for the user to act freely in the VR space and converse with other characters.

[0040] Server Processing

[0041] Data collection and database construction

[0042] The server uses APIs and web scraping to collect historical data, including economic indicators, social statistics, scientific research, etc. The collected data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[0043] Development of future prediction models

[0044] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[0045] VR space generation

[0046] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[0047] Terminal handling

[0048] VR space rendering

[0049] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[0050] User operations

[0051] Access to VR space

[0052] Users put on a VR headset and controllers and access the VR space through the device. If it's their first time experiencing it, the system will display basic operation instructions and a usage guide.

[0053] Freedom of movement and conversation

[0054] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to AI characters. Depending on the user's choices, the story will unfold in different directions, allowing them to enjoy multiple scenarios.

[0055] Specific examples

[0056] For example, consider a scenario that predicts Tokyo in 2025 based on Tokyo's economic indicators, transportation data, and social statistics from the past five years. The server collects and cleans this data, and uses RNN to develop a model that predicts Tokyo in 2025. A video generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device. Users can access this VR space, explore new landmarks and transportation systems, and enjoy conversations with AI characters about life in the future. The scenario branches depending on the user's choices, allowing them to experience different futures.

[0057] The processing flow will be explained below.

[0058] Server Processing

[0059] Step 1: Data collection

[0060] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[0061] Step 2: Data cleansing

[0062] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[0063] Step 3: Store in the database

[0064] The server stores the cleansed data in a database.

[0065] Step 4: Preprocessing the data

[0066] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[0067] Step 5: Develop a future prediction model

[0068] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, for example, using RNNs and GANs.

[0069] Step 6: Evaluate and tune the model

[0070] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[0071] Step 7: Generate the scenario

[0072] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[0073] Step 8: Use video generation AI

[0074] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[0075] Step 9: Sending Data

[0076] The server transmits the generated VR space data to the terminal.

[0077] Terminal handling

[0078] Step 1: Receiving and analyzing data

[0079] The device analyzes the VR space data received from the server and prepares for rendering.

[0080] Step 2: Rendering the VR space

[0081] The device uses a rendering engine such as a GPU to render the VR space in real time.

[0082] Step 3: Providing a User Interface

[0083] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[0084] User operations

[0085] Step 1: Put on the VR equipment

[0086] The user puts on a VR headset and controllers and prepares to access the VR space.

[0087] Step 2: Access the VR space

[0088] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[0089] Step 3: Free movement and conversation

[0090] Users can freely move around in the VR space and interact with the objects and characters around them. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era.

[0091] Step 4: Branching scenarios

[0092] The story changes depending on the user's choices. For example, if the user visits a specific location, different events will occur depending on the choices made. This allows the user to experience multiple future scenarios.

[0093] Specific examples

[0094] Example: A scenario looking at Tokyo in the future in 2025

[0095] Server Processing

[0096] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[0097] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[0098] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[0099] Terminal handling

[0100] 1. The device analyzes the received VR space data and prepares for rendering.

[0101] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[0102] 3. Provide landing points and navigation tools as a user interface.

[0103] User operations

[0104] 1. The user puts on a VR headset and enters the VR space via the device.

[0105] 2. Freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[0106] 3. Talk to AI characters at specific locations and hear about new technologies and social trends. The scenario branches depending on the user's choices, allowing users to enjoy different futures.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] In conventional technologies, it has been difficult to effectively integrate a series of processes, from data collection and cleansing, to the development of future prediction models and the generation of virtual reality spaces, in systems that use past data to make future predictions. Furthermore, there has been a lack of systems that allow users to experience scenarios that change in real time while freely acting and interacting in a virtual space. This has meant that no environment has been provided in which users can realistically experience future events and dynamically change those scenarios.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for collecting past data, means for cleansing the collected data and storing it in a database, means for developing a future prediction model using machine learning and deep learning technologies, means for creating a virtual reality space based on a future scenario generated from the future prediction model, means for dynamically generating events in the virtual reality space in response to user operations and branching the story, and means for the user to act freely in the virtual reality space and converse with other characters. This enables the realization of a future prediction system in which users can realistically experience the future predicted from past data and can be interactively involved.

[0112] Define "data" in the following format:

[0113] "Historical data" refers to time-series information and numerical data such as economic indicators, social statistics, and scientific research.

[0114] "Means of collection" refers to methods of obtaining data from outside using technologies such as APIs and web scraping.

[0115] "Means of storing in a database" refers to the method of organizing collected data into a certain format and storing it in a database such as a relational database management system (RDBMS).

[0116] "Cleansing" refers to the process of improving data quality by preparing collected data through methods such as filling in missing values ​​and removing outliers.

[0117] "Machine learning and deep learning technologies" refer to technologies in which computers use large amounts of data to learn patterns and then use those patterns to make predictions and classifications.

[0118] A "future prediction model" refers to an algorithm or model that uses statistical methods and machine learning technology to generate future scenarios based on collected past data.

[0119] "Virtual reality space" refers to a three-dimensional computer-generated virtual environment that users can experience interactively.

[0120] "Dynamic generation" refers to a method of changing events and scenarios in real time in response to user operations and input.

[0121] "Means for branching the story" refers to a function that allows the scenario to unfold in different directions depending on the user's choices and actions.

[0122] "Means for free movement and conversation with other characters" refers to a method by which a user can freely move within a virtual reality space and interact and converse with characters.

[0123] MODE FOR CARRYING OUT THE INVENTION

[0124] The present invention relates to a system that predicts the future based on past data and allows users to experience the scenario in a virtual reality space (VR space). The system of the present invention is composed of components such as a server, a terminal, and a user, and an embodiment thereof will be described in detail below.

[0125] Server Processing

[0126] Data collection and database construction

[0127] The server uses APIs and web scraping to collect historical data. This data includes economic indicators, social statistics, scientific research, and more. Specifically, it uses APIs (e.g., APIs provided by international organizations) and web scraping technology (e.g., Python's BeautifulSoup library). The server organizes the collected data using a data cleansing tool such as Pandas and stores it in a relational database management system (e.g., MySQL). Missing values ​​are imputed and outliers are removed at this stage.

[0128] Development of future prediction models

[0129] The server uses TensorFlow or PyTorch to develop a future prediction model based on the cleansed data. For example, it trains a model that generates future scenarios from the data using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The trained model is then appropriately evaluated and tuned to build the optimal prediction model.

[0130] VR space generation

[0131] The server creates a VR space using a virtual reality engine (e.g., Unreal Engine or Unity) based on the scenario generated by the future prediction model. For example, it generates 3D models that reflect future cityscapes or new technological concepts, and includes visual, auditory, and tactile elements to add realism. The generated VR space data is then sent to the user's device via WebSocket.

[0132] Terminal handling

[0133] VR space rendering

[0134] The device renders the VR space data received from the server in real time. Specifically, it uses a high-speed rendering engine (e.g., NVIDIA RTX series GPU) to provide a realistic experience for the user. The device also uses Unity or Unreal Engine to provide a user interface, displaying menus and options that allow the user to operate within the VR space.

[0135] User operations

[0136] Access to VR space

[0137] Users put on a VR headset (such as Oculus Quest or HTC Vive) and controllers and access the VR space through their device. When using it for the first time, the system will provide basic operation instructions and a usage guide.

[0138] Freedom of movement and conversation

[0139] Users can move freely within the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to an AI character. Natural language processing technology (e.g., IBM Watson) is used to converse with the AI ​​character. Depending on the user's choices, the scenario will unfold in different directions, allowing users to enjoy multiple scenarios.

[0140] Specific examples

[0141] For example, consider a scenario in which a city's economic indicators, traffic data, and social statistics from the past five years are used to predict what it will be like in 2025. The server collects this data using the IMF's API and the World Bank's open data, cleansing it with Pandas, and developing a 2025 prediction model (RNN model) using TensorFlow. A 3D model of the city in 2025 is created using Unreal Engine and sent to the device via WebSocket. Users can access the VR space by wearing an Oculus Quest, exploring new landmarks and transportation systems, and enjoying conversations with AI characters powered by IBM Watson about life in the future. Furthermore, the scenario branches depending on the user's choices, allowing them to experience different futures.

[0142] Prompt Sentence Examples

[0143] By inputting prompts like the following into a generative AI model, we can generate future scenarios:

[0144] "You will create a virtual cityscape of 2025 and generate scenarios that include future urban landscapes, transportation systems, and new technological applications. You will also include new landmarks for users to explore and specific interaction scenes with AI characters."

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1: Data collection

[0147] Server Processing

[0148] The server uses APIs and web scraping to collect historical data, such as economic indicators, social statistics, and scientific research. Specifically, it uses APIs (e.g., public sector APIs) and web scraping techniques (e.g., Python's BeautifulSoup library).

[0149] Input: API endpoint or website URL

[0150] Output: Collected data (e.g., economic indicator data in JSON format)

[0151] Specific operation: The server sends a request to the API endpoint, saves the returned JSON data in internal storage, and performs web scraping to obtain the required information in text format.

[0152] Step 2: Data cleansing and database construction

[0153] Server Processing

[0154] The server cleanses the collected data and stores it in a database. Specifically, it completes missing values ​​and removes outliers, and stores the organized data in a relational database management system (e.g., MySQL).

[0155] Input: Collected data (output of step 1)

[0156] Output: Cleansed data (structured data)

[0157] How it works: The server uses the Pandas library to analyze the collected data, impute missing values ​​with the mean, and detect and remove outliers. It then uses SQL INSERT statements to store the cleansed data in a MySQL database.

[0158] Step 3: Develop a future prediction model

[0159] Server Processing

[0160] The server uses the cleansed data to develop future prediction models, specifically training RNNs and GANs using TensorFlow and PyTorch.

[0161] Input: Cleansed data (output of Step 2)

[0162] Output: A trained future prediction model

[0163] Specific operation: The server uses the TensorFlow library to build an RNN model, inputs the cleansed dataset into the model to train it, evaluates the model's performance after training, and saves the optimized prediction model.

[0164] Step 4: Creating the VR space

[0165] Server Processing

[0166] The server creates a VR space based on the scenario generated by the future prediction model, using a virtual reality engine (e.g., Unreal Engine or Unity).

[0167] Input: Forecast model and scenario data (output of Step 3)

[0168] Output: VR space data

[0169] How it works: The server inputs scenario data into Unreal Engine and uses 3D modeling tools to generate a model to recreate the future cityscape. The generated 3D model and other VR data are then sent to the user's device via WebSocket.

[0170] Step 5: Sending VR space data

[0171] Server Processing

[0172] The server sends the generated VR space data to the user's device.

[0173] Input: VR space data (output of step 4)

[0174] Output: Transmitted VR space data

[0175] Specific operation: The server opens a WebSocket and establishes a connection to the device. The generated VR space data is compressed and sent to the device via the WebSocket.

[0176] Step 6: Rendering the VR space

[0177] Terminal handling

[0178] The device renders the VR space data received from the server in real time, using a high-speed rendering engine.

[0179] Input: Received VR space data (output of step 5)

[0180] Output: Rendered VR scene

[0181] How it works: The device decompresses the received data and uses an NVIDIA RTX series GPU to perform high-speed rendering. The rendered scene is then displayed on the VR headset.

[0182] Step 7: Access the VR space

[0183] User operations

[0184] Users access the VR space by wearing a VR headset and controllers.

[0185] Input: Wearing and operating a VR headset and controllers

[0186] Output: Access to VR space

[0187] Specific operation: The user puts on the VR headset and launches the application on the device. When launched for the first time, a tutorial on basic operation is played.

[0188] Step 8: Free movement and conversation

[0189] User operations

[0190] Users can move freely within the VR space and interact with AI characters.

[0191] Input: User actions and commands

[0192] Output: Interactive VR experience

[0193] How it works: The user operates the VR controller to move around the virtual space, approach the AI ​​character, and start a conversation. A natural language processing engine is used to converse with the AI ​​character, and the scenario changes dynamically depending on the user's choices.

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] As autonomous vehicles become more widespread, simulations are required to predict future changes in the transportation environment and infrastructure and adapt to them. However, previous simulation technologies did not provide realistic experiences, making it difficult to properly understand the impact of actual autonomous vehicle operations and infrastructure. In addition, there was a lack of visualization methods that allowed users to experience specific future scenarios and intuitively understand them.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes a means for collecting past data and storing it in a database, a means for developing a future prediction model based on the collected data, and a means for creating a VR space based on the generated future scenario, allowing users to realistically experience future operation scenarios of autonomous vehicles in the VR space and specifically understand changes in traffic conditions and infrastructure.

[0199] "Past data" refers to information that shows events or records that have already occurred, and is detailed information based on time and space.

[0200] A "database" is a collection of information that is systematically managed so that past data can be efficiently stored, searched, and used.

[0201] A "future prediction model" is a mathematical model that uses statistical methods and machine learning techniques to predict future events and conditions based on past data.

[0202] "VR space" is a computer-generated virtual reality space, an interactive 3D environment in which users can feel as if they are actually there.

[0203] "User operation" refers to an input operation that a user performs on the system via an interface, and is an action that triggers an implemented function or operation.

[0204] An "event" refers to a specific occurrence or action that occurs within the VR space, and is an element that causes the story to progress or the environment to change.

[0205] "Branching the story" means generating a scenario with different developments depending on the user's choices and actions, and events in the VR space changing accordingly.

[0206] "Acting freely" means that the user can move around in the VR space, perform specific actions, and interact with characters and the environment of their own volition.

[0207] "Character" refers to any other virtual entity that the user controls or interacts with within the VR space, and can take the form of a person, animal, robot, or other entity.

[0208] An "autonomous vehicle" is a vehicle equipped with technology that allows it to recognize its surroundings and operate autonomously without the need for human driving.

[0209] "Future operation scenarios" are specific simulations of how autonomous vehicles will operate in the future, based on future prediction models.

[0210] "Traffic conditions" refers to comprehensive information that indicates the state of traffic flow, congestion, accidents, etc. at a specific time and location.

[0211] "Infrastructure" refers to basic public facilities for transportation and logistics, such as roads, bridges, tunnels, and signal systems.

[0212] "Experience" refers to the user actually operating within a virtual space and experiencing it realistically through their senses of sight, hearing, touch, etc.

[0213] This invention is a system that recreates operational scenarios for autonomous vehicles in future cities in a VR space. This system collects past data, builds a future prediction model, and implements a series of specific steps to create a VR space based on the generated future scenario. The details are described below.

[0214] Server Processing

[0215] Data collection and database construction

[0216] The server collects historical traffic data, accident rates, data on new infrastructure projects, and more. This data is collected using APIs and web scraping. The collected data undergoes a cleansing process, including filling in missing values ​​and removing outliers, before being stored in a database. Specifically, the data is shaped and cleansed using Python and the Pandas library.

[0217] Development of future prediction models

[0218] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it uses a recurrent neural network (LSTM) to train a model that generates future scenarios from past traffic data. The TensorFlow and Keras libraries are used for this process.

[0219] VR space generation

[0220] Based on the generated future prediction scenario, a VR space is created using image generation AI. This uses libraries such as OpenVR. The server creates 3D models of future cityscapes and new infrastructure, generating realistic VR space data that includes visual, auditory, and tactile elements. This data is then sent to the user's device.

[0221] Terminal handling

[0222] VR space rendering

[0223] The device renders the VR space in real time based on the received VR space data. A high-speed rendering engine such as a GPU is used to ensure the user has a realistic experience. The device also provides an interface, displaying menus and options for the user to operate within the VR space.

[0224] User operations

[0225] Access to VR space

[0226] Users access the VR space using a head-mounted display (HMD) and controllers. The system displays basic operation instructions and usage guides.

[0227] Freedom of movement and conversation

[0228] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can drive through a futuristic city and explore predicted traffic conditions and new infrastructure projects. Depending on the user's choices, the story branches, allowing them to experience different future scenarios.

[0229] Specific examples

[0230] For example, to simulate traffic conditions in Tokyo in 2025, we collect past traffic data, accident data, and information on new infrastructure projects, and then use LSTM to predict future scenarios. Based on the generated scenarios, we create a 3D model of Tokyo in 2025 in a VR space, and users can explore the space through a VR headset.

[0231] Prompt Sentence Examples

[0232] "Create a VR application that simulates traffic conditions in Tokyo in 2025. Explain how you can use past traffic data to predict the future and recreate that data in a VR space."

[0233] In this way, a system is created that allows users to intuitively understand specific traffic conditions and infrastructure changes by experiencing future operating scenarios for autonomous vehicles in a VR space.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] The server collects historical traffic data, accident rates, data on new infrastructure projects, etc. using APIs and web scraping. Before storing the collected data in a database, it performs cleansing processes such as filling in missing values ​​and removing outliers. The input data is raw traffic data, accident data, and infrastructure data, and the output is cleansed data. Specific operations use Python and the Pandas library.

[0237] Step 2:

[0238] The server builds a future prediction model based on the cleansed data. This process uses machine learning algorithms such as recurrent neural networks (LSTM). The input is the cleansed historical data, and the output is future traffic scenarios. The model is trained using the TensorFlow and Keras libraries. Specifically, the data is reformatted and an LSTM model is trained.

[0239] Step 3:

[0240] The server creates a VR space based on a scenario generated from a future prediction model. This process uses image generation AI and the OpenVR library. The input is future scenario data, and the output is the completed VR space data. Specifically, it generates a 3D city model and creates realistic data that includes visual, auditory, and tactile elements.

[0241] Step 4:

[0242] The device renders the VR space data received from the server in real time. The input is the VR space data sent from the server, and the output is the VR space experienced by the user through the HMD. Specifically, it uses a high-speed rendering engine (GPU) to provide a user interface.

[0243] Step 5:

[0244] Users access the VR space using a head-mounted display (HMD) and controllers. When a user experiences the system for the first time, the system displays basic operating instructions and a usage guide. The input is an operation instruction from the user, and the output is the system's response based on that instruction. Specific operations include initial setup of the HMD and controllers and display of the usage guide.

[0245] Step 6:

[0246] Users can move freely within the VR space and interact with the surrounding environment and characters. Input is the user's movement and control instructions, and output is changes to the environment within the VR space and the character's reactions. Specific actions include exploring the VR space, conversing with AI characters, and branching storylines.

[0247] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0248] The present invention is a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions, and is realized by the following program: This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely within the VR space and converse with other characters, and an emotion engine that recognizes the user's emotions.

[0249] Server Processing

[0250] Data collection and database construction

[0251] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping. This data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[0252] Development of future prediction models

[0253] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[0254] VR space generation

[0255] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[0256] Terminal handling

[0257] VR space rendering

[0258] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[0259] emotion recognition

[0260] The device uses an emotion engine to recognize the user's emotions. This information is input into the system as the user's current emotional state.

[0261] User operations

[0262] Wearing VR equipment

[0263] The user puts on a VR headset and controllers and prepares to access the VR space.

[0264] Access to VR space

[0265] Users enter the VR space through a device. If it is their first time experiencing it, the system will display basic operating instructions and a usage guide.

[0266] Freedom of movement and conversation

[0267] Users can freely move around in the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user makes a surprised expression, the surrounding characters will sense the user's surprise and respond accordingly.

[0268] Branching scenarios

[0269] The story changes depending on the user's choices. The user's emotional state also affects the branching of the scenario. For example, if the user has an inquisitive expression, a new event will occur based on that emotion. This allows the user to experience multiple future scenarios.

[0270] Specific examples

[0271] For example, consider a scenario in which Tokyo in 2025 is predicted based on the past five years' worth of economic indicators, traffic data, and social statistics. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. Image generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device.

[0272] Users put on a VR headset and enter the VR space via their device, freely exploring Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system for 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive experience of the future.

[0273] The processing flow will be explained below.

[0274] Server Processing

[0275] Step 1: Data collection

[0276] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[0277] Step 2: Data cleansing

[0278] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[0279] Step 3: Store in the database

[0280] The server stores the cleansed data in a database.

[0281] Step 4: Preprocessing the data

[0282] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[0283] Step 5: Develop a future prediction model

[0284] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs).

[0285] Step 6: Evaluate and tune the model

[0286] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[0287] Step 7: Generate the scenario

[0288] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[0289] Step 8: Use video generation AI

[0290] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[0291] Step 9: Sending Data

[0292] The server transmits the generated VR space data to the terminal.

[0293] Terminal handling

[0294] Step 1: Receiving and analyzing data

[0295] The device analyzes the VR space data received from the server and prepares for rendering.

[0296] Step 2: Rendering the VR space

[0297] The device uses a rendering engine such as a GPU to render the VR space in real time.

[0298] Step 3: Providing a User Interface

[0299] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[0300] Step 4: Start Emotion Recognition

[0301] The device uses devices such as a camera, microphone, and biosensor to analyze the user's facial expressions, tone of voice, heart rate, etc. in real time, and an emotion engine determines the user's emotional state.

[0302] User operations

[0303] Step 1: Put on the VR equipment

[0304] The user puts on a VR headset and controllers and prepares to access the VR space.

[0305] Step 2: Access the VR space

[0306] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[0307] Step 3: Free movement and conversation

[0308] Users can move freely within the VR space and interact with surrounding objects and characters. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era. The emotion engine recognizes the user's emotions, and the characters respond accordingly.

[0309] Step 4: Branching scenarios

[0310] The story changes depending on the user's choices and emotional state. For example, if the user is surprised or excited, new events are dynamically generated based on that emotion. The user can experience multiple future scenarios based on different emotions.

[0311] Specific examples

[0312] Example: A scenario looking at Tokyo in the future in 2025

[0313] Server Processing

[0314] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[0315] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[0316] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[0317] Terminal handling

[0318] 1. The device analyzes the received VR space data and prepares for rendering.

[0319] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[0320] 3. Provide landing points and navigation tools for the user interface.

[0321] 4. The device uses a camera and microphone to recognize the user's facial expressions and tone of voice, and the emotion engine analyzes this data to determine the user's emotions.

[0322] User operations

[0323] 1. The user puts on a VR headset and enters the VR space via the device.

[0324] 2. Users can freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[0325] 3. If the user expresses excitement or surprise, the emotion engine will recognize that emotion and the character in VR will begin explaining, "Are you surprised? This is the latest transportation system of 2025." In this way, interactions based on the user's emotions are realized.

[0326] 4. Depending on the user's emotional state, the scenario branches and new events occur. For example, if a user visits a particular location in a curious state, more detailed information or other new events will be provided.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Currently, there is no technology that can simulate future events in a virtual reality space and dynamically change interactions based on the user's emotions. Current systems lack the ability to simulate events using future prediction models or to recognize the user's emotions and change interactions accordingly, so new technologies are needed to provide a more realistic and immersive experience.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a virtual reality space based on the generated future scenario. This enables advanced simulations using the future prediction model. The server also includes means for dynamically generating events in the virtual reality space in response to user operations and branching the story, means for allowing the user to freely act in the virtual reality space and converse with other characters, an emotion engine for recognizing the user's emotions, and means for dynamically changing the reactions of events and characters in the virtual reality space based on the emotion recognition. This realizes interactions tailored to the user's emotions, enabling a more realistic and immersive experience.

[0331] "Historical data" refers to information or records collected historically, including statistical information such as economic indicators, traffic data, and social statistics within a specific period of time.

[0332] A "database" is a collection of information that is organized and stored to efficiently manage collected data and make it easy to search and retrieve.

[0333] A "future forecasting model" is a mathematical or machine learning algorithm designed to predict future events or conditions based on past data.

[0334] A "virtual reality space" is a primarily three-dimensional virtual environment generated using computer technology, which allows users to experience it realistically through their senses of sight, hearing, and touch.

[0335] "User operation" refers to the inputs and actions that a user makes within a virtual reality space using VR equipment or a controller.

[0336] An "event" refers to an occurrence or action that occurs within a virtual reality space.

[0337] "Branching of the story" refers to the function in which the scenario changes depending on the user's operations and choices.

[0338] "Character" refers to a person, creature, or other entity that appears in a virtual reality space.

[0339] An "emotion engine" is a technology for recognizing a user's emotional state, and includes facial recognition, voice analysis, and analysis of biometric sensor data.

[0340] "Interaction" refers to a two-way exchange between a user and a virtual reality space and the characters present within it.

[0341] The present invention provides a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions. This system is realized by the following means.

[0342] The server first collects historical data through APIs or web scraping. The collected data ranges from economic indicators, traffic data, and social statistics. This data collection uses hardware and software such as data acquisition interfaces and analysis programs. For example, Google Analytics API or general web scraping tools are used. The data is then cleansed by completing processes such as filling in missing values ​​and removing outliers, and the organized data is stored in a database. A general SQL database is often used for database management.

[0343] The server then uses machine learning and deep learning techniques to develop a future prediction model based on the cleansed data. For example, it uses recurrent neural networks (RNNs) and generative artificial neural networks (GANs). To do this, it builds a model using libraries such as Keras and TensorFlow and trains it using the training data. After training is complete, it evaluates the model using test data and tunes its hyperparameters to improve accuracy.

[0344] Based on the scenarios generated by the future prediction model, the server uses image generation AI to create a virtual reality space. For example, it uses Unreal Engine or Unity to create 3D models that reflect future cityscapes and new technology concepts. This VR space data is designed to include visual, auditory, and tactile elements. The generated VR space data is then sent to the user's device. Data is transmitted using the HTTP protocol or WebSocket.

[0345] The device performs real-time rendering based on the VR space data received from the server. For example, a device equipped with an NVIDIA GPU can use Unity or Unreal Engine to draw high-quality images. The device also provides a user interface, displaying menus and options that allow the user to operate within the VR space.

[0346] The user's emotions are recognized using an emotion engine built into the device. Emotion recognition is achieved by facial recognition, voice analysis, and biometric sensor data analysis. For example, facial expressions are recognized using a camera, and voice analysis is performed using a microphone. Biometric sensor data such as heart rate and electrodermal activity are also analyzed. This information is input into the system as the user's current emotional state and sent to the server.

[0347] The user then accesses the VR space by putting on a VR headset and controllers. For example, they start the virtual reality experience using an Oculus Quest 2. First-time users are shown instructions on how to operate the VR space and a usage guide. Within the virtual reality space, users can move freely and converse with other characters. Furthermore, events and character reactions within the VR space change dynamically based on emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will respond accordingly.

[0348] As a specific example, it is possible to predict future scenarios based on Tokyo's economic indicators, traffic data, and social statistics from the past five years, and recreate Tokyo in 2025. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. It then uses image generation AI to create a 3D model of Tokyo in 2025, and sends the VR space data to the device.

[0349] Users put on a VR headset and enter the VR space via their device, where they can freely explore Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system in 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive future experience.

[0350] An example of a prompt is: "Tell me the steps to develop a future prediction model to create a VR space of Tokyo in 2025 based on Tokyo's economic indicators, traffic data, and social statistics for the past five years."

[0351] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0352] Step 1: Collect historical data

[0353] The server uses APIs and web scraping tools to collect data such as historical economic indicators, traffic data, and social statistics. For example, it uses Google Analytics APIs and web scraping tools to obtain relevant data. The input is a data collection API or a web page URL. The output is raw historical data.

[0354] Step 2: Data cleansing

[0355] The server cleanses the collected data. This process includes imputing missing values ​​and removing outliers. For example, it uses the Python Pandas library to impute missing values ​​using the median and detect and remove outliers using Z-scores. The input is the raw historical data. The output is the cleansed data.

[0356] Step 3: Data storage

[0357] The server stores the cleansed data in a database. A general SQL database is used to efficiently manage data. The input is the cleansed data. The output is the data stored in the database.

[0358] Step 4: Develop a future prediction model

[0359] The server uses machine learning and deep learning techniques to develop a future prediction model based on the data stored in the database. For example, it uses the Keras or TensorFlow library to train a recurrent neural network (RNN) or a generative artificial neural network (GAN). The input is the data stored in the database. The output is the trained future prediction model.

[0360] Step 5: Generate future scenarios

[0361] The server generates future scenarios using a trained future prediction model. For example, it uses an RNN to predict the future economic and social conditions of Tokyo and generates specific scenarios. The input is the trained future prediction model. The output is the future scenario.

[0362] Step 6: Creating the VR space

[0363] The server uses image generation AI to create a VR space based on the generated future scenario. For example, it uses Unreal Engine or Unity to create 3D modeling of a future cityscape and creates VR content that includes visual, auditory, and tactile elements. The input is the future scenario. The output is the generated VR space data.

[0364] Step 7: Streaming VR space data

[0365] The server compresses the generated VR space data and sends it to the device. The data is transferred in real time via HTTP or WebSocket. The input is the generated VR space data. The output is the VR space data sent to the device.

[0366] Step 8: Rendering the VR space

[0367] The device performs real-time rendering based on the VR space data received from the server. Using Unity or Unreal Engine, high-quality images are drawn on devices equipped with NVIDIA GPUs. The device also provides a user interface, displaying menus and options for users to operate within the VR space. The input is the VR space data sent to the device. The output is the rendered VR space.

[0368] Step 9: Emotion Recognition

[0369] The device recognizes the user's emotions using a built-in emotion engine. It uses a camera to recognize facial expressions and a microphone to perform voice analysis. It also analyzes biosensor data such as heart rate and electrodermal activity. The inputs are the user's facial image, voice, and biosensor data. The output is the emotion recognition results.

[0370] Step 10: User's VR experience

[0371] Users access the VR space by wearing a VR headset and controllers. For example, they start a virtual reality experience using Oculus Quest 2. First-time users are shown instructions on how to use the device and a usage guide. The input is the state of wearing the VR headset and controllers. The output is the state of access to the VR space.

[0372] Step 11: Interaction and Scenario Branching

[0373] The device dynamically changes events and character reactions in the VR space based on the user's emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will react accordingly. The input is the emotion recognition results. The output is emotion-based interactions and scenario branching.

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional virtual reality (VR) systems have been unable to predict and recreate future events or dynamically change interactions based on the user's emotions. This has made it difficult to provide interactive VR experiences that allow users to have a more immersive experience. Furthermore, they have been unable to provide personalized interactions based on emotions, such as a futuristic shopping experience. Therefore, there has been a demand for improving the quality of the user experience and providing a more realistic and personal VR space.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a VR space based on the generated future scenario. This makes it possible to recreate future events in a virtual reality space. The server also includes means for dynamically generating events in the VR space and branching the story in response to user operations, means for the user to freely act and converse with other characters in the VR space, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions. This improves the quality of the user experience and makes it possible to provide personalized interactions based on emotions, such as future shopping experiences.

[0378] "Past data" refers to existing information such as economic indicators, social statistics, and scientific research, and serves as the basic data for predicting the future.

[0379] A "database" is a collection of information that stores collected past data in an orderly manner so that it can be used efficiently later.

[0380] A "future prediction model" is a predictive algorithm that uses machine learning and deep learning technologies to generate future scenarios based on collected past data.

[0381] "VR space" is a three-dimensional space generated on a computer using virtual reality technology, allowing users to experience a virtual environment that is different from the physical world.

[0382] "User operation" refers to the actions and choices the user makes using controllers and gestures within the VR space.

[0383] An "event" refers to a specific occurrence or action that occurs within the VR space, and changes dynamically depending on the user's actions and emotions.

[0384] "Branching of the story" means that the scenario in the VR space changes in different directions depending on the user's choices and actions.

[0385] "Character" refers to a virtual person or creature that interacts with the user in the VR space.

[0386] "Emotion recognition" refers to the technology of analyzing a user's facial expressions, voice, and biometric sensor data to identify their current emotional state.

[0387] "Interaction" refers to the interaction between the user and the environment and characters in the VR space, which changes dynamically based on emotion recognition.

[0388] "Guide" is a function that provides information and instructions to the user using voice and text.

[0389] "Additional information" refers to additional information or explanations provided based on the user's emotions or actions.

[0390] This invention is a system that recreates future events in a virtual reality space (VR space), recognizes a user's emotions, and dynamically changes interactions based on those emotions. The invention includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely in the VR space and converse with other characters, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions.

[0391] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping, cleansing it by filling in missing values ​​and removing outliers, and then stores it in a database. Next, based on this cleansed data, a future prediction model is developed using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The model is evaluated and tuned to create the optimal future prediction model. Based on the generated future scenario, a VR space is created using video generation AI. This VR space includes visual, auditory, and tactile elements.

[0392] The user device performs real-time rendering based on the VR space data received from the server, providing a realistic experience to the user wearing the VR headset. The device uses an emotion engine to recognize the user's emotions through facial recognition, voice analysis, and biometric sensor data analysis. This information is input into the system as the user's current emotional state.

[0393] Users put on a VR headset and controllers and access the VR space. They can move freely within the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, such as surprise or excitement, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user is surprised by a futuristic transportation system, a character in the VR space will provide guidance by explaining, "Are you surprised? This is the latest transportation system of the future."

[0394] As a concrete example, consider a scenario in which the future of a city is predicted based on the city's economic indicators, traffic data, and social statistics from the past five years. The server collects and cleans the data, and uses RNN to develop a model to predict the future city. A video generation AI is used to create a 3D model of the future city, and the VR space data is sent to the device. The user puts on a VR headset and enters the VR space via the device, freely exploring the future city. The emotion engine recognizes the user's emotions, and the interaction changes accordingly.

[0395] An example of a prompt sentence is, "When the user expresses surprise at a particular product in the future shopping mall, please provide guidance or additional information based on that emotion." In this way, the present invention improves the quality of the user experience and provides a more immersive future experience.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping. The input for the collection is multiple public databases and websites, and the data is obtained using these. Then, cleansing processes such as imputing missing values ​​and removing outliers are performed on the data, and the cleansed data is stored in a database. The output is the cleansed historical data.

[0399] Step 2:

[0400] The server uses the cleansed historical data stored in the database to develop a future prediction model. It takes the cleansed data as input and trains the model using machine learning algorithms such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs). It processes the data by performing feature engineering and preprocessing, converting it into a suitable format. The output is a trained future prediction model.

[0401] Step 3:

[0402] The server generates future scenarios based on the trained future prediction model. It takes the future prediction model and actual data as input, runs the prediction algorithm to simulate future events, and outputs future scenario data.

[0403] Step 4:

[0404] The server creates a VR space using image generation AI based on a future scenario. It takes future scenario data as input and generates the VR space using 3D modeling and image rendering software. Specific operations include generating 3D models, applying textures, and adding sound effects. The output is the generated VR space data.

[0405] Step 5:

[0406] The device receives VR space data from the server and renders it in real time. It takes the VR space data from the server as input and uses the GPU to perform rendering processing. Specific operations include drawing 3D graphics, playing audio, and generating haptic feedback. The output is a real-time VR view provided to the user.

[0407] Step 6:

[0408] Users access the VR space using a VR headset and controllers and act freely. The system receives user operations and motion inputs as input, and reflects those movements within the VR space. Specific movements include movement, item selection, and character interaction. The output is a change within the VR space based on user operations.

[0409] Step 7:

[0410] The device uses an emotion engine to recognize the user's emotions. User data is acquired as input from the camera, microphone, and biometric sensors, and analyzed using emotion recognition algorithms. Data processing involves analyzing facial expressions through image processing, analyzing voice, and interpreting biometric data. The output is the recognized emotional state of the user.

[0411] Step 8:

[0412] The device dynamically changes interactions in the VR space based on the recognized emotions. It takes the user's emotional state as input and executes an interaction update algorithm. Specific actions include changing the character's reaction, generating new events, and displaying guides or additional information. The output is the dynamically changed interaction.

[0413] Step 9:

[0414] The device provides guidance and additional information according to the user's emotions. It takes the recognized emotional state and current scenario information as input and generates appropriate guidance messages and additional information. Specific operations include generating audio guidance, displaying text guidance, and displaying additional information panels. The output is the guidance and additional information provided to the user.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] Understood. The following is a description of the "Mode for carrying out the invention."

[0432] The present invention is a system for recreating future events in a virtual reality space (VR space), and is realized by the following program. This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, and means for the user to act freely in the VR space and converse with other characters.

[0433] Server Processing

[0434] Data collection and database construction

[0435] The server uses APIs and web scraping to collect historical data, including economic indicators, social statistics, scientific research, etc. The collected data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[0436] Development of future prediction models

[0437] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[0438] VR space generation

[0439] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[0440] Terminal handling

[0441] VR space rendering

[0442] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[0443] User operations

[0444] Access to VR space

[0445] Users put on a VR headset and controllers and access the VR space through the device. If it's their first time experiencing it, the system will display basic operation instructions and a usage guide.

[0446] Freedom of movement and conversation

[0447] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to AI characters. Depending on the user's choices, the story will unfold in different directions, allowing them to enjoy multiple scenarios.

[0448] Specific examples

[0449] For example, consider a scenario that predicts Tokyo in 2025 based on Tokyo's economic indicators, transportation data, and social statistics from the past five years. The server collects and cleans this data, and uses RNN to develop a model that predicts Tokyo in 2025. A video generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device. Users can access this VR space, explore new landmarks and transportation systems, and enjoy conversations with AI characters about life in the future. The scenario branches depending on the user's choices, allowing them to experience different futures.

[0450] The processing flow will be explained below.

[0451] Server Processing

[0452] Step 1: Data collection

[0453] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[0454] Step 2: Data cleansing

[0455] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[0456] Step 3: Store in the database

[0457] The server stores the cleansed data in a database.

[0458] Step 4: Preprocessing the data

[0459] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[0460] Step 5: Develop a future prediction model

[0461] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, for example, using RNNs and GANs.

[0462] Step 6: Evaluate and tune the model

[0463] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[0464] Step 7: Generate the scenario

[0465] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[0466] Step 8: Use video generation AI

[0467] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[0468] Step 9: Sending Data

[0469] The server transmits the generated VR space data to the terminal.

[0470] Terminal handling

[0471] Step 1: Receiving and analyzing data

[0472] The device analyzes the VR space data received from the server and prepares for rendering.

[0473] Step 2: Rendering the VR space

[0474] The device uses a rendering engine such as a GPU to render the VR space in real time.

[0475] Step 3: Providing a User Interface

[0476] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[0477] User operations

[0478] Step 1: Put on the VR equipment

[0479] The user puts on a VR headset and controllers and prepares to access the VR space.

[0480] Step 2: Access the VR space

[0481] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[0482] Step 3: Free movement and conversation

[0483] Users can freely move around in the VR space and interact with the objects and characters around them. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era.

[0484] Step 4: Branching scenarios

[0485] The story changes depending on the user's choices. For example, if the user visits a specific location, different events will occur depending on the choices made. This allows the user to experience multiple future scenarios.

[0486] Specific examples

[0487] Example: A scenario looking at Tokyo in the future in 2025

[0488] Server Processing

[0489] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[0490] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[0491] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[0492] Terminal handling

[0493] 1. The device analyzes the received VR space data and prepares for rendering.

[0494] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[0495] 3. Provide landing points and navigation tools as a user interface.

[0496] User operations

[0497] 1. The user puts on a VR headset and enters the VR space via the device.

[0498] 2. Freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[0499] 3. Talk to AI characters at specific locations and hear about new technologies and social trends. The scenario branches depending on the user's choices, allowing users to enjoy different futures.

[0500] Example 1

[0501] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0502] In conventional technologies, it has been difficult to effectively integrate a series of processes, from data collection and cleansing, to the development of future prediction models and the generation of virtual reality spaces, in systems that use past data to make future predictions. Furthermore, there has been a lack of systems that allow users to experience scenarios that change in real time while freely acting and interacting in a virtual space. This has meant that no environment has been provided in which users can realistically experience future events and dynamically change those scenarios.

[0503] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0504] In this invention, the server includes means for collecting past data, means for cleansing the collected data and storing it in a database, means for developing a future prediction model using machine learning and deep learning technologies, means for creating a virtual reality space based on a future scenario generated from the future prediction model, means for dynamically generating events in the virtual reality space in response to user operations and branching the story, and means for the user to act freely in the virtual reality space and converse with other characters. This enables the realization of a future prediction system in which users can realistically experience the future predicted from past data and can be interactively involved.

[0505] Define "data" in the following format:

[0506] "Historical data" refers to time-series information and numerical data such as economic indicators, social statistics, and scientific research.

[0507] "Means of collection" refers to methods of obtaining data from outside using technologies such as APIs and web scraping.

[0508] "Means of storing in a database" refers to the method of organizing collected data into a certain format and storing it in a database such as a relational database management system (RDBMS).

[0509] "Cleansing" refers to the process of improving data quality by preparing collected data through methods such as filling in missing values ​​and removing outliers.

[0510] "Machine learning and deep learning technologies" refer to technologies in which computers use large amounts of data to learn patterns and then use those patterns to make predictions and classifications.

[0511] A "future prediction model" refers to an algorithm or model that uses statistical methods and machine learning technology to generate future scenarios based on collected past data.

[0512] "Virtual reality space" refers to a three-dimensional computer-generated virtual environment that users can experience interactively.

[0513] "Dynamic generation" refers to a method of changing events and scenarios in real time in response to user operations and input.

[0514] "Means for branching the story" refers to a function that allows the scenario to unfold in different directions depending on the user's choices and actions.

[0515] "Means for free movement and conversation with other characters" refers to a method by which a user can freely move within a virtual reality space and interact and converse with characters.

[0516] MODE FOR CARRYING OUT THE INVENTION

[0517] The present invention relates to a system that predicts the future based on past data and allows users to experience the scenario in a virtual reality space (VR space). The system of the present invention is composed of components such as a server, a terminal, and a user, and an embodiment thereof will be described in detail below.

[0518] Server Processing

[0519] Data collection and database construction

[0520] The server uses APIs and web scraping to collect historical data. This data includes economic indicators, social statistics, scientific research, and more. Specifically, it uses APIs (e.g., APIs provided by international organizations) and web scraping technology (e.g., Python's BeautifulSoup library). The server organizes the collected data using a data cleansing tool such as Pandas and stores it in a relational database management system (e.g., MySQL). Missing values ​​are imputed and outliers are removed at this stage.

[0521] Development of future prediction models

[0522] The server uses TensorFlow or PyTorch to develop a future prediction model based on the cleansed data. For example, it trains a model that generates future scenarios from the data using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The trained model is then appropriately evaluated and tuned to build the optimal prediction model.

[0523] VR space generation

[0524] The server creates a VR space using a virtual reality engine (e.g., Unreal Engine or Unity) based on the scenario generated by the future prediction model. For example, it generates 3D models that reflect future cityscapes or new technological concepts, and includes visual, auditory, and tactile elements to add realism. The generated VR space data is then sent to the user's device via WebSocket.

[0525] Terminal handling

[0526] VR space rendering

[0527] The device renders the VR space data received from the server in real time. Specifically, it uses a high-speed rendering engine (e.g., NVIDIA RTX series GPU) to provide a realistic experience for the user. The device also uses Unity or Unreal Engine to provide a user interface, displaying menus and options that allow the user to operate within the VR space.

[0528] User operations

[0529] Access to VR space

[0530] Users put on a VR headset (such as Oculus Quest or HTC Vive) and controllers and access the VR space through their device. When using it for the first time, the system will provide basic operation instructions and a usage guide.

[0531] Freedom of movement and conversation

[0532] Users can move freely within the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to an AI character. Natural language processing technology (e.g., IBM Watson) is used to converse with the AI ​​character. Depending on the user's choices, the scenario will unfold in different directions, allowing users to enjoy multiple scenarios.

[0533] Specific examples

[0534] For example, consider a scenario in which a city's economic indicators, traffic data, and social statistics from the past five years are used to predict what it will be like in 2025. The server collects this data using the IMF's API and the World Bank's open data, cleansing it with Pandas, and developing a 2025 prediction model (RNN model) using TensorFlow. A 3D model of the city in 2025 is created using Unreal Engine and sent to the device via WebSocket. Users can access the VR space by wearing an Oculus Quest, exploring new landmarks and transportation systems, and enjoying conversations with AI characters powered by IBM Watson about life in the future. Furthermore, the scenario branches depending on the user's choices, allowing them to experience different futures.

[0535] Prompt Sentence Examples

[0536] By inputting prompts like the following into a generative AI model, we can generate future scenarios:

[0537] "You will create a virtual cityscape of 2025 and generate scenarios that include future urban landscapes, transportation systems, and new technological applications. You will also include new landmarks for users to explore and specific interaction scenes with AI characters."

[0538] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0539] Step 1: Data collection

[0540] Server Processing

[0541] The server uses APIs and web scraping to collect historical data, such as economic indicators, social statistics, and scientific research. Specifically, it uses APIs (e.g., public sector APIs) and web scraping techniques (e.g., Python's BeautifulSoup library).

[0542] Input: API endpoint or website URL

[0543] Output: Collected data (e.g., economic indicator data in JSON format)

[0544] Specific operation: The server sends a request to the API endpoint, saves the returned JSON data in internal storage, and performs web scraping to obtain the required information in text format.

[0545] Step 2: Data cleansing and database construction

[0546] Server Processing

[0547] The server cleanses the collected data and stores it in a database. Specifically, it completes missing values ​​and removes outliers, and stores the organized data in a relational database management system (e.g., MySQL).

[0548] Input: Collected data (output of step 1)

[0549] Output: Cleansed data (structured data)

[0550] How it works: The server uses the Pandas library to analyze the collected data, impute missing values ​​with the mean, and detect and remove outliers. It then uses SQL INSERT statements to store the cleansed data in a MySQL database.

[0551] Step 3: Develop a future prediction model

[0552] Server Processing

[0553] The server uses the cleansed data to develop future prediction models, specifically training RNNs and GANs using TensorFlow and PyTorch.

[0554] Input: Cleansed data (output of Step 2)

[0555] Output: A trained future prediction model

[0556] Specific operation: The server uses the TensorFlow library to build an RNN model, inputs the cleansed dataset into the model to train it, evaluates the model's performance after training, and saves the optimized prediction model.

[0557] Step 4: Creating the VR space

[0558] Server Processing

[0559] The server creates a VR space based on the scenario generated by the future prediction model, using a virtual reality engine (e.g., Unreal Engine or Unity).

[0560] Input: Forecast model and scenario data (output of Step 3)

[0561] Output: VR space data

[0562] How it works: The server inputs scenario data into Unreal Engine and uses 3D modeling tools to generate a model to recreate the future cityscape. The generated 3D model and other VR data are then sent to the user's device via WebSocket.

[0563] Step 5: Sending VR space data

[0564] Server Processing

[0565] The server sends the generated VR space data to the user's device.

[0566] Input: VR space data (output of step 4)

[0567] Output: Transmitted VR space data

[0568] Specific operation: The server opens a WebSocket and establishes a connection to the device. The generated VR space data is compressed and sent to the device via the WebSocket.

[0569] Step 6: Rendering the VR space

[0570] Terminal handling

[0571] The device renders the VR space data received from the server in real time, using a high-speed rendering engine.

[0572] Input: Received VR space data (output of step 5)

[0573] Output: Rendered VR scene

[0574] How it works: The device decompresses the received data and uses an NVIDIA RTX series GPU to perform high-speed rendering. The rendered scene is then displayed on the VR headset.

[0575] Step 7: Access the VR space

[0576] User operations

[0577] Users access the VR space by wearing a VR headset and controllers.

[0578] Input: Wearing and operating a VR headset and controllers

[0579] Output: Access to VR space

[0580] Specific operation: The user puts on the VR headset and launches the application on the device. When launched for the first time, a tutorial on basic operation is played.

[0581] Step 8: Free movement and conversation

[0582] User operations

[0583] Users can move freely within the VR space and interact with AI characters.

[0584] Input: User actions and commands

[0585] Output: Interactive VR experience

[0586] How it works: The user operates the VR controller to move around the virtual space, approach the AI ​​character, and start a conversation. A natural language processing engine is used to converse with the AI ​​character, and the scenario changes dynamically depending on the user's choices.

[0587] (Application example 1)

[0588] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0589] As autonomous vehicles become more widespread, simulations are required to predict future changes in the transportation environment and infrastructure and adapt to them. However, previous simulation technologies did not provide realistic experiences, making it difficult to properly understand the impact of actual autonomous vehicle operations and infrastructure. In addition, there was a lack of visualization methods that allowed users to experience specific future scenarios and intuitively understand them.

[0590] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0591] In this invention, the server includes a means for collecting past data and storing it in a database, a means for developing a future prediction model based on the collected data, and a means for creating a VR space based on the generated future scenario, allowing users to realistically experience future operation scenarios of autonomous vehicles in the VR space and specifically understand changes in traffic conditions and infrastructure.

[0592] "Past data" refers to information that shows events or records that have already occurred, and is detailed information based on time and space.

[0593] A "database" is a collection of information that is systematically managed so that past data can be efficiently stored, searched, and used.

[0594] A "future prediction model" is a mathematical model that uses statistical methods and machine learning techniques to predict future events and conditions based on past data.

[0595] "VR space" is a computer-generated virtual reality space, an interactive 3D environment in which users can feel as if they are actually there.

[0596] "User operation" refers to an input operation that a user performs on the system via an interface, and is an action that triggers an implemented function or operation.

[0597] An "event" refers to a specific occurrence or action that occurs within the VR space, and is an element that causes the story to progress or the environment to change.

[0598] "Branching the story" means generating a scenario with different developments depending on the user's choices and actions, and events in the VR space changing accordingly.

[0599] "Acting freely" means that the user can move around in the VR space, perform specific actions, and interact with characters and the environment of their own volition.

[0600] "Character" refers to any other virtual entity that the user controls or interacts with within the VR space, and can take the form of a person, animal, robot, or other entity.

[0601] An "autonomous vehicle" is a vehicle equipped with technology that allows it to recognize its surroundings and operate autonomously without the need for human driving.

[0602] "Future operation scenarios" are specific simulations of how autonomous vehicles will operate in the future, based on future prediction models.

[0603] "Traffic conditions" refers to comprehensive information that indicates the state of traffic flow, congestion, accidents, etc. at a specific time and location.

[0604] "Infrastructure" refers to basic public facilities for transportation and logistics, such as roads, bridges, tunnels, and signal systems.

[0605] "Experience" refers to the user actually operating within a virtual space and experiencing it realistically through their senses of sight, hearing, touch, etc.

[0606] This invention is a system that recreates operational scenarios for autonomous vehicles in future cities in a VR space. This system collects past data, builds a future prediction model, and implements a series of specific steps to create a VR space based on the generated future scenario. The details are described below.

[0607] Server Processing

[0608] Data collection and database construction

[0609] The server collects historical traffic data, accident rates, data on new infrastructure projects, and more. This data is collected using APIs and web scraping. The collected data undergoes a cleansing process, including filling in missing values ​​and removing outliers, before being stored in a database. Specifically, the data is shaped and cleansed using Python and the Pandas library.

[0610] Development of future prediction models

[0611] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it uses a recurrent neural network (LSTM) to train a model that generates future scenarios from past traffic data. The TensorFlow and Keras libraries are used for this process.

[0612] VR space generation

[0613] Based on the generated future prediction scenario, a VR space is created using image generation AI. This uses libraries such as OpenVR. The server creates 3D models of future cityscapes and new infrastructure, generating realistic VR space data that includes visual, auditory, and tactile elements. This data is then sent to the user's device.

[0614] Terminal handling

[0615] VR space rendering

[0616] The device renders the VR space in real time based on the received VR space data. A high-speed rendering engine such as a GPU is used to ensure the user has a realistic experience. The device also provides an interface, displaying menus and options for the user to operate within the VR space.

[0617] User operations

[0618] Access to VR space

[0619] Users access the VR space using a head-mounted display (HMD) and controllers. The system displays basic operation instructions and usage guides.

[0620] Freedom of movement and conversation

[0621] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can drive through a futuristic city and explore predicted traffic conditions and new infrastructure projects. Depending on the user's choices, the story branches, allowing them to experience different future scenarios.

[0622] Specific examples

[0623] For example, to simulate traffic conditions in Tokyo in 2025, we collect past traffic data, accident data, and information on new infrastructure projects, and then use LSTM to predict future scenarios. Based on the generated scenarios, we create a 3D model of Tokyo in 2025 in a VR space, and users can explore the space through a VR headset.

[0624] Prompt Sentence Examples

[0625] "Create a VR application that simulates traffic conditions in Tokyo in 2025. Explain how you can use past traffic data to predict the future and recreate that data in a VR space."

[0626] In this way, a system is created that allows users to intuitively understand specific traffic conditions and infrastructure changes by experiencing future operating scenarios for autonomous vehicles in a VR space.

[0627] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0628] Step 1:

[0629] The server collects historical traffic data, accident rates, data on new infrastructure projects, etc. using APIs and web scraping. Before storing the collected data in a database, it performs cleansing processes such as filling in missing values ​​and removing outliers. The input data is raw traffic data, accident data, and infrastructure data, and the output is cleansed data. Specific operations use Python and the Pandas library.

[0630] Step 2:

[0631] The server builds a future prediction model based on the cleansed data. This process uses machine learning algorithms such as recurrent neural networks (LSTM). The input is the cleansed historical data, and the output is future traffic scenarios. The model is trained using the TensorFlow and Keras libraries. Specifically, the data is reformatted and an LSTM model is trained.

[0632] Step 3:

[0633] The server creates a VR space based on a scenario generated from a future prediction model. This process uses image generation AI and the OpenVR library. The input is future scenario data, and the output is the completed VR space data. Specifically, it generates a 3D city model and creates realistic data that includes visual, auditory, and tactile elements.

[0634] Step 4:

[0635] The device renders the VR space data received from the server in real time. The input is the VR space data sent from the server, and the output is the VR space experienced by the user through the HMD. Specifically, it uses a high-speed rendering engine (GPU) to provide a user interface.

[0636] Step 5:

[0637] Users access the VR space using a head-mounted display (HMD) and controllers. When a user experiences the system for the first time, the system displays basic operating instructions and a usage guide. The input is an operation instruction from the user, and the output is the system's response based on that instruction. Specific operations include initial setup of the HMD and controllers and display of the usage guide.

[0638] Step 6:

[0639] Users can move freely within the VR space and interact with the surrounding environment and characters. Input is the user's movement and control instructions, and output is changes to the environment within the VR space and the character's reactions. Specific actions include exploring the VR space, conversing with AI characters, and branching storylines.

[0640] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0641] The present invention is a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions, and is realized by the following program: This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely within the VR space and converse with other characters, and an emotion engine that recognizes the user's emotions.

[0642] Server Processing

[0643] Data collection and database construction

[0644] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping. This data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[0645] Development of future prediction models

[0646] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[0647] VR space generation

[0648] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[0649] Terminal handling

[0650] VR space rendering

[0651] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[0652] emotion recognition

[0653] The device uses an emotion engine to recognize the user's emotions. This information is input into the system as the user's current emotional state.

[0654] User operations

[0655] Wearing VR equipment

[0656] The user puts on a VR headset and controllers and prepares to access the VR space.

[0657] Access to VR space

[0658] Users enter the VR space through a device. If it is their first time experiencing it, the system will display basic operating instructions and a usage guide.

[0659] Freedom of movement and conversation

[0660] Users can freely move around in the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user makes a surprised expression, the surrounding characters will sense the user's surprise and respond accordingly.

[0661] Branching scenarios

[0662] The story changes depending on the user's choices. The user's emotional state also affects the branching of the scenario. For example, if the user has an inquisitive expression, a new event will occur based on that emotion. This allows the user to experience multiple future scenarios.

[0663] Specific examples

[0664] For example, consider a scenario in which Tokyo in 2025 is predicted based on the past five years' worth of economic indicators, traffic data, and social statistics. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. Image generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device.

[0665] Users put on a VR headset and enter the VR space via their device, freely exploring Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system for 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive experience of the future.

[0666] The processing flow will be explained below.

[0667] Server Processing

[0668] Step 1: Data collection

[0669] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[0670] Step 2: Data cleansing

[0671] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[0672] Step 3: Store in the database

[0673] The server stores the cleansed data in a database.

[0674] Step 4: Preprocessing the data

[0675] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[0676] Step 5: Develop a future prediction model

[0677] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs).

[0678] Step 6: Evaluate and tune the model

[0679] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[0680] Step 7: Generate the scenario

[0681] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[0682] Step 8: Use video generation AI

[0683] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[0684] Step 9: Sending Data

[0685] The server transmits the generated VR space data to the terminal.

[0686] Terminal handling

[0687] Step 1: Receiving and analyzing data

[0688] The device analyzes the VR space data received from the server and prepares for rendering.

[0689] Step 2: Rendering the VR space

[0690] The device uses a rendering engine such as a GPU to render the VR space in real time.

[0691] Step 3: Providing a User Interface

[0692] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[0693] Step 4: Start Emotion Recognition

[0694] The device uses devices such as a camera, microphone, and biosensor to analyze the user's facial expressions, tone of voice, heart rate, etc. in real time, and an emotion engine determines the user's emotional state.

[0695] User operations

[0696] Step 1: Put on the VR equipment

[0697] The user puts on a VR headset and controllers and prepares to access the VR space.

[0698] Step 2: Access the VR space

[0699] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[0700] Step 3: Free movement and conversation

[0701] Users can move freely within the VR space and interact with surrounding objects and characters. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era. The emotion engine recognizes the user's emotions, and the characters respond accordingly.

[0702] Step 4: Branching scenarios

[0703] The story changes depending on the user's choices and emotional state. For example, if the user is surprised or excited, new events are dynamically generated based on that emotion. The user can experience multiple future scenarios based on different emotions.

[0704] Specific examples

[0705] Example: A scenario looking at Tokyo in the future in 2025

[0706] Server Processing

[0707] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[0708] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[0709] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[0710] Terminal handling

[0711] 1. The device analyzes the received VR space data and prepares for rendering.

[0712] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[0713] 3. Provide landing points and navigation tools for the user interface.

[0714] 4. The device uses a camera and microphone to recognize the user's facial expressions and tone of voice, and the emotion engine analyzes this data to determine the user's emotions.

[0715] User operations

[0716] 1. The user puts on a VR headset and enters the VR space via the device.

[0717] 2. Users can freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[0718] 3. If the user expresses excitement or surprise, the emotion engine will recognize that emotion and the character in VR will begin explaining, "Are you surprised? This is the latest transportation system of 2025." In this way, interactions based on the user's emotions are realized.

[0719] 4. Depending on the user's emotional state, the scenario branches and new events occur. For example, if a user visits a particular location in a curious state, more detailed information or other new events will be provided.

[0720] Example 2

[0721] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] Currently, there is no technology that can simulate future events in a virtual reality space and dynamically change interactions based on the user's emotions. Current systems lack the ability to simulate events using future prediction models or to recognize the user's emotions and change interactions accordingly, so new technologies are needed to provide a more realistic and immersive experience.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a virtual reality space based on the generated future scenario. This enables advanced simulations using the future prediction model. The server also includes means for dynamically generating events in the virtual reality space in response to user operations and branching the story, means for allowing the user to freely act in the virtual reality space and converse with other characters, an emotion engine for recognizing the user's emotions, and means for dynamically changing the reactions of events and characters in the virtual reality space based on the emotion recognition. This realizes interactions tailored to the user's emotions, enabling a more realistic and immersive experience.

[0724] "Historical data" refers to information or records collected historically, including statistical information such as economic indicators, traffic data, and social statistics within a specific period of time.

[0725] A "database" is a collection of information that is organized and stored to efficiently manage collected data and make it easy to search and retrieve.

[0726] A "future forecasting model" is a mathematical or machine learning algorithm designed to predict future events or conditions based on past data.

[0727] A "virtual reality space" is a primarily three-dimensional virtual environment generated using computer technology, which allows users to experience it realistically through their senses of sight, hearing, and touch.

[0728] "User operation" refers to the inputs and actions that a user makes within a virtual reality space using VR equipment or a controller.

[0729] An "event" refers to an occurrence or action that occurs within a virtual reality space.

[0730] "Branching of the story" refers to the function in which the scenario changes depending on the user's operations and choices.

[0731] "Character" refers to a person, creature, or other entity that appears in a virtual reality space.

[0732] An "emotion engine" is a technology for recognizing a user's emotional state, and includes facial recognition, voice analysis, and analysis of biometric sensor data.

[0733] "Interaction" refers to a two-way exchange between a user and a virtual reality space and the characters present within it.

[0734] The present invention provides a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions. This system is realized by the following means.

[0735] The server first collects historical data through APIs or web scraping. The collected data ranges from economic indicators, traffic data, and social statistics. This data collection uses hardware and software such as data acquisition interfaces and analysis programs. For example, Google Analytics API or general web scraping tools are used. The data is then cleansed by completing processes such as filling in missing values ​​and removing outliers, and the organized data is stored in a database. A general SQL database is often used for database management.

[0736] The server then uses machine learning and deep learning techniques to develop a future prediction model based on the cleansed data. For example, it uses recurrent neural networks (RNNs) and generative artificial neural networks (GANs). To do this, it builds a model using libraries such as Keras and TensorFlow and trains it using the training data. After training is complete, it evaluates the model using test data and tunes its hyperparameters to improve accuracy.

[0737] Based on the scenarios generated by the future prediction model, the server uses image generation AI to create a virtual reality space. For example, it uses Unreal Engine or Unity to create 3D models that reflect future cityscapes and new technology concepts. This VR space data is designed to include visual, auditory, and tactile elements. The generated VR space data is then sent to the user's device. Data is transmitted using the HTTP protocol or WebSocket.

[0738] The device performs real-time rendering based on the VR space data received from the server. For example, a device equipped with an NVIDIA GPU can use Unity or Unreal Engine to draw high-quality images. The device also provides a user interface, displaying menus and options that allow the user to operate within the VR space.

[0739] The user's emotions are recognized using an emotion engine built into the device. Emotion recognition is achieved by facial recognition, voice analysis, and biometric sensor data analysis. For example, facial expressions are recognized using a camera, and voice analysis is performed using a microphone. Biometric sensor data such as heart rate and electrodermal activity are also analyzed. This information is input into the system as the user's current emotional state and sent to the server.

[0740] The user then accesses the VR space by putting on a VR headset and controllers. For example, they start the virtual reality experience using an Oculus Quest 2. First-time users are shown instructions on how to operate the VR space and a usage guide. Within the virtual reality space, users can move freely and converse with other characters. Furthermore, events and character reactions within the VR space change dynamically based on emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will respond accordingly.

[0741] As a specific example, it is possible to predict future scenarios based on Tokyo's economic indicators, traffic data, and social statistics from the past five years, and recreate Tokyo in 2025. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. It then uses image generation AI to create a 3D model of Tokyo in 2025, and sends the VR space data to the device.

[0742] Users put on a VR headset and enter the VR space via their device, where they can freely explore Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system in 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive future experience.

[0743] An example of a prompt is: "Tell me the steps to develop a future prediction model to create a VR space of Tokyo in 2025 based on Tokyo's economic indicators, traffic data, and social statistics for the past five years."

[0744] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0745] Step 1: Collect historical data

[0746] The server uses APIs and web scraping tools to collect data such as historical economic indicators, traffic data, and social statistics. For example, it uses Google Analytics APIs and web scraping tools to obtain relevant data. The input is a data collection API or a web page URL. The output is raw historical data.

[0747] Step 2: Data cleansing

[0748] The server cleanses the collected data. This process includes imputing missing values ​​and removing outliers. For example, it uses the Python Pandas library to impute missing values ​​using the median and detect and remove outliers using Z-scores. The input is the raw historical data. The output is the cleansed data.

[0749] Step 3: Data storage

[0750] The server stores the cleansed data in a database. A general SQL database is used to efficiently manage data. The input is the cleansed data. The output is the data stored in the database.

[0751] Step 4: Develop a future prediction model

[0752] The server uses machine learning and deep learning techniques to develop a future prediction model based on the data stored in the database. For example, it uses the Keras or TensorFlow library to train a recurrent neural network (RNN) or a generative artificial neural network (GAN). The input is the data stored in the database. The output is the trained future prediction model.

[0753] Step 5: Generate future scenarios

[0754] The server generates future scenarios using a trained future prediction model. For example, it uses an RNN to predict the future economic and social conditions of Tokyo and generates specific scenarios. The input is the trained future prediction model. The output is the future scenario.

[0755] Step 6: Creating the VR space

[0756] The server uses image generation AI to create a VR space based on the generated future scenario. For example, it uses Unreal Engine or Unity to create 3D modeling of a future cityscape and creates VR content that includes visual, auditory, and tactile elements. The input is the future scenario. The output is the generated VR space data.

[0757] Step 7: Streaming VR space data

[0758] The server compresses the generated VR space data and sends it to the device. The data is transferred in real time via HTTP or WebSocket. The input is the generated VR space data. The output is the VR space data sent to the device.

[0759] Step 8: Rendering the VR space

[0760] The device performs real-time rendering based on the VR space data received from the server. Using Unity or Unreal Engine, high-quality images are drawn on devices equipped with NVIDIA GPUs. The device also provides a user interface, displaying menus and options for users to operate within the VR space. The input is the VR space data sent to the device. The output is the rendered VR space.

[0761] Step 9: Emotion Recognition

[0762] The device recognizes the user's emotions using a built-in emotion engine. It uses a camera to recognize facial expressions and a microphone to perform voice analysis. It also analyzes biosensor data such as heart rate and electrodermal activity. The inputs are the user's facial image, voice, and biosensor data. The output is the emotion recognition results.

[0763] Step 10: User's VR experience

[0764] Users access the VR space by wearing a VR headset and controllers. For example, they start a virtual reality experience using Oculus Quest 2. First-time users are shown instructions on how to use the device and a usage guide. The input is the state of wearing the VR headset and controllers. The output is the state of access to the VR space.

[0765] Step 11: Interaction and Scenario Branching

[0766] The device dynamically changes events and character reactions in the VR space based on the user's emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will react accordingly. The input is the emotion recognition results. The output is emotion-based interactions and scenario branching.

[0767] (Application example 2)

[0768] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0769] Conventional virtual reality (VR) systems have been unable to predict and recreate future events or dynamically change interactions based on the user's emotions. This has made it difficult to provide interactive VR experiences that allow users to have a more immersive experience. Furthermore, they have been unable to provide personalized interactions based on emotions, such as a futuristic shopping experience. Therefore, there has been a demand for improving the quality of the user experience and providing a more realistic and personal VR space.

[0770] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a VR space based on the generated future scenario. This makes it possible to recreate future events in a virtual reality space. The server also includes means for dynamically generating events in the VR space and branching the story in response to user operations, means for the user to freely act and converse with other characters in the VR space, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions. This improves the quality of the user experience and makes it possible to provide personalized interactions based on emotions, such as future shopping experiences.

[0771] "Past data" refers to existing information such as economic indicators, social statistics, and scientific research, and serves as the basic data for predicting the future.

[0772] A "database" is a collection of information that stores collected past data in an orderly manner so that it can be used efficiently later.

[0773] A "future prediction model" is a predictive algorithm that uses machine learning and deep learning technologies to generate future scenarios based on collected past data.

[0774] "VR space" is a three-dimensional space generated on a computer using virtual reality technology, allowing users to experience a virtual environment that is different from the physical world.

[0775] "User operation" refers to the actions and choices the user makes using controllers and gestures within the VR space.

[0776] An "event" refers to a specific occurrence or action that occurs within the VR space, and changes dynamically depending on the user's actions and emotions.

[0777] "Branching of the story" means that the scenario in the VR space changes in different directions depending on the user's choices and actions.

[0778] "Character" refers to a virtual person or creature that interacts with the user in the VR space.

[0779] "Emotion recognition" refers to the technology of analyzing a user's facial expressions, voice, and biometric sensor data to identify their current emotional state.

[0780] "Interaction" refers to the interaction between the user and the environment and characters in the VR space, which changes dynamically based on emotion recognition.

[0781] "Guide" is a function that provides information and instructions to the user using voice and text.

[0782] "Additional information" refers to additional information or explanations provided based on the user's emotions or actions.

[0783] This invention is a system that recreates future events in a virtual reality space (VR space), recognizes a user's emotions, and dynamically changes interactions based on those emotions. The invention includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely in the VR space and converse with other characters, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions.

[0784] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping, cleansing it by filling in missing values ​​and removing outliers, and then stores it in a database. Next, based on this cleansed data, a future prediction model is developed using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The model is evaluated and tuned to create the optimal future prediction model. Based on the generated future scenario, a VR space is created using video generation AI. This VR space includes visual, auditory, and tactile elements.

[0785] The user device performs real-time rendering based on the VR space data received from the server, providing a realistic experience to the user wearing the VR headset. The device uses an emotion engine to recognize the user's emotions through facial recognition, voice analysis, and biometric sensor data analysis. This information is input into the system as the user's current emotional state.

[0786] Users put on a VR headset and controllers and access the VR space. They can move freely within the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, such as surprise or excitement, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user is surprised by a futuristic transportation system, a character in the VR space will provide guidance by explaining, "Are you surprised? This is the latest transportation system of the future."

[0787] As a concrete example, consider a scenario in which the future of a city is predicted based on the city's economic indicators, traffic data, and social statistics from the past five years. The server collects and cleans the data, and uses RNN to develop a model to predict the future city. A video generation AI is used to create a 3D model of the future city, and the VR space data is sent to the device. The user puts on a VR headset and enters the VR space via the device, freely exploring the future city. The emotion engine recognizes the user's emotions, and the interaction changes accordingly.

[0788] An example of a prompt sentence is, "When the user expresses surprise at a particular product in the future shopping mall, please provide guidance or additional information based on that emotion." In this way, the present invention improves the quality of the user experience and provides a more immersive future experience.

[0789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0790] Step 1:

[0791] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping. The input for the collection is multiple public databases and websites, and the data is obtained using these. Then, cleansing processes such as imputing missing values ​​and removing outliers are performed on the data, and the cleansed data is stored in a database. The output is the cleansed historical data.

[0792] Step 2:

[0793] The server uses the cleansed historical data stored in the database to develop a future prediction model. It takes the cleansed data as input and trains the model using machine learning algorithms such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs). It processes the data by performing feature engineering and preprocessing, converting it into a suitable format. The output is a trained future prediction model.

[0794] Step 3:

[0795] The server generates future scenarios based on the trained future prediction model. It takes the future prediction model and actual data as input, runs the prediction algorithm to simulate future events, and outputs future scenario data.

[0796] Step 4:

[0797] The server creates a VR space using image generation AI based on a future scenario. It takes future scenario data as input and generates the VR space using 3D modeling and image rendering software. Specific operations include generating 3D models, applying textures, and adding sound effects. The output is the generated VR space data.

[0798] Step 5:

[0799] The device receives VR space data from the server and renders it in real time. It takes the VR space data from the server as input and uses the GPU to perform rendering processing. Specific operations include drawing 3D graphics, playing audio, and generating haptic feedback. The output is a real-time VR view provided to the user.

[0800] Step 6:

[0801] Users access the VR space using a VR headset and controllers and act freely. The system receives user operations and motion inputs as input, and reflects those movements within the VR space. Specific movements include movement, item selection, and character interaction. The output is a change within the VR space based on user operations.

[0802] Step 7:

[0803] The device uses an emotion engine to recognize the user's emotions. User data is acquired as input from the camera, microphone, and biometric sensors, and analyzed using emotion recognition algorithms. Data processing involves analyzing facial expressions through image processing, analyzing voice, and interpreting biometric data. The output is the recognized emotional state of the user.

[0804] Step 8:

[0805] The device dynamically changes interactions in the VR space based on the recognized emotions. It takes the user's emotional state as input and executes an interaction update algorithm. Specific actions include changing the character's reaction, generating new events, and displaying guides or additional information. The output is the dynamically changed interaction.

[0806] Step 9:

[0807] The device provides guidance and additional information according to the user's emotions. It takes the recognized emotional state and current scenario information as input and generates appropriate guidance messages and additional information. Specific operations include generating audio guidance, displaying text guidance, and displaying additional information panels. The output is the guidance and additional information provided to the user.

[0808] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0809] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0810] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0811] [Third embodiment]

[0812] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0813] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0814] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0815] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0816] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0817] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0818] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0819] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0820] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0821] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0822] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0823] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0824] Understood. The following is a description of the "Mode for carrying out the invention."

[0825] The present invention is a system for recreating future events in a virtual reality space (VR space), and is realized by the following program. This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, and means for the user to act freely in the VR space and converse with other characters.

[0826] Server Processing

[0827] Data collection and database construction

[0828] The server uses APIs and web scraping to collect historical data, including economic indicators, social statistics, scientific research, etc. The collected data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[0829] Development of future prediction models

[0830] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[0831] VR space generation

[0832] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[0833] Terminal handling

[0834] VR space rendering

[0835] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[0836] User operations

[0837] Access to VR space

[0838] Users put on a VR headset and controllers and access the VR space through the device. If it's their first time experiencing it, the system will display basic operation instructions and a usage guide.

[0839] Freedom of movement and conversation

[0840] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to AI characters. Depending on the user's choices, the story will unfold in different directions, allowing them to enjoy multiple scenarios.

[0841] Specific examples

[0842] For example, consider a scenario that predicts Tokyo in 2025 based on Tokyo's economic indicators, transportation data, and social statistics from the past five years. The server collects and cleans this data, and uses RNN to develop a model that predicts Tokyo in 2025. A video generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device. Users can access this VR space, explore new landmarks and transportation systems, and enjoy conversations with AI characters about life in the future. The scenario branches depending on the user's choices, allowing them to experience different futures.

[0843] The processing flow will be explained below.

[0844] Server Processing

[0845] Step 1: Data collection

[0846] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[0847] Step 2: Data cleansing

[0848] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[0849] Step 3: Store in the database

[0850] The server stores the cleansed data in a database.

[0851] Step 4: Preprocessing the data

[0852] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[0853] Step 5: Develop a future prediction model

[0854] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, for example, using RNNs and GANs.

[0855] Step 6: Evaluate and tune the model

[0856] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[0857] Step 7: Generate the scenario

[0858] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[0859] Step 8: Use video generation AI

[0860] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[0861] Step 9: Sending Data

[0862] The server transmits the generated VR space data to the terminal.

[0863] Terminal handling

[0864] Step 1: Receiving and analyzing data

[0865] The device analyzes the VR space data received from the server and prepares for rendering.

[0866] Step 2: Rendering the VR space

[0867] The device uses a rendering engine such as a GPU to render the VR space in real time.

[0868] Step 3: Providing a User Interface

[0869] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[0870] User operations

[0871] Step 1: Put on the VR equipment

[0872] The user puts on a VR headset and controllers and prepares to access the VR space.

[0873] Step 2: Access the VR space

[0874] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[0875] Step 3: Free movement and conversation

[0876] Users can freely move around in the VR space and interact with the objects and characters around them. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era.

[0877] Step 4: Branching scenarios

[0878] The story changes depending on the user's choices. For example, if the user visits a specific location, different events will occur depending on the choices made. This allows the user to experience multiple future scenarios.

[0879] Specific examples

[0880] Example: A scenario looking at Tokyo in the future in 2025

[0881] Server Processing

[0882] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[0883] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[0884] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[0885] Terminal handling

[0886] 1. The device analyzes the received VR space data and prepares for rendering.

[0887] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[0888] 3. Provide landing points and navigation tools as a user interface.

[0889] User operations

[0890] 1. The user puts on a VR headset and enters the VR space via the device.

[0891] 2. Freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[0892] 3. Talk to AI characters at specific locations and hear about new technologies and social trends. The scenario branches depending on the user's choices, allowing users to enjoy different futures.

[0893] Example 1

[0894] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] In conventional technologies, it has been difficult to effectively integrate a series of processes, from data collection and cleansing, to the development of future prediction models and the generation of virtual reality spaces, in systems that use past data to make future predictions. Furthermore, there has been a lack of systems that allow users to experience scenarios that change in real time while freely acting and interacting in a virtual space. This has meant that no environment has been provided in which users can realistically experience future events and dynamically change those scenarios.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0897] In this invention, the server includes means for collecting past data, means for cleansing the collected data and storing it in a database, means for developing a future prediction model using machine learning and deep learning technologies, means for creating a virtual reality space based on a future scenario generated from the future prediction model, means for dynamically generating events in the virtual reality space in response to user operations and branching the story, and means for the user to act freely in the virtual reality space and converse with other characters. This enables the realization of a future prediction system in which users can realistically experience the future predicted from past data and can be interactively involved.

[0898] Define "data" in the following format:

[0899] "Historical data" refers to time-series information and numerical data such as economic indicators, social statistics, and scientific research.

[0900] "Means of collection" refers to methods of obtaining data from outside using technologies such as APIs and web scraping.

[0901] "Means of storing in a database" refers to the method of organizing collected data into a certain format and storing it in a database such as a relational database management system (RDBMS).

[0902] "Cleansing" refers to the process of improving data quality by preparing collected data through methods such as filling in missing values ​​and removing outliers.

[0903] "Machine learning and deep learning technologies" refer to technologies in which computers use large amounts of data to learn patterns and then use those patterns to make predictions and classifications.

[0904] A "future prediction model" refers to an algorithm or model that uses statistical methods and machine learning technology to generate future scenarios based on collected past data.

[0905] "Virtual reality space" refers to a three-dimensional computer-generated virtual environment that users can experience interactively.

[0906] "Dynamic generation" refers to a method of changing events and scenarios in real time in response to user operations and input.

[0907] "Means for branching the story" refers to a function that allows the scenario to unfold in different directions depending on the user's choices and actions.

[0908] "Means for free movement and conversation with other characters" refers to a method by which a user can freely move within a virtual reality space and interact and converse with characters.

[0909] MODE FOR CARRYING OUT THE INVENTION

[0910] The present invention relates to a system that predicts the future based on past data and allows users to experience the scenario in a virtual reality space (VR space). The system of the present invention is composed of components such as a server, a terminal, and a user, and an embodiment thereof will be described in detail below.

[0911] Server Processing

[0912] Data collection and database construction

[0913] The server uses APIs and web scraping to collect historical data. This data includes economic indicators, social statistics, scientific research, and more. Specifically, it uses APIs (e.g., APIs provided by international organizations) and web scraping technology (e.g., Python's BeautifulSoup library). The server organizes the collected data using a data cleansing tool such as Pandas and stores it in a relational database management system (e.g., MySQL). Missing values ​​are imputed and outliers are removed at this stage.

[0914] Development of future prediction models

[0915] The server uses TensorFlow or PyTorch to develop a future prediction model based on the cleansed data. For example, it trains a model that generates future scenarios from the data using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The trained model is then appropriately evaluated and tuned to build the optimal prediction model.

[0916] VR space generation

[0917] The server creates a VR space using a virtual reality engine (e.g., Unreal Engine or Unity) based on the scenario generated by the future prediction model. For example, it generates 3D models that reflect future cityscapes or new technological concepts, and includes visual, auditory, and tactile elements to add realism. The generated VR space data is then sent to the user's device via WebSocket.

[0918] Terminal handling

[0919] VR space rendering

[0920] The device renders the VR space data received from the server in real time. Specifically, it uses a high-speed rendering engine (e.g., NVIDIA RTX series GPU) to provide a realistic experience for the user. The device also uses Unity or Unreal Engine to provide a user interface, displaying menus and options that allow the user to operate within the VR space.

[0921] User operations

[0922] Access to VR space

[0923] Users put on a VR headset (such as Oculus Quest or HTC Vive) and controllers and access the VR space through their device. When using it for the first time, the system will provide basic operation instructions and a usage guide.

[0924] Freedom of movement and conversation

[0925] Users can move freely within the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to an AI character. Natural language processing technology (e.g., IBM Watson) is used to converse with the AI ​​character. Depending on the user's choices, the scenario will unfold in different directions, allowing users to enjoy multiple scenarios.

[0926] Specific examples

[0927] For example, consider a scenario in which a city's economic indicators, traffic data, and social statistics from the past five years are used to predict what it will be like in 2025. The server collects this data using the IMF's API and the World Bank's open data, cleansing it with Pandas, and developing a 2025 prediction model (RNN model) using TensorFlow. A 3D model of the city in 2025 is created using Unreal Engine and sent to the device via WebSocket. Users can access the VR space by wearing an Oculus Quest, exploring new landmarks and transportation systems, and enjoying conversations with AI characters powered by IBM Watson about life in the future. Furthermore, the scenario branches depending on the user's choices, allowing them to experience different futures.

[0928] Prompt Sentence Examples

[0929] By inputting prompts like the following into a generative AI model, we can generate future scenarios:

[0930] "You will create a virtual cityscape of 2025 and generate scenarios that include future urban landscapes, transportation systems, and new technological applications. You will also include new landmarks for users to explore and specific interaction scenes with AI characters."

[0931] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0932] Step 1: Data collection

[0933] Server Processing

[0934] The server uses APIs and web scraping to collect historical data, such as economic indicators, social statistics, and scientific research. Specifically, it uses APIs (e.g., public sector APIs) and web scraping techniques (e.g., Python's BeautifulSoup library).

[0935] Input: API endpoint or website URL

[0936] Output: Collected data (e.g., economic indicator data in JSON format)

[0937] Specific operation: The server sends a request to the API endpoint, saves the returned JSON data in internal storage, and performs web scraping to obtain the required information in text format.

[0938] Step 2: Data cleansing and database construction

[0939] Server Processing

[0940] The server cleanses the collected data and stores it in a database. Specifically, it completes missing values ​​and removes outliers, and stores the organized data in a relational database management system (e.g., MySQL).

[0941] Input: Collected data (output of step 1)

[0942] Output: Cleansed data (structured data)

[0943] How it works: The server uses the Pandas library to analyze the collected data, impute missing values ​​with the mean, and detect and remove outliers. It then uses SQL INSERT statements to store the cleansed data in a MySQL database.

[0944] Step 3: Develop a future prediction model

[0945] Server Processing

[0946] The server uses the cleansed data to develop future prediction models, specifically training RNNs and GANs using TensorFlow and PyTorch.

[0947] Input: Cleansed data (output of Step 2)

[0948] Output: A trained future prediction model

[0949] Specific operation: The server uses the TensorFlow library to build an RNN model, inputs the cleansed dataset into the model to train it, evaluates the model's performance after training, and saves the optimized prediction model.

[0950] Step 4: Creating the VR space

[0951] Server Processing

[0952] The server creates a VR space based on the scenario generated by the future prediction model, using a virtual reality engine (e.g., Unreal Engine or Unity).

[0953] Input: Forecast model and scenario data (output of Step 3)

[0954] Output: VR space data

[0955] How it works: The server inputs scenario data into Unreal Engine and uses 3D modeling tools to generate a model to recreate the future cityscape. The generated 3D model and other VR data are then sent to the user's device via WebSocket.

[0956] Step 5: Sending VR space data

[0957] Server Processing

[0958] The server sends the generated VR space data to the user's device.

[0959] Input: VR space data (output of step 4)

[0960] Output: Transmitted VR space data

[0961] Specific operation: The server opens a WebSocket and establishes a connection to the device. The generated VR space data is compressed and sent to the device via the WebSocket.

[0962] Step 6: Rendering the VR space

[0963] Terminal handling

[0964] The device renders the VR space data received from the server in real time, using a high-speed rendering engine.

[0965] Input: Received VR space data (output of step 5)

[0966] Output: Rendered VR scene

[0967] How it works: The device decompresses the received data and uses an NVIDIA RTX series GPU to perform high-speed rendering. The rendered scene is then displayed on the VR headset.

[0968] Step 7: Access the VR space

[0969] User operations

[0970] Users access the VR space by wearing a VR headset and controllers.

[0971] Input: Wearing and operating a VR headset and controllers

[0972] Output: Access to VR space

[0973] Specific operation: The user puts on the VR headset and launches the application on the device. When launched for the first time, a tutorial on basic operation is played.

[0974] Step 8: Free movement and conversation

[0975] User operations

[0976] Users can move freely within the VR space and interact with AI characters.

[0977] Input: User actions and commands

[0978] Output: Interactive VR experience

[0979] How it works: The user operates the VR controller to move around the virtual space, approach the AI ​​character, and start a conversation. A natural language processing engine is used to converse with the AI ​​character, and the scenario changes dynamically depending on the user's choices.

[0980] (Application example 1)

[0981] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0982] As autonomous vehicles become more widespread, simulations are required to predict future changes in the transportation environment and infrastructure and adapt to them. However, previous simulation technologies did not provide realistic experiences, making it difficult to properly understand the impact of actual autonomous vehicle operations and infrastructure. In addition, there was a lack of visualization methods that allowed users to experience specific future scenarios and intuitively understand them.

[0983] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0984] In this invention, the server includes a means for collecting past data and storing it in a database, a means for developing a future prediction model based on the collected data, and a means for creating a VR space based on the generated future scenario, allowing users to realistically experience future operation scenarios of autonomous vehicles in the VR space and specifically understand changes in traffic conditions and infrastructure.

[0985] "Past data" refers to information that shows events or records that have already occurred, and is detailed information based on time and space.

[0986] A "database" is a collection of information that is systematically managed so that past data can be efficiently stored, searched, and used.

[0987] A "future prediction model" is a mathematical model that uses statistical methods and machine learning techniques to predict future events and conditions based on past data.

[0988] "VR space" is a computer-generated virtual reality space, an interactive 3D environment in which users can feel as if they are actually there.

[0989] "User operation" refers to an input operation that a user performs on the system via an interface, and is an action that triggers an implemented function or operation.

[0990] An "event" refers to a specific occurrence or action that occurs within the VR space, and is an element that causes the story to progress or the environment to change.

[0991] "Branching the story" means generating a scenario with different developments depending on the user's choices and actions, and events in the VR space changing accordingly.

[0992] "Acting freely" means that the user can move around in the VR space, perform specific actions, and interact with characters and the environment of their own volition.

[0993] "Character" refers to any other virtual entity that the user controls or interacts with within the VR space, and can take the form of a person, animal, robot, or other entity.

[0994] An "autonomous vehicle" is a vehicle equipped with technology that allows it to recognize its surroundings and operate autonomously without the need for human driving.

[0995] "Future operation scenarios" are specific simulations of how autonomous vehicles will operate in the future, based on future prediction models.

[0996] "Traffic conditions" refers to comprehensive information that indicates the state of traffic flow, congestion, accidents, etc. at a specific time and location.

[0997] "Infrastructure" refers to basic public facilities for transportation and logistics, such as roads, bridges, tunnels, and signal systems.

[0998] "Experience" refers to the user actually operating within a virtual space and experiencing it realistically through their senses of sight, hearing, touch, etc.

[0999] This invention is a system that recreates operational scenarios for autonomous vehicles in future cities in a VR space. This system collects past data, builds a future prediction model, and implements a series of specific steps to create a VR space based on the generated future scenario. The details are described below.

[1000] Server Processing

[1001] Data collection and database construction

[1002] The server collects historical traffic data, accident rates, data on new infrastructure projects, and more. This data is collected using APIs and web scraping. The collected data undergoes a cleansing process, including filling in missing values ​​and removing outliers, before being stored in a database. Specifically, the data is shaped and cleansed using Python and the Pandas library.

[1003] Development of future prediction models

[1004] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it uses a recurrent neural network (LSTM) to train a model that generates future scenarios from past traffic data. The TensorFlow and Keras libraries are used for this process.

[1005] VR space generation

[1006] Based on the generated future prediction scenario, a VR space is created using image generation AI. This uses libraries such as OpenVR. The server creates 3D models of future cityscapes and new infrastructure, generating realistic VR space data that includes visual, auditory, and tactile elements. This data is then sent to the user's device.

[1007] Terminal handling

[1008] VR space rendering

[1009] The device renders the VR space in real time based on the received VR space data. A high-speed rendering engine such as a GPU is used to ensure the user has a realistic experience. The device also provides an interface, displaying menus and options for the user to operate within the VR space.

[1010] User operations

[1011] Access to VR space

[1012] Users access the VR space using a head-mounted display (HMD) and controllers. The system displays basic operation instructions and usage guides.

[1013] Freedom of movement and conversation

[1014] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can drive through a futuristic city and explore predicted traffic conditions and new infrastructure projects. Depending on the user's choices, the story branches, allowing them to experience different future scenarios.

[1015] Specific examples

[1016] For example, to simulate traffic conditions in Tokyo in 2025, we collect past traffic data, accident data, and information on new infrastructure projects, and then use LSTM to predict future scenarios. Based on the generated scenarios, we create a 3D model of Tokyo in 2025 in a VR space, and users can explore the space through a VR headset.

[1017] Prompt Sentence Examples

[1018] "Create a VR application that simulates traffic conditions in Tokyo in 2025. Explain how you can use past traffic data to predict the future and recreate that data in a VR space."

[1019] In this way, a system is created that allows users to intuitively understand specific traffic conditions and infrastructure changes by experiencing future operating scenarios for autonomous vehicles in a VR space.

[1020] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1021] Step 1:

[1022] The server collects historical traffic data, accident rates, data on new infrastructure projects, etc. using APIs and web scraping. Before storing the collected data in a database, it performs cleansing processes such as filling in missing values ​​and removing outliers. The input data is raw traffic data, accident data, and infrastructure data, and the output is cleansed data. Specific operations use Python and the Pandas library.

[1023] Step 2:

[1024] The server builds a future prediction model based on the cleansed data. This process uses machine learning algorithms such as recurrent neural networks (LSTM). The input is the cleansed historical data, and the output is future traffic scenarios. The model is trained using the TensorFlow and Keras libraries. Specifically, the data is reformatted and an LSTM model is trained.

[1025] Step 3:

[1026] The server creates a VR space based on a scenario generated from a future prediction model. This process uses image generation AI and the OpenVR library. The input is future scenario data, and the output is the completed VR space data. Specifically, it generates a 3D city model and creates realistic data that includes visual, auditory, and tactile elements.

[1027] Step 4:

[1028] The device renders the VR space data received from the server in real time. The input is the VR space data sent from the server, and the output is the VR space experienced by the user through the HMD. Specifically, it uses a high-speed rendering engine (GPU) to provide a user interface.

[1029] Step 5:

[1030] Users access the VR space using a head-mounted display (HMD) and controllers. When a user experiences the system for the first time, the system displays basic operating instructions and a usage guide. The input is an operation instruction from the user, and the output is the system's response based on that instruction. Specific operations include initial setup of the HMD and controllers and display of the usage guide.

[1031] Step 6:

[1032] Users can move freely within the VR space and interact with the surrounding environment and characters. Input is the user's movement and control instructions, and output is changes to the environment within the VR space and the character's reactions. Specific actions include exploring the VR space, conversing with AI characters, and branching storylines.

[1033] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1034] The present invention is a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions, and is realized by the following program: This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely within the VR space and converse with other characters, and an emotion engine that recognizes the user's emotions.

[1035] Server Processing

[1036] Data collection and database construction

[1037] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping. This data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[1038] Development of future prediction models

[1039] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[1040] VR space generation

[1041] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[1042] Terminal handling

[1043] VR space rendering

[1044] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[1045] emotion recognition

[1046] The device uses an emotion engine to recognize the user's emotions. This information is input into the system as the user's current emotional state.

[1047] User operations

[1048] Wearing VR equipment

[1049] The user puts on a VR headset and controllers and prepares to access the VR space.

[1050] Access to VR space

[1051] Users enter the VR space through a device. If it is their first time experiencing it, the system will display basic operating instructions and a usage guide.

[1052] Freedom of movement and conversation

[1053] Users can freely move around in the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user makes a surprised expression, the surrounding characters will sense the user's surprise and respond accordingly.

[1054] Branching scenarios

[1055] The story changes depending on the user's choices. The user's emotional state also affects the branching of the scenario. For example, if the user has an inquisitive expression, a new event will occur based on that emotion. This allows the user to experience multiple future scenarios.

[1056] Specific examples

[1057] For example, consider a scenario in which Tokyo in 2025 is predicted based on the past five years' worth of economic indicators, traffic data, and social statistics. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. Image generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device.

[1058] Users put on a VR headset and enter the VR space via their device, freely exploring Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system for 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive experience of the future.

[1059] The processing flow will be explained below.

[1060] Server Processing

[1061] Step 1: Data collection

[1062] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[1063] Step 2: Data cleansing

[1064] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[1065] Step 3: Store in the database

[1066] The server stores the cleansed data in a database.

[1067] Step 4: Preprocessing the data

[1068] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[1069] Step 5: Develop a future prediction model

[1070] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs).

[1071] Step 6: Evaluate and tune the model

[1072] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[1073] Step 7: Generate the scenario

[1074] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[1075] Step 8: Use video generation AI

[1076] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[1077] Step 9: Sending Data

[1078] The server transmits the generated VR space data to the terminal.

[1079] Terminal handling

[1080] Step 1: Receiving and analyzing data

[1081] The device analyzes the VR space data received from the server and prepares for rendering.

[1082] Step 2: Rendering the VR space

[1083] The device uses a rendering engine such as a GPU to render the VR space in real time.

[1084] Step 3: Providing a User Interface

[1085] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[1086] Step 4: Start Emotion Recognition

[1087] The device uses devices such as a camera, microphone, and biosensor to analyze the user's facial expressions, tone of voice, heart rate, etc. in real time, and an emotion engine determines the user's emotional state.

[1088] User operations

[1089] Step 1: Put on the VR equipment

[1090] The user puts on a VR headset and controllers and prepares to access the VR space.

[1091] Step 2: Access the VR space

[1092] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[1093] Step 3: Free movement and conversation

[1094] Users can move freely within the VR space and interact with surrounding objects and characters. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era. The emotion engine recognizes the user's emotions, and the characters respond accordingly.

[1095] Step 4: Branching scenarios

[1096] The story changes depending on the user's choices and emotional state. For example, if the user is surprised or excited, new events are dynamically generated based on that emotion. The user can experience multiple future scenarios based on different emotions.

[1097] Specific examples

[1098] Example: A scenario looking at Tokyo in the future in 2025

[1099] Server Processing

[1100] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[1101] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[1102] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[1103] Terminal handling

[1104] 1. The device analyzes the received VR space data and prepares for rendering.

[1105] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[1106] 3. Provide landing points and navigation tools for the user interface.

[1107] 4. The device uses a camera and microphone to recognize the user's facial expressions and tone of voice, and the emotion engine analyzes this data to determine the user's emotions.

[1108] User operations

[1109] 1. The user puts on a VR headset and enters the VR space via the device.

[1110] 2. Users can freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[1111] 3. If the user expresses excitement or surprise, the emotion engine will recognize that emotion and the character in VR will begin explaining, "Are you surprised? This is the latest transportation system of 2025." In this way, interactions based on the user's emotions are realized.

[1112] 4. Depending on the user's emotional state, the scenario branches and new events occur. For example, if a user visits a particular location in a curious state, more detailed information or other new events will be provided.

[1113] Example 2

[1114] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1115] Currently, there is no technology that can simulate future events in a virtual reality space and dynamically change interactions based on the user's emotions. Current systems lack the ability to simulate events using future prediction models or to recognize the user's emotions and change interactions accordingly, so new technologies are needed to provide a more realistic and immersive experience.

[1116] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a virtual reality space based on the generated future scenario. This enables advanced simulations using the future prediction model. The server also includes means for dynamically generating events in the virtual reality space in response to user operations and branching the story, means for allowing the user to freely act in the virtual reality space and converse with other characters, an emotion engine for recognizing the user's emotions, and means for dynamically changing the reactions of events and characters in the virtual reality space based on the emotion recognition. This realizes interactions tailored to the user's emotions, enabling a more realistic and immersive experience.

[1117] "Historical data" refers to information or records collected historically, including statistical information such as economic indicators, traffic data, and social statistics within a specific period of time.

[1118] A "database" is a collection of information that is organized and stored to efficiently manage collected data and make it easy to search and retrieve.

[1119] A "future forecasting model" is a mathematical or machine learning algorithm designed to predict future events or conditions based on past data.

[1120] A "virtual reality space" is a primarily three-dimensional virtual environment generated using computer technology, which allows users to experience it realistically through their senses of sight, hearing, and touch.

[1121] "User operation" refers to the inputs and actions that a user makes within a virtual reality space using VR equipment or a controller.

[1122] An "event" refers to an occurrence or action that occurs within a virtual reality space.

[1123] "Branching of the story" refers to the function in which the scenario changes depending on the user's operations and choices.

[1124] "Character" refers to a person, creature, or other entity that appears in a virtual reality space.

[1125] An "emotion engine" is a technology for recognizing a user's emotional state, and includes facial recognition, voice analysis, and analysis of biometric sensor data.

[1126] "Interaction" refers to a two-way exchange between a user and a virtual reality space and the characters present within it.

[1127] The present invention provides a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions. This system is realized by the following means.

[1128] The server first collects historical data through APIs or web scraping. The collected data ranges from economic indicators, traffic data, and social statistics. This data collection uses hardware and software such as data acquisition interfaces and analysis programs. For example, Google Analytics API or general web scraping tools are used. The data is then cleansed by completing processes such as filling in missing values ​​and removing outliers, and the organized data is stored in a database. A general SQL database is often used for database management.

[1129] The server then uses machine learning and deep learning techniques to develop a future prediction model based on the cleansed data. For example, it uses recurrent neural networks (RNNs) and generative artificial neural networks (GANs). To do this, it builds a model using libraries such as Keras and TensorFlow and trains it using the training data. After training is complete, it evaluates the model using test data and tunes its hyperparameters to improve accuracy.

[1130] Based on the scenarios generated by the future prediction model, the server uses image generation AI to create a virtual reality space. For example, it uses Unreal Engine or Unity to create 3D models that reflect future cityscapes and new technology concepts. This VR space data is designed to include visual, auditory, and tactile elements. The generated VR space data is then sent to the user's device. Data is transmitted using the HTTP protocol or WebSocket.

[1131] The device performs real-time rendering based on the VR space data received from the server. For example, a device equipped with an NVIDIA GPU can use Unity or Unreal Engine to draw high-quality images. The device also provides a user interface, displaying menus and options that allow the user to operate within the VR space.

[1132] The user's emotions are recognized using an emotion engine built into the device. Emotion recognition is achieved by facial recognition, voice analysis, and biometric sensor data analysis. For example, facial expressions are recognized using a camera, and voice analysis is performed using a microphone. Biometric sensor data such as heart rate and electrodermal activity are also analyzed. This information is input into the system as the user's current emotional state and sent to the server.

[1133] The user then accesses the VR space by putting on a VR headset and controllers. For example, they start the virtual reality experience using an Oculus Quest 2. First-time users are shown instructions on how to operate the VR space and a usage guide. Within the virtual reality space, users can move freely and converse with other characters. Furthermore, events and character reactions within the VR space change dynamically based on emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will respond accordingly.

[1134] As a specific example, it is possible to predict future scenarios based on Tokyo's economic indicators, traffic data, and social statistics from the past five years, and recreate Tokyo in 2025. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. It then uses image generation AI to create a 3D model of Tokyo in 2025, and sends the VR space data to the device.

[1135] Users put on a VR headset and enter the VR space via their device, where they can freely explore Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system in 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive future experience.

[1136] An example of a prompt is: "Tell me the steps to develop a future prediction model to create a VR space of Tokyo in 2025 based on Tokyo's economic indicators, traffic data, and social statistics for the past five years."

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1: Collect historical data

[1139] The server uses APIs and web scraping tools to collect data such as historical economic indicators, traffic data, and social statistics. For example, it uses Google Analytics APIs and web scraping tools to obtain relevant data. The input is a data collection API or a web page URL. The output is raw historical data.

[1140] Step 2: Data cleansing

[1141] The server cleanses the collected data. This process includes imputing missing values ​​and removing outliers. For example, it uses the Python Pandas library to impute missing values ​​using the median and detect and remove outliers using Z-scores. The input is the raw historical data. The output is the cleansed data.

[1142] Step 3: Data storage

[1143] The server stores the cleansed data in a database. A general SQL database is used to efficiently manage data. The input is the cleansed data. The output is the data stored in the database.

[1144] Step 4: Develop a future prediction model

[1145] The server uses machine learning and deep learning techniques to develop a future prediction model based on the data stored in the database. For example, it uses the Keras or TensorFlow library to train a recurrent neural network (RNN) or a generative artificial neural network (GAN). The input is the data stored in the database. The output is the trained future prediction model.

[1146] Step 5: Generate future scenarios

[1147] The server generates future scenarios using a trained future prediction model. For example, it uses an RNN to predict the future economic and social conditions of Tokyo and generates specific scenarios. The input is the trained future prediction model. The output is the future scenario.

[1148] Step 6: Creating the VR space

[1149] The server uses image generation AI to create a VR space based on the generated future scenario. For example, it uses Unreal Engine or Unity to create 3D modeling of a future cityscape and creates VR content that includes visual, auditory, and tactile elements. The input is the future scenario. The output is the generated VR space data.

[1150] Step 7: Streaming VR space data

[1151] The server compresses the generated VR space data and sends it to the device. The data is transferred in real time via HTTP or WebSocket. The input is the generated VR space data. The output is the VR space data sent to the device.

[1152] Step 8: Rendering the VR space

[1153] The device performs real-time rendering based on the VR space data received from the server. Using Unity or Unreal Engine, high-quality images are drawn on devices equipped with NVIDIA GPUs. The device also provides a user interface, displaying menus and options for users to operate within the VR space. The input is the VR space data sent to the device. The output is the rendered VR space.

[1154] Step 9: Emotion Recognition

[1155] The device recognizes the user's emotions using a built-in emotion engine. It uses a camera to recognize facial expressions and a microphone to perform voice analysis. It also analyzes biosensor data such as heart rate and electrodermal activity. The inputs are the user's facial image, voice, and biosensor data. The output is the emotion recognition results.

[1156] Step 10: User's VR experience

[1157] Users access the VR space by wearing a VR headset and controllers. For example, they start a virtual reality experience using Oculus Quest 2. First-time users are shown instructions on how to use the device and a usage guide. The input is the state of wearing the VR headset and controllers. The output is the state of access to the VR space.

[1158] Step 11: Interaction and Scenario Branching

[1159] The device dynamically changes events and character reactions in the VR space based on the user's emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will react accordingly. The input is the emotion recognition results. The output is emotion-based interactions and scenario branching.

[1160] (Application example 2)

[1161] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1162] Conventional virtual reality (VR) systems have been unable to predict and recreate future events or dynamically change interactions based on the user's emotions. This has made it difficult to provide interactive VR experiences that allow users to have a more immersive experience. Furthermore, they have been unable to provide personalized interactions based on emotions, such as a futuristic shopping experience. Therefore, there has been a demand for improving the quality of the user experience and providing a more realistic and personal VR space.

[1163] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a VR space based on the generated future scenario. This makes it possible to recreate future events in a virtual reality space. The server also includes means for dynamically generating events in the VR space and branching the story in response to user operations, means for the user to freely act and converse with other characters in the VR space, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions. This improves the quality of the user experience and makes it possible to provide personalized interactions based on emotions, such as future shopping experiences.

[1164] "Past data" refers to existing information such as economic indicators, social statistics, and scientific research, and serves as the basic data for predicting the future.

[1165] A "database" is a collection of information that stores collected past data in an orderly manner so that it can be used efficiently later.

[1166] A "future prediction model" is a predictive algorithm that uses machine learning and deep learning technologies to generate future scenarios based on collected past data.

[1167] "VR space" is a three-dimensional space generated on a computer using virtual reality technology, allowing users to experience a virtual environment that is different from the physical world.

[1168] "User operation" refers to the actions and choices the user makes using controllers and gestures within the VR space.

[1169] An "event" refers to a specific occurrence or action that occurs within the VR space, and changes dynamically depending on the user's actions and emotions.

[1170] "Branching of the story" means that the scenario in the VR space changes in different directions depending on the user's choices and actions.

[1171] "Character" refers to a virtual person or creature that interacts with the user in the VR space.

[1172] "Emotion recognition" refers to the technology of analyzing a user's facial expressions, voice, and biometric sensor data to identify their current emotional state.

[1173] "Interaction" refers to the interaction between the user and the environment and characters in the VR space, which changes dynamically based on emotion recognition.

[1174] "Guide" is a function that provides information and instructions to the user using voice and text.

[1175] "Additional information" refers to additional information or explanations provided based on the user's emotions or actions.

[1176] This invention is a system that recreates future events in a virtual reality space (VR space), recognizes a user's emotions, and dynamically changes interactions based on those emotions. The invention includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely in the VR space and converse with other characters, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions.

[1177] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping, cleansing it by filling in missing values ​​and removing outliers, and then stores it in a database. Next, based on this cleansed data, a future prediction model is developed using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The model is evaluated and tuned to create the optimal future prediction model. Based on the generated future scenario, a VR space is created using video generation AI. This VR space includes visual, auditory, and tactile elements.

[1178] The user device performs real-time rendering based on the VR space data received from the server, providing a realistic experience to the user wearing the VR headset. The device uses an emotion engine to recognize the user's emotions through facial recognition, voice analysis, and biometric sensor data analysis. This information is input into the system as the user's current emotional state.

[1179] Users put on a VR headset and controllers and access the VR space. They can move freely within the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, such as surprise or excitement, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user is surprised by a futuristic transportation system, a character in the VR space will provide guidance by explaining, "Are you surprised? This is the latest transportation system of the future."

[1180] As a concrete example, consider a scenario in which the future of a city is predicted based on the city's economic indicators, traffic data, and social statistics from the past five years. The server collects and cleans the data, and uses RNN to develop a model to predict the future city. A video generation AI is used to create a 3D model of the future city, and the VR space data is sent to the device. The user puts on a VR headset and enters the VR space via the device, freely exploring the future city. The emotion engine recognizes the user's emotions, and the interaction changes accordingly.

[1181] An example of a prompt sentence is, "When the user expresses surprise at a particular product in the future shopping mall, please provide guidance or additional information based on that emotion." In this way, the present invention improves the quality of the user experience and provides a more immersive future experience.

[1182] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1183] Step 1:

[1184] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping. The input for the collection is multiple public databases and websites, and the data is obtained using these. Then, cleansing processes such as imputing missing values ​​and removing outliers are performed on the data, and the cleansed data is stored in a database. The output is the cleansed historical data.

[1185] Step 2:

[1186] The server uses the cleansed historical data stored in the database to develop a future prediction model. It takes the cleansed data as input and trains the model using machine learning algorithms such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs). It processes the data by performing feature engineering and preprocessing, converting it into a suitable format. The output is a trained future prediction model.

[1187] Step 3:

[1188] The server generates future scenarios based on the trained future prediction model. It takes the future prediction model and actual data as input, runs the prediction algorithm to simulate future events, and outputs future scenario data.

[1189] Step 4:

[1190] The server creates a VR space using image generation AI based on a future scenario. It takes future scenario data as input and generates the VR space using 3D modeling and image rendering software. Specific operations include generating 3D models, applying textures, and adding sound effects. The output is the generated VR space data.

[1191] Step 5:

[1192] The device receives VR space data from the server and renders it in real time. It takes the VR space data from the server as input and uses the GPU to perform rendering processing. Specific operations include drawing 3D graphics, playing audio, and generating haptic feedback. The output is a real-time VR view provided to the user.

[1193] Step 6:

[1194] Users access the VR space using a VR headset and controllers and act freely. The system receives user operations and motion inputs as input, and reflects those movements within the VR space. Specific movements include movement, item selection, and character interaction. The output is a change within the VR space based on user operations.

[1195] Step 7:

[1196] The device uses an emotion engine to recognize the user's emotions. User data is acquired as input from the camera, microphone, and biometric sensors, and analyzed using emotion recognition algorithms. Data processing involves analyzing facial expressions through image processing, analyzing voice, and interpreting biometric data. The output is the recognized emotional state of the user.

[1197] Step 8:

[1198] The device dynamically changes interactions in the VR space based on the recognized emotions. It takes the user's emotional state as input and executes an interaction update algorithm. Specific actions include changing the character's reaction, generating new events, and displaying guides or additional information. The output is the dynamically changed interaction.

[1199] Step 9:

[1200] The device provides guidance and additional information according to the user's emotions. It takes the recognized emotional state and current scenario information as input and generates appropriate guidance messages and additional information. Specific operations include generating audio guidance, displaying text guidance, and displaying additional information panels. The output is the guidance and additional information provided to the user.

[1201] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1202] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1203] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1204] [Fourth embodiment]

[1205] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1206] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1207] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1208] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1209] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1210] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1211] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1212] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1213] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1214] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1215] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1216] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1217] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1218] Understood. The following is a description of the "Mode for carrying out the invention."

[1219] The present invention is a system for recreating future events in a virtual reality space (VR space), and is realized by the following program. This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, and means for the user to act freely in the VR space and converse with other characters.

[1220] Server Processing

[1221] Data collection and database construction

[1222] The server uses APIs and web scraping to collect historical data, including economic indicators, social statistics, scientific research, etc. The collected data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[1223] Development of future prediction models

[1224] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[1225] VR space generation

[1226] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[1227] Terminal handling

[1228] VR space rendering

[1229] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[1230] User operations

[1231] Access to VR space

[1232] Users put on a VR headset and controllers and access the VR space through the device. If it's their first time experiencing it, the system will display basic operation instructions and a usage guide.

[1233] Freedom of movement and conversation

[1234] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to AI characters. Depending on the user's choices, the story will unfold in different directions, allowing them to enjoy multiple scenarios.

[1235] Specific examples

[1236] For example, consider a scenario that predicts Tokyo in 2025 based on Tokyo's economic indicators, transportation data, and social statistics from the past five years. The server collects and cleans this data, and uses RNN to develop a model that predicts Tokyo in 2025. A video generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device. Users can access this VR space, explore new landmarks and transportation systems, and enjoy conversations with AI characters about life in the future. The scenario branches depending on the user's choices, allowing them to experience different futures.

[1237] The processing flow will be explained below.

[1238] Server Processing

[1239] Step 1: Data collection

[1240] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[1241] Step 2: Data cleansing

[1242] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[1243] Step 3: Store in the database

[1244] The server stores the cleansed data in a database.

[1245] Step 4: Preprocessing the data

[1246] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[1247] Step 5: Develop a future prediction model

[1248] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, for example, using RNNs and GANs.

[1249] Step 6: Evaluate and tune the model

[1250] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[1251] Step 7: Generate the scenario

[1252] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[1253] Step 8: Use video generation AI

[1254] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[1255] Step 9: Sending Data

[1256] The server transmits the generated VR space data to the terminal.

[1257] Terminal handling

[1258] Step 1: Receiving and analyzing data

[1259] The device analyzes the VR space data received from the server and prepares for rendering.

[1260] Step 2: Rendering the VR space

[1261] The device uses a rendering engine such as a GPU to render the VR space in real time.

[1262] Step 3: Providing a User Interface

[1263] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[1264] User operations

[1265] Step 1: Put on the VR equipment

[1266] The user puts on a VR headset and controllers and prepares to access the VR space.

[1267] Step 2: Access the VR space

[1268] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[1269] Step 3: Free movement and conversation

[1270] Users can freely move around in the VR space and interact with the objects and characters around them. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era.

[1271] Step 4: Branching scenarios

[1272] The story changes depending on the user's choices. For example, if the user visits a specific location, different events will occur depending on the choices made. This allows the user to experience multiple future scenarios.

[1273] Specific examples

[1274] Example: A scenario looking at Tokyo in the future in 2025

[1275] Server Processing

[1276] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[1277] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[1278] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[1279] Terminal handling

[1280] 1. The device analyzes the received VR space data and prepares for rendering.

[1281] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[1282] 3. Provide landing points and navigation tools as a user interface.

[1283] User operations

[1284] 1. The user puts on a VR headset and enters the VR space via the device.

[1285] 2. Freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[1286] 3. Talk to AI characters at specific locations and hear about new technologies and social trends. The scenario branches depending on the user's choices, allowing users to enjoy different futures.

[1287] Example 1

[1288] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] In conventional technologies, it has been difficult to effectively integrate a series of processes, from data collection and cleansing, to the development of future prediction models and the generation of virtual reality spaces, in systems that use past data to make future predictions. Furthermore, there has been a lack of systems that allow users to experience scenarios that change in real time while freely acting and interacting in a virtual space. This has meant that no environment has been provided in which users can realistically experience future events and dynamically change those scenarios.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1291] In this invention, the server includes means for collecting past data, means for cleansing the collected data and storing it in a database, means for developing a future prediction model using machine learning and deep learning technologies, means for creating a virtual reality space based on a future scenario generated from the future prediction model, means for dynamically generating events in the virtual reality space in response to user operations and branching the story, and means for the user to act freely in the virtual reality space and converse with other characters. This enables the realization of a future prediction system in which users can realistically experience the future predicted from past data and can be interactively involved.

[1292] Define "data" in the following format:

[1293] "Historical data" refers to time-series information and numerical data such as economic indicators, social statistics, and scientific research.

[1294] "Means of collection" refers to methods of obtaining data from outside using technologies such as APIs and web scraping.

[1295] "Means of storing in a database" refers to the method of organizing collected data into a certain format and storing it in a database such as a relational database management system (RDBMS).

[1296] "Cleansing" refers to the process of improving data quality by preparing collected data through methods such as filling in missing values ​​and removing outliers.

[1297] "Machine learning and deep learning technologies" refer to technologies in which computers use large amounts of data to learn patterns and then use those patterns to make predictions and classifications.

[1298] A "future prediction model" refers to an algorithm or model that uses statistical methods and machine learning technology to generate future scenarios based on collected past data.

[1299] "Virtual reality space" refers to a three-dimensional computer-generated virtual environment that users can experience interactively.

[1300] "Dynamic generation" refers to a method of changing events and scenarios in real time in response to user operations and input.

[1301] "Means for branching the story" refers to a function that allows the scenario to unfold in different directions depending on the user's choices and actions.

[1302] "Means for free movement and conversation with other characters" refers to a method by which a user can freely move within a virtual reality space and interact and converse with characters.

[1303] MODE FOR CARRYING OUT THE INVENTION

[1304] The present invention relates to a system that predicts the future based on past data and allows users to experience the scenario in a virtual reality space (VR space). The system of the present invention is composed of components such as a server, a terminal, and a user, and an embodiment thereof will be described in detail below.

[1305] Server Processing

[1306] Data collection and database construction

[1307] The server uses APIs and web scraping to collect historical data. This data includes economic indicators, social statistics, scientific research, and more. Specifically, it uses APIs (e.g., APIs provided by international organizations) and web scraping technology (e.g., Python's BeautifulSoup library). The server organizes the collected data using a data cleansing tool such as Pandas and stores it in a relational database management system (e.g., MySQL). Missing values ​​are imputed and outliers are removed at this stage.

[1308] Development of future prediction models

[1309] The server uses TensorFlow or PyTorch to develop a future prediction model based on the cleansed data. For example, it trains a model that generates future scenarios from the data using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The trained model is then appropriately evaluated and tuned to build the optimal prediction model.

[1310] VR space generation

[1311] The server creates a VR space using a virtual reality engine (e.g., Unreal Engine or Unity) based on the scenario generated by the future prediction model. For example, it generates 3D models that reflect future cityscapes or new technological concepts, and includes visual, auditory, and tactile elements to add realism. The generated VR space data is then sent to the user's device via WebSocket.

[1312] Terminal handling

[1313] VR space rendering

[1314] The device renders the VR space data received from the server in real time. Specifically, it uses a high-speed rendering engine (e.g., NVIDIA RTX series GPU) to provide a realistic experience for the user. The device also uses Unity or Unreal Engine to provide a user interface, displaying menus and options that allow the user to operate within the VR space.

[1315] User operations

[1316] Access to VR space

[1317] Users put on a VR headset (such as Oculus Quest or HTC Vive) and controllers and access the VR space through their device. When using it for the first time, the system will provide basic operation instructions and a usage guide.

[1318] Freedom of movement and conversation

[1319] Users can move freely within the VR space and interact with the surrounding environment and characters. For example, they can explore a futuristic city and learn more about that era by talking to an AI character. Natural language processing technology (e.g., IBM Watson) is used to converse with the AI ​​character. Depending on the user's choices, the scenario will unfold in different directions, allowing users to enjoy multiple scenarios.

[1320] Specific examples

[1321] For example, consider a scenario in which a city's economic indicators, traffic data, and social statistics from the past five years are used to predict what it will be like in 2025. The server collects this data using the IMF's API and the World Bank's open data, cleansing it with Pandas, and developing a 2025 prediction model (RNN model) using TensorFlow. A 3D model of the city in 2025 is created using Unreal Engine and sent to the device via WebSocket. Users can access the VR space by wearing an Oculus Quest, exploring new landmarks and transportation systems, and enjoying conversations with AI characters powered by IBM Watson about life in the future. Furthermore, the scenario branches depending on the user's choices, allowing them to experience different futures.

[1322] Prompt Sentence Examples

[1323] By inputting prompts like the following into a generative AI model, we can generate future scenarios:

[1324] "You will create a virtual cityscape of 2025 and generate scenarios that include future urban landscapes, transportation systems, and new technological applications. You will also include new landmarks for users to explore and specific interaction scenes with AI characters."

[1325] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1326] Step 1: Data collection

[1327] Server Processing

[1328] The server uses APIs and web scraping to collect historical data, such as economic indicators, social statistics, and scientific research. Specifically, it uses APIs (e.g., public sector APIs) and web scraping techniques (e.g., Python's BeautifulSoup library).

[1329] Input: API endpoint or website URL

[1330] Output: Collected data (e.g., economic indicator data in JSON format)

[1331] Specific operation: The server sends a request to the API endpoint, saves the returned JSON data in internal storage, and performs web scraping to obtain the required information in text format.

[1332] Step 2: Data cleansing and database construction

[1333] Server Processing

[1334] The server cleanses the collected data and stores it in a database. Specifically, it completes missing values ​​and removes outliers, and stores the organized data in a relational database management system (e.g., MySQL).

[1335] Input: Collected data (output of step 1)

[1336] Output: Cleansed data (structured data)

[1337] How it works: The server uses the Pandas library to analyze the collected data, impute missing values ​​with the mean, and detect and remove outliers. It then uses SQL INSERT statements to store the cleansed data in a MySQL database.

[1338] Step 3: Develop a future prediction model

[1339] Server Processing

[1340] The server uses the cleansed data to develop future prediction models, specifically training RNNs and GANs using TensorFlow and PyTorch.

[1341] Input: Cleansed data (output of Step 2)

[1342] Output: A trained future prediction model

[1343] Specific operation: The server uses the TensorFlow library to build an RNN model, inputs the cleansed dataset into the model to train it, evaluates the model's performance after training, and saves the optimized prediction model.

[1344] Step 4: Creating the VR space

[1345] Server Processing

[1346] The server creates a VR space based on the scenario generated by the future prediction model, using a virtual reality engine (e.g., Unreal Engine or Unity).

[1347] Input: Forecast model and scenario data (output of Step 3)

[1348] Output: VR space data

[1349] How it works: The server inputs scenario data into Unreal Engine and uses 3D modeling tools to generate a model to recreate the future cityscape. The generated 3D model and other VR data are then sent to the user's device via WebSocket.

[1350] Step 5: Sending VR space data

[1351] Server Processing

[1352] The server sends the generated VR space data to the user's device.

[1353] Input: VR space data (output of step 4)

[1354] Output: Transmitted VR space data

[1355] Specific operation: The server opens a WebSocket and establishes a connection to the device. The generated VR space data is compressed and sent to the device via the WebSocket.

[1356] Step 6: Rendering the VR space

[1357] Terminal handling

[1358] The device renders the VR space data received from the server in real time, using a high-speed rendering engine.

[1359] Input: Received VR space data (output of step 5)

[1360] Output: Rendered VR scene

[1361] How it works: The device decompresses the received data and uses an NVIDIA RTX series GPU to perform high-speed rendering. The rendered scene is then displayed on the VR headset.

[1362] Step 7: Access the VR space

[1363] User operations

[1364] Users access the VR space by wearing a VR headset and controllers.

[1365] Input: Wearing and operating a VR headset and controllers

[1366] Output: Access to VR space

[1367] Specific operation: The user puts on the VR headset and launches the application on the device. When launched for the first time, a tutorial on basic operation is played.

[1368] Step 8: Free movement and conversation

[1369] User operations

[1370] Users can move freely within the VR space and interact with AI characters.

[1371] Input: User actions and commands

[1372] Output: Interactive VR experience

[1373] How it works: The user operates the VR controller to move around the virtual space, approach the AI ​​character, and start a conversation. A natural language processing engine is used to converse with the AI ​​character, and the scenario changes dynamically depending on the user's choices.

[1374] (Application example 1)

[1375] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1376] As autonomous vehicles become more widespread, simulations are required to predict future changes in the transportation environment and infrastructure and adapt to them. However, previous simulation technologies did not provide realistic experiences, making it difficult to properly understand the impact of actual autonomous vehicle operations and infrastructure. In addition, there was a lack of visualization methods that allowed users to experience specific future scenarios and intuitively understand them.

[1377] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1378] In this invention, the server includes a means for collecting past data and storing it in a database, a means for developing a future prediction model based on the collected data, and a means for creating a VR space based on the generated future scenario, allowing users to realistically experience future operation scenarios of autonomous vehicles in the VR space and specifically understand changes in traffic conditions and infrastructure.

[1379] "Past data" refers to information that shows events or records that have already occurred, and is detailed information based on time and space.

[1380] A "database" is a collection of information that is systematically managed so that past data can be efficiently stored, searched, and used.

[1381] A "future prediction model" is a mathematical model that uses statistical methods and machine learning techniques to predict future events and conditions based on past data.

[1382] "VR space" is a computer-generated virtual reality space, an interactive 3D environment in which users can feel as if they are actually there.

[1383] "User operation" refers to an input operation that a user performs on the system via an interface, and is an action that triggers an implemented function or operation.

[1384] An "event" refers to a specific occurrence or action that occurs within the VR space, and is an element that causes the story to progress or the environment to change.

[1385] "Branching the story" means generating a scenario with different developments depending on the user's choices and actions, and events in the VR space changing accordingly.

[1386] "Acting freely" means that the user can move around in the VR space, perform specific actions, and interact with characters and the environment of their own volition.

[1387] "Character" refers to any other virtual entity that the user controls or interacts with within the VR space, and can take the form of a person, animal, robot, or other entity.

[1388] An "autonomous vehicle" is a vehicle equipped with technology that allows it to recognize its surroundings and operate autonomously without the need for human driving.

[1389] "Future operation scenarios" are specific simulations of how autonomous vehicles will operate in the future, based on future prediction models.

[1390] "Traffic conditions" refers to comprehensive information that indicates the state of traffic flow, congestion, accidents, etc. at a specific time and location.

[1391] "Infrastructure" refers to basic public facilities for transportation and logistics, such as roads, bridges, tunnels, and signal systems.

[1392] "Experience" refers to the user actually operating within a virtual space and experiencing it realistically through their senses of sight, hearing, touch, etc.

[1393] This invention is a system that recreates operational scenarios for autonomous vehicles in future cities in a VR space. This system collects past data, builds a future prediction model, and implements a series of specific steps to create a VR space based on the generated future scenario. The details are described below.

[1394] Server Processing

[1395] Data collection and database construction

[1396] The server collects historical traffic data, accident rates, data on new infrastructure projects, and more. This data is collected using APIs and web scraping. The collected data undergoes a cleansing process, including filling in missing values ​​and removing outliers, before being stored in a database. Specifically, the data is shaped and cleansed using Python and the Pandas library.

[1397] Development of future prediction models

[1398] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it uses a recurrent neural network (LSTM) to train a model that generates future scenarios from past traffic data. The TensorFlow and Keras libraries are used for this process.

[1399] VR space generation

[1400] Based on the generated future prediction scenario, a VR space is created using image generation AI. This uses libraries such as OpenVR. The server creates 3D models of future cityscapes and new infrastructure, generating realistic VR space data that includes visual, auditory, and tactile elements. This data is then sent to the user's device.

[1401] Terminal handling

[1402] VR space rendering

[1403] The device renders the VR space in real time based on the received VR space data. A high-speed rendering engine such as a GPU is used to ensure the user has a realistic experience. The device also provides an interface, displaying menus and options for the user to operate within the VR space.

[1404] User operations

[1405] Access to VR space

[1406] Users access the VR space using a head-mounted display (HMD) and controllers. The system displays basic operation instructions and usage guides.

[1407] Freedom of movement and conversation

[1408] Users can freely move around in the VR space and interact with the surrounding environment and characters. For example, they can drive through a futuristic city and explore predicted traffic conditions and new infrastructure projects. Depending on the user's choices, the story branches, allowing them to experience different future scenarios.

[1409] Specific examples

[1410] For example, to simulate traffic conditions in Tokyo in 2025, we collect past traffic data, accident data, and information on new infrastructure projects, and then use LSTM to predict future scenarios. Based on the generated scenarios, we create a 3D model of Tokyo in 2025 in a VR space, and users can explore the space through a VR headset.

[1411] Prompt Sentence Examples

[1412] "Create a VR application that simulates traffic conditions in Tokyo in 2025. Explain how you can use past traffic data to predict the future and recreate that data in a VR space."

[1413] In this way, a system is created that allows users to intuitively understand specific traffic conditions and infrastructure changes by experiencing future operating scenarios for autonomous vehicles in a VR space.

[1414] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1415] Step 1:

[1416] The server collects historical traffic data, accident rates, data on new infrastructure projects, etc. using APIs and web scraping. Before storing the collected data in a database, it performs cleansing processes such as filling in missing values ​​and removing outliers. The input data is raw traffic data, accident data, and infrastructure data, and the output is cleansed data. Specific operations use Python and the Pandas library.

[1417] Step 2:

[1418] The server builds a future prediction model based on the cleansed data. This process uses machine learning algorithms such as recurrent neural networks (LSTM). The input is the cleansed historical data, and the output is future traffic scenarios. The model is trained using the TensorFlow and Keras libraries. Specifically, the data is reformatted and an LSTM model is trained.

[1419] Step 3:

[1420] The server creates a VR space based on a scenario generated from a future prediction model. This process uses image generation AI and the OpenVR library. The input is future scenario data, and the output is the completed VR space data. Specifically, it generates a 3D city model and creates realistic data that includes visual, auditory, and tactile elements.

[1421] Step 4:

[1422] The device renders the VR space data received from the server in real time. The input is the VR space data sent from the server, and the output is the VR space experienced by the user through the HMD. Specifically, it uses a high-speed rendering engine (GPU) to provide a user interface.

[1423] Step 5:

[1424] Users access the VR space using a head-mounted display (HMD) and controllers. When a user experiences the system for the first time, the system displays basic operating instructions and a usage guide. The input is an operation instruction from the user, and the output is the system's response based on that instruction. Specific operations include initial setup of the HMD and controllers and display of the usage guide.

[1425] Step 6:

[1426] Users can move freely within the VR space and interact with the surrounding environment and characters. Input is the user's movement and control instructions, and output is changes to the environment within the VR space and the character's reactions. Specific actions include exploring the VR space, conversing with AI characters, and branching storylines.

[1427] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1428] The present invention is a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions, and is realized by the following program: This system includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely within the VR space and converse with other characters, and an emotion engine that recognizes the user's emotions.

[1429] Server Processing

[1430] Data collection and database construction

[1431] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping. This data undergoes a cleansing process, such as imputing missing values ​​and removing outliers, before being stored in a database.

[1432] Development of future prediction models

[1433] The server uses machine learning and deep learning techniques to develop future prediction models based on the cleansed data. For example, it trains models that generate future scenarios from data using recurrent neural networks (RNNs) and generative artificial neural networks (GANs). After learning, it evaluates and tunes the models to build the optimal prediction model.

[1434] VR space generation

[1435] The server uses image generation AI to create a VR space based on a scenario generated by a future prediction model. For example, it creates a 3D model that reflects a future cityscape or a new technology concept. This VR space data includes visual, auditory, and tactile elements to give it a sense of reality. The generated VR space data is then sent to the user's device.

[1436] Terminal handling

[1437] VR space rendering

[1438] The device renders the VR space data received from the server in real time, using a high-speed rendering engine such as a GPU to ensure the user has a realistic experience. The device also provides a user interface, displaying menus and options for the user to operate within the VR space.

[1439] emotion recognition

[1440] The device uses an emotion engine to recognize the user's emotions. This information is input into the system as the user's current emotional state.

[1441] User operations

[1442] Wearing VR equipment

[1443] The user puts on a VR headset and controllers and prepares to access the VR space.

[1444] Access to VR space

[1445] Users enter the VR space through a device. If it is their first time experiencing it, the system will display basic operating instructions and a usage guide.

[1446] Freedom of movement and conversation

[1447] Users can freely move around in the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user makes a surprised expression, the surrounding characters will sense the user's surprise and respond accordingly.

[1448] Branching scenarios

[1449] The story changes depending on the user's choices. The user's emotional state also affects the branching of the scenario. For example, if the user has an inquisitive expression, a new event will occur based on that emotion. This allows the user to experience multiple future scenarios.

[1450] Specific examples

[1451] For example, consider a scenario in which Tokyo in 2025 is predicted based on the past five years' worth of economic indicators, traffic data, and social statistics. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. Image generation AI is used to create a 3D model of Tokyo in 2025, and the VR space data is sent to the device.

[1452] Users put on a VR headset and enter the VR space via their device, freely exploring Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system for 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive experience of the future.

[1453] The processing flow will be explained below.

[1454] Server Processing

[1455] Step 1: Data collection

[1456] The server collects historical data (economic indicators, social statistics, scientific research, etc.) through APIs and web scraping.

[1457] Step 2: Data cleansing

[1458] The server cleanses the collected data, filling in missing values, removing outliers, and standardizing the data format.

[1459] Step 3: Store in the database

[1460] The server stores the cleansed data in a database.

[1461] Step 4: Preprocessing the data

[1462] The server performs preprocessing such as normalizing, standardizing, and feature selection of the data for future prediction models.

[1463] Step 5: Develop a future prediction model

[1464] The server uses machine learning and deep learning algorithms to develop future prediction models and train them using training data, such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs).

[1465] Step 6: Evaluate and tune the model

[1466] The server evaluates the learned model and adjusts hyperparameters to improve accuracy.

[1467] Step 7: Generate the scenario

[1468] The server uses the developed future prediction model to generate future scenarios by specifying specific dates and conditions.

[1469] Step 8: Use video generation AI

[1470] The server uses image generation AI to create VR space data based on the generated future scenario, generating 3D data for cityscapes, buildings, transportation systems, and more.

[1471] Step 9: Sending Data

[1472] The server transmits the generated VR space data to the terminal.

[1473] Terminal handling

[1474] Step 1: Receiving and analyzing data

[1475] The device analyzes the VR space data received from the server and prepares for rendering.

[1476] Step 2: Rendering the VR space

[1477] The device uses a rendering engine such as a GPU to render the VR space in real time.

[1478] Step 3: Providing a User Interface

[1479] The device provides an interface (menus, options, navigation tools, etc.) for the user to operate within the VR space.

[1480] Step 4: Start Emotion Recognition

[1481] The device uses devices such as a camera, microphone, and biosensor to analyze the user's facial expressions, tone of voice, heart rate, etc. in real time, and an emotion engine determines the user's emotional state.

[1482] User operations

[1483] Step 1: Put on the VR equipment

[1484] The user puts on a VR headset and controllers and prepares to access the VR space.

[1485] Step 2: Access the VR space

[1486] Users enter the VR space through a device, and when they enter an area for the first time, the system will provide basic operating instructions and a usage guide.

[1487] Step 3: Free movement and conversation

[1488] Users can move freely within the VR space and interact with surrounding objects and characters. For example, they can walk around a futuristic city and talk to AI characters to learn more about that era. The emotion engine recognizes the user's emotions, and the characters respond accordingly.

[1489] Step 4: Branching scenarios

[1490] The story changes depending on the user's choices and emotional state. For example, if the user is surprised or excited, new events are dynamically generated based on that emotion. The user can experience multiple future scenarios based on different emotions.

[1491] Specific examples

[1492] Example: A scenario looking at Tokyo in the future in 2025

[1493] Server Processing

[1494] 1. The server collects Tokyo's economic indicators, traffic data, social statistics, etc. for the past five years, cleans them, and stores them in a database.

[1495] 2. Based on the collected data, a future prediction model is developed using RNN to generate a scenario that predicts Tokyo in 2025.

[1496] 3. Create a 3D model of Tokyo in 2025 using image generation AI, create VR space data, and send it to the device.

[1497] Terminal handling

[1498] 1. The device analyzes the received VR space data and prepares for rendering.

[1499] 2. Using a GPU, we will render a VR space of Tokyo in 2025 in real time.

[1500] 3. Provide landing points and navigation tools for the user interface.

[1501] 4. The device uses a camera and microphone to recognize the user's facial expressions and tone of voice, and the emotion engine analyzes this data to determine the user's emotions.

[1502] User operations

[1503] 1. The user puts on a VR headset and enters the VR space via the device.

[1504] 2. Users can freely explore Tokyo in 2025 and experience new landmarks and transportation systems.

[1505] 3. If the user expresses excitement or surprise, the emotion engine will recognize that emotion and the character in VR will begin explaining, "Are you surprised? This is the latest transportation system of 2025." In this way, interactions based on the user's emotions are realized.

[1506] 4. Depending on the user's emotional state, the scenario branches and new events occur. For example, if a user visits a particular location in a curious state, more detailed information or other new events will be provided.

[1507] Example 2

[1508] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Currently, there is no technology that can simulate future events in a virtual reality space and dynamically change interactions based on the user's emotions. Current systems lack the ability to simulate events using future prediction models or to recognize the user's emotions and change interactions accordingly, so new technologies are needed to provide a more realistic and immersive experience.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a virtual reality space based on the generated future scenario. This enables advanced simulations using the future prediction model. The server also includes means for dynamically generating events in the virtual reality space in response to user operations and branching the story, means for allowing the user to freely act in the virtual reality space and converse with other characters, an emotion engine for recognizing the user's emotions, and means for dynamically changing the reactions of events and characters in the virtual reality space based on the emotion recognition. This realizes interactions tailored to the user's emotions, enabling a more realistic and immersive experience.

[1511] "Historical data" refers to information or records collected historically, including statistical information such as economic indicators, traffic data, and social statistics within a specific period of time.

[1512] A "database" is a collection of information that is organized and stored to efficiently manage collected data and make it easy to search and retrieve.

[1513] A "future forecasting model" is a mathematical or machine learning algorithm designed to predict future events or conditions based on past data.

[1514] A "virtual reality space" is a primarily three-dimensional virtual environment generated using computer technology, which allows users to experience it realistically through their senses of sight, hearing, and touch.

[1515] "User operation" refers to the inputs and actions that a user makes within a virtual reality space using VR equipment or a controller.

[1516] An "event" refers to an occurrence or action that occurs within a virtual reality space.

[1517] "Branching of the story" refers to the function in which the scenario changes depending on the user's operations and choices.

[1518] "Character" refers to a person, creature, or other entity that appears in a virtual reality space.

[1519] An "emotion engine" is a technology for recognizing a user's emotional state, and includes facial recognition, voice analysis, and analysis of biometric sensor data.

[1520] "Interaction" refers to a two-way exchange between a user and a virtual reality space and the characters present within it.

[1521] The present invention provides a system that recreates future events in a virtual reality space (VR space), recognizes the user's emotions, and dynamically changes interactions based on those emotions. This system is realized by the following means.

[1522] The server first collects historical data through APIs or web scraping. The collected data ranges from economic indicators, traffic data, and social statistics. This data collection uses hardware and software such as data acquisition interfaces and analysis programs. For example, Google Analytics API or general web scraping tools are used. The data is then cleansed by completing processes such as filling in missing values ​​and removing outliers, and the organized data is stored in a database. A general SQL database is often used for database management.

[1523] The server then uses machine learning and deep learning techniques to develop a future prediction model based on the cleansed data. For example, it uses recurrent neural networks (RNNs) and generative artificial neural networks (GANs). To do this, it builds a model using libraries such as Keras and TensorFlow and trains it using the training data. After training is complete, it evaluates the model using test data and tunes its hyperparameters to improve accuracy.

[1524] Based on the scenarios generated by the future prediction model, the server uses image generation AI to create a virtual reality space. For example, it uses Unreal Engine or Unity to create 3D models that reflect future cityscapes and new technology concepts. This VR space data is designed to include visual, auditory, and tactile elements. The generated VR space data is then sent to the user's device. Data is transmitted using the HTTP protocol or WebSocket.

[1525] The device performs real-time rendering based on the VR space data received from the server. For example, a device equipped with an NVIDIA GPU can use Unity or Unreal Engine to draw high-quality images. The device also provides a user interface, displaying menus and options that allow the user to operate within the VR space.

[1526] The user's emotions are recognized using an emotion engine built into the device. Emotion recognition is achieved by facial recognition, voice analysis, and biometric sensor data analysis. For example, facial expressions are recognized using a camera, and voice analysis is performed using a microphone. Biometric sensor data such as heart rate and electrodermal activity are also analyzed. This information is input into the system as the user's current emotional state and sent to the server.

[1527] The user then accesses the VR space by putting on a VR headset and controllers. For example, they start the virtual reality experience using an Oculus Quest 2. First-time users are shown instructions on how to operate the VR space and a usage guide. Within the virtual reality space, users can move freely and converse with other characters. Furthermore, events and character reactions within the VR space change dynamically based on emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will respond accordingly.

[1528] As a specific example, it is possible to predict future scenarios based on Tokyo's economic indicators, traffic data, and social statistics from the past five years, and recreate Tokyo in 2025. The server collects and cleanses this data, and uses RNN to develop a model that predicts Tokyo in 2025. It then uses image generation AI to create a 3D model of Tokyo in 2025, and sends the VR space data to the device.

[1529] Users put on a VR headset and enter the VR space via their device, where they can freely explore Tokyo in 2025. The emotion engine recognizes the user's surprise or excitement, causing the characters in the VR to react accordingly or triggering new events. For example, if the user shows surprise at the future transportation system, the character in the system will begin explaining, "Are you surprised? This is the latest transportation system in 2025." In this way, scenarios branch based on the user's emotions, providing a more immersive future experience.

[1530] An example of a prompt is: "Tell me the steps to develop a future prediction model to create a VR space of Tokyo in 2025 based on Tokyo's economic indicators, traffic data, and social statistics for the past five years."

[1531] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1532] Step 1: Collect historical data

[1533] The server uses APIs and web scraping tools to collect data such as historical economic indicators, traffic data, and social statistics. For example, it uses Google Analytics APIs and web scraping tools to obtain relevant data. The input is a data collection API or a web page URL. The output is raw historical data.

[1534] Step 2: Data cleansing

[1535] The server cleanses the collected data. This process includes imputing missing values ​​and removing outliers. For example, it uses the Python Pandas library to impute missing values ​​using the median and detect and remove outliers using Z-scores. The input is the raw historical data. The output is the cleansed data.

[1536] Step 3: Data storage

[1537] The server stores the cleansed data in a database. A general SQL database is used to efficiently manage data. The input is the cleansed data. The output is the data stored in the database.

[1538] Step 4: Develop a future prediction model

[1539] The server uses machine learning and deep learning techniques to develop a future prediction model based on the data stored in the database. For example, it uses the Keras or TensorFlow library to train a recurrent neural network (RNN) or a generative artificial neural network (GAN). The input is the data stored in the database. The output is the trained future prediction model.

[1540] Step 5: Generate future scenarios

[1541] The server generates future scenarios using a trained future prediction model. For example, it uses an RNN to predict the future economic and social conditions of Tokyo and generates specific scenarios. The input is the trained future prediction model. The output is the future scenario.

[1542] Step 6: Creating the VR space

[1543] The server uses image generation AI to create a VR space based on the generated future scenario. For example, it uses Unreal Engine or Unity to create 3D modeling of a future cityscape and creates VR content that includes visual, auditory, and tactile elements. The input is the future scenario. The output is the generated VR space data.

[1544] Step 7: Streaming VR space data

[1545] The server compresses the generated VR space data and sends it to the device. The data is transferred in real time via HTTP or WebSocket. The input is the generated VR space data. The output is the VR space data sent to the device.

[1546] Step 8: Rendering the VR space

[1547] The device performs real-time rendering based on the VR space data received from the server. Using Unity or Unreal Engine, high-quality images are drawn on devices equipped with NVIDIA GPUs. The device also provides a user interface, displaying menus and options for users to operate within the VR space. The input is the VR space data sent to the device. The output is the rendered VR space.

[1548] Step 9: Emotion Recognition

[1549] The device recognizes the user's emotions using a built-in emotion engine. It uses a camera to recognize facial expressions and a microphone to perform voice analysis. It also analyzes biosensor data such as heart rate and electrodermal activity. The inputs are the user's facial image, voice, and biosensor data. The output is the emotion recognition results.

[1550] Step 10: User's VR experience

[1551] Users access the VR space by wearing a VR headset and controllers. For example, they start a virtual reality experience using Oculus Quest 2. First-time users are shown instructions on how to use the device and a usage guide. The input is the state of wearing the VR headset and controllers. The output is the state of access to the VR space.

[1552] Step 11: Interaction and Scenario Branching

[1553] The device dynamically changes events and character reactions in the VR space based on the user's emotions recognized by the emotion engine. For example, if the user shows a surprised expression, the surrounding characters will react accordingly. The input is the emotion recognition results. The output is emotion-based interactions and scenario branching.

[1554] (Application example 2)

[1555] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1556] Conventional virtual reality (VR) systems have been unable to predict and recreate future events or dynamically change interactions based on the user's emotions. This has made it difficult to provide interactive VR experiences that allow users to have a more immersive experience. Furthermore, they have been unable to provide personalized interactions based on emotions, such as a futuristic shopping experience. Therefore, there has been a demand for improving the quality of the user experience and providing a more realistic and personal VR space.

[1557] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, and means for creating a VR space based on the generated future scenario. This makes it possible to recreate future events in a virtual reality space. The server also includes means for dynamically generating events in the VR space and branching the story in response to user operations, means for the user to freely act and converse with other characters in the VR space, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions. This improves the quality of the user experience and makes it possible to provide personalized interactions based on emotions, such as future shopping experiences.

[1558] "Past data" refers to existing information such as economic indicators, social statistics, and scientific research, and serves as the basic data for predicting the future.

[1559] A "database" is a collection of information that stores collected past data in an orderly manner so that it can be used efficiently later.

[1560] A "future prediction model" is a predictive algorithm that uses machine learning and deep learning technologies to generate future scenarios based on collected past data.

[1561] "VR space" is a three-dimensional space generated on a computer using virtual reality technology, allowing users to experience a virtual environment that is different from the physical world.

[1562] "User operation" refers to the actions and choices the user makes using controllers and gestures within the VR space.

[1563] An "event" refers to a specific occurrence or action that occurs within the VR space, and changes dynamically depending on the user's actions and emotions.

[1564] "Branching of the story" means that the scenario in the VR space changes in different directions depending on the user's choices and actions.

[1565] "Character" refers to a virtual person or creature that interacts with the user in the VR space.

[1566] "Emotion recognition" refers to the technology of analyzing a user's facial expressions, voice, and biometric sensor data to identify their current emotional state.

[1567] "Interaction" refers to the interaction between the user and the environment and characters in the VR space, which changes dynamically based on emotion recognition.

[1568] "Guide" is a function that provides information and instructions to the user using voice and text.

[1569] "Additional information" refers to additional information or explanations provided based on the user's emotions or actions.

[1570] This invention is a system that recreates future events in a virtual reality space (VR space), recognizes a user's emotions, and dynamically changes interactions based on those emotions. The invention includes means for collecting past data and storing it in a database, means for developing a future prediction model based on the collected data, means for creating a VR space based on the generated future scenario, means for dynamically generating events in the VR space in response to user operations and branching the story, means for the user to act freely in the VR space and converse with other characters, means for recognizing emotions and dynamically changing interactions based on those emotions, and means for providing guides and additional information in response to the user's emotions.

[1571] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping, cleansing it by filling in missing values ​​and removing outliers, and then stores it in a database. Next, based on this cleansed data, a future prediction model is developed using a recurrent neural network (RNN) or a generative artificial neural network (GAN). The model is evaluated and tuned to create the optimal future prediction model. Based on the generated future scenario, a VR space is created using video generation AI. This VR space includes visual, auditory, and tactile elements.

[1572] The user device performs real-time rendering based on the VR space data received from the server, providing a realistic experience to the user wearing the VR headset. The device uses an emotion engine to recognize the user's emotions through facial recognition, voice analysis, and biometric sensor data analysis. This information is input into the system as the user's current emotional state.

[1573] Users put on a VR headset and controllers and access the VR space. They can move freely within the VR space and interact with the surrounding environment and characters. The emotion engine recognizes the user's emotions, such as surprise or excitement, and the reactions of events and characters in the VR space change dynamically accordingly. For example, if the user is surprised by a futuristic transportation system, a character in the VR space will provide guidance by explaining, "Are you surprised? This is the latest transportation system of the future."

[1574] As a concrete example, consider a scenario in which the future of a city is predicted based on the city's economic indicators, traffic data, and social statistics from the past five years. The server collects and cleans the data, and uses RNN to develop a model to predict the future city. A video generation AI is used to create a 3D model of the future city, and the VR space data is sent to the device. The user puts on a VR headset and enters the VR space via the device, freely exploring the future city. The emotion engine recognizes the user's emotions, and the interaction changes accordingly.

[1575] An example of a prompt sentence is, "When the user expresses surprise at a particular product in the future shopping mall, please provide guidance or additional information based on that emotion." In this way, the present invention improves the quality of the user experience and provides a more immersive future experience.

[1576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1577] Step 1:

[1578] The server collects historical data such as economic indicators, social statistics, and scientific research through APIs and web scraping. The input for the collection is multiple public databases and websites, and the data is obtained using these. Then, cleansing processes such as imputing missing values ​​and removing outliers are performed on the data, and the cleansed data is stored in a database. The output is the cleansed historical data.

[1579] Step 2:

[1580] The server uses the cleansed historical data stored in the database to develop a future prediction model. It takes the cleansed data as input and trains the model using machine learning algorithms such as recurrent neural networks (RNNs) and generative artificial neural networks (GANs). It processes the data by performing feature engineering and preprocessing, converting it into a suitable format. The output is a trained future prediction model.

[1581] Step 3:

[1582] The server generates future scenarios based on the trained future prediction model. It takes the future prediction model and actual data as input, runs the prediction algorithm to simulate future events, and outputs future scenario data.

[1583] Step 4:

[1584] The server creates a VR space using image generation AI based on a future scenario. It takes future scenario data as input and generates the VR space using 3D modeling and image rendering software. Specific operations include generating 3D models, applying textures, and adding sound effects. The output is the generated VR space data.

[1585] Step 5:

[1586] The device receives VR space data from the server and renders it in real time. It takes the VR space data from the server as input and uses the GPU to perform rendering processing. Specific operations include drawing 3D graphics, playing audio, and generating haptic feedback. The output is a real-time VR view provided to the user.

[1587] Step 6:

[1588] Users access the VR space using a VR headset and controllers and act freely. The system receives user operations and motion inputs as input, and reflects those movements within the VR space. Specific movements include movement, item selection, and character interaction. The output is a change within the VR space based on user operations.

[1589] Step 7:

[1590] The device uses an emotion engine to recognize the user's emotions. User data is acquired as input from the camera, microphone, and biometric sensors, and analyzed using emotion recognition algorithms. Data processing involves analyzing facial expressions through image processing, analyzing voice, and interpreting biometric data. The output is the recognized emotional state of the user.

[1591] Step 8:

[1592] The device dynamically changes interactions in the VR space based on the recognized emotions. It takes the user's emotional state as input and executes an interaction update algorithm. Specific actions include changing the character's reaction, generating new events, and displaying guides or additional information. The output is the dynamically changed interaction.

[1593] Step 9:

[1594] The device provides guidance and additional information according to the user's emotions. It takes the recognized emotional state and current scenario information as input and generates appropriate guidance messages and additional information. Specific operations include generating audio guidance, displaying text guidance, and displaying additional information panels. The output is the guidance and additional information provided to the user.

[1595] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1597] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1598] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1599] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1600] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1601] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1602] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1603] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1604] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1605] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1606] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1607] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1608] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1609] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1610] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1611] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1612] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1613] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Theref...

Claims

1. a means for collecting and storing historical data in a database; A means of developing future prediction models based on collected data; A means for creating a VR space based on the generated future scenario; A means to dynamically generate events in the VR space according to user operations and branch the story. A system that includes a means for users to move freely within a VR space and converse with other characters.

2. 2. The system according to claim 1, further comprising means for cleansing necessary information from past data and storing the cleansed information in the database.

3. 10. The system of claim 1, further comprising means for using machine learning and deep learning techniques in developing the future prediction model.

Images