System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting sensor data; means for analyzing the sensor data and modeling city activity and traffic flow; means for building a virtual environment based on the generated model; means for generating human and vehicle behavior in the virtual environment; and means for running a simulation in the virtual environment and analyzing the results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In autonomous driving technology and urban development, it is sometimes difficult to conduct demonstration experiments in real environments due to safety and legal restrictions. There is also a growing need to improve the efficiency and quality of urban development, logistics, and disaster prevention. To address these challenges, there is a demand for virtual spaces that can recreate complex environments similar to those in the real world. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. It provides a means for collecting sensor data and a means for analyzing the sensor data to model urban activities and traffic flows. It also provides a system including a means for constructing a virtual environment based on the generated model, a means for generating the behavior of people and vehicles in the virtual environment, and a means for running simulations in the virtual environment and analyzing the results. This makes it possible to conduct demonstration experiments and simulations in a low-risk environment, improving the accuracy of autonomous driving technology and streamlining urban development and disaster prevention measures.

[0006] "Sensor data" is a general term for various data such as position information, speed information, and environmental information obtained from sensors.

[0007] "Analytical tools" refer to algorithms or processes used to analyze collected data and extract specific information or patterns.

[0008] "Modeling" is the process of generating virtual structures and patterns based on real-world data.

[0009] A "virtual environment" is an environment that mimics real-world urban and traffic conditions and is simulated on a computer.

[0010] "Generative means" refers to the algorithms and processes used to create various behaviors and phenomena within a virtual environment.

[0011] "Simulation" is a method of recreating realistic situations in a virtual environment to predict behavior and outcomes.

[0012] "Means of analysis" means the process or method for evaluating the simulation results and creating a report. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system that collects and analyzes sensor data, generates a virtual environment, and executes a simulation within it. This system has three main functions: a server, a terminal, and a user.

[0035] Server Processing

[0036] 1. Data Collection

[0037] The server collects GPS data, people flow data, purchasing data, and other data in real time from IoT sensors and mobile devices.

[0038] The server performs preprocessing on the collected data, such as noise removal and missing value completion, and converts it into a format that is easy to analyze.

[0039] 2. Data Analysis

[0040] The server classifies the collected data, for example, into traffic data, people flow data, etc.

[0041] The server uses analytical algorithms to model traffic and pedestrian flow patterns in a city.

[0042] For example, the server analyzes traffic congestion patterns by time of day and the number of customers in a particular area.

[0043] 3. Building a Virtual Environment Using Generative AI

[0044] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc.

[0045] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[0046] 4. Run the simulation

[0047] The server sets the simulation parameters and scenarios, including settings based on user input and predefined conditions.

[0048] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[0049] The server collects and analyzes the simulation results and generates a report of the analysis results.

[0050] Terminal handling

[0051] 1. Data transmission

[0052] Terminals (e.g., sensors and IoT devices) collect data in real time from GPS sensors and cameras.

[0053] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[0054] 2. Running the simulation

[0055] The terminal receives the information about the virtual environment sent from the server.

[0056] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[0057] Data generated during the simulation is sent from the device to a server for further analysis.

[0058] User Action

[0059] 1. Enter settings

[0060] The user uses a dedicated interface to input simulation parameters, such as scenario conditions and goals.

[0061] After completing the settings, the user instructs the start of the simulation.

[0062] 2. Check the results

[0063] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[0064] The user evaluates the simulation results, sets new conditions as necessary, and instructs the simulation to be executed again.

[0065] Specific examples

[0066] For example, in a self-driving car simulation, the server collects and analyzes traffic and pedestrian flow data within a city. Using the model derived from the analysis, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, and the user can review the results to evaluate the performance of the self-driving car.

[0067] The processing flow will be explained below.

[0068] Server Processing

[0069] Step 1:

[0070] The server collects GPS data, people flow data, and purchase data in real time from IoT sensors and mobile devices. When a data reception trigger is activated, new data is transferred to the collection system.

[0071] Step 2:

[0072] The collected data is preprocessed to remove noise and fill in missing values. The server detects outliers in the data and executes algorithms to correct them.

[0073] Step 3:

[0074] The server classifies the pre-processed data into traffic data, people flow data, etc. The data classification module does this automatically.

[0075] Step 4:

[0076] Analytical algorithms are applied to analyze and model urban activity and traffic flow patterns, utilizing statistical methods and machine learning algorithms.

[0077] Step 5:

[0078] Based on the analyzed model, a virtual environment is generated. The server creates a 3D model and simulation space, and places roads, buildings, traffic signals, etc. in it.

[0079] Step 6:

[0080] Generative AI is used to simulate the movement of people and vehicles in a virtual environment. The AI ​​model runs real-time simulations and generates scenarios such as a child suddenly jumping out.

[0081] Step 7:

[0082] Sets simulation parameters and prepares various scenarios, including unexpected events, using user input and pre-defined scenario files.

[0083] Step 8:

[0084] The server runs the simulation engine in the virtual environment and executes the simulation with the set parameters.

[0085] Step 9:

[0086] Collects and analyzes simulation results and generates reports. The server collects the result data and the analysis module performs detailed analysis.

[0087] Terminal handling

[0088] Step 1:

[0089] Terminals (sensors and IoT devices) collect GPS data and camera footage in real time, and the data collection module operates to record new data.

[0090] Step 2:

[0091] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[0092] Step 3:

[0093] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[0094] Step 4:

[0095] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[0096] User Action

[0097] Step 1:

[0098] The user uses a dedicated interface to input simulation parameters (e.g., scenario conditions and goals).

[0099] Step 2:

[0100] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[0101] Step 3:

[0102] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[0103] Step 4:

[0104] If necessary, the user can set new conditions based on the simulation results and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] In modern cities, it is extremely important to simulate and predict complex environments such as traffic flow and human movement. However, conventional systems face the challenge of complex and inefficient processes, from collecting sensor data to constructing a virtual environment and running a simulation. In particular, it is difficult to immediately perform useful simulations based on the generated data and to visually analyze the results.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for collecting sensor data, means for preprocessing the sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating a scenario in the virtual environment based on a prompt sentence using the generative AI model, means for running a simulation in the virtual environment and analyzing the results, and means for visualizing the results of the simulation, thereby enabling a series of processes from collecting sensor data to running the simulation and visualizing the results to be performed in an integrated manner.

[0110] "Sensor data" refers to information such as physical phenomena and environmental information collected by various sensors.

[0111] "Preprocessing" is the process of converting collected sensor data into a format that is easier to analyze, such as by removing noise and filling in missing values.

[0112] "Analysis" is the process of sorting collected data, extracting information, and finding trends and patterns in the data.

[0113] "Modeling" is the mathematical or logical representation of a specific phenomenon or system based on analyzed data.

[0114] A "virtual environment" is a simulation environment that recreates a real-world environment in a digital space.

[0115] A "generative AI model" is an algorithm that uses artificial intelligence technology to automatically generate digital content and scenarios based on input sentences (prompt sentences).

[0116] A "prompt" is text that serves as instructions for a generative AI model to generate content.

[0117] A "scenario" is a particular occurrence or sequence of events that occurs within a virtual environment.

[0118] "Simulation" is the process of reproducing a modeled phenomenon or system in a virtual environment and verifying its behavior and results.

[0119] "Visualization" refers to the display of analysis and simulation results using visual means such as graphs and charts.

[0120] This invention is a system that collects and analyzes sensor data, generates a virtual environment based on that data, and executes a simulation. This system has three main functions: server, terminal, and user. Each function will be explained below.

[0121] server

[0122] The server first collects sensor data, which can be obtained from devices such as GPS sensors and cameras that collect information about physical phenomena and the environment. The server uses the Python Pandas library to perform preprocessing on the collected data, such as removing noise and filling in missing values.

[0123] The server then performs data analysis. The collected data is classified using distributed processing frameworks such as Hadoop and Apache Spark. Specifically, it is segmented into traffic data, people flow data, etc. Then, machine learning algorithms such as Scikit-learn and TensorFlow are used to build models of urban activity and traffic flow.

[0124] The server then builds a virtual environment based on the generated model, using game engines such as Unity or Unreal Engine to recreate the physical environment, including roads, buildings, and traffic lights.

[0125] The server also uses a generative AI model, such as OpenAI's GPT-3, to generate a scenario in the virtual environment based on a prompt entered by the user, such as "Scenario for a vehicle making an unexpected left turn."

[0126] Finally, the server runs the simulation in the virtual environment and analyzes the results, using Python's Matplotlib and Seaborn to visualize and generate reports based on the generated data.

[0127] Terminal

[0128] The terminal consists of hardware such as sensors and IoT devices. The terminal collects data from GPS sensors and cameras in real time and performs preprocessing. The collected data is packetized according to the MQTT protocol and sent to the server.

[0129] Next, the device receives the virtual environment information sent from the server, launches a simulation execution module, such as a Unity execution engine, and executes a simulation within the received virtual environment.

[0130] The data generated during the simulation is sent back from the device to the server, allowing for advanced analysis.

[0131] User

[0132] The user inputs simulation parameters using a browser or a dedicated app, specifically setting specific scenario conditions and goals. Once the settings are complete, the user can start the simulation.

[0133] The results of the simulation are displayed as a dashboard, allowing users to review the results in charts and graphs. Users can evaluate these results, set new conditions as needed, and run the simulation again.

[0134] Specific examples

[0135] For example, consider a simulation of a self-driving car. The user sets traffic conditions, such as "peak-hour city traffic congestion," through a browser. The server uses a cloud platform to collect traffic data in real time and uses an analysis module to model traffic congestion patterns. A generative AI (e.g., GPT-3) generates a specific scenario based on a prompt, such as "a scenario involving a vehicle making a sudden left turn." The server then builds a virtual environment using Unity based on this. The device receives this virtual environment information, runs the simulation, and sends the results to the server. The user can then view the results through a dashboard and evaluate the performance of the self-driving car.

[0136] This invention efficiently integrates a series of processes from collecting sensor data to running simulations and analyzing the results, and enables rapid implementation.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] The server collects sensor data and uses a cloud platform to receive data in real time from devices such as GPS sensors and cameras. The input is sensor data such as GPS data, people flow data, and purchase data, and the output is a list of the collected data.

[0140] Step 2:

[0141] The server preprocesses the collected sensor data. It uses Python's Pandas library to remove noise and impute missing values. Specific operations include imputing missing values ​​using KNN and smoothing filters. The input here is the raw data collected in step 1, and the output is a preprocessed data frame.

[0142] Step 3:

[0143] The server analyzes the preprocessed data and models city activity and traffic flow. It uses Hadoop or Apache Spark to classify the data into categories. It applies machine learning algorithms using Scikit-learn or TensorFlow. For example, it uses clustering techniques to create traffic congestion patterns for each time period. The input is the preprocessed data frame, and the output is the modeled traffic and pedestrian flow patterns.

[0144] Step 4:

[0145] The server then constructs a virtual environment based on the generated model. Using Unity or Unreal Engine, it recreates a digital urban environment based on the analysis results. This includes road networks, buildings, traffic signals, and more. The input here is the modeled traffic flow and pedestrian flow patterns, and the output is environmental data for the virtual city.

[0146] Step 5:

[0147] The server uses a generative AI model to generate a scenario based on a prompt. For example, OpenAI's GPT-3 is used to input a prompt such as "Scenario for a vehicle making an unexpected left turn." The input here is the prompt set by the user, and the output is scenario data in the virtual environment.

[0148] Step 6:

[0149] The server runs the simulation in the virtual environment. It uses Unity's execution engine to recreate events in the virtual environment based on the generated scenario and parameters. The inputs are scenario data and virtual environment data, and the output is simulation data.

[0150] Step 7:

[0151] The server analyzes and visualizes the simulation results, using Python's Matplotlib and Seaborn to convert the data into graphs and charts. The final output is a visualized report that can be evaluated by the user.

[0152] Step 8:

[0153] The terminal sends the collected sensor data to the server in real time. The data is packetized and sent according to the MQTT protocol. The input is the real-time data acquired from the sensor, and the output is the pre-processed data sent to the server.

[0154] Step 9:

[0155] The device receives the virtual environment information sent from the server. It starts the Unity execution engine and runs a simulation based on the received information. The input is the virtual environment and scenario data from the server, and the output is the simulation results.

[0156] Step 10:

[0157] The user inputs simulation parameters using a browser or a dedicated app. Specifically, they set scenario conditions, goals, etc. The system begins operation when the user clicks a button to start the simulation. The input here is the simulation parameters set by the user, and the output is command instructions to the server.

[0158] Step 11:

[0159] The user checks the simulation results through a dashboard, viewing and evaluating the displayed charts and graphs. The input is the visualized report provided by the server, and the output is the user's evaluation results.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] Inventory management and optimization of transport routes are major challenges for modern logistics centers. In particular, there is a need to check inventory information in real time, respond to sudden inventory demands, and select efficient transport routes. To solve these challenges, advanced data analysis and simulation technologies are required.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating the behavior of people and vehicles in the virtual environment, means for running a simulation in the virtual environment and analyzing the results, means for collecting and analyzing inventory control data, means for building a virtual warehouse based on the inventory control data and generating a scenario, and means for running a simulation of inventory control and transportation routes in the virtual warehouse. This enables real-time inventory control and efficient transportation route selection in a logistics center.

[0165] "Sensor data" refers to real-time information collected from IoT sensors and mobile devices.

[0166] "Analysis" is the process of classifying and modeling collected sensor data and inventory control data.

[0167] A "model" is a virtual structure that represents patterns of urban activity, traffic flow, and inventory management, built based on analyzed data.

[0168] A "virtual environment" refers to a virtual city or warehouse environment for simulation that is constructed based on the generated model.

[0169] "Behavior" is the part of the simulation that represents how people, vehicles, and inventory behave within the virtual environment.

[0170] "Simulation" is the process of executing the movements of people and vehicles, inventory management, and delivery routes under specific conditions in a virtual environment and analyzing the results.

[0171] "Inventory management data" refers to data collected in real time, including information on the number, location, temperature, and humidity of inventory items.

[0172] A "virtual warehouse" is a virtual model constructed based on inventory management data that represents the layout of shelves, the location of items, and transportation routes within a warehouse.

[0173] A "scenario" refers to a specific situation or condition created within a simulation, such as an unexpected inventory need or equipment failure.

[0174] A "transport route" is a route optimized for efficiently moving inventory within a warehouse.

[0175] This invention is a system for realizing inventory management and transport route optimization in a logistics center. The system has three main functions: server, terminal, and user.

[0176] Server Processing

[0177] 1. Data Collection and Preprocessing:

[0178] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, and inventory location. This data is preprocessed to remove noise and impute missing values. The hardware used includes GPS sensors, temperature sensors, humidity sensors, RFID readers, and surveillance cameras.

[0179] 2. Data analysis and modeling:

[0180] The collected data is sorted and analyzed based on inventory and environmental data to determine inventory quantity, location, and environmental conditions. The software used here includes Python and analytical algorithms.

[0181] 3. Virtual warehouse construction and scenario generation:

[0182] A virtual warehouse is generated based on the analysis data, virtually representing shelf layout, product locations, and transport routes. The server uses a generative AI model to generate scenarios for unexpected inventory demands and equipment failures.

[0183] 4. Run the simulation:

[0184] Simulations of inventory management and transport routes are run in the virtual warehouse, and the results are analyzed. This makes it possible to propose optimized inventory management and transport routes. The simulation results are generated as a report and provided to the user.

[0185] Terminal handling

[0186] 1. Data transmission:

[0187] The devices collect data in real time from IoT sensors and cameras, and transmit the pre-processed data to a server. The devices used here include smartphones and head-mounted displays.

[0188] 2. Run the simulation:

[0189] The terminals run simulations based on the virtual warehouse information received from the server, and the data generated by the terminals is sent to the server for further detailed analysis.

[0190] User Action

[0191] 1. Enter the settings:

[0192] Users input simulation parameters through a smartphone or head-mounted display interface, such as inventory levels, warehouse layout, and anticipated scenarios (such as unexpected inventory demands or equipment failures).

[0193] 2. Check the results:

[0194] Users can check the results of the executed simulation on a dashboard and view the analysis report, which provides information on specific areas for improvement and optimized transport routes.

[0195] Specific examples

[0196] The server starts the analysis and simulation when the user enters a prompt into the interface, such as:

[0197] Warehouse configuration: 3 floors, 20 shelves per floor

[0198] Temperature: 22°C

[0199] Humidity: 55%

[0200] Stock items: Electronic components, textiles

[0201] Number in stock: 1000, 1500

[0202] Transport route: Shortened

[0203] Expected scenarios: sudden inventory demand, equipment failure

[0204] Based on this prompt, the generative AI model simulates unexpected situations and provides the results to the user, enabling efficient inventory management and optimization of transportation routes.

[0205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0206] Step 1: Data collection and preprocessing

[0207] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, inventory location, etc. The collected data undergoes preprocessing to remove noise and fill in missing values. This converts the data into a format suitable for analysis. The input is sensor data, and the output is preprocessed data.

[0208] Step 2: Data analysis and modeling

[0209] The server analyzes the preprocessed data and determines the inventory quantity, location, and environmental conditions based on the inventory data and environmental data. Based on the analysis results, a model of inventory placement and environmental conditions is constructed. The software used here includes Python and analytical algorithms. The input is the preprocessed data, and the output is the analysis results and model.

[0210] Step 3: Building a virtual warehouse and generating scenarios

[0211] The server generates a virtual warehouse based on the analysis results. It virtually represents shelf layout, item locations, and transport routes. The server uses the generative AI model to generate scenarios for sudden inventory demands, equipment failures, and other situations. The inputs are the analysis results and the model, and the output is the virtual warehouse and scenarios.

[0212] Step 4: Run the simulation

[0213] The server runs simulations of inventory management and transport routes within the virtual warehouse. The scenarios run include unexpected inventory demands and equipment failures. The server analyzes the simulation results and proposes optimized inventory management and transport routes. The inputs are the virtual warehouse and scenarios, and the outputs are the simulation results and an analysis report.

[0214] Step 5: Send data

[0215] The terminal collects data in real time from IoT sensors and cameras and sends the pre-processed data to the server, which then runs the simulation again based on the new data. The input is the sensor data, and the output is the data sent to the server.

[0216] Step 6: Run the simulation (terminal)

[0217] The terminal executes a simulation based on the virtual warehouse information received from the server. The data generated by the terminal is sent to the server for further detailed analysis. The input is the virtual warehouse information, and the output is the simulation results.

[0218] Step 7: Enter your settings

[0219] The user inputs simulation parameters through a smartphone or head-mounted display interface. For example, they set inventory quantities, warehouse layouts, and anticipated scenarios (such as unexpected inventory demands or equipment failures). The input is the user's configuration information, and the output is data sent to the server.

[0220] Step 8: Check the results

[0221] Users can check the results of the executed simulation on the dashboard and view the analysis report, which allows them to obtain information on specific areas for improvement and optimized transport routes. The input is the simulation results and analysis report, and the output is the information viewed by the user.

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] This invention is a system that not only collects and analyzes sensor data, generates a virtual environment, and runs a simulation within it, but also combines it with an emotion engine that recognizes the user's emotions. This system has three main functions: a server, a terminal, and a user.

[0224] Server Processing

[0225] 1. Data Collection

[0226] The server collects real-time GPS data, people flow data, and purchasing data from IoT sensors and mobile devices, as well as facial expression and voice data from users.

[0227] The server removes noise from the collected data, fills in missing values, and converts it into a format that is easy to analyze.

[0228] 2. Data Analysis

[0229] The server categorizes the collected data, dividing it into categories such as traffic data, people flow data, and emotional data. For emotional data, it applies facial expression analysis algorithms and voice analysis algorithms.

[0230] The server uses analytical algorithms to model urban traffic and pedestrian flow patterns, including changes in user emotions.

[0231] 3. Building a Virtual Environment Using Generative AI

[0232] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc. It also generates an environment that takes emotion data into account.

[0233] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[0234] 4. Run the simulation

[0235] The server sets the simulation parameters and scenarios, including scenarios that take into account the user's emotions.

[0236] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[0237] The server collects and analyzes the simulation results and generates a report.

[0238] Terminal handling

[0239] 1. Data transmission

[0240] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time.

[0241] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[0242] 2. Running the simulation

[0243] The device receives information about the virtual environment sent from the server, including emotion data.

[0244] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[0245] Data generated during the simulation is sent from the device to a server for further analysis.

[0246] User Action

[0247] 1. Enter settings

[0248] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the user's emotions are also captured in real time.

[0249] The user checks the set parameters and gives an instruction to start the simulation.

[0250] 2. Check the results

[0251] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[0252] The user's emotional data is also included in the report and used to evaluate the simulation, for example to check stress levels in response to sudden changes in the situation.

[0253] 3. Providing Feedback

[0254] Based on the simulation results, the user can provide feedback on new conditions and emotion data and instruct the simulation to be carried out again.

[0255] Specific examples

[0256] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs a simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[0257] The processing flow will be explained below.

[0258] Server Processing

[0259] Step 1:

[0260] The server collects GPS data, people flow data, purchase data, and user facial image and voice data in real time from IoT sensors and mobile devices. It activates a data reception trigger and collects data from connected devices.

[0261] Step 2:

[0262] The server preprocesses the received data, removing noise and imputing missing values. The preprocessing module detects outliers and applies filtering algorithms to clean the data.

[0263] Step 3:

[0264] The server analyzes the pre-processed data, applying emotion recognition algorithms, particularly to facial images and voice data. Using facial expression analysis and voice emotion recognition algorithms, the data is classified into emotion categories such as "joy" and "stress."

[0265] Step 4:

[0266] The server sorts through traffic, people flow, and emotion data, and uses analytical algorithms to model urban activity and traffic flows, using time series analysis and machine learning models to identify patterns and generate a model of the virtual environment.

[0267] Step 5:

[0268] Based on the generated model, the server builds a virtual city with roads, buildings, and traffic lights, overlaid with the user's emotional data.

[0269] Step 6:

[0270] The server uses a generative AI to generate the behavior of people and cars in the virtual environment, such as vehicles that suddenly stop or pedestrians that suddenly appear on a crosswalk.

[0271] Step 7:

[0272] The server sets the simulation parameters and prepares scenarios based on the user's emotions. Emotional data is used as a trigger to create a scenario in which, for example, traffic volume increases when stress increases.

[0273] Step 8:

[0274] The simulation is executed in the virtual environment. The server starts the simulation engine and runs the simulation with the set parameters.

[0275] Step 9:

[0276] Collect, analyze and report simulation results: The analysis module collects the result data, performs detailed analysis and reports the results in a user-friendly format.

[0277] Terminal handling

[0278] Step 1:

[0279] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time. The data collection module operates and records new data.

[0280] Step 2:

[0281] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[0282] Step 3:

[0283] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[0284] Step 4:

[0285] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[0286] User Action

[0287] Step 1:

[0288] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface. The system collects the user's facial images and voice data in real time and sends them to the emotion engine.

[0289] Step 2:

[0290] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[0291] Step 3:

[0292] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[0293] Step 4:

[0294] Based on the simulation results, the user can provide feedback on new conditions and emotional data and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[0295] Specific examples

[0296] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, while also continuously collecting user emotional data. The user can view the simulation results, including the emotional data, on a dashboard and consider specific measures to take in response to, for example, rising stress levels.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] Modern urban environments require smooth management of traffic and pedestrian flows, but their complexity makes it difficult to perform simulations based on real-world data. Furthermore, conventional technologies are not yet capable of performing simulations and analyses that take user emotions into account. This creates a need for highly accurate traffic management and activity prediction, as well as measures to reduce users' psychological burden.

[0300] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data by removing noise and completing missing values, means for modeling urban activities, traffic flow, and user emotions based on the analysis results, means for constructing a virtual environment based on a generative AI model using the model, means for generating unexpected events in the virtual environment, and means for running a simulation in the virtual environment and analyzing the results. This makes it possible to run a simulation close to reality and obtain analysis results that take user emotions into consideration.

[0301] Understood. Create the definition below.

[0302] "Sensor data" refers to various types of measurement data collected from devices that monitor the environment or situation.

[0303] "Noise removal" is a process for removing unnecessary noise contained in data.

[0304] "Missing value imputation" is a technique for estimating or completing missing values ​​in a dataset.

[0305] "Analysis" is the act of processing data to extract useful information from collected data.

[0306] "Modeling" is the creation of mathematical or digital models to reproduce real-world phenomena based on collected data.

[0307] A "generative AI model" is an algorithm that uses artificial intelligence technology to learn from data and generate new data or scenarios.

[0308] A "virtual environment" is a digital space created by computer simulation that mimics the real world.

[0309] A "sudden event" refers to an unexpected situation or occurrence, a scenario that occurs suddenly within a simulation.

[0310] "Simulation" refers to the reproduction of real-world phenomena or systems based on models and the experimental investigation of their behavior.

[0311] "Results analysis" is the process of evaluating simulation and experimental data and drawing meaningful conclusions.

[0312] I understand. Below is the "Form for carrying out the invention."

[0313] This invention is a system that collects sensor data, analyzes it, generates a virtual environment, and runs a simulation within it. Furthermore, it features an emotion engine that recognizes the user's emotions, improving the user experience. This system is primarily composed of three elements: a server, a terminal, and a user.

[0314] Server embodiment

[0315] 1. Data Collection

[0316] The server collects GPS data, people flow data, purchasing data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. HTTP requests and MQTT are used as communication protocols. For example, the server receives a "GET" request, the body of which contains the user's current location. Techniques such as a moving average filter are used to remove noise, and linear interpolation is used to fill in missing values.

[0317] 2. Data Analysis

[0318] The server categorizes the collected data into traffic data, people flow data, emotion data, etc. Emotion data analysis uses OpenCV and LibROSA algorithms. It also applies multiple machine learning algorithms to model urban traffic flow, people flow patterns, and emotion changes. For example, a clustering algorithm is used to identify people flow patterns.

[0319] 3. Building a Virtual Environment Using Generative AI

[0320] The server uses 3D modeling software (e.g., Blender) to build a virtual city based on the analysis data. This virtual city includes roads, buildings, traffic signals, etc. It also uses a generation AI to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). An example of a prompt for the generation AI could be, "Please generate a scenario for a vehicle making an unexpected left turn."

[0321] 4. Run the simulation

[0322] The server sets simulation parameters and scenarios, and runs a simulation of an autonomous vehicle in a virtual environment. Abnormal situations are also simulated, and user emotional data is analyzed. Finally, the server collects the simulation results, performs detailed analysis, and generates a dashboard or Excel report.

[0323] Terminal embodiment

[0324] 1. Data transmission

[0325] The device collects GPS data, camera footage, and audio data in real time. The preprocessed data is packetized and sent to the server via a communication protocol (e.g., HTTP POST or WebSocket).

[0326] 2. Running the simulation

[0327] The device receives the virtual environment data sent from the server and runs the simulation. The data generated during the simulation is sent from the device to the server, allowing for detailed analysis. For example, the device can use the Unity engine to run the simulation and send the results to the server.

[0328] User's embodiment

[0329] 1. Enter settings

[0330] The user inputs simulation parameters (e.g., scenario conditions and goals) through a dedicated interface, which also captures the user's emotions in real time.

[0331] 2. Starting the simulation

[0332] The user confirms the set parameters and clicks the "Start Simulation" button to start the simulation.

[0333] 3. Check the results

[0334] Users can check the results of the simulation on a dashboard and view analysis reports, which also include emotional data, allowing them to check, for example, stress levels in response to sudden changes in the situation.

[0335] 4. Providing Feedback

[0336] Based on the simulation results, the user can input new conditions and emotional data and run the simulation again. For example, a user can instruct a re-simulation by changing the condition "sudden left turn" to "right turn."

[0337] In this way, the system operates in cooperation with the server, terminal, and user, enabling simulation and analysis that is very close to the real environment, and generating highly accurate reports that include user emotional data.

[0338] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0339] Step 1: Collect data

[0340] The server collects GPS data, people flow data, purchase data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. The collected data is sent to the server using HTTP requests or the MQTT protocol. For example, the server receives a "GET" request, and the body of the request contains the user's current location.

[0341] Input: GPS data, people flow data, purchasing data, facial expression data, voice data.

[0342] Output: The raw data collected.

[0343] Step 2: Preprocessing the data

[0344] The server performs noise removal and missing value completion on the collected data. It uses a moving average filter to remove noise and linear interpolation to complete missing values, converting the data into a format that is easier to analyze. For example, the server applies a moving average filter to the GPS data it receives to reduce noise in the coordinates.

[0345] Input: Collected raw data.

[0346] Output: Denoised and missing value imputed data.

[0347] Step 3: Classify the data

[0348] The server classifies the preprocessed data into traffic data, people flow data, emotion data, etc. Emotion data is analyzed using a facial expression analysis algorithm (e.g., OpenCV) or a voice analysis algorithm (e.g., LibROSA). For example, the server processes camera footage with OpenCV and identifies emotions from facial expressions.

[0349] Input: Denoised and missing value imputed data.

[0350] Output: Classified traffic data, people flow data, emotion data, etc.

[0351] Step 4: Modeling

[0352] The server models urban traffic flow, pedestrian flow patterns, and user emotional changes based on the classified data. This modeling uses machine learning algorithms such as clustering and regression analysis. For example, the server uses a clustering algorithm to identify pedestrian flow patterns.

[0353] Input: Classified traffic data, people flow data, and emotion data.

[0354] Output: Traffic flow model, people flow pattern model, emotion change model.

[0355] Step 5: Build a virtual environment

[0356] The server uses a generative AI model to build a virtual city based on the model. This virtual city includes roads, buildings, traffic signals, etc. The generative AI is also used to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). For example, the server gives the generative AI a prompt: "Generate a scenario of a vehicle making an unexpected left turn."

[0357] Input: Traffic flow model, people flow pattern model, emotion change model.

[0358] Output: Virtual city, sudden event scenario.

[0359] Step 6: Run the simulation

[0360] The server sets the parameters and scenarios necessary to run a self-driving car simulation in a virtual environment, including the user's emotional data. During the simulation, unexpected events and abnormal situations are also reproduced. For example, the server runs a self-driving car simulation program in Python.

[0361] Input: Virtual city, sudden event scenario, simulation parameters.

[0362] Output: Simulation results.

[0363] Step 7: Collect and analyze results

[0364] The server collects the simulation results and performs detailed analysis. Based on the resulting numerical data and logs, it generates reports in dashboard or Excel format. For example, the server analyzes the simulation log and outputs it in a report format.

[0365] Input: Simulation results.

[0366] Output: Analyzed data, report.

[0367] Step 8: Send and display data

[0368] The terminal receives the simulation results and analysis reports and provides them to the user. Through the terminal interface, the user can check the analysis results on a dashboard. For example, the terminal receives and displays simulation data in real time from the server using WebSocket.

[0369] Input: Analyzed data, report.

[0370] Output: Displayed analysis results, reports.

[0371] Step 9: User Feedback

[0372] The user can run the simulation again by inputting new conditions and emotional data based on the simulation results and analysis report. For example, the user can change the condition from "sudden left turn" to "right turn" and instruct the simulation to be run again.

[0373] Input: Analysis results, feedback data.

[0374] Output: The new simulation parameters.

[0375] (Application example 2)

[0376] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] Conventional systems primarily simulate traffic and pedestrian flow patterns, but do not take into account user emotion data, resulting in simulations that are not necessarily realistic. Furthermore, there is a lack of means to evaluate how user emotion affects simulation results. As a result, simulations can lack accuracy and reliability.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, emotion engine means for collecting and analyzing user emotion data, means for generating a simulation scenario that takes the user emotion data into account, and means for running a simulation in the virtual environment and analyzing the results. This enables a more realistic simulation that takes the user's emotional state into account.

[0379] "Sensor data" is digital or analog data collected from a physical sensor.

[0380] "Analysis" is the process of classifying, evaluating, and processing collected data to extract useful information.

[0381] "Modeling" refers to the mathematical and statistical representation of real-world phenomena and converting them into a form that can be used for prediction and simulation.

[0382] A "virtual environment" is an artificial environment generated on a computer and used to recreate real-world situations.

[0383] "Emotion engine" is a general term for algorithms and systems that recognize and analyze emotions from a user's facial expressions, voice, etc.

[0384] A "simulation scenario" is a detailed definition of events or situations that occur within a virtual environment and a plan for recreating them.

[0385] "Simulation" is the process of performing calculations or mock experiments based on scenarios set up within a virtual environment.

[0386] An "analysis report" is a document or data that details and analyzes the results of a simulation.

[0387] This invention realizes a detailed simulation that takes into account the user's emotions in a system that collects and analyzes sensor and user data. In the following, a detailed embodiment of the system will be described based on the roles of the server, terminal, and user.

[0388] Server Processing

[0389] The server collects and analyzes data and generates a virtual environment using the following procedure.

[0390] 1. Data Collection:

[0391] The server collects GPS data, people flow data, purchasing data, and user emotion data such as facial expressions and voice data in real time from physical sensors (e.g., IoT sensors, GPS devices) and mobile devices, making it possible to integrate and obtain information from a variety of data sources.

[0392] 2. Data Analysis:

[0393] The server removes noise from the collected data and fills in missing values. It uses analysis algorithms (e.g., facial expression analysis algorithms, voice analysis algorithms) to classify emotional data and analyze it separately into traffic, pedestrian flow, and emotional data. It also models urban traffic and pedestrian flow patterns and performs a comprehensive analysis, including changes in user emotions.

[0394] 3. Building virtual environments using generative AI:

[0395] The server builds a basic virtual city based on the analyzed data, using a generative AI model to generate scenarios such as unexpected traffic conditions and pedestrians crossing the street, and also incorporates user emotional data to create a realistic virtual environment.

[0396] 4. Run the simulation:

[0397] The server runs simulations in the virtual environment, recreating scenarios that include unexpected events and abnormal situations. The results are analyzed in detail and a report is generated. The report, which also includes user emotional data, is used to evaluate the simulation.

[0398] Terminal handling

[0399] The terminal transmits data and executes the simulation as follows:

[0400] 1. Data transmission:

[0401] The device (e.g., smartphone, smart glasses) collects GPS data, camera footage, and audio data in real time and transmits the pre-processed data to the server.

[0402] 2. Run the simulation:

[0403] The device receives information about the virtual environment sent from the server and runs a simulation in that environment. The data generated by the simulation is sent to the server for detailed analysis.

[0404] User Action

[0405] The user operates the simulation by performing the following operations.

[0406] 1. Enter the settings:

[0407] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the system also captures the user's emotional data in real time.

[0408] 2. Check the results:

[0409] Users can check the results of the simulation on a dashboard and view analysis reports, which include things like checking stress levels in response to sudden changes in the situation, and the reports also include data on the user's emotions.

[0410] 3. Providing Feedback:

[0411] Based on the simulation results, the user provides feedback and instructs the simulation to be repeated with new conditions and emotional data.

[0412] Specific examples

[0413] For example, in a food delivery simulation, the server collects and analyzes weather, traffic information, and user emotional data. Based on the analysis results, a virtual city is constructed, and a generation AI simulates unexpected traffic conditions. The device then runs the simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[0414] Example prompt sentence:

[0415] Generate a virtual environment and simulate a food delivery using user location, weather, and emotion data as input. Suggest the optimal delivery route when traffic conditions are "heavy," the weather is "rainy," and the user's emotion is "stressed."

[0416] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0417] Step 1:

[0418] Data collection:

[0419] The server collects real-time GPS data, people flow data, purchasing data, and user facial and voice data from sensors and mobile devices. Specifically, it uses an IoT sensor network to obtain traffic information and people flow within the city, and collects user location information and emotional data through a smartphone application. The collected data is then sent to the server.

[0420] Step 2:

[0421] Data preprocessing:

[0422] The server removes noise from the collected data and completes missing values. Specifically, it uses a data completion algorithm to complete missing values ​​and filtering techniques to remove noise. This results in clean data that is easy to analyze. It receives multiple collected datasets (e.g., GPS data, people flow data, emotion data) as input data and outputs a clean, preprocessed dataset.

[0423] Step 3:

[0424] Data Analysis:

[0425] The server classifies the preprocessed data and analyzes it separately into traffic data, people flow data, and emotion data. Specifically, it analyzes the emotion data using facial expression analysis algorithms and voice analysis algorithms, and models traffic flow and people flow patterns using machine learning models. It receives the preprocessed dataset as input data and outputs the modeled traffic, people flow, and emotion data as the analysis results.

[0426] Step 4:

[0427] Building virtual environments with generative AI:

[0428] The server builds a basic virtual city based on the analyzed data. Specifically, it recreates traffic infrastructure (e.g., roads, buildings, and traffic lights) in the virtual space and uses a generative AI model to simulate unexpected traffic conditions and pedestrian behavior. It receives the analysis results as input data and outputs a city model as a virtual environment.

[0429] Step 5:

[0430] Simulation scenario generation:

[0431] The server generates a simulation scenario within the virtual environment, taking into account the user's emotional data. Specifically, it incorporates the user's emotional state into the scenario parameters and recreates stress-inducing traffic conditions. It receives a virtual city model and emotional data as input data, and outputs a simulation scenario.

[0432] Step 6:

[0433] Run the simulation:

[0434] The server executes a simulation in a virtual environment based on a simulation scenario. Specifically, it calculates the movement of vehicles and pedestrians in the virtual environment and reproduces sudden events and abnormal situations. It receives the simulation scenario as input data and outputs the simulation results.

[0435] Step 7:

[0436] Analysis of simulation results:

[0437] The server analyzes the results of the executed simulation. Specifically, it evaluates the data collected during the simulation in detail and interprets the results taking into account the user's emotional data. It receives the simulation results as input data and outputs an analysis report.

[0438] Step 8:

[0439] Collecting user feedback:

[0440] The user views the analysis report and provides feedback based on the simulation results. Specifically, the user inputs new scenarios and conditions using a dedicated interface, and updates the emotion data. The system receives the analysis report as input data and sends new simulation conditions to the server.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0444] [Second embodiment]

[0445] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0446] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0452] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0453] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0456] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0457] This invention is a system that collects and analyzes sensor data, generates a virtual environment, and executes a simulation within it. This system has three main functions: a server, a terminal, and a user.

[0458] Server Processing

[0459] 1. Data Collection

[0460] The server collects GPS data, people flow data, purchasing data, and other data in real time from IoT sensors and mobile devices.

[0461] The server performs preprocessing on the collected data, such as noise removal and missing value completion, and converts it into a format that is easy to analyze.

[0462] 2. Data Analysis

[0463] The server classifies the collected data, for example, into traffic data, people flow data, etc.

[0464] The server uses analytical algorithms to model traffic and pedestrian flow patterns in a city.

[0465] For example, the server analyzes traffic congestion patterns by time of day and the number of customers in a particular area.

[0466] 3. Building a Virtual Environment Using Generative AI

[0467] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc.

[0468] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[0469] 4. Run the simulation

[0470] The server sets the simulation parameters and scenarios, including settings based on user input and predefined conditions.

[0471] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[0472] The server collects and analyzes the simulation results and generates a report of the analysis results.

[0473] Terminal handling

[0474] 1. Data transmission

[0475] Terminals (e.g., sensors and IoT devices) collect data in real time from GPS sensors and cameras.

[0476] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[0477] 2. Running the simulation

[0478] The terminal receives the information about the virtual environment sent from the server.

[0479] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[0480] Data generated during the simulation is sent from the device to a server for further analysis.

[0481] User Action

[0482] 1. Enter settings

[0483] The user uses a dedicated interface to input simulation parameters, such as scenario conditions and goals.

[0484] After completing the settings, the user instructs the start of the simulation.

[0485] 2. Check the results

[0486] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[0487] The user evaluates the simulation results, sets new conditions as necessary, and instructs the simulation to be executed again.

[0488] Specific examples

[0489] For example, in a self-driving car simulation, the server collects and analyzes traffic and pedestrian flow data within a city. Using the model derived from the analysis, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, and the user can review the results to evaluate the performance of the self-driving car.

[0490] The processing flow will be explained below.

[0491] Server Processing

[0492] Step 1:

[0493] The server collects GPS data, people flow data, and purchase data in real time from IoT sensors and mobile devices. When a data reception trigger is activated, new data is transferred to the collection system.

[0494] Step 2:

[0495] The collected data is preprocessed to remove noise and fill in missing values. The server detects outliers in the data and executes algorithms to correct them.

[0496] Step 3:

[0497] The server classifies the pre-processed data into traffic data, people flow data, etc. The data classification module does this automatically.

[0498] Step 4:

[0499] Analytical algorithms are applied to analyze and model urban activity and traffic flow patterns, utilizing statistical methods and machine learning algorithms.

[0500] Step 5:

[0501] Based on the analyzed model, a virtual environment is generated. The server creates a 3D model and simulation space, and places roads, buildings, traffic signals, etc. in it.

[0502] Step 6:

[0503] Generative AI is used to simulate the movement of people and vehicles in a virtual environment. The AI ​​model runs real-time simulations and generates scenarios such as a child suddenly jumping out.

[0504] Step 7:

[0505] Sets simulation parameters and prepares various scenarios, including unexpected events, using user input and pre-defined scenario files.

[0506] Step 8:

[0507] The server runs the simulation engine in the virtual environment and executes the simulation with the set parameters.

[0508] Step 9:

[0509] Collects and analyzes simulation results and generates reports. The server collects the result data and the analysis module performs detailed analysis.

[0510] Terminal handling

[0511] Step 1:

[0512] Terminals (sensors and IoT devices) collect GPS data and camera footage in real time, and the data collection module operates to record new data.

[0513] Step 2:

[0514] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[0515] Step 3:

[0516] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[0517] Step 4:

[0518] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[0519] User Action

[0520] Step 1:

[0521] The user uses a dedicated interface to input simulation parameters (e.g., scenario conditions and goals).

[0522] Step 2:

[0523] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[0524] Step 3:

[0525] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[0526] Step 4:

[0527] If necessary, the user can set new conditions based on the simulation results and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[0528] Example 1

[0529] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] In modern cities, it is extremely important to simulate and predict complex environments such as traffic flow and human movement. However, conventional systems face the challenge of complex and inefficient processes, from collecting sensor data to constructing a virtual environment and running a simulation. In particular, it is difficult to immediately perform useful simulations based on the generated data and to visually analyze the results.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0532] In this invention, the server includes means for collecting sensor data, means for preprocessing the sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating a scenario in the virtual environment based on a prompt sentence using the generative AI model, means for running a simulation in the virtual environment and analyzing the results, and means for visualizing the results of the simulation, thereby enabling a series of processes from collecting sensor data to running the simulation and visualizing the results to be performed in an integrated manner.

[0533] "Sensor data" refers to information such as physical phenomena and environmental information collected by various sensors.

[0534] "Preprocessing" is the process of converting collected sensor data into a format that is easier to analyze, such as by removing noise and filling in missing values.

[0535] "Analysis" is the process of sorting collected data, extracting information, and finding trends and patterns in the data.

[0536] "Modeling" is the mathematical or logical representation of a specific phenomenon or system based on analyzed data.

[0537] A "virtual environment" is a simulation environment that recreates a real-world environment in a digital space.

[0538] A "generative AI model" is an algorithm that uses artificial intelligence technology to automatically generate digital content and scenarios based on input sentences (prompt sentences).

[0539] A "prompt" is text that serves as instructions for a generative AI model to generate content.

[0540] A "scenario" is a particular occurrence or sequence of events that occurs within a virtual environment.

[0541] "Simulation" is the process of reproducing a modeled phenomenon or system in a virtual environment and verifying its behavior and results.

[0542] "Visualization" refers to the display of analysis and simulation results using visual means such as graphs and charts.

[0543] This invention is a system that collects and analyzes sensor data, generates a virtual environment based on that data, and executes a simulation. This system has three main functions: server, terminal, and user. Each function will be explained below.

[0544] server

[0545] The server first collects sensor data, which can be obtained from devices such as GPS sensors and cameras that collect information about physical phenomena and the environment. The server uses the Python Pandas library to perform preprocessing on the collected data, such as removing noise and filling in missing values.

[0546] The server then performs data analysis. The collected data is classified using distributed processing frameworks such as Hadoop and Apache Spark. Specifically, it is segmented into traffic data, people flow data, etc. Then, machine learning algorithms such as Scikit-learn and TensorFlow are used to build models of urban activity and traffic flow.

[0547] The server then builds a virtual environment based on the generated model, using game engines such as Unity or Unreal Engine to recreate the physical environment, including roads, buildings, and traffic lights.

[0548] The server also uses a generative AI model, such as OpenAI's GPT-3, to generate a scenario in the virtual environment based on a prompt entered by the user, such as "Scenario for a vehicle making an unexpected left turn."

[0549] Finally, the server runs the simulation in the virtual environment and analyzes the results, using Python's Matplotlib and Seaborn to visualize and generate reports based on the generated data.

[0550] Terminal

[0551] The terminal consists of hardware such as sensors and IoT devices. The terminal collects data from GPS sensors and cameras in real time and performs preprocessing. The collected data is packetized according to the MQTT protocol and sent to the server.

[0552] Next, the device receives the virtual environment information sent from the server, launches a simulation execution module, such as a Unity execution engine, and executes a simulation within the received virtual environment.

[0553] The data generated during the simulation is sent back from the device to the server, allowing for advanced analysis.

[0554] User

[0555] The user inputs simulation parameters using a browser or a dedicated app, specifically setting specific scenario conditions and goals. Once the settings are complete, the user can start the simulation.

[0556] The results of the simulation are displayed as a dashboard, allowing users to review the results in charts and graphs. Users can evaluate these results, set new conditions as needed, and run the simulation again.

[0557] Specific examples

[0558] For example, consider a simulation of a self-driving car. The user sets traffic conditions, such as "peak-hour city traffic congestion," through a browser. The server uses a cloud platform to collect traffic data in real time and uses an analysis module to model traffic congestion patterns. A generative AI (e.g., GPT-3) generates a specific scenario based on a prompt, such as "a scenario involving a vehicle making a sudden left turn." The server then builds a virtual environment using Unity based on this. The device receives this virtual environment information, runs the simulation, and sends the results to the server. The user can then view the results through a dashboard and evaluate the performance of the self-driving car.

[0559] This invention efficiently integrates a series of processes from collecting sensor data to running simulations and analyzing the results, and enables rapid implementation.

[0560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0561] Step 1:

[0562] The server collects sensor data and uses a cloud platform to receive data in real time from devices such as GPS sensors and cameras. The input is sensor data such as GPS data, people flow data, and purchase data, and the output is a list of the collected data.

[0563] Step 2:

[0564] The server preprocesses the collected sensor data. It uses Python's Pandas library to remove noise and impute missing values. Specific operations include imputing missing values ​​using KNN and smoothing filters. The input here is the raw data collected in step 1, and the output is a preprocessed data frame.

[0565] Step 3:

[0566] The server analyzes the preprocessed data and models city activity and traffic flow. It uses Hadoop or Apache Spark to classify the data into categories. It applies machine learning algorithms using Scikit-learn or TensorFlow. For example, it uses clustering techniques to create traffic congestion patterns for each time period. The input is the preprocessed data frame, and the output is the modeled traffic and pedestrian flow patterns.

[0567] Step 4:

[0568] The server then constructs a virtual environment based on the generated model. Using Unity or Unreal Engine, it recreates a digital urban environment based on the analysis results. This includes road networks, buildings, traffic signals, and more. The input here is the modeled traffic flow and pedestrian flow patterns, and the output is environmental data for the virtual city.

[0569] Step 5:

[0570] The server uses a generative AI model to generate a scenario based on a prompt. For example, OpenAI's GPT-3 is used to input a prompt such as "Scenario for a vehicle making an unexpected left turn." The input here is the prompt set by the user, and the output is scenario data in the virtual environment.

[0571] Step 6:

[0572] The server runs the simulation in the virtual environment. It uses Unity's execution engine to recreate events in the virtual environment based on the generated scenario and parameters. The inputs are scenario data and virtual environment data, and the output is simulation data.

[0573] Step 7:

[0574] The server analyzes and visualizes the simulation results, using Python's Matplotlib and Seaborn to convert the data into graphs and charts. The final output is a visualized report that can be evaluated by the user.

[0575] Step 8:

[0576] The terminal sends the collected sensor data to the server in real time. The data is packetized and sent according to the MQTT protocol. The input is the real-time data acquired from the sensor, and the output is the pre-processed data sent to the server.

[0577] Step 9:

[0578] The device receives the virtual environment information sent from the server. It starts the Unity execution engine and runs a simulation based on the received information. The input is the virtual environment and scenario data from the server, and the output is the simulation results.

[0579] Step 10:

[0580] The user inputs simulation parameters using a browser or a dedicated app. Specifically, they set scenario conditions, goals, etc. The system begins operation when the user clicks a button to start the simulation. The input here is the simulation parameters set by the user, and the output is command instructions to the server.

[0581] Step 11:

[0582] The user checks the simulation results through a dashboard, viewing and evaluating the displayed charts and graphs. The input is the visualized report provided by the server, and the output is the user's evaluation results.

[0583] (Application example 1)

[0584] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] Inventory management and optimization of transport routes are major challenges for modern logistics centers. In particular, there is a need to check inventory information in real time, respond to sudden inventory demands, and select efficient transport routes. To solve these challenges, advanced data analysis and simulation technologies are required.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0587] In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating the behavior of people and vehicles in the virtual environment, means for running a simulation in the virtual environment and analyzing the results, means for collecting and analyzing inventory control data, means for building a virtual warehouse based on the inventory control data and generating a scenario, and means for running a simulation of inventory control and transportation routes in the virtual warehouse. This enables real-time inventory control and efficient transportation route selection in a logistics center.

[0588] "Sensor data" refers to real-time information collected from IoT sensors and mobile devices.

[0589] "Analysis" is the process of classifying and modeling collected sensor data and inventory control data.

[0590] A "model" is a virtual structure that represents patterns of urban activity, traffic flow, and inventory management, built based on analyzed data.

[0591] A "virtual environment" refers to a virtual city or warehouse environment for simulation that is constructed based on the generated model.

[0592] "Behavior" is the part of the simulation that represents how people, vehicles, and inventory behave within the virtual environment.

[0593] "Simulation" is the process of executing the movements of people and vehicles, inventory management, and delivery routes under specific conditions in a virtual environment and analyzing the results.

[0594] "Inventory management data" refers to data collected in real time, including information on the number, location, temperature, and humidity of inventory items.

[0595] A "virtual warehouse" is a virtual model constructed based on inventory management data that represents the layout of shelves, the location of items, and transportation routes within a warehouse.

[0596] A "scenario" refers to a specific situation or condition created within a simulation, such as an unexpected inventory need or equipment failure.

[0597] A "transport route" is a route optimized for efficiently moving inventory within a warehouse.

[0598] This invention is a system for realizing inventory management and transport route optimization in a logistics center. The system has three main functions: server, terminal, and user.

[0599] Server Processing

[0600] 1. Data Collection and Preprocessing:

[0601] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, and inventory location. This data is preprocessed to remove noise and impute missing values. The hardware used includes GPS sensors, temperature sensors, humidity sensors, RFID readers, and surveillance cameras.

[0602] 2. Data analysis and modeling:

[0603] The collected data is sorted and analyzed based on inventory and environmental data to determine inventory quantity, location, and environmental conditions. The software used here includes Python and analytical algorithms.

[0604] 3. Virtual warehouse construction and scenario generation:

[0605] A virtual warehouse is generated based on the analysis data, virtually representing shelf layout, product locations, and transport routes. The server uses a generative AI model to generate scenarios for unexpected inventory demands and equipment failures.

[0606] 4. Run the simulation:

[0607] Simulations of inventory management and transport routes are run in the virtual warehouse, and the results are analyzed. This makes it possible to propose optimized inventory management and transport routes. The simulation results are generated as a report and provided to the user.

[0608] Terminal handling

[0609] 1. Data transmission:

[0610] The devices collect data in real time from IoT sensors and cameras, and transmit the pre-processed data to a server. The devices used here include smartphones and head-mounted displays.

[0611] 2. Run the simulation:

[0612] The terminals run simulations based on the virtual warehouse information received from the server, and the data generated by the terminals is sent to the server for further detailed analysis.

[0613] User Action

[0614] 1. Enter the settings:

[0615] Users input simulation parameters through a smartphone or head-mounted display interface, such as inventory levels, warehouse layout, and anticipated scenarios (such as unexpected inventory demands or equipment failures).

[0616] 2. Check the results:

[0617] Users can check the results of the executed simulation on a dashboard and view the analysis report, which provides information on specific areas for improvement and optimized transport routes.

[0618] Specific examples

[0619] The server starts the analysis and simulation when the user enters a prompt into the interface, such as:

[0620] Warehouse configuration: 3 floors, 20 shelves per floor

[0621] Temperature: 22°C

[0622] Humidity: 55%

[0623] Stock items: Electronic components, textiles

[0624] Number in stock: 1000, 1500

[0625] Transport route: Shortened

[0626] Expected scenarios: sudden inventory demand, equipment failure

[0627] Based on this prompt, the generative AI model simulates unexpected situations and provides the results to the user, enabling efficient inventory management and optimization of transportation routes.

[0628] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0629] Step 1: Data collection and preprocessing

[0630] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, inventory location, etc. The collected data undergoes preprocessing to remove noise and fill in missing values. This converts the data into a format suitable for analysis. The input is sensor data, and the output is preprocessed data.

[0631] Step 2: Data analysis and modeling

[0632] The server analyzes the preprocessed data and determines the inventory quantity, location, and environmental conditions based on the inventory data and environmental data. Based on the analysis results, a model of inventory placement and environmental conditions is constructed. The software used here includes Python and analytical algorithms. The input is the preprocessed data, and the output is the analysis results and model.

[0633] Step 3: Building a virtual warehouse and generating scenarios

[0634] The server generates a virtual warehouse based on the analysis results. It virtually represents shelf layout, item locations, and transport routes. The server uses the generative AI model to generate scenarios for sudden inventory demands, equipment failures, and other situations. The inputs are the analysis results and the model, and the output is the virtual warehouse and scenarios.

[0635] Step 4: Run the simulation

[0636] The server runs simulations of inventory management and transport routes within the virtual warehouse. The scenarios run include unexpected inventory demands and equipment failures. The server analyzes the simulation results and proposes optimized inventory management and transport routes. The inputs are the virtual warehouse and scenarios, and the outputs are the simulation results and an analysis report.

[0637] Step 5: Send data

[0638] The terminal collects data in real time from IoT sensors and cameras and sends the pre-processed data to the server, which then runs the simulation again based on the new data. The input is the sensor data, and the output is the data sent to the server.

[0639] Step 6: Run the simulation (terminal)

[0640] The terminal executes a simulation based on the virtual warehouse information received from the server. The data generated by the terminal is sent to the server for further detailed analysis. The input is the virtual warehouse information, and the output is the simulation results.

[0641] Step 7: Enter your settings

[0642] The user inputs simulation parameters through a smartphone or head-mounted display interface. For example, they set inventory quantities, warehouse layouts, and anticipated scenarios (such as unexpected inventory demands or equipment failures). The input is the user's configuration information, and the output is data sent to the server.

[0643] Step 8: Check the results

[0644] Users can check the results of the executed simulation on the dashboard and view the analysis report, which allows them to obtain information on specific areas for improvement and optimized transport routes. The input is the simulation results and analysis report, and the output is the information viewed by the user.

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] This invention is a system that not only collects and analyzes sensor data, generates a virtual environment, and runs a simulation within it, but also combines it with an emotion engine that recognizes the user's emotions. This system has three main functions: a server, a terminal, and a user.

[0647] Server Processing

[0648] 1. Data Collection

[0649] The server collects real-time GPS data, people flow data, and purchasing data from IoT sensors and mobile devices, as well as facial expression and voice data from users.

[0650] The server removes noise from the collected data, fills in missing values, and converts it into a format that is easy to analyze.

[0651] 2. Data Analysis

[0652] The server categorizes the collected data, dividing it into categories such as traffic data, people flow data, and emotional data. For emotional data, it applies facial expression analysis algorithms and voice analysis algorithms.

[0653] The server uses analytical algorithms to model urban traffic and pedestrian flow patterns, including changes in user emotions.

[0654] 3. Building a Virtual Environment Using Generative AI

[0655] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc. It also generates an environment that takes emotion data into account.

[0656] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[0657] 4. Run the simulation

[0658] The server sets the simulation parameters and scenarios, including scenarios that take into account the user's emotions.

[0659] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[0660] The server collects and analyzes the simulation results and generates a report.

[0661] Terminal handling

[0662] 1. Data transmission

[0663] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time.

[0664] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[0665] 2. Running the simulation

[0666] The device receives information about the virtual environment sent from the server, including emotion data.

[0667] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[0668] Data generated during the simulation is sent from the device to a server for further analysis.

[0669] User Action

[0670] 1. Enter settings

[0671] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the user's emotions are also captured in real time.

[0672] The user checks the set parameters and gives an instruction to start the simulation.

[0673] 2. Check the results

[0674] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[0675] The user's emotional data is also included in the report and used to evaluate the simulation, for example to check stress levels in response to sudden changes in the situation.

[0676] 3. Providing Feedback

[0677] Based on the simulation results, the user can provide feedback on new conditions and emotion data and instruct the simulation to be carried out again.

[0678] Specific examples

[0679] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs a simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[0680] The processing flow will be explained below.

[0681] Server Processing

[0682] Step 1:

[0683] The server collects GPS data, people flow data, purchase data, and user facial image and voice data in real time from IoT sensors and mobile devices. It activates a data reception trigger and collects data from connected devices.

[0684] Step 2:

[0685] The server preprocesses the received data, removing noise and imputing missing values. The preprocessing module detects outliers and applies filtering algorithms to clean the data.

[0686] Step 3:

[0687] The server analyzes the pre-processed data, applying emotion recognition algorithms, particularly to facial images and voice data. Using facial expression analysis and voice emotion recognition algorithms, the data is classified into emotion categories such as "joy" and "stress."

[0688] Step 4:

[0689] The server sorts through traffic, people flow, and emotion data, and uses analytical algorithms to model urban activity and traffic flows, using time series analysis and machine learning models to identify patterns and generate a model of the virtual environment.

[0690] Step 5:

[0691] Based on the generated model, the server builds a virtual city with roads, buildings, and traffic lights, overlaid with the user's emotional data.

[0692] Step 6:

[0693] The server uses a generative AI to generate the behavior of people and cars in the virtual environment, such as vehicles that suddenly stop or pedestrians that suddenly appear on a crosswalk.

[0694] Step 7:

[0695] The server sets the simulation parameters and prepares scenarios based on the user's emotions. Emotional data is used as a trigger to create a scenario in which, for example, traffic volume increases when stress increases.

[0696] Step 8:

[0697] The simulation is executed in the virtual environment. The server starts the simulation engine and runs the simulation with the set parameters.

[0698] Step 9:

[0699] Collect, analyze and report simulation results: The analysis module collects the result data, performs detailed analysis and reports the results in a user-friendly format.

[0700] Terminal handling

[0701] Step 1:

[0702] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time. The data collection module operates and records new data.

[0703] Step 2:

[0704] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[0705] Step 3:

[0706] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[0707] Step 4:

[0708] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[0709] User Action

[0710] Step 1:

[0711] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface. The system collects the user's facial images and voice data in real time and sends them to the emotion engine.

[0712] Step 2:

[0713] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[0714] Step 3:

[0715] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[0716] Step 4:

[0717] Based on the simulation results, the user can provide feedback on new conditions and emotional data and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[0718] Specific examples

[0719] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, while also continuously collecting user emotional data. The user can view the simulation results, including the emotional data, on a dashboard and consider specific measures to take in response to, for example, rising stress levels.

[0720] Example 2

[0721] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] Modern urban environments require smooth management of traffic and pedestrian flows, but their complexity makes it difficult to perform simulations based on real-world data. Furthermore, conventional technologies are not yet capable of performing simulations and analyses that take user emotions into account. This creates a need for highly accurate traffic management and activity prediction, as well as measures to reduce users' psychological burden.

[0723] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data by removing noise and completing missing values, means for modeling urban activities, traffic flow, and user emotions based on the analysis results, means for constructing a virtual environment based on a generative AI model using the model, means for generating unexpected events in the virtual environment, and means for running a simulation in the virtual environment and analyzing the results. This makes it possible to run a simulation close to reality and obtain analysis results that take user emotions into consideration.

[0724] Understood. Create the definition below.

[0725] "Sensor data" refers to various types of measurement data collected from devices that monitor the environment or situation.

[0726] "Noise removal" is a process for removing unnecessary noise contained in data.

[0727] "Missing value imputation" is a technique for estimating or completing missing values ​​in a dataset.

[0728] "Analysis" is the act of processing data to extract useful information from collected data.

[0729] "Modeling" is the creation of mathematical or digital models to reproduce real-world phenomena based on collected data.

[0730] A "generative AI model" is an algorithm that uses artificial intelligence technology to learn from data and generate new data or scenarios.

[0731] A "virtual environment" is a digital space created by computer simulation that mimics the real world.

[0732] A "sudden event" refers to an unexpected situation or occurrence, a scenario that occurs suddenly within a simulation.

[0733] "Simulation" refers to the reproduction of real-world phenomena or systems based on models and the experimental investigation of their behavior.

[0734] "Results analysis" is the process of evaluating simulation and experimental data and drawing meaningful conclusions.

[0735] I understand. Below is the "Form for carrying out the invention."

[0736] This invention is a system that collects sensor data, analyzes it, generates a virtual environment, and runs a simulation within it. Furthermore, it features an emotion engine that recognizes the user's emotions, improving the user experience. This system is primarily composed of three elements: a server, a terminal, and a user.

[0737] Server embodiment

[0738] 1. Data Collection

[0739] The server collects GPS data, people flow data, purchasing data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. HTTP requests and MQTT are used as communication protocols. For example, the server receives a "GET" request, the body of which contains the user's current location. Techniques such as a moving average filter are used to remove noise, and linear interpolation is used to fill in missing values.

[0740] 2. Data Analysis

[0741] The server categorizes the collected data into traffic data, people flow data, emotion data, etc. Emotion data analysis uses OpenCV and LibROSA algorithms. It also applies multiple machine learning algorithms to model urban traffic flow, people flow patterns, and emotion changes. For example, a clustering algorithm is used to identify people flow patterns.

[0742] 3. Building a Virtual Environment Using Generative AI

[0743] The server uses 3D modeling software (e.g., Blender) to build a virtual city based on the analysis data. This virtual city includes roads, buildings, traffic signals, etc. It also uses a generation AI to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). An example of a prompt for the generation AI could be, "Please generate a scenario for a vehicle making an unexpected left turn."

[0744] 4. Run the simulation

[0745] The server sets simulation parameters and scenarios, and runs a simulation of an autonomous vehicle in a virtual environment. Abnormal situations are also simulated, and user emotional data is analyzed. Finally, the server collects the simulation results, performs detailed analysis, and generates a dashboard or Excel report.

[0746] Terminal embodiment

[0747] 1. Data transmission

[0748] The device collects GPS data, camera footage, and audio data in real time. The preprocessed data is packetized and sent to the server via a communication protocol (e.g., HTTP POST or WebSocket).

[0749] 2. Running the simulation

[0750] The device receives the virtual environment data sent from the server and runs the simulation. The data generated during the simulation is sent from the device to the server, allowing for detailed analysis. For example, the device can use the Unity engine to run the simulation and send the results to the server.

[0751] User's embodiment

[0752] 1. Enter settings

[0753] The user inputs simulation parameters (e.g., scenario conditions and goals) through a dedicated interface, which also captures the user's emotions in real time.

[0754] 2. Starting the simulation

[0755] The user confirms the set parameters and clicks the "Start Simulation" button to start the simulation.

[0756] 3. Check the results

[0757] Users can check the results of the simulation on a dashboard and view analysis reports, which also include emotional data, allowing them to check, for example, stress levels in response to sudden changes in the situation.

[0758] 4. Providing Feedback

[0759] Based on the simulation results, the user can input new conditions and emotional data and run the simulation again. For example, a user can instruct a re-simulation by changing the condition "sudden left turn" to "right turn."

[0760] In this way, the system operates in cooperation with the server, terminal, and user, enabling simulation and analysis that is very close to the real environment, and generating highly accurate reports that include user emotional data.

[0761] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0762] Step 1: Collect data

[0763] The server collects GPS data, people flow data, purchase data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. The collected data is sent to the server using HTTP requests or the MQTT protocol. For example, the server receives a "GET" request, and the body of the request contains the user's current location.

[0764] Input: GPS data, people flow data, purchasing data, facial expression data, voice data.

[0765] Output: The raw data collected.

[0766] Step 2: Preprocessing the data

[0767] The server performs noise removal and missing value completion on the collected data. It uses a moving average filter to remove noise and linear interpolation to complete missing values, converting the data into a format that is easier to analyze. For example, the server applies a moving average filter to the GPS data it receives to reduce noise in the coordinates.

[0768] Input: Collected raw data.

[0769] Output: Denoised and missing value imputed data.

[0770] Step 3: Classify the data

[0771] The server classifies the preprocessed data into traffic data, people flow data, emotion data, etc. Emotion data is analyzed using a facial expression analysis algorithm (e.g., OpenCV) or a voice analysis algorithm (e.g., LibROSA). For example, the server processes camera footage with OpenCV and identifies emotions from facial expressions.

[0772] Input: Denoised and missing value imputed data.

[0773] Output: Classified traffic data, people flow data, emotion data, etc.

[0774] Step 4: Modeling

[0775] The server models urban traffic flow, pedestrian flow patterns, and user emotional changes based on the classified data. This modeling uses machine learning algorithms such as clustering and regression analysis. For example, the server uses a clustering algorithm to identify pedestrian flow patterns.

[0776] Input: Classified traffic data, people flow data, and emotion data.

[0777] Output: Traffic flow model, people flow pattern model, emotion change model.

[0778] Step 5: Build a virtual environment

[0779] The server uses a generative AI model to build a virtual city based on the model. This virtual city includes roads, buildings, traffic signals, etc. The generative AI is also used to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). For example, the server gives the generative AI a prompt: "Generate a scenario of a vehicle making an unexpected left turn."

[0780] Input: Traffic flow model, people flow pattern model, emotion change model.

[0781] Output: Virtual city, sudden event scenario.

[0782] Step 6: Run the simulation

[0783] The server sets the parameters and scenarios necessary to run a self-driving car simulation in a virtual environment, including the user's emotional data. During the simulation, unexpected events and abnormal situations are also reproduced. For example, the server runs a self-driving car simulation program in Python.

[0784] Input: Virtual city, sudden event scenario, simulation parameters.

[0785] Output: Simulation results.

[0786] Step 7: Collect and analyze results

[0787] The server collects the simulation results and performs detailed analysis. Based on the resulting numerical data and logs, it generates reports in dashboard or Excel format. For example, the server analyzes the simulation log and outputs it in a report format.

[0788] Input: Simulation results.

[0789] Output: Analyzed data, report.

[0790] Step 8: Send and display data

[0791] The terminal receives the simulation results and analysis reports and provides them to the user. Through the terminal interface, the user can check the analysis results on a dashboard. For example, the terminal receives and displays simulation data in real time from the server using WebSocket.

[0792] Input: Analyzed data, report.

[0793] Output: Displayed analysis results, reports.

[0794] Step 9: User Feedback

[0795] The user can run the simulation again by inputting new conditions and emotional data based on the simulation results and analysis report. For example, the user can change the condition from "sudden left turn" to "right turn" and instruct the simulation to be run again.

[0796] Input: Analysis results, feedback data.

[0797] Output: The new simulation parameters.

[0798] (Application example 2)

[0799] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] Conventional systems primarily simulate traffic and pedestrian flow patterns, but do not take into account user emotion data, resulting in simulations that are not necessarily realistic. Furthermore, there is a lack of means to evaluate how user emotion affects simulation results. As a result, simulations can lack accuracy and reliability.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, emotion engine means for collecting and analyzing user emotion data, means for generating a simulation scenario that takes the user emotion data into account, and means for running a simulation in the virtual environment and analyzing the results. This enables a more realistic simulation that takes the user's emotional state into account.

[0802] "Sensor data" is digital or analog data collected from a physical sensor.

[0803] "Analysis" is the process of classifying, evaluating, and processing collected data to extract useful information.

[0804] "Modeling" refers to the mathematical and statistical representation of real-world phenomena and converting them into a form that can be used for prediction and simulation.

[0805] A "virtual environment" is an artificial environment generated on a computer and used to recreate real-world situations.

[0806] "Emotion engine" is a general term for algorithms and systems that recognize and analyze emotions from a user's facial expressions, voice, etc.

[0807] A "simulation scenario" is a detailed definition of events or situations that occur within a virtual environment and a plan for recreating them.

[0808] "Simulation" is the process of performing calculations or mock experiments based on scenarios set up within a virtual environment.

[0809] An "analysis report" is a document or data that details and analyzes the results of a simulation.

[0810] This invention realizes a detailed simulation that takes into account the user's emotions in a system that collects and analyzes sensor and user data. In the following, a detailed embodiment of the system will be described based on the roles of the server, terminal, and user.

[0811] Server Processing

[0812] The server collects and analyzes data and generates a virtual environment using the following procedure.

[0813] 1. Data Collection:

[0814] The server collects GPS data, people flow data, purchasing data, and user emotion data such as facial expressions and voice data in real time from physical sensors (e.g., IoT sensors, GPS devices) and mobile devices, making it possible to integrate and obtain information from a variety of data sources.

[0815] 2. Data Analysis:

[0816] The server removes noise from the collected data and fills in missing values. It uses analysis algorithms (e.g., facial expression analysis algorithms, voice analysis algorithms) to classify emotional data and analyze it separately into traffic, pedestrian flow, and emotional data. It also models urban traffic and pedestrian flow patterns and performs a comprehensive analysis, including changes in user emotions.

[0817] 3. Building virtual environments using generative AI:

[0818] The server builds a basic virtual city based on the analyzed data, using a generative AI model to generate scenarios such as unexpected traffic conditions and pedestrians crossing the street, and also incorporates user emotional data to create a realistic virtual environment.

[0819] 4. Run the simulation:

[0820] The server runs simulations in the virtual environment, recreating scenarios that include unexpected events and abnormal situations. The results are analyzed in detail and a report is generated. The report, which also includes user emotional data, is used to evaluate the simulation.

[0821] Terminal handling

[0822] The terminal transmits data and executes the simulation as follows:

[0823] 1. Data transmission:

[0824] The device (e.g., smartphone, smart glasses) collects GPS data, camera footage, and audio data in real time and transmits the pre-processed data to the server.

[0825] 2. Run the simulation:

[0826] The device receives information about the virtual environment sent from the server and runs a simulation in that environment. The data generated by the simulation is sent to the server for detailed analysis.

[0827] User Action

[0828] The user operates the simulation by performing the following operations.

[0829] 1. Enter the settings:

[0830] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the system also captures the user's emotional data in real time.

[0831] 2. Check the results:

[0832] Users can check the results of the simulation on a dashboard and view analysis reports, which include things like checking stress levels in response to sudden changes in the situation, and the reports also include data on the user's emotions.

[0833] 3. Providing Feedback:

[0834] Based on the simulation results, the user provides feedback and instructs the simulation to be repeated with new conditions and emotional data.

[0835] Specific examples

[0836] For example, in a food delivery simulation, the server collects and analyzes weather, traffic information, and user emotional data. Based on the analysis results, a virtual city is constructed, and a generation AI simulates unexpected traffic conditions. The device then runs the simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[0837] Example prompt sentence:

[0838] Generate a virtual environment and simulate a food delivery using user location, weather, and emotion data as input. Suggest the optimal delivery route when traffic conditions are "heavy," the weather is "rainy," and the user's emotion is "stressed."

[0839] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0840] Step 1:

[0841] Data collection:

[0842] The server collects real-time GPS data, people flow data, purchasing data, and user facial and voice data from sensors and mobile devices. Specifically, it uses an IoT sensor network to obtain traffic information and people flow within the city, and collects user location information and emotional data through a smartphone application. The collected data is then sent to the server.

[0843] Step 2:

[0844] Data preprocessing:

[0845] The server removes noise from the collected data and completes missing values. Specifically, it uses a data completion algorithm to complete missing values ​​and filtering techniques to remove noise. This results in clean data that is easy to analyze. It receives multiple collected datasets (e.g., GPS data, people flow data, emotion data) as input data and outputs a clean, preprocessed dataset.

[0846] Step 3:

[0847] Data Analysis:

[0848] The server classifies the preprocessed data and analyzes it separately into traffic data, people flow data, and emotion data. Specifically, it analyzes the emotion data using facial expression analysis algorithms and voice analysis algorithms, and models traffic flow and people flow patterns using machine learning models. It receives the preprocessed dataset as input data and outputs the modeled traffic, people flow, and emotion data as the analysis results.

[0849] Step 4:

[0850] Building virtual environments with generative AI:

[0851] The server builds a basic virtual city based on the analyzed data. Specifically, it recreates traffic infrastructure (e.g., roads, buildings, and traffic lights) in the virtual space and uses a generative AI model to simulate unexpected traffic conditions and pedestrian behavior. It receives the analysis results as input data and outputs a city model as a virtual environment.

[0852] Step 5:

[0853] Simulation scenario generation:

[0854] The server generates a simulation scenario within the virtual environment, taking into account the user's emotional data. Specifically, it incorporates the user's emotional state into the scenario parameters and recreates stress-inducing traffic conditions. It receives a virtual city model and emotional data as input data, and outputs a simulation scenario.

[0855] Step 6:

[0856] Run the simulation:

[0857] The server executes a simulation in a virtual environment based on a simulation scenario. Specifically, it calculates the movement of vehicles and pedestrians in the virtual environment and reproduces sudden events and abnormal situations. It receives the simulation scenario as input data and outputs the simulation results.

[0858] Step 7:

[0859] Analysis of simulation results:

[0860] The server analyzes the results of the executed simulation. Specifically, it evaluates the data collected during the simulation in detail and interprets the results taking into account the user's emotional data. It receives the simulation results as input data and outputs an analysis report.

[0861] Step 8:

[0862] Collecting user feedback:

[0863] The user views the analysis report and provides feedback based on the simulation results. Specifically, the user inputs new scenarios and conditions using a dedicated interface, and updates the emotion data. The system receives the analysis report as input data and sends new simulation conditions to the server.

[0864] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0865] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0866] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0867] [Third embodiment]

[0868] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0869] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0870] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0871] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0872] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0873] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0874] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0875] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0876] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0877] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0878] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0879] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0880] This invention is a system that collects and analyzes sensor data, generates a virtual environment, and executes a simulation within it. This system has three main functions: a server, a terminal, and a user.

[0881] Server Processing

[0882] 1. Data Collection

[0883] The server collects GPS data, people flow data, purchasing data, and other data in real time from IoT sensors and mobile devices.

[0884] The server performs preprocessing on the collected data, such as noise removal and missing value completion, and converts it into a format that is easy to analyze.

[0885] 2. Data Analysis

[0886] The server classifies the collected data, for example, into traffic data, people flow data, etc.

[0887] The server uses analytical algorithms to model traffic and pedestrian flow patterns in a city.

[0888] For example, the server analyzes traffic congestion patterns by time of day and the number of customers in a particular area.

[0889] 3. Building a Virtual Environment Using Generative AI

[0890] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc.

[0891] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[0892] 4. Run the simulation

[0893] The server sets the simulation parameters and scenarios, including settings based on user input and predefined conditions.

[0894] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[0895] The server collects and analyzes the simulation results and generates a report of the analysis results.

[0896] Terminal handling

[0897] 1. Data transmission

[0898] Terminals (e.g., sensors and IoT devices) collect data in real time from GPS sensors and cameras.

[0899] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[0900] 2. Running the simulation

[0901] The terminal receives the information about the virtual environment sent from the server.

[0902] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[0903] Data generated during the simulation is sent from the device to a server for further analysis.

[0904] User Action

[0905] 1. Enter settings

[0906] The user uses a dedicated interface to input simulation parameters, such as scenario conditions and goals.

[0907] After completing the settings, the user instructs the start of the simulation.

[0908] 2. Check the results

[0909] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[0910] The user evaluates the simulation results, sets new conditions as necessary, and instructs the simulation to be executed again.

[0911] Specific examples

[0912] For example, in a self-driving car simulation, the server collects and analyzes traffic and pedestrian flow data within a city. Using the model derived from the analysis, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, and the user can review the results to evaluate the performance of the self-driving car.

[0913] The processing flow will be explained below.

[0914] Server Processing

[0915] Step 1:

[0916] The server collects GPS data, people flow data, and purchase data in real time from IoT sensors and mobile devices. When a data reception trigger is activated, new data is transferred to the collection system.

[0917] Step 2:

[0918] The collected data is preprocessed to remove noise and fill in missing values. The server detects outliers in the data and executes algorithms to correct them.

[0919] Step 3:

[0920] The server classifies the pre-processed data into traffic data, people flow data, etc. The data classification module does this automatically.

[0921] Step 4:

[0922] Analytical algorithms are applied to analyze and model urban activity and traffic flow patterns, utilizing statistical methods and machine learning algorithms.

[0923] Step 5:

[0924] Based on the analyzed model, a virtual environment is generated. The server creates a 3D model and simulation space, and places roads, buildings, traffic signals, etc. in it.

[0925] Step 6:

[0926] Generative AI is used to simulate the movement of people and vehicles in a virtual environment. The AI ​​model runs real-time simulations and generates scenarios such as a child suddenly jumping out.

[0927] Step 7:

[0928] Sets simulation parameters and prepares various scenarios, including unexpected events, using user input and pre-defined scenario files.

[0929] Step 8:

[0930] The server runs the simulation engine in the virtual environment and executes the simulation with the set parameters.

[0931] Step 9:

[0932] Collects and analyzes simulation results and generates reports. The server collects the result data and the analysis module performs detailed analysis.

[0933] Terminal handling

[0934] Step 1:

[0935] Terminals (sensors and IoT devices) collect GPS data and camera footage in real time, and the data collection module operates to record new data.

[0936] Step 2:

[0937] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[0938] Step 3:

[0939] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[0940] Step 4:

[0941] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[0942] User Action

[0943] Step 1:

[0944] The user uses a dedicated interface to input simulation parameters (e.g., scenario conditions and goals).

[0945] Step 2:

[0946] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[0947] Step 3:

[0948] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[0949] Step 4:

[0950] If necessary, the user can set new conditions based on the simulation results and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[0951] Example 1

[0952] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] In modern cities, it is extremely important to simulate and predict complex environments such as traffic flow and human movement. However, conventional systems face the challenge of complex and inefficient processes, from collecting sensor data to constructing a virtual environment and running a simulation. In particular, it is difficult to immediately perform useful simulations based on the generated data and to visually analyze the results.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0955] In this invention, the server includes means for collecting sensor data, means for preprocessing the sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating a scenario in the virtual environment based on a prompt sentence using the generative AI model, means for running a simulation in the virtual environment and analyzing the results, and means for visualizing the results of the simulation, thereby enabling a series of processes from collecting sensor data to running the simulation and visualizing the results to be performed in an integrated manner.

[0956] "Sensor data" refers to information such as physical phenomena and environmental information collected by various sensors.

[0957] "Preprocessing" is the process of converting collected sensor data into a format that is easier to analyze, such as by removing noise and filling in missing values.

[0958] "Analysis" is the process of sorting collected data, extracting information, and finding trends and patterns in the data.

[0959] "Modeling" is the mathematical or logical representation of a specific phenomenon or system based on analyzed data.

[0960] A "virtual environment" is a simulation environment that recreates a real-world environment in a digital space.

[0961] A "generative AI model" is an algorithm that uses artificial intelligence technology to automatically generate digital content and scenarios based on input sentences (prompt sentences).

[0962] A "prompt" is text that serves as instructions for a generative AI model to generate content.

[0963] A "scenario" is a particular occurrence or sequence of events that occurs within a virtual environment.

[0964] "Simulation" is the process of reproducing a modeled phenomenon or system in a virtual environment and verifying its behavior and results.

[0965] "Visualization" refers to the display of analysis and simulation results using visual means such as graphs and charts.

[0966] This invention is a system that collects and analyzes sensor data, generates a virtual environment based on that data, and executes a simulation. This system has three main functions: server, terminal, and user. Each function will be explained below.

[0967] server

[0968] The server first collects sensor data, which can be obtained from devices such as GPS sensors and cameras that collect information about physical phenomena and the environment. The server uses the Python Pandas library to perform preprocessing on the collected data, such as removing noise and filling in missing values.

[0969] The server then performs data analysis. The collected data is classified using distributed processing frameworks such as Hadoop and Apache Spark. Specifically, it is segmented into traffic data, people flow data, etc. Then, machine learning algorithms such as Scikit-learn and TensorFlow are used to build models of urban activity and traffic flow.

[0970] The server then builds a virtual environment based on the generated model, using game engines such as Unity or Unreal Engine to recreate the physical environment, including roads, buildings, and traffic lights.

[0971] The server also uses a generative AI model, such as OpenAI's GPT-3, to generate a scenario in the virtual environment based on a prompt entered by the user, such as "Scenario for a vehicle making an unexpected left turn."

[0972] Finally, the server runs the simulation in the virtual environment and analyzes the results, using Python's Matplotlib and Seaborn to visualize and generate reports based on the generated data.

[0973] Terminal

[0974] The terminal consists of hardware such as sensors and IoT devices. The terminal collects data from GPS sensors and cameras in real time and performs preprocessing. The collected data is packetized according to the MQTT protocol and sent to the server.

[0975] Next, the device receives the virtual environment information sent from the server, launches a simulation execution module, such as a Unity execution engine, and executes a simulation within the received virtual environment.

[0976] The data generated during the simulation is sent back from the device to the server, allowing for advanced analysis.

[0977] User

[0978] The user inputs simulation parameters using a browser or a dedicated app, specifically setting specific scenario conditions and goals. Once the settings are complete, the user can start the simulation.

[0979] The results of the simulation are displayed as a dashboard, allowing users to review the results in charts and graphs. Users can evaluate these results, set new conditions as needed, and run the simulation again.

[0980] Specific examples

[0981] For example, consider a simulation of a self-driving car. The user sets traffic conditions, such as "peak-hour city traffic congestion," through a browser. The server uses a cloud platform to collect traffic data in real time and uses an analysis module to model traffic congestion patterns. A generative AI (e.g., GPT-3) generates a specific scenario based on a prompt, such as "a scenario involving a vehicle making a sudden left turn." The server then builds a virtual environment using Unity based on this. The device receives this virtual environment information, runs the simulation, and sends the results to the server. The user can then view the results through a dashboard and evaluate the performance of the self-driving car.

[0982] This invention efficiently integrates a series of processes from collecting sensor data to running simulations and analyzing the results, and enables rapid implementation.

[0983] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0984] Step 1:

[0985] The server collects sensor data and uses a cloud platform to receive data in real time from devices such as GPS sensors and cameras. The input is sensor data such as GPS data, people flow data, and purchase data, and the output is a list of the collected data.

[0986] Step 2:

[0987] The server preprocesses the collected sensor data. It uses Python's Pandas library to remove noise and impute missing values. Specific operations include imputing missing values ​​using KNN and smoothing filters. The input here is the raw data collected in step 1, and the output is a preprocessed data frame.

[0988] Step 3:

[0989] The server analyzes the preprocessed data and models city activity and traffic flow. It uses Hadoop or Apache Spark to classify the data into categories. It applies machine learning algorithms using Scikit-learn or TensorFlow. For example, it uses clustering techniques to create traffic congestion patterns for each time period. The input is the preprocessed data frame, and the output is the modeled traffic and pedestrian flow patterns.

[0990] Step 4:

[0991] The server then constructs a virtual environment based on the generated model. Using Unity or Unreal Engine, it recreates a digital urban environment based on the analysis results. This includes road networks, buildings, traffic signals, and more. The input here is the modeled traffic flow and pedestrian flow patterns, and the output is environmental data for the virtual city.

[0992] Step 5:

[0993] The server uses a generative AI model to generate a scenario based on a prompt. For example, OpenAI's GPT-3 is used to input a prompt such as "Scenario for a vehicle making an unexpected left turn." The input here is the prompt set by the user, and the output is scenario data in the virtual environment.

[0994] Step 6:

[0995] The server runs the simulation in the virtual environment. It uses Unity's execution engine to recreate events in the virtual environment based on the generated scenario and parameters. The inputs are scenario data and virtual environment data, and the output is simulation data.

[0996] Step 7:

[0997] The server analyzes and visualizes the simulation results, using Python's Matplotlib and Seaborn to convert the data into graphs and charts. The final output is a visualized report that can be evaluated by the user.

[0998] Step 8:

[0999] The terminal sends the collected sensor data to the server in real time. The data is packetized and sent according to the MQTT protocol. The input is the real-time data acquired from the sensor, and the output is the pre-processed data sent to the server.

[1000] Step 9:

[1001] The device receives the virtual environment information sent from the server. It starts the Unity execution engine and runs a simulation based on the received information. The input is the virtual environment and scenario data from the server, and the output is the simulation results.

[1002] Step 10:

[1003] The user inputs simulation parameters using a browser or a dedicated app. Specifically, they set scenario conditions, goals, etc. The system begins operation when the user clicks a button to start the simulation. The input here is the simulation parameters set by the user, and the output is command instructions to the server.

[1004] Step 11:

[1005] The user checks the simulation results through a dashboard, viewing and evaluating the displayed charts and graphs. The input is the visualized report provided by the server, and the output is the user's evaluation results.

[1006] (Application example 1)

[1007] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1008] Inventory management and optimization of transport routes are major challenges for modern logistics centers. In particular, there is a need to check inventory information in real time, respond to sudden inventory demands, and select efficient transport routes. To solve these challenges, advanced data analysis and simulation technologies are required.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1010] In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating the behavior of people and vehicles in the virtual environment, means for running a simulation in the virtual environment and analyzing the results, means for collecting and analyzing inventory control data, means for building a virtual warehouse based on the inventory control data and generating a scenario, and means for running a simulation of inventory control and transportation routes in the virtual warehouse. This enables real-time inventory control and efficient transportation route selection in a logistics center.

[1011] "Sensor data" refers to real-time information collected from IoT sensors and mobile devices.

[1012] "Analysis" is the process of classifying and modeling collected sensor data and inventory control data.

[1013] A "model" is a virtual structure that represents patterns of urban activity, traffic flow, and inventory management, built based on analyzed data.

[1014] A "virtual environment" refers to a virtual city or warehouse environment for simulation that is constructed based on the generated model.

[1015] "Behavior" is the part of the simulation that represents how people, vehicles, and inventory behave within the virtual environment.

[1016] "Simulation" is the process of executing the movements of people and vehicles, inventory management, and delivery routes under specific conditions in a virtual environment and analyzing the results.

[1017] "Inventory management data" refers to data collected in real time, including information on the number, location, temperature, and humidity of inventory items.

[1018] A "virtual warehouse" is a virtual model constructed based on inventory management data that represents the layout of shelves, the location of items, and transportation routes within a warehouse.

[1019] A "scenario" refers to a specific situation or condition created within a simulation, such as an unexpected inventory need or equipment failure.

[1020] A "transport route" is a route optimized for efficiently moving inventory within a warehouse.

[1021] This invention is a system for realizing inventory management and transport route optimization in a logistics center. The system has three main functions: server, terminal, and user.

[1022] Server Processing

[1023] 1. Data Collection and Preprocessing:

[1024] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, and inventory location. This data is preprocessed to remove noise and impute missing values. The hardware used includes GPS sensors, temperature sensors, humidity sensors, RFID readers, and surveillance cameras.

[1025] 2. Data analysis and modeling:

[1026] The collected data is sorted and analyzed based on inventory and environmental data to determine inventory quantity, location, and environmental conditions. The software used here includes Python and analytical algorithms.

[1027] 3. Virtual warehouse construction and scenario generation:

[1028] A virtual warehouse is generated based on the analysis data, virtually representing shelf layout, product locations, and transport routes. The server uses a generative AI model to generate scenarios for unexpected inventory demands and equipment failures.

[1029] 4. Run the simulation:

[1030] Simulations of inventory management and transport routes are run in the virtual warehouse, and the results are analyzed. This makes it possible to propose optimized inventory management and transport routes. The simulation results are generated as a report and provided to the user.

[1031] Terminal handling

[1032] 1. Data transmission:

[1033] The devices collect data in real time from IoT sensors and cameras, and transmit the pre-processed data to a server. The devices used here include smartphones and head-mounted displays.

[1034] 2. Run the simulation:

[1035] The terminals run simulations based on the virtual warehouse information received from the server, and the data generated by the terminals is sent to the server for further detailed analysis.

[1036] User Action

[1037] 1. Enter the settings:

[1038] Users input simulation parameters through a smartphone or head-mounted display interface, such as inventory levels, warehouse layout, and anticipated scenarios (such as unexpected inventory demands or equipment failures).

[1039] 2. Check the results:

[1040] Users can check the results of the executed simulation on a dashboard and view the analysis report, which provides information on specific areas for improvement and optimized transport routes.

[1041] Specific examples

[1042] The server starts the analysis and simulation when the user enters a prompt into the interface, such as:

[1043] Warehouse configuration: 3 floors, 20 shelves per floor

[1044] Temperature: 22°C

[1045] Humidity: 55%

[1046] Stock items: Electronic components, textiles

[1047] Number in stock: 1000, 1500

[1048] Transport route: Shortened

[1049] Expected scenarios: sudden inventory demand, equipment failure

[1050] Based on this prompt, the generative AI model simulates unexpected situations and provides the results to the user, enabling efficient inventory management and optimization of transportation routes.

[1051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1052] Step 1: Data collection and preprocessing

[1053] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, inventory location, etc. The collected data undergoes preprocessing to remove noise and fill in missing values. This converts the data into a format suitable for analysis. The input is sensor data, and the output is preprocessed data.

[1054] Step 2: Data analysis and modeling

[1055] The server analyzes the preprocessed data and determines the inventory quantity, location, and environmental conditions based on the inventory data and environmental data. Based on the analysis results, a model of inventory placement and environmental conditions is constructed. The software used here includes Python and analytical algorithms. The input is the preprocessed data, and the output is the analysis results and model.

[1056] Step 3: Building a virtual warehouse and generating scenarios

[1057] The server generates a virtual warehouse based on the analysis results. It virtually represents shelf layout, item locations, and transport routes. The server uses the generative AI model to generate scenarios for sudden inventory demands, equipment failures, and other situations. The inputs are the analysis results and the model, and the output is the virtual warehouse and scenarios.

[1058] Step 4: Run the simulation

[1059] The server runs simulations of inventory management and transport routes within the virtual warehouse. The scenarios run include unexpected inventory demands and equipment failures. The server analyzes the simulation results and proposes optimized inventory management and transport routes. The inputs are the virtual warehouse and scenarios, and the outputs are the simulation results and an analysis report.

[1060] Step 5: Send data

[1061] The terminal collects data in real time from IoT sensors and cameras and sends the pre-processed data to the server, which then runs the simulation again based on the new data. The input is the sensor data, and the output is the data sent to the server.

[1062] Step 6: Run the simulation (terminal)

[1063] The terminal executes a simulation based on the virtual warehouse information received from the server. The data generated by the terminal is sent to the server for further detailed analysis. The input is the virtual warehouse information, and the output is the simulation results.

[1064] Step 7: Enter your settings

[1065] The user inputs simulation parameters through a smartphone or head-mounted display interface. For example, they set inventory quantities, warehouse layouts, and anticipated scenarios (such as unexpected inventory demands or equipment failures). The input is the user's configuration information, and the output is data sent to the server.

[1066] Step 8: Check the results

[1067] Users can check the results of the executed simulation on the dashboard and view the analysis report, which allows them to obtain information on specific areas for improvement and optimized transport routes. The input is the simulation results and analysis report, and the output is the information viewed by the user.

[1068] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1069] This invention is a system that not only collects and analyzes sensor data, generates a virtual environment, and runs a simulation within it, but also combines it with an emotion engine that recognizes the user's emotions. This system has three main functions: a server, a terminal, and a user.

[1070] Server Processing

[1071] 1. Data Collection

[1072] The server collects real-time GPS data, people flow data, and purchasing data from IoT sensors and mobile devices, as well as facial expression and voice data from users.

[1073] The server removes noise from the collected data, fills in missing values, and converts it into a format that is easy to analyze.

[1074] 2. Data Analysis

[1075] The server categorizes the collected data, dividing it into categories such as traffic data, people flow data, and emotional data. For emotional data, it applies facial expression analysis algorithms and voice analysis algorithms.

[1076] The server uses analytical algorithms to model urban traffic and pedestrian flow patterns, including changes in user emotions.

[1077] 3. Building a Virtual Environment Using Generative AI

[1078] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc. It also generates an environment that takes emotion data into account.

[1079] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[1080] 4. Run the simulation

[1081] The server sets the simulation parameters and scenarios, including scenarios that take into account the user's emotions.

[1082] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[1083] The server collects and analyzes the simulation results and generates a report.

[1084] Terminal handling

[1085] 1. Data transmission

[1086] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time.

[1087] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[1088] 2. Running the simulation

[1089] The device receives information about the virtual environment sent from the server, including emotion data.

[1090] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[1091] Data generated during the simulation is sent from the device to a server for further analysis.

[1092] User Action

[1093] 1. Enter settings

[1094] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the user's emotions are also captured in real time.

[1095] The user checks the set parameters and gives an instruction to start the simulation.

[1096] 2. Check the results

[1097] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[1098] The user's emotional data is also included in the report and used to evaluate the simulation, for example to check stress levels in response to sudden changes in the situation.

[1099] 3. Providing Feedback

[1100] Based on the simulation results, the user can provide feedback on new conditions and emotion data and instruct the simulation to be carried out again.

[1101] Specific examples

[1102] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs a simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[1103] The processing flow will be explained below.

[1104] Server Processing

[1105] Step 1:

[1106] The server collects GPS data, people flow data, purchase data, and user facial image and voice data in real time from IoT sensors and mobile devices. It activates a data reception trigger and collects data from connected devices.

[1107] Step 2:

[1108] The server preprocesses the received data, removing noise and imputing missing values. The preprocessing module detects outliers and applies filtering algorithms to clean the data.

[1109] Step 3:

[1110] The server analyzes the pre-processed data, applying emotion recognition algorithms, particularly to facial images and voice data. Using facial expression analysis and voice emotion recognition algorithms, the data is classified into emotion categories such as "joy" and "stress."

[1111] Step 4:

[1112] The server sorts through traffic, people flow, and emotion data, and uses analytical algorithms to model urban activity and traffic flows, using time series analysis and machine learning models to identify patterns and generate a model of the virtual environment.

[1113] Step 5:

[1114] Based on the generated model, the server builds a virtual city with roads, buildings, and traffic lights, overlaid with the user's emotional data.

[1115] Step 6:

[1116] The server uses a generative AI to generate the behavior of people and cars in the virtual environment, such as vehicles that suddenly stop or pedestrians that suddenly appear on a crosswalk.

[1117] Step 7:

[1118] The server sets the simulation parameters and prepares scenarios based on the user's emotions. Emotional data is used as a trigger to create a scenario in which, for example, traffic volume increases when stress increases.

[1119] Step 8:

[1120] The simulation is executed in the virtual environment. The server starts the simulation engine and runs the simulation with the set parameters.

[1121] Step 9:

[1122] Collect, analyze and report simulation results: The analysis module collects the result data, performs detailed analysis and reports the results in a user-friendly format.

[1123] Terminal handling

[1124] Step 1:

[1125] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time. The data collection module operates and records new data.

[1126] Step 2:

[1127] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[1128] Step 3:

[1129] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[1130] Step 4:

[1131] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[1132] User Action

[1133] Step 1:

[1134] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface. The system collects the user's facial images and voice data in real time and sends them to the emotion engine.

[1135] Step 2:

[1136] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[1137] Step 3:

[1138] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[1139] Step 4:

[1140] Based on the simulation results, the user can provide feedback on new conditions and emotional data and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[1141] Specific examples

[1142] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, while also continuously collecting user emotional data. The user can view the simulation results, including the emotional data, on a dashboard and consider specific measures to take in response to, for example, rising stress levels.

[1143] Example 2

[1144] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] Modern urban environments require smooth management of traffic and pedestrian flows, but their complexity makes it difficult to perform simulations based on real-world data. Furthermore, conventional technologies are not yet capable of performing simulations and analyses that take user emotions into account. This creates a need for highly accurate traffic management and activity prediction, as well as measures to reduce users' psychological burden.

[1146] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data by removing noise and completing missing values, means for modeling urban activities, traffic flow, and user emotions based on the analysis results, means for constructing a virtual environment based on a generative AI model using the model, means for generating unexpected events in the virtual environment, and means for running a simulation in the virtual environment and analyzing the results. This makes it possible to run a simulation close to reality and obtain analysis results that take user emotions into consideration.

[1147] Understood. Create the definition below.

[1148] "Sensor data" refers to various types of measurement data collected from devices that monitor the environment or situation.

[1149] "Noise removal" is a process for removing unnecessary noise contained in data.

[1150] "Missing value imputation" is a technique for estimating or completing missing values ​​in a dataset.

[1151] "Analysis" is the act of processing data to extract useful information from collected data.

[1152] "Modeling" is the creation of mathematical or digital models to reproduce real-world phenomena based on collected data.

[1153] A "generative AI model" is an algorithm that uses artificial intelligence technology to learn from data and generate new data or scenarios.

[1154] A "virtual environment" is a digital space created by computer simulation that mimics the real world.

[1155] A "sudden event" refers to an unexpected situation or occurrence, a scenario that occurs suddenly within a simulation.

[1156] "Simulation" refers to the reproduction of real-world phenomena or systems based on models and the experimental investigation of their behavior.

[1157] "Results analysis" is the process of evaluating simulation and experimental data and drawing meaningful conclusions.

[1158] I understand. Below is the "Form for carrying out the invention."

[1159] This invention is a system that collects sensor data, analyzes it, generates a virtual environment, and runs a simulation within it. Furthermore, it features an emotion engine that recognizes the user's emotions, improving the user experience. This system is primarily composed of three elements: a server, a terminal, and a user.

[1160] Server embodiment

[1161] 1. Data Collection

[1162] The server collects GPS data, people flow data, purchasing data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. HTTP requests and MQTT are used as communication protocols. For example, the server receives a "GET" request, the body of which contains the user's current location. Techniques such as a moving average filter are used to remove noise, and linear interpolation is used to fill in missing values.

[1163] 2. Data Analysis

[1164] The server categorizes the collected data into traffic data, people flow data, emotion data, etc. Emotion data analysis uses OpenCV and LibROSA algorithms. It also applies multiple machine learning algorithms to model urban traffic flow, people flow patterns, and emotion changes. For example, a clustering algorithm is used to identify people flow patterns.

[1165] 3. Building a Virtual Environment Using Generative AI

[1166] The server uses 3D modeling software (e.g., Blender) to build a virtual city based on the analysis data. This virtual city includes roads, buildings, traffic signals, etc. It also uses a generation AI to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). An example of a prompt for the generation AI could be, "Please generate a scenario for a vehicle making an unexpected left turn."

[1167] 4. Run the simulation

[1168] The server sets simulation parameters and scenarios, and runs a simulation of an autonomous vehicle in a virtual environment. Abnormal situations are also simulated, and user emotional data is analyzed. Finally, the server collects the simulation results, performs detailed analysis, and generates a dashboard or Excel report.

[1169] Terminal embodiment

[1170] 1. Data transmission

[1171] The device collects GPS data, camera footage, and audio data in real time. The preprocessed data is packetized and sent to the server via a communication protocol (e.g., HTTP POST or WebSocket).

[1172] 2. Running the simulation

[1173] The device receives the virtual environment data sent from the server and runs the simulation. The data generated during the simulation is sent from the device to the server, allowing for detailed analysis. For example, the device can use the Unity engine to run the simulation and send the results to the server.

[1174] User's embodiment

[1175] 1. Enter settings

[1176] The user inputs simulation parameters (e.g., scenario conditions and goals) through a dedicated interface, which also captures the user's emotions in real time.

[1177] 2. Starting the simulation

[1178] The user confirms the set parameters and clicks the "Start Simulation" button to start the simulation.

[1179] 3. Check the results

[1180] Users can check the results of the simulation on a dashboard and view analysis reports, which also include emotional data, allowing them to check, for example, stress levels in response to sudden changes in the situation.

[1181] 4. Providing Feedback

[1182] Based on the simulation results, the user can input new conditions and emotional data and run the simulation again. For example, a user can instruct a re-simulation by changing the condition "sudden left turn" to "right turn."

[1183] In this way, the system operates in cooperation with the server, terminal, and user, enabling simulation and analysis that is very close to the real environment, and generating highly accurate reports that include user emotional data.

[1184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1185] Step 1: Collect data

[1186] The server collects GPS data, people flow data, purchase data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. The collected data is sent to the server using HTTP requests or the MQTT protocol. For example, the server receives a "GET" request, and the body of the request contains the user's current location.

[1187] Input: GPS data, people flow data, purchasing data, facial expression data, voice data.

[1188] Output: The raw data collected.

[1189] Step 2: Preprocessing the data

[1190] The server performs noise removal and missing value completion on the collected data. It uses a moving average filter to remove noise and linear interpolation to complete missing values, converting the data into a format that is easier to analyze. For example, the server applies a moving average filter to the GPS data it receives to reduce noise in the coordinates.

[1191] Input: Collected raw data.

[1192] Output: Denoised and missing value imputed data.

[1193] Step 3: Classify the data

[1194] The server classifies the preprocessed data into traffic data, people flow data, emotion data, etc. Emotion data is analyzed using a facial expression analysis algorithm (e.g., OpenCV) or a voice analysis algorithm (e.g., LibROSA). For example, the server processes camera footage with OpenCV and identifies emotions from facial expressions.

[1195] Input: Denoised and missing value imputed data.

[1196] Output: Classified traffic data, people flow data, emotion data, etc.

[1197] Step 4: Modeling

[1198] The server models urban traffic flow, pedestrian flow patterns, and user emotional changes based on the classified data. This modeling uses machine learning algorithms such as clustering and regression analysis. For example, the server uses a clustering algorithm to identify pedestrian flow patterns.

[1199] Input: Classified traffic data, people flow data, and emotion data.

[1200] Output: Traffic flow model, people flow pattern model, emotion change model.

[1201] Step 5: Build a virtual environment

[1202] The server uses a generative AI model to build a virtual city based on the model. This virtual city includes roads, buildings, traffic signals, etc. The generative AI is also used to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). For example, the server gives the generative AI a prompt: "Generate a scenario of a vehicle making an unexpected left turn."

[1203] Input: Traffic flow model, people flow pattern model, emotion change model.

[1204] Output: Virtual city, sudden event scenario.

[1205] Step 6: Run the simulation

[1206] The server sets the parameters and scenarios necessary to run a self-driving car simulation in a virtual environment, including the user's emotional data. During the simulation, unexpected events and abnormal situations are also reproduced. For example, the server runs a self-driving car simulation program in Python.

[1207] Input: Virtual city, sudden event scenario, simulation parameters.

[1208] Output: Simulation results.

[1209] Step 7: Collect and analyze results

[1210] The server collects the simulation results and performs detailed analysis. Based on the resulting numerical data and logs, it generates reports in dashboard or Excel format. For example, the server analyzes the simulation log and outputs it in a report format.

[1211] Input: Simulation results.

[1212] Output: Analyzed data, report.

[1213] Step 8: Send and display data

[1214] The terminal receives the simulation results and analysis reports and provides them to the user. Through the terminal interface, the user can check the analysis results on a dashboard. For example, the terminal receives and displays simulation data in real time from the server using WebSocket.

[1215] Input: Analyzed data, report.

[1216] Output: Displayed analysis results, reports.

[1217] Step 9: User Feedback

[1218] The user can run the simulation again by inputting new conditions and emotional data based on the simulation results and analysis report. For example, the user can change the condition from "sudden left turn" to "right turn" and instruct the simulation to be run again.

[1219] Input: Analysis results, feedback data.

[1220] Output: The new simulation parameters.

[1221] (Application example 2)

[1222] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1223] Conventional systems primarily simulate traffic and pedestrian flow patterns, but do not take into account user emotion data, resulting in simulations that are not necessarily realistic. Furthermore, there is a lack of means to evaluate how user emotion affects simulation results. As a result, simulations can lack accuracy and reliability.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, emotion engine means for collecting and analyzing user emotion data, means for generating a simulation scenario that takes the user emotion data into account, and means for running a simulation in the virtual environment and analyzing the results. This enables a more realistic simulation that takes the user's emotional state into account.

[1225] "Sensor data" is digital or analog data collected from a physical sensor.

[1226] "Analysis" is the process of classifying, evaluating, and processing collected data to extract useful information.

[1227] "Modeling" refers to the mathematical and statistical representation of real-world phenomena and converting them into a form that can be used for prediction and simulation.

[1228] A "virtual environment" is an artificial environment generated on a computer and used to recreate real-world situations.

[1229] "Emotion engine" is a general term for algorithms and systems that recognize and analyze emotions from a user's facial expressions, voice, etc.

[1230] A "simulation scenario" is a detailed definition of events or situations that occur within a virtual environment and a plan for recreating them.

[1231] "Simulation" is the process of performing calculations or mock experiments based on scenarios set up within a virtual environment.

[1232] An "analysis report" is a document or data that details and analyzes the results of a simulation.

[1233] This invention realizes a detailed simulation that takes into account the user's emotions in a system that collects and analyzes sensor and user data. In the following, a detailed embodiment of the system will be described based on the roles of the server, terminal, and user.

[1234] Server Processing

[1235] The server collects and analyzes data and generates a virtual environment using the following procedure.

[1236] 1. Data Collection:

[1237] The server collects GPS data, people flow data, purchasing data, and user emotion data such as facial expressions and voice data in real time from physical sensors (e.g., IoT sensors, GPS devices) and mobile devices, making it possible to integrate and obtain information from a variety of data sources.

[1238] 2. Data Analysis:

[1239] The server removes noise from the collected data and fills in missing values. It uses analysis algorithms (e.g., facial expression analysis algorithms, voice analysis algorithms) to classify emotional data and analyze it separately into traffic, pedestrian flow, and emotional data. It also models urban traffic and pedestrian flow patterns and performs a comprehensive analysis, including changes in user emotions.

[1240] 3. Building virtual environments using generative AI:

[1241] The server builds a basic virtual city based on the analyzed data, using a generative AI model to generate scenarios such as unexpected traffic conditions and pedestrians crossing the street, and also incorporates user emotional data to create a realistic virtual environment.

[1242] 4. Run the simulation:

[1243] The server runs simulations in the virtual environment, recreating scenarios that include unexpected events and abnormal situations. The results are analyzed in detail and a report is generated. The report, which also includes user emotional data, is used to evaluate the simulation.

[1244] Terminal handling

[1245] The terminal transmits data and executes the simulation as follows:

[1246] 1. Data transmission:

[1247] The device (e.g., smartphone, smart glasses) collects GPS data, camera footage, and audio data in real time and transmits the pre-processed data to the server.

[1248] 2. Run the simulation:

[1249] The device receives information about the virtual environment sent from the server and runs a simulation in that environment. The data generated by the simulation is sent to the server for detailed analysis.

[1250] User Action

[1251] The user operates the simulation by performing the following operations.

[1252] 1. Enter the settings:

[1253] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the system also captures the user's emotional data in real time.

[1254] 2. Check the results:

[1255] Users can check the results of the simulation on a dashboard and view analysis reports, which include things like checking stress levels in response to sudden changes in the situation, and the reports also include data on the user's emotions.

[1256] 3. Providing Feedback:

[1257] Based on the simulation results, the user provides feedback and instructs the simulation to be repeated with new conditions and emotional data.

[1258] Specific examples

[1259] For example, in a food delivery simulation, the server collects and analyzes weather, traffic information, and user emotional data. Based on the analysis results, a virtual city is constructed, and a generation AI simulates unexpected traffic conditions. The device then runs the simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[1260] Example prompt sentence:

[1261] Generate a virtual environment and simulate a food delivery using user location, weather, and emotion data as input. Suggest the optimal delivery route when traffic conditions are "heavy," the weather is "rainy," and the user's emotion is "stressed."

[1262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1263] Step 1:

[1264] Data collection:

[1265] The server collects real-time GPS data, people flow data, purchasing data, and user facial and voice data from sensors and mobile devices. Specifically, it uses an IoT sensor network to obtain traffic information and people flow within the city, and collects user location information and emotional data through a smartphone application. The collected data is then sent to the server.

[1266] Step 2:

[1267] Data preprocessing:

[1268] The server removes noise from the collected data and completes missing values. Specifically, it uses a data completion algorithm to complete missing values ​​and filtering techniques to remove noise. This results in clean data that is easy to analyze. It receives multiple collected datasets (e.g., GPS data, people flow data, emotion data) as input data and outputs a clean, preprocessed dataset.

[1269] Step 3:

[1270] Data Analysis:

[1271] The server classifies the preprocessed data and analyzes it separately into traffic data, people flow data, and emotion data. Specifically, it analyzes the emotion data using facial expression analysis algorithms and voice analysis algorithms, and models traffic flow and people flow patterns using machine learning models. It receives the preprocessed dataset as input data and outputs the modeled traffic, people flow, and emotion data as the analysis results.

[1272] Step 4:

[1273] Building virtual environments with generative AI:

[1274] The server builds a basic virtual city based on the analyzed data. Specifically, it recreates traffic infrastructure (e.g., roads, buildings, and traffic lights) in the virtual space and uses a generative AI model to simulate unexpected traffic conditions and pedestrian behavior. It receives the analysis results as input data and outputs a city model as a virtual environment.

[1275] Step 5:

[1276] Simulation scenario generation:

[1277] The server generates a simulation scenario within the virtual environment, taking into account the user's emotional data. Specifically, it incorporates the user's emotional state into the scenario parameters and recreates stress-inducing traffic conditions. It receives a virtual city model and emotional data as input data, and outputs a simulation scenario.

[1278] Step 6:

[1279] Run the simulation:

[1280] The server executes a simulation in a virtual environment based on a simulation scenario. Specifically, it calculates the movement of vehicles and pedestrians in the virtual environment and reproduces sudden events and abnormal situations. It receives the simulation scenario as input data and outputs the simulation results.

[1281] Step 7:

[1282] Analysis of simulation results:

[1283] The server analyzes the results of the executed simulation. Specifically, it evaluates the data collected during the simulation in detail and interprets the results taking into account the user's emotional data. It receives the simulation results as input data and outputs an analysis report.

[1284] Step 8:

[1285] Collecting user feedback:

[1286] The user views the analysis report and provides feedback based on the simulation results. Specifically, the user inputs new scenarios and conditions using a dedicated interface, and updates the emotion data. The system receives the analysis report as input data and sends new simulation conditions to the server.

[1287] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1288] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1289] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1290] [Fourth embodiment]

[1291] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1292] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1293] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1294] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1295] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1296] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1297] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1298] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1299] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1300] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1301] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1302] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1303] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1304] This invention is a system that collects and analyzes sensor data, generates a virtual environment, and executes a simulation within it. This system has three main functions: a server, a terminal, and a user.

[1305] Server Processing

[1306] 1. Data Collection

[1307] The server collects GPS data, people flow data, purchasing data, and other data in real time from IoT sensors and mobile devices.

[1308] The server performs preprocessing on the collected data, such as noise removal and missing value completion, and converts it into a format that is easy to analyze.

[1309] 2. Data Analysis

[1310] The server classifies the collected data, for example, into traffic data, people flow data, etc.

[1311] The server uses analytical algorithms to model traffic and pedestrian flow patterns in a city.

[1312] For example, the server analyzes traffic congestion patterns by time of day and the number of customers in a particular area.

[1313] 3. Building a Virtual Environment Using Generative AI

[1314] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc.

[1315] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[1316] 4. Run the simulation

[1317] The server sets the simulation parameters and scenarios, including settings based on user input and predefined conditions.

[1318] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[1319] The server collects and analyzes the simulation results and generates a report of the analysis results.

[1320] Terminal handling

[1321] 1. Data transmission

[1322] Terminals (e.g., sensors and IoT devices) collect data in real time from GPS sensors and cameras.

[1323] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[1324] 2. Running the simulation

[1325] The terminal receives the information about the virtual environment sent from the server.

[1326] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[1327] Data generated during the simulation is sent from the device to a server for further analysis.

[1328] User Action

[1329] 1. Enter settings

[1330] The user uses a dedicated interface to input simulation parameters, such as scenario conditions and goals.

[1331] After completing the settings, the user instructs the start of the simulation.

[1332] 2. Check the results

[1333] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[1334] The user evaluates the simulation results, sets new conditions as necessary, and instructs the simulation to be executed again.

[1335] Specific examples

[1336] For example, in a self-driving car simulation, the server collects and analyzes traffic and pedestrian flow data within a city. Using the model derived from the analysis, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, and the user can review the results to evaluate the performance of the self-driving car.

[1337] The processing flow will be explained below.

[1338] Server Processing

[1339] Step 1:

[1340] The server collects GPS data, people flow data, and purchase data in real time from IoT sensors and mobile devices. When a data reception trigger is activated, new data is transferred to the collection system.

[1341] Step 2:

[1342] The collected data is preprocessed to remove noise and fill in missing values. The server detects outliers in the data and executes algorithms to correct them.

[1343] Step 3:

[1344] The server classifies the pre-processed data into traffic data, people flow data, etc. The data classification module does this automatically.

[1345] Step 4:

[1346] Analytical algorithms are applied to analyze and model urban activity and traffic flow patterns, utilizing statistical methods and machine learning algorithms.

[1347] Step 5:

[1348] Based on the analyzed model, a virtual environment is generated. The server creates a 3D model and simulation space, and places roads, buildings, traffic signals, etc. in it.

[1349] Step 6:

[1350] Generative AI is used to simulate the movement of people and vehicles in a virtual environment. The AI ​​model runs real-time simulations and generates scenarios such as a child suddenly jumping out.

[1351] Step 7:

[1352] Sets simulation parameters and prepares various scenarios, including unexpected events, using user input and pre-defined scenario files.

[1353] Step 8:

[1354] The server runs the simulation engine in the virtual environment and executes the simulation with the set parameters.

[1355] Step 9:

[1356] Collects and analyzes simulation results and generates reports. The server collects the result data and the analysis module performs detailed analysis.

[1357] Terminal handling

[1358] Step 1:

[1359] Terminals (sensors and IoT devices) collect GPS data and camera footage in real time, and the data collection module operates to record new data.

[1360] Step 2:

[1361] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[1362] Step 3:

[1363] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[1364] Step 4:

[1365] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[1366] User Action

[1367] Step 1:

[1368] The user uses a dedicated interface to input simulation parameters (e.g., scenario conditions and goals).

[1369] Step 2:

[1370] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[1371] Step 3:

[1372] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[1373] Step 4:

[1374] If necessary, the user can set new conditions based on the simulation results and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[1375] Example 1

[1376] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1377] In modern cities, it is extremely important to simulate and predict complex environments such as traffic flow and human movement. However, conventional systems face the challenge of complex and inefficient processes, from collecting sensor data to constructing a virtual environment and running a simulation. In particular, it is difficult to immediately perform useful simulations based on the generated data and to visually analyze the results.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1379] In this invention, the server includes means for collecting sensor data, means for preprocessing the sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating a scenario in the virtual environment based on a prompt sentence using the generative AI model, means for running a simulation in the virtual environment and analyzing the results, and means for visualizing the results of the simulation, thereby enabling a series of processes from collecting sensor data to running the simulation and visualizing the results to be performed in an integrated manner.

[1380] "Sensor data" refers to information such as physical phenomena and environmental information collected by various sensors.

[1381] "Preprocessing" is the process of converting collected sensor data into a format that is easier to analyze, such as by removing noise and filling in missing values.

[1382] "Analysis" is the process of sorting collected data, extracting information, and finding trends and patterns in the data.

[1383] "Modeling" is the mathematical or logical representation of a specific phenomenon or system based on analyzed data.

[1384] A "virtual environment" is a simulation environment that recreates a real-world environment in a digital space.

[1385] A "generative AI model" is an algorithm that uses artificial intelligence technology to automatically generate digital content and scenarios based on input sentences (prompt sentences).

[1386] A "prompt" is text that serves as instructions for a generative AI model to generate content.

[1387] A "scenario" is a particular occurrence or sequence of events that occurs within a virtual environment.

[1388] "Simulation" is the process of reproducing a modeled phenomenon or system in a virtual environment and verifying its behavior and results.

[1389] "Visualization" refers to the display of analysis and simulation results using visual means such as graphs and charts.

[1390] This invention is a system that collects and analyzes sensor data, generates a virtual environment based on that data, and executes a simulation. This system has three main functions: server, terminal, and user. Each function will be explained below.

[1391] server

[1392] The server first collects sensor data, which can be obtained from devices such as GPS sensors and cameras that collect information about physical phenomena and the environment. The server uses the Python Pandas library to perform preprocessing on the collected data, such as removing noise and filling in missing values.

[1393] The server then performs data analysis. The collected data is classified using distributed processing frameworks such as Hadoop and Apache Spark. Specifically, it is segmented into traffic data, people flow data, etc. Then, machine learning algorithms such as Scikit-learn and TensorFlow are used to build models of urban activity and traffic flow.

[1394] The server then builds a virtual environment based on the generated model, using game engines such as Unity or Unreal Engine to recreate the physical environment, including roads, buildings, and traffic lights.

[1395] The server also uses a generative AI model, such as OpenAI's GPT-3, to generate a scenario in the virtual environment based on a prompt entered by the user, such as "Scenario for a vehicle making an unexpected left turn."

[1396] Finally, the server runs the simulation in the virtual environment and analyzes the results, using Python's Matplotlib and Seaborn to visualize and generate reports based on the generated data.

[1397] Terminal

[1398] The terminal consists of hardware such as sensors and IoT devices. The terminal collects data from GPS sensors and cameras in real time and performs preprocessing. The collected data is packetized according to the MQTT protocol and sent to the server.

[1399] Next, the device receives the virtual environment information sent from the server, launches a simulation execution module, such as a Unity execution engine, and executes a simulation within the received virtual environment.

[1400] The data generated during the simulation is sent back from the device to the server, allowing for advanced analysis.

[1401] User

[1402] The user inputs simulation parameters using a browser or a dedicated app, specifically setting specific scenario conditions and goals. Once the settings are complete, the user can start the simulation.

[1403] The results of the simulation are displayed as a dashboard, allowing users to review the results in charts and graphs. Users can evaluate these results, set new conditions as needed, and run the simulation again.

[1404] Specific examples

[1405] For example, consider a simulation of a self-driving car. The user sets traffic conditions, such as "peak-hour city traffic congestion," through a browser. The server uses a cloud platform to collect traffic data in real time and uses an analysis module to model traffic congestion patterns. A generative AI (e.g., GPT-3) generates a specific scenario based on a prompt, such as "a scenario involving a vehicle making a sudden left turn." The server then builds a virtual environment using Unity based on this. The device receives this virtual environment information, runs the simulation, and sends the results to the server. The user can then view the results through a dashboard and evaluate the performance of the self-driving car.

[1406] This invention efficiently integrates a series of processes from collecting sensor data to running simulations and analyzing the results, and enables rapid implementation.

[1407] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1408] Step 1:

[1409] The server collects sensor data and uses a cloud platform to receive data in real time from devices such as GPS sensors and cameras. The input is sensor data such as GPS data, people flow data, and purchase data, and the output is a list of the collected data.

[1410] Step 2:

[1411] The server preprocesses the collected sensor data. It uses Python's Pandas library to remove noise and impute missing values. Specific operations include imputing missing values ​​using KNN and smoothing filters. The input here is the raw data collected in step 1, and the output is a preprocessed data frame.

[1412] Step 3:

[1413] The server analyzes the preprocessed data and models city activity and traffic flow. It uses Hadoop or Apache Spark to classify the data into categories. It applies machine learning algorithms using Scikit-learn or TensorFlow. For example, it uses clustering techniques to create traffic congestion patterns for each time period. The input is the preprocessed data frame, and the output is the modeled traffic and pedestrian flow patterns.

[1414] Step 4:

[1415] The server then constructs a virtual environment based on the generated model. Using Unity or Unreal Engine, it recreates a digital urban environment based on the analysis results. This includes road networks, buildings, traffic signals, and more. The input here is the modeled traffic flow and pedestrian flow patterns, and the output is environmental data for the virtual city.

[1416] Step 5:

[1417] The server uses a generative AI model to generate a scenario based on a prompt. For example, OpenAI's GPT-3 is used to input a prompt such as "Scenario for a vehicle making an unexpected left turn." The input here is the prompt set by the user, and the output is scenario data in the virtual environment.

[1418] Step 6:

[1419] The server runs the simulation in the virtual environment. It uses Unity's execution engine to recreate events in the virtual environment based on the generated scenario and parameters. The inputs are scenario data and virtual environment data, and the output is simulation data.

[1420] Step 7:

[1421] The server analyzes and visualizes the simulation results, using Python's Matplotlib and Seaborn to convert the data into graphs and charts. The final output is a visualized report that can be evaluated by the user.

[1422] Step 8:

[1423] The terminal sends the collected sensor data to the server in real time. The data is packetized and sent according to the MQTT protocol. The input is the real-time data acquired from the sensor, and the output is the pre-processed data sent to the server.

[1424] Step 9:

[1425] The device receives the virtual environment information sent from the server. It starts the Unity execution engine and runs a simulation based on the received information. The input is the virtual environment and scenario data from the server, and the output is the simulation results.

[1426] Step 10:

[1427] The user inputs simulation parameters using a browser or a dedicated app. Specifically, they set scenario conditions, goals, etc. The system begins operation when the user clicks a button to start the simulation. The input here is the simulation parameters set by the user, and the output is command instructions to the server.

[1428] Step 11:

[1429] The user checks the simulation results through a dashboard, viewing and evaluating the displayed charts and graphs. The input is the visualized report provided by the server, and the output is the user's evaluation results.

[1430] (Application example 1)

[1431] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] Inventory management and optimization of transport routes are major challenges for modern logistics centers. In particular, there is a need to check inventory information in real time, respond to sudden inventory demands, and select efficient transport routes. To solve these challenges, advanced data analysis and simulation technologies are required.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1434] In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, means for generating the behavior of people and vehicles in the virtual environment, means for running a simulation in the virtual environment and analyzing the results, means for collecting and analyzing inventory control data, means for building a virtual warehouse based on the inventory control data and generating a scenario, and means for running a simulation of inventory control and transportation routes in the virtual warehouse. This enables real-time inventory control and efficient transportation route selection in a logistics center.

[1435] "Sensor data" refers to real-time information collected from IoT sensors and mobile devices.

[1436] "Analysis" is the process of classifying and modeling collected sensor data and inventory control data.

[1437] A "model" is a virtual structure that represents patterns of urban activity, traffic flow, and inventory management, built based on analyzed data.

[1438] A "virtual environment" refers to a virtual city or warehouse environment for simulation that is constructed based on the generated model.

[1439] "Behavior" is the part of the simulation that represents how people, vehicles, and inventory behave within the virtual environment.

[1440] "Simulation" is the process of executing the movements of people and vehicles, inventory management, and delivery routes under specific conditions in a virtual environment and analyzing the results.

[1441] "Inventory management data" refers to data collected in real time, including information on the number, location, temperature, and humidity of inventory items.

[1442] A "virtual warehouse" is a virtual model constructed based on inventory management data that represents the layout of shelves, the location of items, and transportation routes within a warehouse.

[1443] A "scenario" refers to a specific situation or condition created within a simulation, such as an unexpected inventory need or equipment failure.

[1444] A "transport route" is a route optimized for efficiently moving inventory within a warehouse.

[1445] This invention is a system for realizing inventory management and transport route optimization in a logistics center. The system has three main functions: server, terminal, and user.

[1446] Server Processing

[1447] 1. Data Collection and Preprocessing:

[1448] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, and inventory location. This data is preprocessed to remove noise and impute missing values. The hardware used includes GPS sensors, temperature sensors, humidity sensors, RFID readers, and surveillance cameras.

[1449] 2. Data analysis and modeling:

[1450] The collected data is sorted and analyzed based on inventory and environmental data to determine inventory quantity, location, and environmental conditions. The software used here includes Python and analytical algorithms.

[1451] 3. Virtual warehouse construction and scenario generation:

[1452] A virtual warehouse is generated based on the analysis data, virtually representing shelf layout, product locations, and transport routes. The server uses a generative AI model to generate scenarios for unexpected inventory demands and equipment failures.

[1453] 4. Run the simulation:

[1454] Simulations of inventory management and transport routes are run in the virtual warehouse, and the results are analyzed. This makes it possible to propose optimized inventory management and transport routes. The simulation results are generated as a report and provided to the user.

[1455] Terminal handling

[1456] 1. Data transmission:

[1457] The devices collect data in real time from IoT sensors and cameras, and transmit the pre-processed data to a server. The devices used here include smartphones and head-mounted displays.

[1458] 2. Run the simulation:

[1459] The terminals run simulations based on the virtual warehouse information received from the server, and the data generated by the terminals is sent to the server for further detailed analysis.

[1460] User Action

[1461] 1. Enter the settings:

[1462] Users input simulation parameters through a smartphone or head-mounted display interface, such as inventory levels, warehouse layout, and anticipated scenarios (such as unexpected inventory demands or equipment failures).

[1463] 2. Check the results:

[1464] Users can check the results of the executed simulation on a dashboard and view the analysis report, which provides information on specific areas for improvement and optimized transport routes.

[1465] Specific examples

[1466] The server starts the analysis and simulation when the user enters a prompt into the interface, such as:

[1467] Warehouse configuration: 3 floors, 20 shelves per floor

[1468] Temperature: 22°C

[1469] Humidity: 55%

[1470] Stock items: Electronic components, textiles

[1471] Number in stock: 1000, 1500

[1472] Transport route: Shortened

[1473] Expected scenarios: sudden inventory demand, equipment failure

[1474] Based on this prompt, the generative AI model simulates unexpected situations and provides the results to the user, enabling efficient inventory management and optimization of transportation routes.

[1475] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1476] Step 1: Data collection and preprocessing

[1477] The server collects data in real time from IoT sensors, RFID readers, and cameras. Sensor data includes GPS information, temperature, humidity, inventory location, etc. The collected data undergoes preprocessing to remove noise and fill in missing values. This converts the data into a format suitable for analysis. The input is sensor data, and the output is preprocessed data.

[1478] Step 2: Data analysis and modeling

[1479] The server analyzes the preprocessed data and determines the inventory quantity, location, and environmental conditions based on the inventory data and environmental data. Based on the analysis results, a model of inventory placement and environmental conditions is constructed. The software used here includes Python and analytical algorithms. The input is the preprocessed data, and the output is the analysis results and model.

[1480] Step 3: Building a virtual warehouse and generating scenarios

[1481] The server generates a virtual warehouse based on the analysis results. It virtually represents shelf layout, item locations, and transport routes. The server uses the generative AI model to generate scenarios for sudden inventory demands, equipment failures, and other situations. The inputs are the analysis results and the model, and the output is the virtual warehouse and scenarios.

[1482] Step 4: Run the simulation

[1483] The server runs simulations of inventory management and transport routes within the virtual warehouse. The scenarios run include unexpected inventory demands and equipment failures. The server analyzes the simulation results and proposes optimized inventory management and transport routes. The inputs are the virtual warehouse and scenarios, and the outputs are the simulation results and an analysis report.

[1484] Step 5: Send data

[1485] The terminal collects data in real time from IoT sensors and cameras and sends the pre-processed data to the server, which then runs the simulation again based on the new data. The input is the sensor data, and the output is the data sent to the server.

[1486] Step 6: Run the simulation (terminal)

[1487] The terminal executes a simulation based on the virtual warehouse information received from the server. The data generated by the terminal is sent to the server for further detailed analysis. The input is the virtual warehouse information, and the output is the simulation results.

[1488] Step 7: Enter your settings

[1489] The user inputs simulation parameters through a smartphone or head-mounted display interface. For example, they set inventory quantities, warehouse layouts, and anticipated scenarios (such as unexpected inventory demands or equipment failures). The input is the user's configuration information, and the output is data sent to the server.

[1490] Step 8: Check the results

[1491] Users can check the results of the executed simulation on the dashboard and view the analysis report, which allows them to obtain information on specific areas for improvement and optimized transport routes. The input is the simulation results and analysis report, and the output is the information viewed by the user.

[1492] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1493] This invention is a system that not only collects and analyzes sensor data, generates a virtual environment, and runs a simulation within it, but also combines it with an emotion engine that recognizes the user's emotions. This system has three main functions: a server, a terminal, and a user.

[1494] Server Processing

[1495] 1. Data Collection

[1496] The server collects real-time GPS data, people flow data, and purchasing data from IoT sensors and mobile devices, as well as facial expression and voice data from users.

[1497] The server removes noise from the collected data, fills in missing values, and converts it into a format that is easy to analyze.

[1498] 2. Data Analysis

[1499] The server categorizes the collected data, dividing it into categories such as traffic data, people flow data, and emotional data. For emotional data, it applies facial expression analysis algorithms and voice analysis algorithms.

[1500] The server uses analytical algorithms to model urban traffic and pedestrian flow patterns, including changes in user emotions.

[1501] 3. Building a Virtual Environment Using Generative AI

[1502] The server uses the collected data to build a basic virtual city, including roads, buildings, traffic lights, etc. It also generates an environment that takes emotion data into account.

[1503] The server uses generative AI to generate scenarios within the virtual environment, such as a vehicle making a sudden left turn or a pedestrian suddenly crossing a crosswalk.

[1504] 4. Run the simulation

[1505] The server sets the simulation parameters and scenarios, including scenarios that take into account the user's emotions.

[1506] The server runs a simulation of an autonomous vehicle in a virtual environment, recreating sudden events and abnormal situations.

[1507] The server collects and analyzes the simulation results and generates a report.

[1508] Terminal handling

[1509] 1. Data transmission

[1510] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time.

[1511] The terminal packetizes the pre-processed data for transmission to the server, and transmits the packetized data in accordance with a communication protocol.

[1512] 2. Running the simulation

[1513] The device receives information about the virtual environment sent from the server, including emotion data.

[1514] The terminal starts the simulation execution module and executes the simulation in the received virtual environment.

[1515] Data generated during the simulation is sent from the device to a server for further analysis.

[1516] User Action

[1517] 1. Enter settings

[1518] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the user's emotions are also captured in real time.

[1519] The user checks the set parameters and gives an instruction to start the simulation.

[1520] 2. Check the results

[1521] Users can check the results of the executed simulation on the dashboard and view the analysis report.

[1522] The user's emotional data is also included in the report and used to evaluate the simulation, for example to check stress levels in response to sudden changes in the situation.

[1523] 3. Providing Feedback

[1524] Based on the simulation results, the user can provide feedback on new conditions and emotion data and instruct the simulation to be carried out again.

[1525] Specific examples

[1526] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs a simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[1527] The processing flow will be explained below.

[1528] Server Processing

[1529] Step 1:

[1530] The server collects GPS data, people flow data, purchase data, and user facial image and voice data in real time from IoT sensors and mobile devices. It activates a data reception trigger and collects data from connected devices.

[1531] Step 2:

[1532] The server preprocesses the received data, removing noise and imputing missing values. The preprocessing module detects outliers and applies filtering algorithms to clean the data.

[1533] Step 3:

[1534] The server analyzes the pre-processed data, applying emotion recognition algorithms, particularly to facial images and voice data. Using facial expression analysis and voice emotion recognition algorithms, the data is classified into emotion categories such as "joy" and "stress."

[1535] Step 4:

[1536] The server sorts through traffic, people flow, and emotion data, and uses analytical algorithms to model urban activity and traffic flows, using time series analysis and machine learning models to identify patterns and generate a model of the virtual environment.

[1537] Step 5:

[1538] Based on the generated model, the server builds a virtual city with roads, buildings, and traffic lights, overlaid with the user's emotional data.

[1539] Step 6:

[1540] The server uses a generative AI to generate the behavior of people and cars in the virtual environment, such as vehicles that suddenly stop or pedestrians that suddenly appear on a crosswalk.

[1541] Step 7:

[1542] The server sets the simulation parameters and prepares scenarios based on the user's emotions. Emotional data is used as a trigger to create a scenario in which, for example, traffic volume increases when stress increases.

[1543] Step 8:

[1544] The simulation is executed in the virtual environment. The server starts the simulation engine and runs the simulation with the set parameters.

[1545] Step 9:

[1546] Collect, analyze and report simulation results: The analysis module collects the result data, performs detailed analysis and reports the results in a user-friendly format.

[1547] Terminal handling

[1548] Step 1:

[1549] Terminals (sensors and IoT devices) collect GPS data, camera footage, and audio data in real time. The data collection module operates and records new data.

[1550] Step 2:

[1551] The terminal preprocesses the collected data, packets it, and sends it to the server. The data packets are then sent over the network according to the communication protocol.

[1552] Step 3:

[1553] The terminal receives the virtual environment information sent from the server, analyzes it, and performs the initial settings to start the simulation execution module.

[1554] Step 4:

[1555] The device executes a simulation in the received virtual environment, records the data generated during the simulation in real time, and transmits it to the server.

[1556] User Action

[1557] Step 1:

[1558] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface. The system collects the user's facial images and voice data in real time and sends them to the emotion engine.

[1559] Step 2:

[1560] The user confirms the set parameters and commands the start of the simulation. The interface sends this information to the server.

[1561] Step 3:

[1562] The user can check the results of the executed simulation through the dashboard, which launches a display module for viewing the simulation results and analysis reports.

[1563] Step 4:

[1564] Based on the simulation results, the user can provide feedback on new conditions and emotional data and instruct the simulation to be run again. The reconfigured information is sent to the server, and the simulation is rerun.

[1565] Specific examples

[1566] For example, in a simulation of an autonomous vehicle, the server collects and analyzes traffic data, people flow data, and user emotional data within a city. Using a model derived from the analysis results, a virtual city is constructed, and a generative AI simulates unexpected traffic conditions. The device then runs the simulation based on this data, while also continuously collecting user emotional data. The user can view the simulation results, including the emotional data, on a dashboard and consider specific measures to take in response to, for example, rising stress levels.

[1567] Example 2

[1568] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1569] Modern urban environments require smooth management of traffic and pedestrian flows, but their complexity makes it difficult to perform simulations based on real-world data. Furthermore, conventional technologies are not yet capable of performing simulations and analyses that take user emotions into account. This creates a need for highly accurate traffic management and activity prediction, as well as measures to reduce users' psychological burden.

[1570] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data by removing noise and completing missing values, means for modeling urban activities, traffic flow, and user emotions based on the analysis results, means for constructing a virtual environment based on a generative AI model using the model, means for generating unexpected events in the virtual environment, and means for running a simulation in the virtual environment and analyzing the results. This makes it possible to run a simulation close to reality and obtain analysis results that take user emotions into consideration.

[1571] Understood. Create the definition below.

[1572] "Sensor data" refers to various types of measurement data collected from devices that monitor the environment or situation.

[1573] "Noise removal" is a process for removing unnecessary noise contained in data.

[1574] "Missing value imputation" is a technique for estimating or completing missing values ​​in a dataset.

[1575] "Analysis" is the act of processing data to extract useful information from collected data.

[1576] "Modeling" is the creation of mathematical or digital models to reproduce real-world phenomena based on collected data.

[1577] A "generative AI model" is an algorithm that uses artificial intelligence technology to learn from data and generate new data or scenarios.

[1578] A "virtual environment" is a digital space created by computer simulation that mimics the real world.

[1579] A "sudden event" refers to an unexpected situation or occurrence, a scenario that occurs suddenly within a simulation.

[1580] "Simulation" refers to the reproduction of real-world phenomena or systems based on models and the experimental investigation of their behavior.

[1581] "Results analysis" is the process of evaluating simulation and experimental data and drawing meaningful conclusions.

[1582] I understand. Below is the "Form for carrying out the invention."

[1583] This invention is a system that collects sensor data, analyzes it, generates a virtual environment, and runs a simulation within it. Furthermore, it features an emotion engine that recognizes the user's emotions, improving the user experience. This system is primarily composed of three elements: a server, a terminal, and a user.

[1584] Server embodiment

[1585] 1. Data Collection

[1586] The server collects GPS data, people flow data, purchasing data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. HTTP requests and MQTT are used as communication protocols. For example, the server receives a "GET" request, the body of which contains the user's current location. Techniques such as a moving average filter are used to remove noise, and linear interpolation is used to fill in missing values.

[1587] 2. Data Analysis

[1588] The server categorizes the collected data into traffic data, people flow data, emotion data, etc. Emotion data analysis uses OpenCV and LibROSA algorithms. It also applies multiple machine learning algorithms to model urban traffic flow, people flow patterns, and emotion changes. For example, a clustering algorithm is used to identify people flow patterns.

[1589] 3. Building a Virtual Environment Using Generative AI

[1590] The server uses 3D modeling software (e.g., Blender) to build a virtual city based on the analysis data. This virtual city includes roads, buildings, traffic signals, etc. It also uses a generation AI to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). An example of a prompt for the generation AI could be, "Please generate a scenario for a vehicle making an unexpected left turn."

[1591] 4. Run the simulation

[1592] The server sets simulation parameters and scenarios, and runs a simulation of an autonomous vehicle in a virtual environment. Abnormal situations are also simulated, and user emotional data is analyzed. Finally, the server collects the simulation results, performs detailed analysis, and generates a dashboard or Excel report.

[1593] Terminal embodiment

[1594] 1. Data transmission

[1595] The device collects GPS data, camera footage, and audio data in real time. The preprocessed data is packetized and sent to the server via a communication protocol (e.g., HTTP POST or WebSocket).

[1596] 2. Running the simulation

[1597] The device receives the virtual environment data sent from the server and runs the simulation. The data generated during the simulation is sent from the device to the server, allowing for detailed analysis. For example, the device can use the Unity engine to run the simulation and send the results to the server.

[1598] User's embodiment

[1599] 1. Enter settings

[1600] The user inputs simulation parameters (e.g., scenario conditions and goals) through a dedicated interface, which also captures the user's emotions in real time.

[1601] 2. Starting the simulation

[1602] The user confirms the set parameters and clicks the "Start Simulation" button to start the simulation.

[1603] 3. Check the results

[1604] Users can check the results of the simulation on a dashboard and view analysis reports, which also include emotional data, allowing them to check, for example, stress levels in response to sudden changes in the situation.

[1605] 4. Providing Feedback

[1606] Based on the simulation results, the user can input new conditions and emotional data and run the simulation again. For example, a user can instruct a re-simulation by changing the condition "sudden left turn" to "right turn."

[1607] In this way, the system operates in cooperation with the server, terminal, and user, enabling simulation and analysis that is very close to the real environment, and generating highly accurate reports that include user emotional data.

[1608] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1609] Step 1: Collect data

[1610] The server collects GPS data, people flow data, purchase data, facial expression data, and voice data in real time from multiple IoT sensors and mobile devices. The collected data is sent to the server using HTTP requests or the MQTT protocol. For example, the server receives a "GET" request, and the body of the request contains the user's current location.

[1611] Input: GPS data, people flow data, purchasing data, facial expression data, voice data.

[1612] Output: The raw data collected.

[1613] Step 2: Preprocessing the data

[1614] The server performs noise removal and missing value completion on the collected data. It uses a moving average filter to remove noise and linear interpolation to complete missing values, converting the data into a format that is easier to analyze. For example, the server applies a moving average filter to the GPS data it receives to reduce noise in the coordinates.

[1615] Input: Collected raw data.

[1616] Output: Denoised and missing value imputed data.

[1617] Step 3: Classify the data

[1618] The server classifies the preprocessed data into traffic data, people flow data, emotion data, etc. Emotion data is analyzed using a facial expression analysis algorithm (e.g., OpenCV) or a voice analysis algorithm (e.g., LibROSA). For example, the server processes camera footage with OpenCV and identifies emotions from facial expressions.

[1619] Input: Denoised and missing value imputed data.

[1620] Output: Classified traffic data, people flow data, emotion data, etc.

[1621] Step 4: Modeling

[1622] The server models urban traffic flow, pedestrian flow patterns, and user emotional changes based on the classified data. This modeling uses machine learning algorithms such as clustering and regression analysis. For example, the server uses a clustering algorithm to identify pedestrian flow patterns.

[1623] Input: Classified traffic data, people flow data, and emotion data.

[1624] Output: Traffic flow model, people flow pattern model, emotion change model.

[1625] Step 5: Build a virtual environment

[1626] The server uses a generative AI model to build a virtual city based on the model. This virtual city includes roads, buildings, traffic signals, etc. The generative AI is also used to simulate unexpected events (e.g., "a vehicle making an unexpected left turn"). For example, the server gives the generative AI a prompt: "Generate a scenario of a vehicle making an unexpected left turn."

[1627] Input: Traffic flow model, people flow pattern model, emotion change model.

[1628] Output: Virtual city, sudden event scenario.

[1629] Step 6: Run the simulation

[1630] The server sets the parameters and scenarios necessary to run a self-driving car simulation in a virtual environment, including the user's emotional data. During the simulation, unexpected events and abnormal situations are also reproduced. For example, the server runs a self-driving car simulation program in Python.

[1631] Input: Virtual city, sudden event scenario, simulation parameters.

[1632] Output: Simulation results.

[1633] Step 7: Collect and analyze results

[1634] The server collects the simulation results and performs detailed analysis. Based on the resulting numerical data and logs, it generates reports in dashboard or Excel format. For example, the server analyzes the simulation log and outputs it in a report format.

[1635] Input: Simulation results.

[1636] Output: Analyzed data, report.

[1637] Step 8: Send and display data

[1638] The terminal receives the simulation results and analysis reports and provides them to the user. Through the terminal interface, the user can check the analysis results on a dashboard. For example, the terminal receives and displays simulation data in real time from the server using WebSocket.

[1639] Input: Analyzed data, report.

[1640] Output: Displayed analysis results, reports.

[1641] Step 9: User Feedback

[1642] The user can run the simulation again by inputting new conditions and emotional data based on the simulation results and analysis report. For example, the user can change the condition from "sudden left turn" to "right turn" and instruct the simulation to be run again.

[1643] Input: Analysis results, feedback data.

[1644] Output: The new simulation parameters.

[1645] (Application example 2)

[1646] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1647] Conventional systems primarily simulate traffic and pedestrian flow patterns, but do not take into account user emotion data, resulting in simulations that are not necessarily realistic. Furthermore, there is a lack of means to evaluate how user emotion affects simulation results. As a result, simulations can lack accuracy and reliability.

[1648] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for analyzing the sensor data and modeling urban activities and traffic flows, means for constructing a virtual environment based on the generated model, emotion engine means for collecting and analyzing user emotion data, means for generating a simulation scenario that takes the user emotion data into account, and means for running a simulation in the virtual environment and analyzing the results. This enables a more realistic simulation that takes the user's emotional state into account.

[1649] "Sensor data" is digital or analog data collected from a physical sensor.

[1650] "Analysis" is the process of classifying, evaluating, and processing collected data to extract useful information.

[1651] "Modeling" refers to the mathematical and statistical representation of real-world phenomena and converting them into a form that can be used for prediction and simulation.

[1652] A "virtual environment" is an artificial environment generated on a computer and used to recreate real-world situations.

[1653] "Emotion engine" is a general term for algorithms and systems that recognize and analyze emotions from a user's facial expressions, voice, etc.

[1654] A "simulation scenario" is a detailed definition of events or situations that occur within a virtual environment and a plan for recreating them.

[1655] "Simulation" is the process of performing calculations or mock experiments based on scenarios set up within a virtual environment.

[1656] An "analysis report" is a document or data that details and analyzes the results of a simulation.

[1657] This invention realizes a detailed simulation that takes into account the user's emotions in a system that collects and analyzes sensor and user data. In the following, a detailed embodiment of the system will be described based on the roles of the server, terminal, and user.

[1658] Server Processing

[1659] The server collects and analyzes data and generates a virtual environment using the following procedure.

[1660] 1. Data Collection:

[1661] The server collects GPS data, people flow data, purchasing data, and user emotion data such as facial expressions and voice data in real time from physical sensors (e.g., IoT sensors, GPS devices) and mobile devices, making it possible to integrate and obtain information from a variety of data sources.

[1662] 2. Data Analysis:

[1663] The server removes noise from the collected data and fills in missing values. It uses analysis algorithms (e.g., facial expression analysis algorithms, voice analysis algorithms) to classify emotional data and analyze it separately into traffic, pedestrian flow, and emotional data. It also models urban traffic and pedestrian flow patterns and performs a comprehensive analysis, including changes in user emotions.

[1664] 3. Building virtual environments using generative AI:

[1665] The server builds a basic virtual city based on the analyzed data, using a generative AI model to generate scenarios such as unexpected traffic conditions and pedestrians crossing the street, and also incorporates user emotional data to create a realistic virtual environment.

[1666] 4. Run the simulation:

[1667] The server runs simulations in the virtual environment, recreating scenarios that include unexpected events and abnormal situations. The results are analyzed in detail and a report is generated. The report, which also includes user emotional data, is used to evaluate the simulation.

[1668] Terminal handling

[1669] The terminal transmits data and executes the simulation as follows:

[1670] 1. Data transmission:

[1671] The device (e.g., smartphone, smart glasses) collects GPS data, camera footage, and audio data in real time and transmits the pre-processed data to the server.

[1672] 2. Run the simulation:

[1673] The device receives information about the virtual environment sent from the server and runs a simulation in that environment. The data generated by the simulation is sent to the server for detailed analysis.

[1674] User Action

[1675] The user operates the simulation by performing the following operations.

[1676] 1. Enter the settings:

[1677] The user inputs simulation parameters (e.g., scenario conditions and goals) using a dedicated interface, and the system also captures the user's emotional data in real time.

[1678] 2. Check the results:

[1679] Users can check the results of the simulation on a dashboard and view analysis reports, which include things like checking stress levels in response to sudden changes in the situation, and the reports also include data on the user's emotions.

[1680] 3. Providing Feedback:

[1681] Based on the simulation results, the user provides feedback and instructs the simulation to be repeated with new conditions and emotional data.

[1682] Specific examples

[1683] For example, in a food delivery simulation, the server collects and analyzes weather, traffic information, and user emotional data. Based on the analysis results, a virtual city is constructed, and a generation AI simulates unexpected traffic conditions. The device then runs the simulation based on this, and the user can view the results, including the emotional data. If the user's emotions indicate stress, the reason can be analyzed and reflected in the next simulation.

[1684] Example prompt sentence:

[1685] Generate a virtual environment and simulate a food delivery using user location, weather, and emotion data as input. Suggest the optimal delivery route when traffic conditions are "heavy," the weather is "rainy," and the user's emotion is "stressed."

[1686] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1687] Step 1:

[1688] Data collection:

[1689] The server collects real-time GPS data, people flow data, purchasing data, and user facial and voice data from sensors and mobile devices. Specifically, it uses an IoT sensor network to obtain traffic information and people flow within the city, and collects user location information and emotional data through a smartphone application. The collected data is then sent to the server.

[1690] Step 2:

[1691] Data preprocessing:

[1692] The server removes noise from the collected data and completes missing values. Specifically, it uses a data completion algorithm to complete missing values ​​and filtering techniques to remove noise. This results in clean data that is easy to analyze. It receives multiple collected datasets (e.g., GPS data, people flow data, emotion data) as input data and outputs a clean, preprocessed dataset.

[1693] Step 3:

[1694] Data Analysis:

[1695] The server classifies the preprocessed data and analyzes it separately into traffic data, people flow data, and emotion data. Specifically, it analyzes the emotion data using facial expression analysis algorithms and voice analysis algorithms, and models traffic flow and people flow patterns using machine learning models. It receives the preprocessed dataset as input data and outputs the modeled traffic, people flow, and emotion data as the analysis results.

[1696] Step 4:

[1697] Building virtual environments with generative AI:

[1698] The server builds a basic virtual city based on the analyzed data. Specifically, it recreates traffic infrastructure (e.g., roads, buildings, and traffic lights) in the virtual space and uses a generative AI model to simulate unexpected traffic conditions and pedestrian behavior. It receives the analysis results as input data and outputs a city model as a virtual environment.

[1699] Step 5:

[1700] Simulation scenario generation:

[1701] The server generates a simulation scenario within the virtual environment, taking into account the user's emotional data. Specifically, it incorporates the user's emotional state into the scenario parameters and recreates stress-inducing traffic conditions. It receives a virtual city model and emotional data as input data, and outputs a simulation scenario.

[1702] Step 6:

[1703] Run the simulation:

[1704] The server executes a simulation in a virtual environment based on a simulation scenario. Specifically, it calculates the movement of vehicles and pedestrians in the virtual environment and reproduces sudden events and abnormal situations. It receives the simulation scenario as input data and outputs the simulation results.

[1705] Step 7:

[1706] Analysis of simulation results:

[1707] The server analyzes the results of the executed simulation. Specifically, it evaluates the data collected during the simulation in detail and interprets the results taking into account the user's emotional data. It receives the simulation results as input data and outputs an analysis report.

[1708] Step 8:

[1709] Collecting user feedback:

[1710] The user views the analysis report and provides feedback based on the simulation results. Specifically, the user inputs new scenarios and conditions using a dedicated interface, and updates the emotion data. The system receives the analysis report as input data and sends new simulation conditions to the server.

[1711] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1712] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1713] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1714] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1715] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1716] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1717] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1718] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1719] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1720] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1721] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1722] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1723] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1724] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1725] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1726] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1727] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1728] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1729] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1730] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1731] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1732] The following is further disclosed regarding the above embodiment.

[1733] (Claim 1)

[1734] means for collecting sensor data;

[1735] means for analyzing said sensor data and modeling urban activity and traffic flow;

[1736] a means for constructing a virtual environment based on the generated model;

[1737] a means for generating human and vehicle behavior within a virtual environment;

[1738] A system including means for running a simulation in a virtual environment and analyzing the results.

[1739] (Claim 2)

[1740] 2. The system of claim 1, further comprising the analysis means for pre-processing data.

[1741] (Claim 3)

[1742] 10. The system of claim 1, further comprising means for generating a report based on the results of the simulation.

[1743] "Example 1"

[1744] (Claim 1)

[1745] means for collecting sensor data;

[1746] means for preprocessing the sensor data;

[1747] means for analyzing said sensor data and modeling urban activity and traffic flow;

[1748] a means for constructing a virtual environment based on the generated model;

[1749] a means for generating a scenario in a virtual environment based on a prompt sentence using a generative AI model;

[1750] means for running simulations in the virtual environment and analyzing the results;

[1751] A system including a means for visualizing the results of the simulation.

[1752] (Claim 2)

[1753] 2. The system of claim 1, further comprising the analysis means for pre-processing data.

[1754] (Claim 3)

[1755] 10. The system of claim 1, further comprising means for generating a report based on the results of the simulation.

[1756] "Application Example 1"

[1757] (Claim 1)

[1758] means for collecting sensor data;

[1759] means for analyzing said sensor data and modeling urban activity and traffic flow;

[1760] a means for constructing a virtual environment based on the generated model;

[1761] a means for generating human and vehicle behavior within a virtual environment;

[1762] means for running simulations in the virtual environment and analyzing the results;

[1763] a means for collecting and analyzing inventory control data;

[1764] a means for constructing a virtual warehouse based on the inventory management data and generating a scenario;

[1765] The system includes means for executing simulations of inventory management and transportation routes within the virtual warehouse.

[1766] (Claim 2)

[1767] 2. The system of claim 1, further comprising the analysis means for pre-processing data.

[1768] (Claim 3)

[1769] 10. The system of claim 1, further comprising means for generating a report based on the results of the simulation.

[1770] "Example 2: Combining Emotion Engines"

[1771] (Claim 1)

[1772] means for collecting sensor data;

[1773] means for analyzing the sensor data by removing noise and completing missing values;

[1774] means for modeling urban activities, traffic flows, and user emotions based on the analysis results;

[1775] a means for using a generative AI model to construct a virtual environment based on said model;

[1776] means for generating a spontaneous event within the virtual environment;

[1777] A system including means for performing a simulation within said virtual environment and analyzing the results.

[1778] (Claim 2)

[1779] 2. The system of claim 1, further comprising the analysis means for pre-processing data.

[1780] (Claim 3)

[1781] 10. The system of claim 1, further comprising means for generating a report including the user's emotional response based on the results of the simulation.

[1782] "Application example 2 when combining emotion engines"

[1783] (Claim 1)

[1784] means for collecting sensor data;

[1785] means for analyzing said sensor data and modeling urban activity and traffic flow;

[1786] a means for constructing a virtual environment based on the generated model;

[1787] a means for generating human and vehicle behavior within a virtual environment;

[1788] emotion engine means for collecting and analyzing user emotion data;

[1789] means for generating a simulation scenario taking into account user emotion data;

[1790] A system including means for running a simulation in a virtual environment and analyzing the results.

[1791] (Claim 2)

[1792] 2. The system according to claim 1, further comprising: said analysis means for preprocessing data; and said emotion engine means for preprocessing said user's emotion data.

[1793] (Claim 3)

[1794] 2. The system according to claim 1, further comprising: means for generating a report based on a result of the simulation; and means for generating an analysis report including emotion data of the user. [Explanation of symbols]

[1795] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting sensor data; means for analyzing said sensor data and modeling urban activity and traffic flow; a means for constructing a virtual environment based on the generated model; a means for generating human and vehicle behavior within a virtual environment; A system including means for running a simulation in a virtual environment and analyzing the results.

2. 2. The system of claim 1, further comprising said analysis means for pre-processing data.

3. 10. The system of claim 1, further comprising means for generating a report based on the results of the simulation.

Images