System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting behavior data of a user; means for collecting situation information from an external source; generation AI means for integrating the behavior data and the situation information and performing future simulation; means for sending a simulation result to a terminal of the user; and means for displaying the simulation result on the terminal for the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many individuals are concerned about their daily lives, work, and health. Rapidly changing environments and information can be particularly challenging to deal with, and this can affect individuals' behavior and decision-making. For example, there are many unpredictable factors, such as delayed commutes, sudden changes in the weather, and traffic jams caused by traffic accidents. To reduce these anxieties, a system is needed that collects and analyzes information in real time and provides individuals with appropriate guidelines for action. [Means for solving the problem]

[0005] The present invention is a system that includes a means for collecting user behavioral data, a means for collecting situational information from external sources, a generating AI means for integrating the behavioral data and situational information to perform a future simulation, a means for transmitting the simulation results to the user's terminal, and a means for displaying the simulation results to the user on the terminal. This allows the user to check the future simulation results based on real-time situational information and obtain appropriate guidelines for action. This reduces the user's anxiety and enables them to live a more planned and secure life.

[0006] "User behavioral data" refers to information about various activities that users engage in in their daily lives and work, including location information, schedule information, and health data.

[0007] "Situation information" refers to external environmental data that may affect user behavior, and includes news information, weather information, traffic information, etc.

[0008] "Generative AI means" refers to artificial intelligence algorithms that analyze collected behavioral data and situational information and perform future simulations.

[0009] "Simulation results" are future situations predicted based on data analyzed by the generating AI means, and include information to provide the user with appropriate guidelines for action.

[0010] A "terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC, that is capable of displaying and communicating information.

[0011] "User" refers to an individual or corporation that uses the system, and in particular, refers to a subject that provides behavioral data and receives simulation results based on that data.

[0012] "External Sources" refers to third-party data providers of real-time situation information, including news agencies, weather data providers, and traffic information services. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention relates to a system that uses user behavior data and situation information to perform future simulations and provides the results to users.

[0035] First, users provide their daily schedule, location, and health data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will record this information.

[0036] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral data.

[0037] The server then combines the behavioral data and situational information and passes it to the generation AI. The generation AI analyzes this data and performs future simulations. For example, if a user plans to leave home at 8:00, it will consider weather forecasts and traffic congestion information to simulate the optimal departure time and route. The generation AI generates multiple scenarios and selects the most appropriate suggestion from among them.

[0038] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX."

[0039] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0040] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commuter route. The server passes this information to the generation AI, which performs a simulation. The generation AI generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." This result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0041] In this way, the present invention is a system that integrates user behavioral data and situational information, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0042] The processing flow will be explained below.

[0043] Step 1:

[0044] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0045] Step 2:

[0046] The device encrypts the collected behavioral data and sends it to the server at regular intervals. For example, by setting it to send data every hour, the latest user behavioral data is always stored on the server.

[0047] Step 3:

[0048] The server receives the user's behavioral data and stores it in a database. This information is organized and managed for each user. For example, behavioral data is stored using the user ID as a key.

[0049] Step 4:

[0050] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0051] Step 5:

[0052] The server stores the collected situation information in a database and prepares it for integration with behavioral data. The situation information is normalized and formatted in a unified format, allowing for smooth integration with behavioral data.

[0053] Step 6:

[0054] The server combines the behavioral data and situational information and passes it to the generation AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0055] Step 7:

[0056] The generative AI calculates the simulation results and generates multiple scenarios, including optimal action suggestions that take into account factors such as weather and traffic conditions. For example, it creates specific suggestions such as "leave at 7:45 and use a different route."

[0057] Step 8:

[0058] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0059] Step 9:

[0060] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0061] Step 10:

[0062] The device notifies the user of the received simulation results. For example, the app's notification function may display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX."

[0063] Step 11:

[0064] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0065] Step 12:

[0066] The device then sends the data on the user's accepted suggestions back to the server, which is then used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0067] Example 1

[0068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0069] In modern society, users seek optimal guidelines for action to respond to complex schedules and changing situations. However, conventional systems have difficulty effectively integrating user behavior data with real-time situation information and providing appropriate suggestions through future simulations. Furthermore, there is a lack of means to continuously improve the accuracy of the system using user feedback. As a result, users face challenges in making quick and appropriate decisions in their daily lives and work.

[0070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0071] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's information processing device, means for displaying the simulation results to the user on the information processing device, and means for collecting feedback information from the user and reflecting it in the next prediction. This enables the user to receive guidelines for action optimized for their individual situation and make efficient and effective decisions in their daily lives and work.

[0072] "User behavioral data" refers to information related to a user's daily activities and habits, including location information, schedule information, and health data.

[0073] "Situation information" refers to real-time information collected from external sources, including news information, weather information, traffic information, etc.

[0074] "Generative AI means" refers to artificial intelligence technology that analyzes user behavioral data and situational information to simulate the future.

[0075] "Simulation Results" refers to future predictions and proposals derived based on data analyzed by the Generative AI Means.

[0076] "Information processing device" refers to a device used by a user, such as a smartphone or tablet, that displays the simulation results.

[0077] "Feedback information" refers to the actions taken by the user based on the simulation results and the evaluation of those actions.

[0078] This invention relates to a system that provides users with appropriate guidelines for action by integrating user behavioral data and situational information and simulating future situations using generative AI.

[0079] First, users use devices such as smartphones or tablets to record their daily schedules, location information, health data, and other information. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will automatically record this information.

[0080] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a secure database (e.g., Amazon RDS). The data is sent using a secure communication protocol (e.g., HTTPS). The server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes news, weather, and traffic information. This situation information is also stored on the server and linked to the behavioral data.

[0081] Next, the server combines the behavioral data and situational information and passes it to the generative AI. The generative AI analyzes this data and runs a future simulation. Specifically, if a user leaves home at 8:00, it will consider weather forecasts and traffic congestion information and simulate the optimal departure time and route. The generative AI model used is GPT-4 (registered trademark) or an equivalently advanced generative AI model.

[0082] The simulation results are then sent back to the device from the server. The device then displays the results in a format that is easy for the user to understand. For example, information such as "Tomorrow's optimal commute time is 7:45, and the suggested route is XX" can be presented via the app's notification function or widget.

[0083] Furthermore, if the user adopts the suggestion, the feedback is collected again and used for the next prediction. This feedback information records whether the user acted as suggested and is sent to the server. This allows the system to continuously learn the user's behavioral patterns and improve the prediction accuracy of the generative AI.

[0084] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the generation AI, which runs a simulation. The generation AI then generates a suggestion that it would be better to leave at 7:45 and use a different route. This result is sent to the device and notifies the user. The user accepts this suggestion, allowing for a smoother commute than usual.

[0085] As described above, the present invention is a system that provides a course of action optimized for the individual situation of a user, enabling efficient and effective decision-making in daily life and work.

[0086] An example of a prompt sentence is, "A user who leaves home to commute to work at 8:00 every morning would like the system to simulate and suggest the optimal departure time and route to avoid heavy rain and traffic congestion."

[0087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0088] Step 1:

[0089] The device records behavioral data from the user.

[0090] Input: Schedule information, location information, and health data that users enter into their devices.

[0091] How it works: An application in the device collects this data and stores it in the internal storage. For example, a user enters, "I leave home every morning at 8:00."

[0092] Output: Behavioral data stored in the device's storage.

[0093] Step 2:

[0094] The terminal transmits the collected behavioral data to the server at regular intervals.

[0095] Input: Behavioral data stored on the device.

[0096] How it works: Every 5 minutes, data is encrypted and sent to a server via a secure communication protocol (e.g. HTTPS).

[0097] Output: Encrypted behavioral data sent to the server.

[0098] Step 3:

[0099] The server decrypts the received data and stores it in a database.

[0100] Input: Encrypted behavioral data sent to the server.

[0101] What happens: The server decrypts the data and executes a SQL query to insert it into a secure database (e.g. Amazon RDS).

[0102] Output: Behavioral data stored in a database.

[0103] Step 4:

[0104] The server collects situational information from external sources.

[0105] Input: Information requests from external sources (News API, Weather API, Traffic API).

[0106] How it works: The server periodically sends requests to external sources, receives the response data, and stores it in a database. For example, "latest news updates from a news API."

[0107] Output: Situation information stored in a database.

[0108] Step 5:

[0109] The server integrates the behavioral data and situational information.

[0110] Input: Behavioral data and situational information obtained from databases.

[0111] How it works: The required data is retrieved from the database using SQL queries and formatted to be passed to the generative AI model.

[0112] Output: The aggregated data that is passed to the generative AI model.

[0113] Step 6:

[0114] The server inputs the integrated data into a generative AI model and runs a simulation of the future.

[0115] Input: Integrated behavioral data and situational information.

[0116] How it works: A generative AI model (e.g., GPT-4) analyzes input data, generates multiple scenarios, and selects the optimal course of action. For example, it generates a prediction result such as, "It would be best to leave tomorrow at 7:45 and take X street."

[0117] Output: Action suggestions as a result of the simulation.

[0118] Step 7:

[0119] The server transmits the simulation results to the terminal.

[0120] Input: Simulation results generated by the generative AI model.

[0121] Operation: The server sends the simulation results to the terminal and converts them into a notification format. For example, "Create a notification message saying 'Tomorrow's best commute time is 7:45, and the suggested route is XX street.'"

[0122] Output: Simulation results sent to the terminal.

[0123] Step 8:

[0124] The terminal notifies the user of the simulation results.

[0125] Input: Simulation results received from the server.

[0126] How it works: Uses the smartphone's notification function to display the simulation results to the user.

[0127] Output: A suggested action notified to the user.

[0128] Step 9:

[0129] The device collects user feedback.

[0130] Input: User feedback input.

[0131] Behavior: Record whether the user followed the suggestion and their rating, and send it back to the server.

[0132] Output: Feedback information sent to the server.

[0133] Step 10:

[0134] The server stores user feedback in a database and uses it for future predictions.

[0135] Input: User feedback information.

[0136] Operation: The feedback information is stored in a database and used as reference data for the next simulation.

[0137] Output: Feedback information stored in a database. This feedback information is used to improve the accuracy of the next simulation.

[0138] (Application example 1)

[0139] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] In order to respond to various situations that users encounter in their daily lives (such as weather changes, traffic congestion, and emergencies), it is important to obtain accurate information in real time and determine appropriate guidelines for action. However, current systems lack sufficient means to comprehensively analyze this information and provide users with optimal guidelines for action. In particular, for autonomous vehicles, it remains difficult to propose optimal routes and departure times in real time. This increases the burden on users and hinders efficient travel and improvements in quality of life.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0142] In this invention, the server includes a means for collecting user behavioral data, a means for collecting situational information from external sources, a generating AI means for integrating the behavioral data and situational information to perform future simulations, a means for transmitting the simulation results to the user's terminal, a means for displaying the simulation results to the user on the terminal, and a means for collecting user feedback and reflecting it in the next prediction. This allows the user to obtain optimal guidelines for action in real time, specifically, to receive suggestions for the optimal route and departure time for an autonomous vehicle. This realizes efficient and smooth travel and improves the quality of life of the user.

[0143] "User behavioral data" refers to information about the behavior of users in their daily lives, and specifically includes location information, schedule information, and health data.

[0144] "Means for collecting situation information from external sources" refers to a system for obtaining information in real time from external data sources such as weather forecasts, traffic information, and news information.

[0145] "Generative AI means" refers to an artificial intelligence system that integrates user behavior data and situational information to simulate the future. Specifically, it uses a generative AI model.

[0146] "Means for transmitting simulation results to the user's device" refers to a mechanism for transmitting the results of future simulations performed on the server to the device used by the user (e.g., smartphone, smart display).

[0147] "Means for displaying simulation results to the user on the device" refers to a function that presents the simulation results visually and in an easy-to-understand manner on the user's device. Specifically, this is done through notification functions and interfaces.

[0148] "Means for collecting user feedback and reflecting it in the next prediction" refers to a mechanism for collecting information on whether the user accepted the proposal and using that information in the next simulation.

[0149] This invention provides a system that integrates user behavior data with situational information from external sources and performs future simulations using generative AI. Specific embodiments for carrying out the invention are described below.

[0150] System Configuration

[0151] The system consists of the following main components:

[0152] 1. How we collect user behavior data

[0153] 2. Gathering situational information from external sources

[0154] 3. Generation AI means

[0155] 4. Means of sending simulation results to the user's device

[0156] 5. A means of displaying the simulation results to the user on the device

[0157] 6. A way to gather user feedback and incorporate it into the next forecast.

[0158] Hardware and Software

[0159] Hardware

[0160] User devices: smartphones, smart displays, etc.

[0161] Server: A server with high-performance computing power

[0162] Communication network: Internet communication network

[0163] software

[0164] Generative AI model: GPT-4 from OpenAI (registered trademark)

[0165] API: Weather forecast API, traffic information API, news information API

[0166] Database management system: PostgreSQL or MySQL (registered trademark)

[0167] Processing Overview

[0168] 1. Data Collection

[0169] Users provide location information, schedule information, and health data via their smartphones or in-car smart displays, and this data is recorded and collected in real time via the device's application.

[0170] 2. Gathering information on the situation

[0171] It collects weather, traffic, and news information in real time from external sources using the Weather API, Traffic API, and News API.

[0172] 3. Simulation using generative AI

[0173] The collected behavioral data and situational information are sent to a server and passed to the generative AI (GPT-4). The generative AI analyzes this data and performs future simulations. For example, given a user's schedule of "leaving home at 8:00 every morning," it integrates weather forecasts and traffic congestion information to suggest the optimal departure time and route.

[0174] Specific prompt examples:

[0175] User schedule information:

[0176] I leave home at 8:00 every morning and go to work.

[0177] Weather Information:

[0178] Heavy rain is forecast for tomorrow.

[0179] Traffic information:

[0180] There are traffic jams on major commuter routes.

[0181] Use this information to suggest the best time and route for the user to depart.

[0182] 4. Sending and displaying simulation results

[0183] The simulation results are sent from the server to the user's device, which then displays the results in a visually easy-to-understand manner, providing a notification function such as, "Tomorrow's best departure time is 7:45, and the proposed route is XX."

[0184] 5. Collecting user feedback

[0185] Feedback is collected from the user on whether they accepted the suggestions and is reflected in the next simulation. This feedback is used to continuously improve the accuracy of the system.

[0186] Specific examples

[0187] Suppose a user leaves home every morning at 8:00 and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the AI ​​generator, which runs a simulation. The AI ​​generator generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." The result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0188] The above is an embodiment of the present invention. This system continuously learns the behavioral patterns of the user and is able to make more accurate suggestions.

[0189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0190] Step 1:

[0191] Users use their smartphones or in-car smart displays to input their location information, schedule information, and health data. This data is recorded and collected in real time by an application on the device. The collected data is encrypted with the user's consent and securely sent to a server. A specific example of input data is schedule information such as "Leave home at 8:00 every morning." The output is encrypted behavioral data.

[0192] Step 2:

[0193] The server uses weather forecast APIs, traffic information APIs, and news information APIs to collect real-time situation information. This includes weather forecasts, traffic congestion information, and emergency alerts. The server also encrypts this data and stores it in a database. Specific examples of input data include "tomorrow's weather forecast is heavy rain" and "traffic congestion on major transportation routes." The output is encrypted situation information.

[0194] Step 3:

[0195] The server integrates the collected user behavioral data and situational information and inputs it into a generative AI model (for example, OpenAI's GPT-4). At this time, an appropriate prompt sentence is passed to the generative AI model. An example of a specific prompt sentence is, "User's schedule information: Leave home at 8:00 every morning and commute to work. Weather forecast information: Heavy rain is predicted for tomorrow. Traffic information: There is traffic congestion on the main commuting route. Based on this information, please suggest the optimal time and route for the user to depart." The input data is the integrated behavioral data and situational information, and the output is a simulation result.

[0196] Step 4:

[0197] The simulation results generated by the generative AI model are sent by the server to the user's device. The device receives these results and displays them in a visually easy-to-understand format. Specifically, the user is notified via a notification function or interface in the form of, "The best departure time tomorrow is 7:45, and the proposed route is XX." The input data are the simulation results from the generative AI, and the output is a notification sent to the user's device.

[0198] Step 5:

[0199] The user decides what to do based on the notification from the device. If the user accepts the suggestion, the feedback is sent from the device to the server. The server stores this feedback in a database and uses it in the next simulation. The input data is the user's feedback, and the output is updated learning data. For example, the user may provide feedback such as "I left at 7:45 as suggested and took a different route."

[0200] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0201] The present invention relates to a system that performs a future simulation based on user behavior data and situation information, and adjusts the simulation results by recognizing the user's emotions.

[0202] First, users provide their daily schedule, location, health, and emotional data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time through an application on the device. For example, if a user inputs, "I leave home at 8:00 every morning," this information is recorded. The emotion engine also uses the smartphone's camera and voice input to collect emotional data from the user's facial expressions and tone of voice.

[0203] The collected behavioral and emotional data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral and emotional data.

[0204] Next, the server combines the behavioral data, emotional data, and situational information and passes it to the generative AI. The generative AI analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00. Furthermore, the generative AI dynamically adjusts the simulation results based on the user's emotional data. For example, if the user is feeling stressed, it might suggest leaving a little earlier and taking some time to relax.

[0205] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0206] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used to make the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0207] As a concrete example, suppose a user leaves home every morning at 8:00 to commute to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. Furthermore, the emotion engine recognizes that the user's stress level is high. The server passes this information to the generation AI, which performs a simulation. The generation AI generates specific suggestions, such as "leave at 7:45, take a different route, and do some stretching to relax before leaving." The results are sent to the device and the user is notified. The user accepts the suggestions, avoiding the congestion and reducing stress, resulting in a smoother and more secure commute than usual.

[0208] In this way, the present invention is a system that integrates user behavioral data, situational information, and emotional data, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0212] Step 2:

[0213] The device uses an emotion engine to collect user emotion data. This includes facial expression recognition and voice analysis using the smartphone's camera and microphone. For example, if the user is feeling stressed, the emotion engine will recognize that state and record it as data.

[0214] Step 3:

[0215] The device encrypts the collected behavioral and emotional data and sends it to the server at regular intervals. For example, by setting the device to send data every hour, the latest user behavioral and emotional data can be stored on the server.

[0216] Step 4:

[0217] The server receives the user's behavioral and emotional data and stores it in a database. This information is organized and managed for each user. For example, the behavioral and emotional data is stored using the user ID as a key.

[0218] Step 5:

[0219] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0220] Step 6:

[0221] The server stores the collected situation information in a database and prepares it for integration with behavioral and emotional data. The situation information is normalized and formatted into a unified format, allowing for smooth integration with behavioral and emotional data.

[0222] Step 7:

[0223] The server combines behavioral data, emotional data, and situational information and passes it to the generative AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0224] Step 8:

[0225] The generative AI calculates the simulation results and generates multiple scenarios. This includes optimal action suggestions that take into account weather, traffic conditions, and the user's emotional state. For example, if the user is feeling highly stressed, the AI ​​will suggest leaving a little earlier and doing some stretching to relax.

[0226] Step 9:

[0227] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0228] Step 10:

[0229] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0230] Step 11:

[0231] The device then notifies the user of the received simulation results. For example, the app's notification function might display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[0232] Step 12:

[0233] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0234] Step 13:

[0235] The device then sends the data on the user's accepted suggestions back to the server, which uses it for the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0236] Example 2

[0237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] In modern society, users' daily lives are hectic, and they need to quickly process a large amount of information to determine appropriate guidelines for action. However, conventional systems have difficulty comprehensively analyzing user behavioral data and situational information to simulate future situations. Furthermore, they lack the ability to adjust simulations to take into account the user's emotional state, making it impossible to provide users with more appropriate guidelines for action. This can lead to unnecessary stress and difficulties.

[0239] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0240] In this invention, the server includes means for collecting user behavioral data and emotional data, means for collecting situation information from external sources, generation AI means for integrating the behavioral data, emotional data, and situation information to perform a future simulation, means for transmitting the simulation results to the user's terminal, and means for displaying the simulation results to the user on the terminal. This makes it possible to comprehensively analyze the user's behavioral data and situation information and perform a future simulation that takes the user's emotional state into consideration, thereby providing the user with more appropriate and useful guidelines for action.

[0241] "Behavioral data" refers to data including the user's daily schedule information, location information, health data, etc., and indicates the user's behavioral patterns.

[0242] "Emotional data" is data obtained by analyzing a user's facial expressions and tone of voice, and indicates the user's emotional state.

[0243] "Situation information" refers to real-time information collected from external sources such as news, weather, and traffic information, and indicates changes in the external environment.

[0244] "Generative AI methods" are methods that use AI models to analyze collected data and simulate future situations.

[0245] "Server" means a computer system that receives, stores, and analyzes data and operates the Generating AI Means.

[0246] A "terminal" is a device used by a user, such as a smartphone or tablet, that collects data and displays simulation results via communication with the server.

[0247] "Simulation results" are the predicted results of future situations and courses of action analyzed by the generative AI means.

[0248] This invention relates to a system that integrates user behavioral data, emotional data, and situational information collected from external sources, and uses a generative AI model to simulate the future. The system is mainly composed of a server and a terminal. Specific embodiments of the system are described below.

[0249] Behavioral data collection

[0250] Users use devices such as smartphones or tablets to provide their daily schedules, location information, and health data. This behavioral data is recorded in real time via an application on the device. For example, if a user inputs "I leave home at 8:00 every morning," this information is recorded on the device. The device's emotion engine also uses the smartphone's camera and voice input to analyze the user's facial expressions and tone of voice to collect emotional data.

[0251] Sending and storing data on the server

[0252] The collected behavioral and emotional data is sent from the device to a server at regular intervals. The server encrypts the data before receiving it and stores it securely in a database. Database management systems such as MySQL and PostgreSQL are used.

[0253] Gathering situational information

[0254] The server collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This situation information includes weather forecasts, traffic congestion information, and emergency alerts. This information is stored on the server and managed in conjunction with behavioral and emotional data.

[0255] Data integration and analysis

[0256] The server integrates behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). This generative AI model performs future simulations based on the integrated data. For example, based on behavioral data such as a user's decision to leave at 8:00, emotional data such as high stress levels, and situational information such as heavy rain and traffic congestion in the weather forecast, the model can suggest the optimal departure time, route, and recommended activities.

[0257] Sending and viewing simulation results

[0258] The simulation results are sent from the server to the device. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0259] Gathering feedback and learning

[0260] Feedback on whether the user accepted the suggestions is also collected and used for the next prediction. The server continuously trains the generative AI model based on this feedback data, enabling more accurate predictions and suggestions.

[0261] Prompt Sentence Examples

[0262] Let's say a user leaves home at 8:00 every morning to commute to work. The weather forecast for that day predicts heavy rain, and traffic information indicates congestion on major commuter routes. The emotion engine also recognizes that the user's stress level is high. Based on this information, the generative AI model generates the following simulation results:

[0263] For example, specific suggestions such as "Leave at 7:45, take an alternative route, and do some relaxing stretches before leaving" are generated and notified to the user.

[0264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0265] Step 1:

[0266] Users use a smartphone or tablet application to input their daily schedule, health data, and location information. The input data is recorded in real time by the application on the device. For example, a user might input a schedule such as "Leave home at 8:00 every morning."

[0267] Input: User schedule information, health data, location information

[0268] Output: Behavioral data recorded in the device

[0269] Step 2:

[0270] The device automatically records the user's real-time location using the built-in GPS sensor. It also uses the device's camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. For example, if the user is smiling while speaking, it is considered to have a "positive emotion."

[0271] Input: Location information from GPS sensor, emotion analysis data from camera and microphone

[0272] Output: Location information and emotion data recorded on the device

[0273] Step 3:

[0274] The device encrypts and transmits the collected behavioral and emotional data to the server at regular intervals, for example, every hour. The transmitted data is then encrypted and received by the server.

[0275] Input: Collected behavioral and emotional data

[0276] Output: Encrypted data is sent to and received from the server

[0277] Step 4:

[0278] The server decrypts the received encrypted data and stores it in a database using a relational database management system such as MySQL or PostgreSQL.

[0279] Input: Encrypted data sent to the server

[0280] Output: Behavioral and emotional data stored in a database

[0281] Step 5:

[0282] The server periodically collects situation information from external sources such as news APIs, weather APIs, and traffic information APIs. The collected information is analyzed within the server and stored in a database. For example, an API request is sent every morning at 6:00.

[0283] Input: Situation information obtained from the news API, weather API, and traffic information API

[0284] Output: Situation information stored in the database

[0285] Step 6:

[0286] The server combines behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). A prompt is generated based on the combined data. For example, the prompt might read, "The user's departure time is 8:00, the emotional data shows a high stress level, and the traffic information shows congestion."

[0287] Input: Integrated behavioral data, emotional data, and situational information

[0288] Output: A prompt to be input to the generative AI model

[0289] Step 7:

[0290] The generative AI model simulates the future based on the prompt text. For example, it simulates the optimal departure time, route, and recommended activities (such as stretching). The generated simulation results are output to the server.

[0291] Input: Prompt sentence for generative AI model

[0292] Output: Simulation results

[0293] Step 8:

[0294] The server sends the generated simulation results to the terminal in real time.

[0295] Input: Generated simulation results

[0296] Output: Simulation results sent to the terminal

[0297] Step 9:

[0298] The device application displays the simulation results in an easy-to-understand format for the user. Specifically, it suggests optimal courses of action through the app's notification function and widgets. For example, it might say, "Tomorrow's best commute time is 7:45, and the suggested route is ____. We also recommend doing a little stretching before you leave."

[0299] Input: Simulation results sent to the terminal

[0300] Output: The course of action displayed to the user

[0301] Step 10:

[0302] The device collects feedback on whether the user accepted the proposed offer. The device stores this feedback data and transmits it to the server at regular intervals.

[0303] Input: User feedback

[0304] Output: Feedback data is stored on the device and sent to the server.

[0305] Step 11:

[0306] The server continuously trains the generative AI model based on the collected feedback data, allowing the system to make more accurate predictions and suggestions.

[0307] Input: Feedback data

[0308] Output: An improved generative AI model

[0309] (Application example 2)

[0310] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] In today's world, it is important to improve the convenience and safety of autonomous vehicles. However, current autonomous vehicle systems are inadequate at proposing optimal driving plans that take into account the individual behavioral patterns and emotional state of the user. As a result, it is difficult to reduce user stress and provide a comfortable driving experience. Another challenge is generating driving plans that appropriately reflect changing situational information in real time.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0313] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's terminal, means for displaying the simulation results to the user on the terminal, means for collecting user emotion data, means for adjusting the simulation results using the emotion data, and means for making suggestions to optimize a driving plan for an autonomous vehicle. This makes it possible to generate an optimal driving plan that reflects the user's emotional state and real-time situational information.

[0314] "Means for collecting user behavioral data" refers to devices and applications that collect users' daily schedule information, location information, health data, etc.

[0315] "Means for collecting situational information from external sources" refers to APIs or interfaces for collecting real-time data from external sources such as news, weather, and traffic information.

[0316] The "generative AI means for integrating the behavioral data and situational information to perform future simulations" is a generative AI model for performing future simulations based on user behavioral data and external situational information.

[0317] The "means for transmitting the simulation results to the user's terminal" refers to a communication means for transmitting the simulation results in real time to the user's terminal such as a smartphone or tablet.

[0318] The "means for displaying the simulation results to the user on the terminal" refers to a user interface for displaying the simulation results on the user's terminal in an easy-to-understand manner.

[0319] The "means for collecting user emotional data" refers to a system that uses a smartphone's camera and microphone to analyze the user's emotional state from their facial expressions and tone of voice, and collects this data.

[0320] The "means for adjusting the simulation results using said emotional data" is an algorithm for dynamically adjusting the simulation results taking into account the collected emotional data.

[0321] The "means for making proposals to optimize driving plans for autonomous vehicles" is a system that provides specific proposals for optimizing the driving time and route of autonomous vehicles based on the generated simulation results.

[0322] This invention is a system that collects user behavior data, emotion data, and situation information and performs future simulations using a generative AI model to optimize driving plans for autonomous vehicles. The following describes in detail the embodiments of the invention.

[0323] Overall system flow

[0324] The server collects user behavioral data (schedule information, location information, health data) and emotional data. It also obtains real-time situational information (news information, weather information, traffic information) from external sources. This data is integrated and a generative AI model is used to perform a simulation. The server then sends the simulation results to the user's device and displays an optimized driving plan on the device.

[0325] Hardware and software used

[0326] Hardware:

[0327] Smartphone (camera, microphone, GPS, internet connection)

[0328] Server (database, generative AI model)

[0329] software:

[0330] EmotionRecognizer (emotion recognition engine)

[0331] API (weather and traffic information acquisition)

[0332] Generative AI Models

[0333] Data collection and integration

[0334] The device collects the user's schedule, location, and health data in real time. Emotion data is captured using the smartphone's camera and microphone, and analyzed by EmotionRecognizer. This data is then sent to a server at regular intervals.

[0335] The server obtains situation information from external sources using weather APIs, traffic information APIs, etc. All data is stored and integrated in a database within the server.

[0336] Simulation using generative AI models

[0337] Based on the integrated data, the server uses a generative AI model to simulate future driving plans. This simulation also takes into account the user's emotional data and dynamically adjusts suggestions to reduce stress. For example, if the user is in a high stress state, the plan can suggest an earlier departure or incorporate breaks for relaxation.

[0338] Presenting simulation results to users

[0339] The generated simulation results are sent from the server to the device. The device displays the results in an easy-to-understand format for the user. Specifically, the app provides users with guidelines for their actions through notifications and widgets. For example, it provides information such as, "The best time to leave tomorrow is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[0340] Examples of concrete examples and prompts

[0341] For example, consider a situation where a user plans to leave at 8:00, but heavy rain is forecast and traffic congestion is occurring along a major commute route. Furthermore, analysis of the user's facial expression using a smartphone camera reveals high stress levels. The system suggests leaving at 7:45, taking a different route, and doing some stretching to relax before leaving.

[0342] Example prompt sentence:

[0343] Use the user's schedule, location, health data, emotional data, and real-time weather and traffic information to simulate optimal driving plans and generate suggestions to reduce stress.

[0344] As described above, by implementing the present invention, the user can obtain an optimal driving plan that reflects the user's emotional state and real-time situation information, enabling comfortable and efficient driving.

[0345] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0346] Step 1:

[0347] The device collects user behavior data. The input is schedule information, location information, and health data, and the output is a collection of these data. The device acquires this data and periodically collects it in the background.

[0348] Step 2:

[0349] The device uses a camera and microphone to collect user emotion data. The input is camera video and audio data, and the output is emotion data (e.g., stress level). The device analyzes the emotion data using EmotionRecognizer.

[0350] Step 3:

[0351] The device sends the collected behavioral and emotional data to the server. The input is behavioral and emotional data, and the output is a transmission packet containing those data. The data is periodically uploaded to the server.

[0352] Step 4:

[0353] The server uses the weather API and traffic information API to collect real-time situation information. The input is the URL of the weather API and traffic information API, and the output is weather data and traffic data. The server sends requests to obtain various situation information.

[0354] Step 5:

[0355] The server integrates the collected behavioral data, emotional data, and situational information. The inputs are behavioral data, emotional data, weather data, and traffic data, and the output is an integrated dataset. The server stores these data in a database and integrates them into a single dataset.

[0356] Step 6:

[0357] The server passes the integrated dataset to the generative AI model to simulate future driving plans. The input is the integrated dataset, and the output is the driving simulation results. The generative AI model analyzes the data and generates an optimal driving plan.

[0358] Step 7:

[0359] The server sends the generated driving simulation results to the user's terminal. The input is the driving simulation results and the output is the transmission packets. The server sends the simulation results in packet format to the terminal.

[0360] Step 8:

[0361] The device displays the received driving simulation results to the user. The input is the driving simulation results, and the output is the display information on the user interface. The device presents the results to the user using notifications and widgets.

[0362] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0363] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0364] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0365] [Second embodiment]

[0366] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0367] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0368] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0369] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0370] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0371] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0372] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0373] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0374] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0375] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0376] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0377] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0378] This invention relates to a system that uses user behavior data and situation information to perform future simulations and provides the results to users.

[0379] First, users provide their daily schedule, location, and health data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will record this information.

[0380] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral data.

[0381] The server then combines the behavioral data and situational information and passes it to the generation AI. The generation AI analyzes this data and performs future simulations. For example, if a user plans to leave home at 8:00, it will consider weather forecasts and traffic congestion information to simulate the optimal departure time and route. The generation AI generates multiple scenarios and selects the most appropriate suggestion from among them.

[0382] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX."

[0383] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0384] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commuter route. The server passes this information to the generation AI, which performs a simulation. The generation AI generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." This result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0385] In this way, the present invention is a system that integrates user behavioral data and situational information, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0386] The processing flow will be explained below.

[0387] Step 1:

[0388] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0389] Step 2:

[0390] The device encrypts the collected behavioral data and sends it to the server at regular intervals. For example, by setting it to send data every hour, the latest user behavioral data is always stored on the server.

[0391] Step 3:

[0392] The server receives the user's behavioral data and stores it in a database. This information is organized and managed for each user. For example, behavioral data is stored using the user ID as a key.

[0393] Step 4:

[0394] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0395] Step 5:

[0396] The server stores the collected situation information in a database and prepares it for integration with behavioral data. The situation information is normalized and formatted in a unified format, allowing for smooth integration with behavioral data.

[0397] Step 6:

[0398] The server combines the behavioral data and situational information and passes it to the generation AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0399] Step 7:

[0400] The generative AI calculates the simulation results and generates multiple scenarios, including optimal action suggestions that take into account factors such as weather and traffic conditions. For example, it creates specific suggestions such as "leave at 7:45 and use a different route."

[0401] Step 8:

[0402] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0403] Step 9:

[0404] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0405] Step 10:

[0406] The device notifies the user of the received simulation results. For example, the app's notification function may display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX."

[0407] Step 11:

[0408] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0409] Step 12:

[0410] The device then sends the data on the user's accepted suggestions back to the server, which is then used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0411] Example 1

[0412] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] In modern society, users seek optimal guidelines for action to respond to complex schedules and changing situations. However, conventional systems have difficulty effectively integrating user behavior data with real-time situation information and providing appropriate suggestions through future simulations. Furthermore, there is a lack of means to continuously improve the accuracy of the system using user feedback. As a result, users face challenges in making quick and appropriate decisions in their daily lives and work.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's information processing device, means for displaying the simulation results to the user on the information processing device, and means for collecting feedback information from the user and reflecting it in the next prediction. This enables the user to receive guidelines for action optimized for their individual situation and make efficient and effective decisions in their daily lives and work.

[0416] "User behavioral data" refers to information related to a user's daily activities and habits, including location information, schedule information, and health data.

[0417] "Situation information" refers to real-time information collected from external sources, including news information, weather information, traffic information, etc.

[0418] "Generative AI means" refers to artificial intelligence technology that analyzes user behavioral data and situational information to simulate the future.

[0419] "Simulation Results" refers to future predictions and proposals derived based on data analyzed by the Generative AI Means.

[0420] "Information processing device" refers to a device used by a user, such as a smartphone or tablet, that displays the simulation results.

[0421] "Feedback information" refers to the actions taken by the user based on the simulation results and the evaluation of those actions.

[0422] This invention relates to a system that provides users with appropriate guidelines for action by integrating user behavioral data and situational information and simulating future situations using generative AI.

[0423] First, users use devices such as smartphones or tablets to record their daily schedules, location information, health data, and other information. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will automatically record this information.

[0424] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a secure database (e.g., Amazon RDS). The data is sent using a secure communication protocol (e.g., HTTPS). The server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes news, weather, and traffic information. This situation information is also stored on the server and linked to the behavioral data.

[0425] The server then combines the behavioral data and situational information and passes it to the generative AI. The generative AI then analyzes this data and runs a future simulation. Specifically, if a user leaves home at 8:00, it will consider weather forecasts and traffic congestion information and simulate the optimal departure time and route. The generative AI model used is GPT-4 or an equally advanced generative AI model.

[0426] The simulation results are then sent back to the device from the server. The device then displays the results in a format that is easy for the user to understand. For example, information such as "Tomorrow's optimal commute time is 7:45, and the suggested route is XX" can be presented via the app's notification function or widget.

[0427] Furthermore, if the user adopts the suggestion, the feedback is collected again and used for the next prediction. This feedback information records whether the user acted as suggested and is sent to the server. This allows the system to continuously learn the user's behavioral patterns and improve the prediction accuracy of the generative AI.

[0428] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the generation AI, which runs a simulation. The generation AI then generates a suggestion that it would be better to leave at 7:45 and use a different route. This result is sent to the device and notifies the user. The user accepts this suggestion, allowing for a smoother commute than usual.

[0429] As described above, the present invention is a system that provides a course of action optimized for the individual situation of a user, enabling efficient and effective decision-making in daily life and work.

[0430] An example of a prompt sentence is, "A user who leaves home to commute to work at 8:00 every morning would like the system to simulate and suggest the optimal departure time and route to avoid heavy rain and traffic congestion."

[0431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] The device records behavioral data from the user.

[0434] Input: Schedule information, location information, and health data that users enter into their devices.

[0435] How it works: An application in the device collects this data and stores it in the internal storage. For example, a user enters, "I leave home every morning at 8:00."

[0436] Output: Behavioral data stored in the device's storage.

[0437] Step 2:

[0438] The terminal transmits the collected behavioral data to the server at regular intervals.

[0439] Input: Behavioral data stored on the device.

[0440] How it works: Every 5 minutes, data is encrypted and sent to a server via a secure communication protocol (e.g. HTTPS).

[0441] Output: Encrypted behavioral data sent to the server.

[0442] Step 3:

[0443] The server decrypts the received data and stores it in a database.

[0444] Input: Encrypted behavioral data sent to the server.

[0445] What happens: The server decrypts the data and executes a SQL query to insert it into a secure database (e.g. Amazon RDS).

[0446] Output: Behavioral data stored in a database.

[0447] Step 4:

[0448] The server collects situational information from external sources.

[0449] Input: Information requests from external sources (News API, Weather API, Traffic API).

[0450] How it works: The server periodically sends requests to external sources, receives the response data, and stores it in a database. For example, "latest news updates from a news API."

[0451] Output: Situation information stored in a database.

[0452] Step 5:

[0453] The server integrates the behavioral data and situational information.

[0454] Input: Behavioral data and situational information obtained from databases.

[0455] How it works: The required data is retrieved from the database using SQL queries and formatted to be passed to the generative AI model.

[0456] Output: The aggregated data that is passed to the generative AI model.

[0457] Step 6:

[0458] The server inputs the integrated data into a generative AI model and runs a simulation of the future.

[0459] Input: Integrated behavioral data and situational information.

[0460] How it works: A generative AI model (e.g., GPT-4) analyzes input data, generates multiple scenarios, and selects the optimal course of action. For example, it generates a prediction result such as, "It would be best to leave tomorrow at 7:45 and take X street."

[0461] Output: Action suggestions as a result of the simulation.

[0462] Step 7:

[0463] The server transmits the simulation results to the terminal.

[0464] Input: Simulation results generated by the generative AI model.

[0465] Operation: The server sends the simulation results to the terminal and converts them into a notification format. For example, "Create a notification message saying 'Tomorrow's best commute time is 7:45, and the suggested route is XX street.'"

[0466] Output: Simulation results sent to the terminal.

[0467] Step 8:

[0468] The terminal notifies the user of the simulation results.

[0469] Input: Simulation results received from the server.

[0470] How it works: Uses the smartphone's notification function to display the simulation results to the user.

[0471] Output: A suggested action notified to the user.

[0472] Step 9:

[0473] The device collects user feedback.

[0474] Input: User feedback input.

[0475] Behavior: Record whether the user followed the suggestion and their rating, and send it back to the server.

[0476] Output: Feedback information sent to the server.

[0477] Step 10:

[0478] The server stores user feedback in a database and uses it for future predictions.

[0479] Input: User feedback information.

[0480] Operation: The feedback information is stored in a database and used as reference data for the next simulation.

[0481] Output: Feedback information stored in a database. This feedback information is used to improve the accuracy of the next simulation.

[0482] (Application example 1)

[0483] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0484] In order to respond to various situations that users encounter in their daily lives (such as weather changes, traffic congestion, and emergencies), it is important to obtain accurate information in real time and determine appropriate guidelines for action. However, current systems lack sufficient means to comprehensively analyze this information and provide users with optimal guidelines for action. In particular, for autonomous vehicles, it remains difficult to propose optimal routes and departure times in real time. This increases the burden on users and hinders efficient travel and improvements in quality of life.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0486] In this invention, the server includes a means for collecting user behavioral data, a means for collecting situational information from external sources, a generating AI means for integrating the behavioral data and situational information to perform future simulations, a means for transmitting the simulation results to the user's terminal, a means for displaying the simulation results to the user on the terminal, and a means for collecting user feedback and reflecting it in the next prediction. This allows the user to obtain optimal guidelines for action in real time, specifically, to receive suggestions for the optimal route and departure time for an autonomous vehicle. This realizes efficient and smooth travel and improves the quality of life of the user.

[0487] "User behavioral data" refers to information about the behavior of users in their daily lives, and specifically includes location information, schedule information, and health data.

[0488] "Means for collecting situation information from external sources" refers to a system for obtaining information in real time from external data sources such as weather forecasts, traffic information, and news information.

[0489] "Generative AI means" refers to an artificial intelligence system that integrates user behavior data and situational information to simulate the future. Specifically, it uses a generative AI model.

[0490] "Means for transmitting simulation results to the user's device" refers to a mechanism for transmitting the results of future simulations performed on the server to the device used by the user (e.g., smartphone, smart display).

[0491] "Means for displaying simulation results to the user on the device" refers to a function that presents the simulation results visually and in an easy-to-understand manner on the user's device. Specifically, this is done through notification functions and interfaces.

[0492] "Means for collecting user feedback and reflecting it in the next prediction" refers to a mechanism for collecting information on whether the user accepted the proposal and using that information in the next simulation.

[0493] This invention provides a system that integrates user behavior data with situational information from external sources and performs future simulations using generative AI. Specific embodiments for carrying out the invention are described below.

[0494] System Configuration

[0495] The system consists of the following main components:

[0496] 1. How we collect user behavior data

[0497] 2. Gathering situational information from external sources

[0498] 3. Generation AI means

[0499] 4. Means of sending simulation results to the user's device

[0500] 5. A means of displaying the simulation results to the user on the device

[0501] 6. A way to gather user feedback and incorporate it into the next forecast.

[0502] Hardware and Software

[0503] Hardware

[0504] User devices: smartphones, smart displays, etc.

[0505] Server: A server with high-performance computing power

[0506] Communication network: Internet communication network

[0507] software

[0508] Generative AI model: OpenAI's GPT-4

[0509] API: Weather forecast API, traffic information API, news information API

[0510] Database management system: PostgreSQL or MySQL

[0511] Processing Overview

[0512] 1. Data Collection

[0513] Users provide location information, schedule information, and health data via their smartphones or in-car smart displays, and this data is recorded and collected in real time via the device's application.

[0514] 2. Gathering information on the situation

[0515] It collects weather, traffic, and news information in real time from external sources using the Weather API, Traffic API, and News API.

[0516] 3. Simulation using generative AI

[0517] The collected behavioral data and situational information are sent to a server and passed to the generative AI (GPT-4). The generative AI analyzes this data and performs future simulations. For example, given a user's schedule of "leaving home at 8:00 every morning," it integrates weather forecasts and traffic congestion information to suggest the optimal departure time and route.

[0518] Specific prompt examples:

[0519] User schedule information:

[0520] I leave home at 8:00 every morning and go to work.

[0521] Weather Information:

[0522] Heavy rain is forecast for tomorrow.

[0523] Traffic information:

[0524] There are traffic jams on major commuter routes.

[0525] Use this information to suggest the best time and route for the user to depart.

[0526] 4. Sending and displaying simulation results

[0527] The simulation results are sent from the server to the user's device, which then displays the results in a visually easy-to-understand manner, providing a notification function such as, "Tomorrow's best departure time is 7:45, and the proposed route is XX."

[0528] 5. Collecting user feedback

[0529] Feedback is collected from the user on whether they accepted the suggestions and is reflected in the next simulation. This feedback is used to continuously improve the accuracy of the system.

[0530] Specific examples

[0531] Suppose a user leaves home every morning at 8:00 and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the AI ​​generator, which runs a simulation. The AI ​​generator generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." The result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0532] The above is an embodiment of the present invention. This system continuously learns the behavioral patterns of the user and is able to make more accurate suggestions.

[0533] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0534] Step 1:

[0535] Users use their smartphones or in-car smart displays to input their location information, schedule information, and health data. This data is recorded and collected in real time by an application on the device. The collected data is encrypted with the user's consent and securely sent to a server. A specific example of input data is schedule information such as "Leave home at 8:00 every morning." The output is encrypted behavioral data.

[0536] Step 2:

[0537] The server uses weather forecast APIs, traffic information APIs, and news information APIs to collect real-time situation information. This includes weather forecasts, traffic congestion information, and emergency alerts. The server also encrypts this data and stores it in a database. Specific examples of input data include "tomorrow's weather forecast is heavy rain" and "traffic congestion on major transportation routes." The output is encrypted situation information.

[0538] Step 3:

[0539] The server integrates the collected user behavioral data and situational information and inputs it into a generative AI model (for example, OpenAI's GPT-4). At this time, an appropriate prompt sentence is passed to the generative AI model. An example of a specific prompt sentence is, "User's schedule information: Leave home at 8:00 every morning and commute to work. Weather forecast information: Heavy rain is predicted for tomorrow. Traffic information: There is traffic congestion on the main commuting route. Based on this information, please suggest the optimal time and route for the user to depart." The input data is the integrated behavioral data and situational information, and the output is a simulation result.

[0540] Step 4:

[0541] The simulation results generated by the generative AI model are sent by the server to the user's device. The device receives these results and displays them in a visually easy-to-understand format. Specifically, the user is notified via a notification function or interface in the form of, "The best departure time tomorrow is 7:45, and the proposed route is XX." The input data are the simulation results from the generative AI, and the output is a notification sent to the user's device.

[0542] Step 5:

[0543] The user decides what to do based on the notification from the device. If the user accepts the suggestion, the feedback is sent from the device to the server. The server stores this feedback in a database and uses it in the next simulation. The input data is the user's feedback, and the output is updated learning data. For example, the user may provide feedback such as "I left at 7:45 as suggested and took a different route."

[0544] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0545] The present invention relates to a system that performs a future simulation based on user behavior data and situation information, and adjusts the simulation results by recognizing the user's emotions.

[0546] First, users provide their daily schedule, location, health, and emotional data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time through an application on the device. For example, if a user inputs, "I leave home at 8:00 every morning," this information is recorded. The emotion engine also uses the smartphone's camera and voice input to collect emotional data from the user's facial expressions and tone of voice.

[0547] The collected behavioral and emotional data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral and emotional data.

[0548] Next, the server combines the behavioral data, emotional data, and situational information and passes it to the generative AI. The generative AI analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00. Furthermore, the generative AI dynamically adjusts the simulation results based on the user's emotional data. For example, if the user is feeling stressed, it might suggest leaving a little earlier and taking some time to relax.

[0549] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0550] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used to make the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0551] As a concrete example, suppose a user leaves home every morning at 8:00 to commute to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. Furthermore, the emotion engine recognizes that the user's stress level is high. The server passes this information to the generation AI, which performs a simulation. The generation AI generates specific suggestions, such as "leave at 7:45, take a different route, and do some stretching to relax before leaving." The results are sent to the device and the user is notified. The user accepts the suggestions, avoiding the congestion and reducing stress, resulting in a smoother and more secure commute than usual.

[0552] In this way, the present invention is a system that integrates user behavioral data, situational information, and emotional data, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0556] Step 2:

[0557] The device uses an emotion engine to collect user emotion data. This includes facial expression recognition and voice analysis using the smartphone's camera and microphone. For example, if the user is feeling stressed, the emotion engine will recognize that state and record it as data.

[0558] Step 3:

[0559] The device encrypts the collected behavioral and emotional data and sends it to the server at regular intervals. For example, by setting the device to send data every hour, the latest user behavioral and emotional data can be stored on the server.

[0560] Step 4:

[0561] The server receives the user's behavioral and emotional data and stores it in a database. This information is organized and managed for each user. For example, the behavioral and emotional data is stored using the user ID as a key.

[0562] Step 5:

[0563] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0564] Step 6:

[0565] The server stores the collected situation information in a database and prepares it for integration with behavioral and emotional data. The situation information is normalized and formatted into a unified format, allowing for smooth integration with behavioral and emotional data.

[0566] Step 7:

[0567] The server combines behavioral data, emotional data, and situational information and passes it to the generative AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0568] Step 8:

[0569] The generative AI calculates the simulation results and generates multiple scenarios. This includes optimal action suggestions that take into account weather, traffic conditions, and the user's emotional state. For example, if the user is feeling highly stressed, the AI ​​will suggest leaving a little earlier and doing some stretching to relax.

[0570] Step 9:

[0571] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0572] Step 10:

[0573] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0574] Step 11:

[0575] The device then notifies the user of the received simulation results. For example, the app's notification function might display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[0576] Step 12:

[0577] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0578] Step 13:

[0579] The device then sends the data on the user's accepted suggestions back to the server, which uses it for the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0580] Example 2

[0581] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] In modern society, users' daily lives are hectic, and they need to quickly process a large amount of information to determine appropriate guidelines for action. However, conventional systems have difficulty comprehensively analyzing user behavioral data and situational information to simulate future situations. Furthermore, they lack the ability to adjust simulations to take into account the user's emotional state, making it impossible to provide users with more appropriate guidelines for action. This can lead to unnecessary stress and difficulties.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0584] In this invention, the server includes means for collecting user behavioral data and emotional data, means for collecting situation information from external sources, generation AI means for integrating the behavioral data, emotional data, and situation information to perform a future simulation, means for transmitting the simulation results to the user's terminal, and means for displaying the simulation results to the user on the terminal. This makes it possible to comprehensively analyze the user's behavioral data and situation information and perform a future simulation that takes the user's emotional state into consideration, thereby providing the user with more appropriate and useful guidelines for action.

[0585] "Behavioral data" refers to data including the user's daily schedule information, location information, health data, etc., and indicates the user's behavioral patterns.

[0586] "Emotional data" is data obtained by analyzing a user's facial expressions and tone of voice, and indicates the user's emotional state.

[0587] "Situation information" refers to real-time information collected from external sources such as news, weather, and traffic information, and indicates changes in the external environment.

[0588] "Generative AI methods" are methods that use AI models to analyze collected data and simulate future situations.

[0589] "Server" means a computer system that receives, stores, and analyzes data and operates the Generating AI Means.

[0590] A "terminal" is a device used by a user, such as a smartphone or tablet, that collects data and displays simulation results via communication with the server.

[0591] "Simulation results" are the predicted results of future situations and courses of action analyzed by the generative AI means.

[0592] This invention relates to a system that integrates user behavioral data, emotional data, and situational information collected from external sources, and uses a generative AI model to simulate the future. The system is mainly composed of a server and a terminal. Specific embodiments of the system are described below.

[0593] Behavioral data collection

[0594] Users use devices such as smartphones or tablets to provide their daily schedules, location information, and health data. This behavioral data is recorded in real time via an application on the device. For example, if a user inputs "I leave home at 8:00 every morning," this information is recorded on the device. The device's emotion engine also uses the smartphone's camera and voice input to analyze the user's facial expressions and tone of voice to collect emotional data.

[0595] Sending and storing data on the server

[0596] The collected behavioral and emotional data is sent from the device to a server at regular intervals. The server encrypts the data before receiving it and stores it securely in a database. Database management systems such as MySQL and PostgreSQL are used.

[0597] Gathering situational information

[0598] The server collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This situation information includes weather forecasts, traffic congestion information, and emergency alerts. This information is stored on the server and managed in conjunction with behavioral and emotional data.

[0599] Data integration and analysis

[0600] The server integrates behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). This generative AI model performs future simulations based on the integrated data. For example, based on behavioral data such as a user's decision to leave at 8:00, emotional data such as high stress levels, and situational information such as heavy rain and traffic congestion in the weather forecast, the model can suggest the optimal departure time, route, and recommended activities.

[0601] Sending and viewing simulation results

[0602] The simulation results are sent from the server to the device. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0603] Gathering feedback and learning

[0604] Feedback on whether the user accepted the suggestions is also collected and used for the next prediction. The server continuously trains the generative AI model based on this feedback data, enabling more accurate predictions and suggestions.

[0605] Prompt Sentence Examples

[0606] Let's say a user leaves home at 8:00 every morning to commute to work. The weather forecast for that day predicts heavy rain, and traffic information indicates congestion on major commuter routes. The emotion engine also recognizes that the user's stress level is high. Based on this information, the generative AI model generates the following simulation results:

[0607] For example, specific suggestions such as "Leave at 7:45, take an alternative route, and do some relaxing stretches before leaving" are generated and notified to the user.

[0608] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0609] Step 1:

[0610] Users use a smartphone or tablet application to input their daily schedule, health data, and location information. The input data is recorded in real time by the application on the device. For example, a user might input a schedule such as "Leave home at 8:00 every morning."

[0611] Input: User schedule information, health data, location information

[0612] Output: Behavioral data recorded in the device

[0613] Step 2:

[0614] The device automatically records the user's real-time location using the built-in GPS sensor. It also uses the device's camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. For example, if the user is smiling while speaking, it is considered to have a "positive emotion."

[0615] Input: Location information from GPS sensor, emotion analysis data from camera and microphone

[0616] Output: Location information and emotion data recorded on the device

[0617] Step 3:

[0618] The device encrypts and transmits the collected behavioral and emotional data to the server at regular intervals, for example, every hour. The transmitted data is then encrypted and received by the server.

[0619] Input: Collected behavioral and emotional data

[0620] Output: Encrypted data is sent to and received from the server

[0621] Step 4:

[0622] The server decrypts the received encrypted data and stores it in a database using a relational database management system such as MySQL or PostgreSQL.

[0623] Input: Encrypted data sent to the server

[0624] Output: Behavioral and emotional data stored in a database

[0625] Step 5:

[0626] The server periodically collects situation information from external sources such as news APIs, weather APIs, and traffic information APIs. The collected information is analyzed within the server and stored in a database. For example, an API request is sent every morning at 6:00.

[0627] Input: Situation information obtained from the news API, weather API, and traffic information API

[0628] Output: Situation information stored in the database

[0629] Step 6:

[0630] The server combines behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). A prompt is generated based on the combined data. For example, the prompt might read, "The user's departure time is 8:00, the emotional data shows a high stress level, and the traffic information shows congestion."

[0631] Input: Integrated behavioral data, emotional data, and situational information

[0632] Output: A prompt to be input to the generative AI model

[0633] Step 7:

[0634] The generative AI model simulates the future based on the prompt text. For example, it simulates the optimal departure time, route, and recommended activities (such as stretching). The generated simulation results are output to the server.

[0635] Input: Prompt sentence for generative AI model

[0636] Output: Simulation results

[0637] Step 8:

[0638] The server sends the generated simulation results to the terminal in real time.

[0639] Input: Generated simulation results

[0640] Output: Simulation results sent to the terminal

[0641] Step 9:

[0642] The device application displays the simulation results in an easy-to-understand format for the user. Specifically, it suggests optimal courses of action through the app's notification function and widgets. For example, it might say, "Tomorrow's best commute time is 7:45, and the suggested route is ____. We also recommend doing a little stretching before you leave."

[0643] Input: Simulation results sent to the terminal

[0644] Output: The course of action displayed to the user

[0645] Step 10:

[0646] The device collects feedback on whether the user accepted the proposed offer. The device stores this feedback data and transmits it to the server at regular intervals.

[0647] Input: User feedback

[0648] Output: Feedback data is stored on the device and sent to the server.

[0649] Step 11:

[0650] The server continuously trains the generative AI model based on the collected feedback data, allowing the system to make more accurate predictions and suggestions.

[0651] Input: Feedback data

[0652] Output: An improved generative AI model

[0653] (Application example 2)

[0654] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0655] In today's world, it is important to improve the convenience and safety of autonomous vehicles. However, current autonomous vehicle systems are inadequate at proposing optimal driving plans that take into account the individual behavioral patterns and emotional state of the user. As a result, it is difficult to reduce user stress and provide a comfortable driving experience. Another challenge is generating driving plans that appropriately reflect changing situational information in real time.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0657] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's terminal, means for displaying the simulation results to the user on the terminal, means for collecting user emotion data, means for adjusting the simulation results using the emotion data, and means for making suggestions to optimize a driving plan for an autonomous vehicle. This makes it possible to generate an optimal driving plan that reflects the user's emotional state and real-time situational information.

[0658] "Means for collecting user behavioral data" refers to devices and applications that collect users' daily schedule information, location information, health data, etc.

[0659] "Means for collecting situational information from external sources" refers to APIs or interfaces for collecting real-time data from external sources such as news, weather, and traffic information.

[0660] The "generative AI means for integrating the behavioral data and situational information to perform future simulations" is a generative AI model for performing future simulations based on user behavioral data and external situational information.

[0661] The "means for transmitting the simulation results to the user's terminal" refers to a communication means for transmitting the simulation results in real time to the user's terminal such as a smartphone or tablet.

[0662] The "means for displaying the simulation results to the user on the terminal" refers to a user interface for displaying the simulation results on the user's terminal in an easy-to-understand manner.

[0663] The "means for collecting user emotional data" refers to a system that uses a smartphone's camera and microphone to analyze the user's emotional state from their facial expressions and tone of voice, and collects this data.

[0664] The "means for adjusting the simulation results using said emotional data" is an algorithm for dynamically adjusting the simulation results taking into account the collected emotional data.

[0665] The "means for making proposals to optimize driving plans for autonomous vehicles" is a system that provides specific proposals for optimizing the driving time and route of autonomous vehicles based on the generated simulation results.

[0666] This invention is a system that collects user behavior data, emotion data, and situation information and performs future simulations using a generative AI model to optimize driving plans for autonomous vehicles. The following describes in detail the embodiments of the invention.

[0667] Overall system flow

[0668] The server collects user behavioral data (schedule information, location information, health data) and emotional data. It also obtains real-time situational information (news information, weather information, traffic information) from external sources. This data is integrated and a generative AI model is used to perform a simulation. The server then sends the simulation results to the user's device and displays an optimized driving plan on the device.

[0669] Hardware and software used

[0670] Hardware:

[0671] Smartphone (camera, microphone, GPS, internet connection)

[0672] Server (database, generative AI model)

[0673] software:

[0674] EmotionRecognizer (emotion recognition engine)

[0675] API (weather and traffic information acquisition)

[0676] Generative AI Models

[0677] Data collection and integration

[0678] The device collects the user's schedule, location, and health data in real time. Emotion data is captured using the smartphone's camera and microphone, and analyzed by EmotionRecognizer. This data is then sent to a server at regular intervals.

[0679] The server obtains situation information from external sources using weather APIs, traffic information APIs, etc. All data is stored and integrated in a database within the server.

[0680] Simulation using generative AI models

[0681] Based on the integrated data, the server uses a generative AI model to simulate future driving plans. This simulation also takes into account the user's emotional data and dynamically adjusts suggestions to reduce stress. For example, if the user is in a high stress state, the plan can suggest an earlier departure or incorporate breaks for relaxation.

[0682] Presenting simulation results to users

[0683] The generated simulation results are sent from the server to the device. The device displays the results in an easy-to-understand format for the user. Specifically, the app provides users with guidelines for their actions through notifications and widgets. For example, it provides information such as, "The best time to leave tomorrow is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[0684] Examples of concrete examples and prompts

[0685] For example, consider a situation where a user plans to leave at 8:00, but heavy rain is forecast and traffic congestion is occurring along a major commute route. Furthermore, analysis of the user's facial expression using a smartphone camera reveals high stress levels. The system suggests leaving at 7:45, taking a different route, and doing some stretching to relax before leaving.

[0686] Example prompt sentence:

[0687] Use the user's schedule, location, health data, emotional data, and real-time weather and traffic information to simulate optimal driving plans and generate suggestions to reduce stress.

[0688] As described above, by implementing the present invention, the user can obtain an optimal driving plan that reflects the user's emotional state and real-time situation information, enabling comfortable and efficient driving.

[0689] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0690] Step 1:

[0691] The device collects user behavior data. The input is schedule information, location information, and health data, and the output is a collection of these data. The device acquires this data and periodically collects it in the background.

[0692] Step 2:

[0693] The device uses a camera and microphone to collect user emotion data. The input is camera video and audio data, and the output is emotion data (e.g., stress level). The device analyzes the emotion data using EmotionRecognizer.

[0694] Step 3:

[0695] The device sends the collected behavioral and emotional data to the server. The input is behavioral and emotional data, and the output is a transmission packet containing those data. The data is periodically uploaded to the server.

[0696] Step 4:

[0697] The server uses the weather API and traffic information API to collect real-time situation information. The input is the URL of the weather API and traffic information API, and the output is weather data and traffic data. The server sends requests to obtain various situation information.

[0698] Step 5:

[0699] The server integrates the collected behavioral data, emotional data, and situational information. The inputs are behavioral data, emotional data, weather data, and traffic data, and the output is an integrated dataset. The server stores these data in a database and integrates them into a single dataset.

[0700] Step 6:

[0701] The server passes the integrated dataset to the generative AI model to simulate future driving plans. The input is the integrated dataset, and the output is the driving simulation results. The generative AI model analyzes the data and generates an optimal driving plan.

[0702] Step 7:

[0703] The server sends the generated driving simulation results to the user's terminal. The input is the driving simulation results and the output is the transmission packets. The server sends the simulation results in packet format to the terminal.

[0704] Step 8:

[0705] The device displays the received driving simulation results to the user. The input is the driving simulation results, and the output is the display information on the user interface. The device presents the results to the user using notifications and widgets.

[0706] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0707] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0708] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0709] [Third embodiment]

[0710] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0711] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0712] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0713] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0714] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0715] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0716] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0717] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0718] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0719] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0720] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0721] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0722] This invention relates to a system that uses user behavior data and situation information to perform future simulations and provides the results to users.

[0723] First, users provide their daily schedule, location, and health data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will record this information.

[0724] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral data.

[0725] The server then combines the behavioral data and situational information and passes it to the generation AI. The generation AI analyzes this data and performs future simulations. For example, if a user plans to leave home at 8:00, it will consider weather forecasts and traffic congestion information to simulate the optimal departure time and route. The generation AI generates multiple scenarios and selects the most appropriate suggestion from among them.

[0726] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX."

[0727] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0728] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commuter route. The server passes this information to the generation AI, which performs a simulation. The generation AI generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." This result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0729] In this way, the present invention is a system that integrates user behavioral data and situational information, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0733] Step 2:

[0734] The device encrypts the collected behavioral data and sends it to the server at regular intervals. For example, by setting it to send data every hour, the latest user behavioral data is always stored on the server.

[0735] Step 3:

[0736] The server receives the user's behavioral data and stores it in a database. This information is organized and managed for each user. For example, behavioral data is stored using the user ID as a key.

[0737] Step 4:

[0738] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0739] Step 5:

[0740] The server stores the collected situation information in a database and prepares it for integration with behavioral data. The situation information is normalized and formatted in a unified format, allowing for smooth integration with behavioral data.

[0741] Step 6:

[0742] The server combines the behavioral data and situational information and passes it to the generation AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0743] Step 7:

[0744] The generative AI calculates the simulation results and generates multiple scenarios, including optimal action suggestions that take into account factors such as weather and traffic conditions. For example, it creates specific suggestions such as "leave at 7:45 and use a different route."

[0745] Step 8:

[0746] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0747] Step 9:

[0748] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0749] Step 10:

[0750] The device notifies the user of the received simulation results. For example, the app's notification function may display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX."

[0751] Step 11:

[0752] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0753] Step 12:

[0754] The device then sends the data on the user's accepted suggestions back to the server, which is then used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[0755] Example 1

[0756] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0757] In modern society, users seek optimal guidelines for action to respond to complex schedules and changing situations. However, conventional systems have difficulty effectively integrating user behavior data with real-time situation information and providing appropriate suggestions through future simulations. Furthermore, there is a lack of means to continuously improve the accuracy of the system using user feedback. As a result, users face challenges in making quick and appropriate decisions in their daily lives and work.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0759] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's information processing device, means for displaying the simulation results to the user on the information processing device, and means for collecting feedback information from the user and reflecting it in the next prediction. This enables the user to receive guidelines for action optimized for their individual situation and make efficient and effective decisions in their daily lives and work.

[0760] "User behavioral data" refers to information related to a user's daily activities and habits, including location information, schedule information, and health data.

[0761] "Situation information" refers to real-time information collected from external sources, including news information, weather information, traffic information, etc.

[0762] "Generative AI means" refers to artificial intelligence technology that analyzes user behavioral data and situational information to simulate the future.

[0763] "Simulation Results" refers to future predictions and proposals derived based on data analyzed by the Generative AI Means.

[0764] "Information processing device" refers to a device used by a user, such as a smartphone or tablet, that displays the simulation results.

[0765] "Feedback information" refers to the actions taken by the user based on the simulation results and the evaluation of those actions.

[0766] This invention relates to a system that provides users with appropriate guidelines for action by integrating user behavioral data and situational information and simulating future situations using generative AI.

[0767] First, users use devices such as smartphones or tablets to record their daily schedules, location information, health data, and other information. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will automatically record this information.

[0768] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a secure database (e.g., Amazon RDS). The data is sent using a secure communication protocol (e.g., HTTPS). The server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes news, weather, and traffic information. This situation information is also stored on the server and linked to the behavioral data.

[0769] The server then combines the behavioral data and situational information and passes it to the generative AI. The generative AI then analyzes this data and runs a future simulation. Specifically, if a user leaves home at 8:00, it will consider weather forecasts and traffic congestion information and simulate the optimal departure time and route. The generative AI model used is GPT-4 or an equally advanced generative AI model.

[0770] The simulation results are then sent back to the device from the server. The device then displays the results in a format that is easy for the user to understand. For example, information such as "Tomorrow's optimal commute time is 7:45, and the suggested route is XX" can be presented via the app's notification function or widget.

[0771] Furthermore, if the user adopts the suggestion, the feedback is collected again and used for the next prediction. This feedback information records whether the user acted as suggested and is sent to the server. This allows the system to continuously learn the user's behavioral patterns and improve the prediction accuracy of the generative AI.

[0772] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the generation AI, which runs a simulation. The generation AI then generates a suggestion that it would be better to leave at 7:45 and use a different route. This result is sent to the device and notifies the user. The user accepts this suggestion, allowing for a smoother commute than usual.

[0773] As described above, the present invention is a system that provides a course of action optimized for the individual situation of a user, enabling efficient and effective decision-making in daily life and work.

[0774] An example of a prompt sentence is, "A user who leaves home to commute to work at 8:00 every morning would like the system to simulate and suggest the optimal departure time and route to avoid heavy rain and traffic congestion."

[0775] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0776] Step 1:

[0777] The device records behavioral data from the user.

[0778] Input: Schedule information, location information, and health data that users enter into their devices.

[0779] How it works: An application in the device collects this data and stores it in the internal storage. For example, a user enters, "I leave home every morning at 8:00."

[0780] Output: Behavioral data stored in the device's storage.

[0781] Step 2:

[0782] The terminal transmits the collected behavioral data to the server at regular intervals.

[0783] Input: Behavioral data stored on the device.

[0784] How it works: Every 5 minutes, data is encrypted and sent to a server via a secure communication protocol (e.g. HTTPS).

[0785] Output: Encrypted behavioral data sent to the server.

[0786] Step 3:

[0787] The server decrypts the received data and stores it in a database.

[0788] Input: Encrypted behavioral data sent to the server.

[0789] What happens: The server decrypts the data and executes a SQL query to insert it into a secure database (e.g. Amazon RDS).

[0790] Output: Behavioral data stored in a database.

[0791] Step 4:

[0792] The server collects situational information from external sources.

[0793] Input: Information requests from external sources (News API, Weather API, Traffic API).

[0794] How it works: The server periodically sends requests to external sources, receives the response data, and stores it in a database. For example, "latest news updates from a news API."

[0795] Output: Situation information stored in a database.

[0796] Step 5:

[0797] The server integrates the behavioral data and situational information.

[0798] Input: Behavioral data and situational information obtained from databases.

[0799] How it works: The required data is retrieved from the database using SQL queries and formatted to be passed to the generative AI model.

[0800] Output: The aggregated data that is passed to the generative AI model.

[0801] Step 6:

[0802] The server inputs the integrated data into a generative AI model and runs a simulation of the future.

[0803] Input: Integrated behavioral data and situational information.

[0804] How it works: A generative AI model (e.g., GPT-4) analyzes input data, generates multiple scenarios, and selects the optimal course of action. For example, it generates a prediction result such as, "It would be best to leave tomorrow at 7:45 and take X street."

[0805] Output: Action suggestions as a result of the simulation.

[0806] Step 7:

[0807] The server transmits the simulation results to the terminal.

[0808] Input: Simulation results generated by the generative AI model.

[0809] Operation: The server sends the simulation results to the terminal and converts them into a notification format. For example, "Create a notification message saying 'Tomorrow's best commute time is 7:45, and the suggested route is XX street.'"

[0810] Output: Simulation results sent to the terminal.

[0811] Step 8:

[0812] The terminal notifies the user of the simulation results.

[0813] Input: Simulation results received from the server.

[0814] How it works: Uses the smartphone's notification function to display the simulation results to the user.

[0815] Output: A suggested action notified to the user.

[0816] Step 9:

[0817] The device collects user feedback.

[0818] Input: User feedback input.

[0819] Behavior: Record whether the user followed the suggestion and their rating, and send it back to the server.

[0820] Output: Feedback information sent to the server.

[0821] Step 10:

[0822] The server stores user feedback in a database and uses it for future predictions.

[0823] Input: User feedback information.

[0824] Operation: The feedback information is stored in a database and used as reference data for the next simulation.

[0825] Output: Feedback information stored in a database. This feedback information is used to improve the accuracy of the next simulation.

[0826] (Application example 1)

[0827] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0828] In order to respond to various situations that users encounter in their daily lives (such as weather changes, traffic congestion, and emergencies), it is important to obtain accurate information in real time and determine appropriate guidelines for action. However, current systems lack sufficient means to comprehensively analyze this information and provide users with optimal guidelines for action. In particular, for autonomous vehicles, it remains difficult to propose optimal routes and departure times in real time. This increases the burden on users and hinders efficient travel and improvements in quality of life.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0830] In this invention, the server includes a means for collecting user behavioral data, a means for collecting situational information from external sources, a generating AI means for integrating the behavioral data and situational information to perform future simulations, a means for transmitting the simulation results to the user's terminal, a means for displaying the simulation results to the user on the terminal, and a means for collecting user feedback and reflecting it in the next prediction. This allows the user to obtain optimal guidelines for action in real time, specifically, to receive suggestions for the optimal route and departure time for an autonomous vehicle. This realizes efficient and smooth travel and improves the quality of life of the user.

[0831] "User behavioral data" refers to information about the behavior of users in their daily lives, and specifically includes location information, schedule information, and health data.

[0832] "Means for collecting situation information from external sources" refers to a system for obtaining information in real time from external data sources such as weather forecasts, traffic information, and news information.

[0833] "Generative AI means" refers to an artificial intelligence system that integrates user behavior data and situational information to simulate the future. Specifically, it uses a generative AI model.

[0834] "Means for transmitting simulation results to the user's device" refers to a mechanism for transmitting the results of future simulations performed on the server to the device used by the user (e.g., smartphone, smart display).

[0835] "Means for displaying simulation results to the user on the device" refers to a function that presents the simulation results visually and in an easy-to-understand manner on the user's device. Specifically, this is done through notification functions and interfaces.

[0836] "Means for collecting user feedback and reflecting it in the next prediction" refers to a mechanism for collecting information on whether the user accepted the proposal and using that information in the next simulation.

[0837] This invention provides a system that integrates user behavior data with situational information from external sources and performs future simulations using generative AI. Specific embodiments for carrying out the invention are described below.

[0838] System Configuration

[0839] The system consists of the following main components:

[0840] 1. How we collect user behavior data

[0841] 2. Gathering situational information from external sources

[0842] 3. Generation AI means

[0843] 4. Means of sending simulation results to the user's device

[0844] 5. A means of displaying the simulation results to the user on the device

[0845] 6. A way to gather user feedback and incorporate it into the next forecast.

[0846] Hardware and Software

[0847] Hardware

[0848] User devices: smartphones, smart displays, etc.

[0849] Server: A server with high-performance computing power

[0850] Communication network: Internet communication network

[0851] software

[0852] Generative AI model: OpenAI's GPT-4

[0853] API: Weather forecast API, traffic information API, news information API

[0854] Database management system: PostgreSQL or MySQL

[0855] Processing Overview

[0856] 1. Data Collection

[0857] Users provide location information, schedule information, and health data via their smartphones or in-car smart displays, and this data is recorded and collected in real time via the device's application.

[0858] 2. Gathering information on the situation

[0859] It collects weather, traffic, and news information in real time from external sources using the Weather API, Traffic API, and News API.

[0860] 3. Simulation using generative AI

[0861] The collected behavioral data and situational information are sent to a server and passed to the generative AI (GPT-4). The generative AI analyzes this data and performs future simulations. For example, given a user's schedule of "leaving home at 8:00 every morning," it integrates weather forecasts and traffic congestion information to suggest the optimal departure time and route.

[0862] Specific prompt examples:

[0863] User schedule information:

[0864] I leave home at 8:00 every morning and go to work.

[0865] Weather Information:

[0866] Heavy rain is forecast for tomorrow.

[0867] Traffic information:

[0868] There are traffic jams on major commuter routes.

[0869] Use this information to suggest the best time and route for the user to depart.

[0870] 4. Sending and displaying simulation results

[0871] The simulation results are sent from the server to the user's device, which then displays the results in a visually easy-to-understand manner, providing a notification function such as, "Tomorrow's best departure time is 7:45, and the proposed route is XX."

[0872] 5. Collecting user feedback

[0873] Feedback is collected from the user on whether they accepted the suggestions and is reflected in the next simulation. This feedback is used to continuously improve the accuracy of the system.

[0874] Specific examples

[0875] Suppose a user leaves home every morning at 8:00 and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the AI ​​generator, which runs a simulation. The AI ​​generator generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." The result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[0876] The above is an embodiment of the present invention. This system continuously learns the behavioral patterns of the user and is able to make more accurate suggestions.

[0877] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0878] Step 1:

[0879] Users use their smartphones or in-car smart displays to input their location information, schedule information, and health data. This data is recorded and collected in real time by an application on the device. The collected data is encrypted with the user's consent and securely sent to a server. A specific example of input data is schedule information such as "Leave home at 8:00 every morning." The output is encrypted behavioral data.

[0880] Step 2:

[0881] The server uses weather forecast APIs, traffic information APIs, and news information APIs to collect real-time situation information. This includes weather forecasts, traffic congestion information, and emergency alerts. The server also encrypts this data and stores it in a database. Specific examples of input data include "tomorrow's weather forecast is heavy rain" and "traffic congestion on major transportation routes." The output is encrypted situation information.

[0882] Step 3:

[0883] The server integrates the collected user behavioral data and situational information and inputs it into a generative AI model (for example, OpenAI's GPT-4). At this time, an appropriate prompt sentence is passed to the generative AI model. An example of a specific prompt sentence is, "User's schedule information: Leave home at 8:00 every morning and commute to work. Weather forecast information: Heavy rain is predicted for tomorrow. Traffic information: There is traffic congestion on the main commuting route. Based on this information, please suggest the optimal time and route for the user to depart." The input data is the integrated behavioral data and situational information, and the output is a simulation result.

[0884] Step 4:

[0885] The simulation results generated by the generative AI model are sent by the server to the user's device. The device receives these results and displays them in a visually easy-to-understand format. Specifically, the user is notified via a notification function or interface in the form of, "The best departure time tomorrow is 7:45, and the proposed route is XX." The input data are the simulation results from the generative AI, and the output is a notification sent to the user's device.

[0886] Step 5:

[0887] The user decides what to do based on the notification from the device. If the user accepts the suggestion, the feedback is sent from the device to the server. The server stores this feedback in a database and uses it in the next simulation. The input data is the user's feedback, and the output is updated learning data. For example, the user may provide feedback such as "I left at 7:45 as suggested and took a different route."

[0888] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0889] The present invention relates to a system that performs a future simulation based on user behavior data and situation information, and adjusts the simulation results by recognizing the user's emotions.

[0890] First, users provide their daily schedule, location, health, and emotional data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time through an application on the device. For example, if a user inputs, "I leave home at 8:00 every morning," this information is recorded. The emotion engine also uses the smartphone's camera and voice input to collect emotional data from the user's facial expressions and tone of voice.

[0891] The collected behavioral and emotional data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral and emotional data.

[0892] Next, the server combines the behavioral data, emotional data, and situational information and passes it to the generative AI. The generative AI analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00. Furthermore, the generative AI dynamically adjusts the simulation results based on the user's emotional data. For example, if the user is feeling stressed, it might suggest leaving a little earlier and taking some time to relax.

[0893] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0894] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used to make the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0895] As a concrete example, suppose a user leaves home every morning at 8:00 to commute to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. Furthermore, the emotion engine recognizes that the user's stress level is high. The server passes this information to the generation AI, which performs a simulation. The generation AI generates specific suggestions, such as "leave at 7:45, take a different route, and do some stretching to relax before leaving." The results are sent to the device and the user is notified. The user accepts the suggestions, avoiding the congestion and reducing stress, resulting in a smoother and more secure commute than usual.

[0896] In this way, the present invention is a system that integrates user behavioral data, situational information, and emotional data, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[0897] The processing flow will be explained below.

[0898] Step 1:

[0899] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[0900] Step 2:

[0901] The device uses an emotion engine to collect user emotion data. This includes facial expression recognition and voice analysis using the smartphone's camera and microphone. For example, if the user is feeling stressed, the emotion engine will recognize that state and record it as data.

[0902] Step 3:

[0903] The device encrypts the collected behavioral and emotional data and sends it to the server at regular intervals. For example, by setting the device to send data every hour, the latest user behavioral and emotional data can be stored on the server.

[0904] Step 4:

[0905] The server receives the user's behavioral and emotional data and stores it in a database. This information is organized and managed for each user. For example, the behavioral and emotional data is stored using the user ID as a key.

[0906] Step 5:

[0907] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[0908] Step 6:

[0909] The server stores the collected situation information in a database and prepares it for integration with behavioral and emotional data. The situation information is normalized and formatted into a unified format, allowing for smooth integration with behavioral and emotional data.

[0910] Step 7:

[0911] The server combines behavioral data, emotional data, and situational information and passes it to the generative AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[0912] Step 8:

[0913] The generative AI calculates the simulation results and generates multiple scenarios. This includes optimal action suggestions that take into account weather, traffic conditions, and the user's emotional state. For example, if the user is feeling highly stressed, the AI ​​will suggest leaving a little earlier and doing some stretching to relax.

[0914] Step 9:

[0915] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[0916] Step 10:

[0917] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[0918] Step 11:

[0919] The device then notifies the user of the received simulation results. For example, the app's notification function might display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[0920] Step 12:

[0921] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[0922] Step 13:

[0923] The device then sends the data on the user's accepted suggestions back to the server, which uses it for the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[0924] Example 2

[0925] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] In modern society, users' daily lives are hectic, and they need to quickly process a large amount of information to determine appropriate guidelines for action. However, conventional systems have difficulty comprehensively analyzing user behavioral data and situational information to simulate future situations. Furthermore, they lack the ability to adjust simulations to take into account the user's emotional state, making it impossible to provide users with more appropriate guidelines for action. This can lead to unnecessary stress and difficulties.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0928] In this invention, the server includes means for collecting user behavioral data and emotional data, means for collecting situation information from external sources, generation AI means for integrating the behavioral data, emotional data, and situation information to perform a future simulation, means for transmitting the simulation results to the user's terminal, and means for displaying the simulation results to the user on the terminal. This makes it possible to comprehensively analyze the user's behavioral data and situation information and perform a future simulation that takes the user's emotional state into consideration, thereby providing the user with more appropriate and useful guidelines for action.

[0929] "Behavioral data" refers to data including the user's daily schedule information, location information, health data, etc., and indicates the user's behavioral patterns.

[0930] "Emotional data" is data obtained by analyzing a user's facial expressions and tone of voice, and indicates the user's emotional state.

[0931] "Situation information" refers to real-time information collected from external sources such as news, weather, and traffic information, and indicates changes in the external environment.

[0932] "Generative AI methods" are methods that use AI models to analyze collected data and simulate future situations.

[0933] "Server" means a computer system that receives, stores, and analyzes data and operates the Generating AI Means.

[0934] A "terminal" is a device used by a user, such as a smartphone or tablet, that collects data and displays simulation results via communication with the server.

[0935] "Simulation results" are the predicted results of future situations and courses of action analyzed by the generative AI means.

[0936] This invention relates to a system that integrates user behavioral data, emotional data, and situational information collected from external sources, and uses a generative AI model to simulate the future. The system is mainly composed of a server and a terminal. Specific embodiments of the system are described below.

[0937] Behavioral data collection

[0938] Users use devices such as smartphones or tablets to provide their daily schedules, location information, and health data. This behavioral data is recorded in real time via an application on the device. For example, if a user inputs "I leave home at 8:00 every morning," this information is recorded on the device. The device's emotion engine also uses the smartphone's camera and voice input to analyze the user's facial expressions and tone of voice to collect emotional data.

[0939] Sending and storing data on the server

[0940] The collected behavioral and emotional data is sent from the device to a server at regular intervals. The server encrypts the data before receiving it and stores it securely in a database. Database management systems such as MySQL and PostgreSQL are used.

[0941] Gathering situational information

[0942] The server collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This situation information includes weather forecasts, traffic congestion information, and emergency alerts. This information is stored on the server and managed in conjunction with behavioral and emotional data.

[0943] Data integration and analysis

[0944] The server integrates behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). This generative AI model performs future simulations based on the integrated data. For example, based on behavioral data such as a user's decision to leave at 8:00, emotional data such as high stress levels, and situational information such as heavy rain and traffic congestion in the weather forecast, the model can suggest the optimal departure time, route, and recommended activities.

[0945] Sending and viewing simulation results

[0946] The simulation results are sent from the server to the device. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[0947] Gathering feedback and learning

[0948] Feedback on whether the user accepted the suggestions is also collected and used for the next prediction. The server continuously trains the generative AI model based on this feedback data, enabling more accurate predictions and suggestions.

[0949] Prompt Sentence Examples

[0950] Let's say a user leaves home at 8:00 every morning to commute to work. The weather forecast for that day predicts heavy rain, and traffic information indicates congestion on major commuter routes. The emotion engine also recognizes that the user's stress level is high. Based on this information, the generative AI model generates the following simulation results:

[0951] For example, specific suggestions such as "Leave at 7:45, take an alternative route, and do some relaxing stretches before leaving" are generated and notified to the user.

[0952] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0953] Step 1:

[0954] Users use a smartphone or tablet application to input their daily schedule, health data, and location information. The input data is recorded in real time by the application on the device. For example, a user might input a schedule such as "Leave home at 8:00 every morning."

[0955] Input: User schedule information, health data, location information

[0956] Output: Behavioral data recorded in the device

[0957] Step 2:

[0958] The device automatically records the user's real-time location using the built-in GPS sensor. It also uses the device's camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. For example, if the user is smiling while speaking, it is considered to have a "positive emotion."

[0959] Input: Location information from GPS sensor, emotion analysis data from camera and microphone

[0960] Output: Location information and emotion data recorded on the device

[0961] Step 3:

[0962] The device encrypts and transmits the collected behavioral and emotional data to the server at regular intervals, for example, every hour. The transmitted data is then encrypted and received by the server.

[0963] Input: Collected behavioral and emotional data

[0964] Output: Encrypted data is sent to and received from the server

[0965] Step 4:

[0966] The server decrypts the received encrypted data and stores it in a database using a relational database management system such as MySQL or PostgreSQL.

[0967] Input: Encrypted data sent to the server

[0968] Output: Behavioral and emotional data stored in a database

[0969] Step 5:

[0970] The server periodically collects situation information from external sources such as news APIs, weather APIs, and traffic information APIs. The collected information is analyzed within the server and stored in a database. For example, an API request is sent every morning at 6:00.

[0971] Input: Situation information obtained from the news API, weather API, and traffic information API

[0972] Output: Situation information stored in the database

[0973] Step 6:

[0974] The server combines behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). A prompt is generated based on the combined data. For example, the prompt might read, "The user's departure time is 8:00, the emotional data shows a high stress level, and the traffic information shows congestion."

[0975] Input: Integrated behavioral data, emotional data, and situational information

[0976] Output: A prompt to be input to the generative AI model

[0977] Step 7:

[0978] The generative AI model simulates the future based on the prompt text. For example, it simulates the optimal departure time, route, and recommended activities (such as stretching). The generated simulation results are output to the server.

[0979] Input: Prompt sentence for generative AI model

[0980] Output: Simulation results

[0981] Step 8:

[0982] The server sends the generated simulation results to the terminal in real time.

[0983] Input: Generated simulation results

[0984] Output: Simulation results sent to the terminal

[0985] Step 9:

[0986] The device application displays the simulation results in an easy-to-understand format for the user. Specifically, it suggests optimal courses of action through the app's notification function and widgets. For example, it might say, "Tomorrow's best commute time is 7:45, and the suggested route is ____. We also recommend doing a little stretching before you leave."

[0987] Input: Simulation results sent to the terminal

[0988] Output: The course of action displayed to the user

[0989] Step 10:

[0990] The device collects feedback on whether the user accepted the proposed offer. The device stores this feedback data and transmits it to the server at regular intervals.

[0991] Input: User feedback

[0992] Output: Feedback data is stored on the device and sent to the server.

[0993] Step 11:

[0994] The server continuously trains the generative AI model based on the collected feedback data, allowing the system to make more accurate predictions and suggestions.

[0995] Input: Feedback data

[0996] Output: An improved generative AI model

[0997] (Application example 2)

[0998] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0999] In today's world, it is important to improve the convenience and safety of autonomous vehicles. However, current autonomous vehicle systems are inadequate at proposing optimal driving plans that take into account the individual behavioral patterns and emotional state of the user. As a result, it is difficult to reduce user stress and provide a comfortable driving experience. Another challenge is generating driving plans that appropriately reflect changing situational information in real time.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1001] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's terminal, means for displaying the simulation results to the user on the terminal, means for collecting user emotion data, means for adjusting the simulation results using the emotion data, and means for making suggestions to optimize a driving plan for an autonomous vehicle. This makes it possible to generate an optimal driving plan that reflects the user's emotional state and real-time situational information.

[1002] "Means for collecting user behavioral data" refers to devices and applications that collect users' daily schedule information, location information, health data, etc.

[1003] "Means for collecting situational information from external sources" refers to APIs or interfaces for collecting real-time data from external sources such as news, weather, and traffic information.

[1004] The "generative AI means for integrating the behavioral data and situational information to perform future simulations" is a generative AI model for performing future simulations based on user behavioral data and external situational information.

[1005] The "means for transmitting the simulation results to the user's terminal" refers to a communication means for transmitting the simulation results in real time to the user's terminal such as a smartphone or tablet.

[1006] The "means for displaying the simulation results to the user on the terminal" refers to a user interface for displaying the simulation results on the user's terminal in an easy-to-understand manner.

[1007] The "means for collecting user emotional data" refers to a system that uses a smartphone's camera and microphone to analyze the user's emotional state from their facial expressions and tone of voice, and collects this data.

[1008] The "means for adjusting the simulation results using said emotional data" is an algorithm for dynamically adjusting the simulation results taking into account the collected emotional data.

[1009] The "means for making proposals to optimize driving plans for autonomous vehicles" is a system that provides specific proposals for optimizing the driving time and route of autonomous vehicles based on the generated simulation results.

[1010] This invention is a system that collects user behavior data, emotion data, and situation information and performs future simulations using a generative AI model to optimize driving plans for autonomous vehicles. The following describes in detail the embodiments of the invention.

[1011] Overall system flow

[1012] The server collects user behavioral data (schedule information, location information, health data) and emotional data. It also obtains real-time situational information (news information, weather information, traffic information) from external sources. This data is integrated and a generative AI model is used to perform a simulation. The server then sends the simulation results to the user's device and displays an optimized driving plan on the device.

[1013] Hardware and software used

[1014] Hardware:

[1015] Smartphone (camera, microphone, GPS, internet connection)

[1016] Server (database, generative AI model)

[1017] software:

[1018] EmotionRecognizer (emotion recognition engine)

[1019] API (weather and traffic information acquisition)

[1020] Generative AI Models

[1021] Data collection and integration

[1022] The device collects the user's schedule, location, and health data in real time. Emotion data is captured using the smartphone's camera and microphone, and analyzed by EmotionRecognizer. This data is then sent to a server at regular intervals.

[1023] The server obtains situation information from external sources using weather APIs, traffic information APIs, etc. All data is stored and integrated in a database within the server.

[1024] Simulation using generative AI models

[1025] Based on the integrated data, the server uses a generative AI model to simulate future driving plans. This simulation also takes into account the user's emotional data and dynamically adjusts suggestions to reduce stress. For example, if the user is in a high stress state, the plan can suggest an earlier departure or incorporate breaks for relaxation.

[1026] Presenting simulation results to users

[1027] The generated simulation results are sent from the server to the device. The device displays the results in an easy-to-understand format for the user. Specifically, the app provides users with guidelines for their actions through notifications and widgets. For example, it provides information such as, "The best time to leave tomorrow is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[1028] Examples of concrete examples and prompts

[1029] For example, consider a situation where a user plans to leave at 8:00, but heavy rain is forecast and traffic congestion is occurring along a major commute route. Furthermore, analysis of the user's facial expression using a smartphone camera reveals high stress levels. The system suggests leaving at 7:45, taking a different route, and doing some stretching to relax before leaving.

[1030] Example prompt sentence:

[1031] Use the user's schedule, location, health data, emotional data, and real-time weather and traffic information to simulate optimal driving plans and generate suggestions to reduce stress.

[1032] As described above, by implementing the present invention, the user can obtain an optimal driving plan that reflects the user's emotional state and real-time situation information, enabling comfortable and efficient driving.

[1033] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1034] Step 1:

[1035] The device collects user behavior data. The input is schedule information, location information, and health data, and the output is a collection of these data. The device acquires this data and periodically collects it in the background.

[1036] Step 2:

[1037] The device uses a camera and microphone to collect user emotion data. The input is camera video and audio data, and the output is emotion data (e.g., stress level). The device analyzes the emotion data using EmotionRecognizer.

[1038] Step 3:

[1039] The device sends the collected behavioral and emotional data to the server. The input is behavioral and emotional data, and the output is a transmission packet containing those data. The data is periodically uploaded to the server.

[1040] Step 4:

[1041] The server uses the weather API and traffic information API to collect real-time situation information. The input is the URL of the weather API and traffic information API, and the output is weather data and traffic data. The server sends requests to obtain various situation information.

[1042] Step 5:

[1043] The server integrates the collected behavioral data, emotional data, and situational information. The inputs are behavioral data, emotional data, weather data, and traffic data, and the output is an integrated dataset. The server stores these data in a database and integrates them into a single dataset.

[1044] Step 6:

[1045] The server passes the integrated dataset to the generative AI model to simulate future driving plans. The input is the integrated dataset, and the output is the driving simulation results. The generative AI model analyzes the data and generates an optimal driving plan.

[1046] Step 7:

[1047] The server sends the generated driving simulation results to the user's terminal. The input is the driving simulation results and the output is the transmission packets. The server sends the simulation results in packet format to the terminal.

[1048] Step 8:

[1049] The device displays the received driving simulation results to the user. The input is the driving simulation results, and the output is the display information on the user interface. The device presents the results to the user using notifications and widgets.

[1050] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1051] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1052] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1053] [Fourth embodiment]

[1054] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1055] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1056] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1057] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1058] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1059] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1060] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1061] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1062] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1063] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1064] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1065] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1066] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1067] This invention relates to a system that uses user behavior data and situation information to perform future simulations and provides the results to users.

[1068] First, users provide their daily schedule, location, and health data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will record this information.

[1069] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral data.

[1070] The server then combines the behavioral data and situational information and passes it to the generation AI. The generation AI analyzes this data and performs future simulations. For example, if a user plans to leave home at 8:00, it will consider weather forecasts and traffic congestion information to simulate the optimal departure time and route. The generation AI generates multiple scenarios and selects the most appropriate suggestion from among them.

[1071] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX."

[1072] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[1073] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commuter route. The server passes this information to the generation AI, which performs a simulation. The generation AI generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." This result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[1074] In this way, the present invention is a system that integrates user behavioral data and situational information, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[1075] The processing flow will be explained below.

[1076] Step 1:

[1077] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[1078] Step 2:

[1079] The device encrypts the collected behavioral data and sends it to the server at regular intervals. For example, by setting it to send data every hour, the latest user behavioral data is always stored on the server.

[1080] Step 3:

[1081] The server receives the user's behavioral data and stores it in a database. This information is organized and managed for each user. For example, behavioral data is stored using the user ID as a key.

[1082] Step 4:

[1083] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[1084] Step 5:

[1085] The server stores the collected situation information in a database and prepares it for integration with behavioral data. The situation information is normalized and formatted in a unified format, allowing for smooth integration with behavioral data.

[1086] Step 6:

[1087] The server combines the behavioral data and situational information and passes it to the generation AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[1088] Step 7:

[1089] The generative AI calculates the simulation results and generates multiple scenarios, including optimal action suggestions that take into account factors such as weather and traffic conditions. For example, it creates specific suggestions such as "leave at 7:45 and use a different route."

[1090] Step 8:

[1091] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[1092] Step 9:

[1093] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[1094] Step 10:

[1095] The device notifies the user of the received simulation results. For example, the app's notification function may display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX."

[1096] Step 11:

[1097] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[1098] Step 12:

[1099] The device then sends the data on the user's accepted suggestions back to the server, which is then used for the next prediction. This allows the system to continuously learn the user's behavioral patterns and make more accurate suggestions.

[1100] Example 1

[1101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1102] In modern society, users seek optimal guidelines for action to respond to complex schedules and changing situations. However, conventional systems have difficulty effectively integrating user behavior data with real-time situation information and providing appropriate suggestions through future simulations. Furthermore, there is a lack of means to continuously improve the accuracy of the system using user feedback. As a result, users face challenges in making quick and appropriate decisions in their daily lives and work.

[1103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1104] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's information processing device, means for displaying the simulation results to the user on the information processing device, and means for collecting feedback information from the user and reflecting it in the next prediction. This enables the user to receive guidelines for action optimized for their individual situation and make efficient and effective decisions in their daily lives and work.

[1105] "User behavioral data" refers to information related to a user's daily activities and habits, including location information, schedule information, and health data.

[1106] "Situation information" refers to real-time information collected from external sources, including news information, weather information, traffic information, etc.

[1107] "Generative AI means" refers to artificial intelligence technology that analyzes user behavioral data and situational information to simulate the future.

[1108] "Simulation Results" refers to future predictions and proposals derived based on data analyzed by the Generative AI Means.

[1109] "Information processing device" refers to a device used by a user, such as a smartphone or tablet, that displays the simulation results.

[1110] "Feedback information" refers to the actions taken by the user based on the simulation results and the evaluation of those actions.

[1111] This invention relates to a system that provides users with appropriate guidelines for action by integrating user behavioral data and situational information and simulating future situations using generative AI.

[1112] First, users use devices such as smartphones or tablets to record their daily schedules, location information, health data, and other information. This behavioral data is recorded and collected in real time via an application on the device. For example, if a user enters a schedule such as "Leave home at 8:00 every morning," the device will automatically record this information.

[1113] The collected behavioral data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a secure database (e.g., Amazon RDS). The data is sent using a secure communication protocol (e.g., HTTPS). The server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes news, weather, and traffic information. This situation information is also stored on the server and linked to the behavioral data.

[1114] The server then combines the behavioral data and situational information and passes it to the generative AI. The generative AI then analyzes this data and runs a future simulation. Specifically, if a user leaves home at 8:00, it will consider weather forecasts and traffic congestion information and simulate the optimal departure time and route. The generative AI model used is GPT-4 or an equally advanced generative AI model.

[1115] The simulation results are then sent back to the device from the server. The device then displays the results in a format that is easy for the user to understand. For example, information such as "Tomorrow's optimal commute time is 7:45, and the suggested route is XX" can be presented via the app's notification function or widget.

[1116] Furthermore, if the user adopts the suggestion, the feedback is collected again and used for the next prediction. This feedback information records whether the user acted as suggested and is sent to the server. This allows the system to continuously learn the user's behavioral patterns and improve the prediction accuracy of the generative AI.

[1117] As a concrete example, suppose a user leaves home at 8:00 every morning and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the generation AI, which runs a simulation. The generation AI then generates a suggestion that it would be better to leave at 7:45 and use a different route. This result is sent to the device and notifies the user. The user accepts this suggestion, allowing for a smoother commute than usual.

[1118] As described above, the present invention is a system that provides a course of action optimized for the individual situation of a user, enabling efficient and effective decision-making in daily life and work.

[1119] An example of a prompt sentence is, "A user who leaves home to commute to work at 8:00 every morning would like the system to simulate and suggest the optimal departure time and route to avoid heavy rain and traffic congestion."

[1120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1121] Step 1:

[1122] The device records behavioral data from the user.

[1123] Input: Schedule information, location information, and health data that users enter into their devices.

[1124] How it works: An application in the device collects this data and stores it in the internal storage. For example, a user enters, "I leave home every morning at 8:00."

[1125] Output: Behavioral data stored in the device's storage.

[1126] Step 2:

[1127] The terminal transmits the collected behavioral data to the server at regular intervals.

[1128] Input: Behavioral data stored on the device.

[1129] How it works: Every 5 minutes, data is encrypted and sent to a server via a secure communication protocol (e.g. HTTPS).

[1130] Output: Encrypted behavioral data sent to the server.

[1131] Step 3:

[1132] The server decrypts the received data and stores it in a database.

[1133] Input: Encrypted behavioral data sent to the server.

[1134] What happens: The server decrypts the data and executes a SQL query to insert it into a secure database (e.g. Amazon RDS).

[1135] Output: Behavioral data stored in a database.

[1136] Step 4:

[1137] The server collects situational information from external sources.

[1138] Input: Information requests from external sources (News API, Weather API, Traffic API).

[1139] How it works: The server periodically sends requests to external sources, receives the response data, and stores it in a database. For example, "latest news updates from a news API."

[1140] Output: Situation information stored in a database.

[1141] Step 5:

[1142] The server integrates the behavioral data and situational information.

[1143] Input: Behavioral data and situational information obtained from databases.

[1144] How it works: The required data is retrieved from the database using SQL queries and formatted to be passed to the generative AI model.

[1145] Output: The aggregated data that is passed to the generative AI model.

[1146] Step 6:

[1147] The server inputs the integrated data into a generative AI model and runs a simulation of the future.

[1148] Input: Integrated behavioral data and situational information.

[1149] How it works: A generative AI model (e.g., GPT-4) analyzes input data, generates multiple scenarios, and selects the optimal course of action. For example, it generates a prediction result such as, "It would be best to leave tomorrow at 7:45 and take X street."

[1150] Output: Action suggestions as a result of the simulation.

[1151] Step 7:

[1152] The server transmits the simulation results to the terminal.

[1153] Input: Simulation results generated by the generative AI model.

[1154] Operation: The server sends the simulation results to the terminal and converts them into a notification format. For example, "Create a notification message saying 'Tomorrow's best commute time is 7:45, and the suggested route is XX street.'"

[1155] Output: Simulation results sent to the terminal.

[1156] Step 8:

[1157] The terminal notifies the user of the simulation results.

[1158] Input: Simulation results received from the server.

[1159] How it works: Uses the smartphone's notification function to display the simulation results to the user.

[1160] Output: A suggested action notified to the user.

[1161] Step 9:

[1162] The device collects user feedback.

[1163] Input: User feedback input.

[1164] Behavior: Record whether the user followed the suggestion and their rating, and send it back to the server.

[1165] Output: Feedback information sent to the server.

[1166] Step 10:

[1167] The server stores user feedback in a database and uses it for future predictions.

[1168] Input: User feedback information.

[1169] Operation: The feedback information is stored in a database and used as reference data for the next simulation.

[1170] Output: Feedback information stored in a database. This feedback information is used to improve the accuracy of the next simulation.

[1171] (Application example 1)

[1172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] In order to respond to various situations that users encounter in their daily lives (such as weather changes, traffic congestion, and emergencies), it is important to obtain accurate information in real time and determine appropriate guidelines for action. However, current systems lack sufficient means to comprehensively analyze this information and provide users with optimal guidelines for action. In particular, for autonomous vehicles, it remains difficult to propose optimal routes and departure times in real time. This increases the burden on users and hinders efficient travel and improvements in quality of life.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1175] In this invention, the server includes a means for collecting user behavioral data, a means for collecting situational information from external sources, a generating AI means for integrating the behavioral data and situational information to perform future simulations, a means for transmitting the simulation results to the user's terminal, a means for displaying the simulation results to the user on the terminal, and a means for collecting user feedback and reflecting it in the next prediction. This allows the user to obtain optimal guidelines for action in real time, specifically, to receive suggestions for the optimal route and departure time for an autonomous vehicle. This realizes efficient and smooth travel and improves the quality of life of the user.

[1176] "User behavioral data" refers to information about the behavior of users in their daily lives, and specifically includes location information, schedule information, and health data.

[1177] "Means for collecting situation information from external sources" refers to a system for obtaining information in real time from external data sources such as weather forecasts, traffic information, and news information.

[1178] "Generative AI means" refers to an artificial intelligence system that integrates user behavior data and situational information to simulate the future. Specifically, it uses a generative AI model.

[1179] "Means for transmitting simulation results to the user's device" refers to a mechanism for transmitting the results of future simulations performed on the server to the device used by the user (e.g., smartphone, smart display).

[1180] "Means for displaying simulation results to the user on the device" refers to a function that presents the simulation results visually and in an easy-to-understand manner on the user's device. Specifically, this is done through notification functions and interfaces.

[1181] "Means for collecting user feedback and reflecting it in the next prediction" refers to a mechanism for collecting information on whether the user accepted the proposal and using that information in the next simulation.

[1182] This invention provides a system that integrates user behavior data with situational information from external sources and performs future simulations using generative AI. Specific embodiments for carrying out the invention are described below.

[1183] System Configuration

[1184] The system consists of the following main components:

[1185] 1. How we collect user behavior data

[1186] 2. Gathering situational information from external sources

[1187] 3. Generation AI means

[1188] 4. Means of sending simulation results to the user's device

[1189] 5. A means of displaying the simulation results to the user on the device

[1190] 6. A way to gather user feedback and incorporate it into the next forecast.

[1191] Hardware and Software

[1192] Hardware

[1193] User devices: smartphones, smart displays, etc.

[1194] Server: A server with high-performance computing power

[1195] Communication network: Internet communication network

[1196] software

[1197] Generative AI model: OpenAI's GPT-4

[1198] API: Weather forecast API, traffic information API, news information API

[1199] Database management system: PostgreSQL or MySQL

[1200] Processing Overview

[1201] 1. Data Collection

[1202] Users provide location information, schedule information, and health data via their smartphones or in-car smart displays, and this data is recorded and collected in real time via the device's application.

[1203] 2. Gathering information on the situation

[1204] It collects weather, traffic, and news information in real time from external sources using the Weather API, Traffic API, and News API.

[1205] 3. Simulation using generative AI

[1206] The collected behavioral data and situational information are sent to a server and passed to the generative AI (GPT-4). The generative AI analyzes this data and performs future simulations. For example, given a user's schedule of "leaving home at 8:00 every morning," it integrates weather forecasts and traffic congestion information to suggest the optimal departure time and route.

[1207] Specific prompt examples:

[1208] User schedule information:

[1209] I leave home at 8:00 every morning and go to work.

[1210] Weather Information:

[1211] Heavy rain is forecast for tomorrow.

[1212] Traffic information:

[1213] There are traffic jams on major commuter routes.

[1214] Use this information to suggest the best time and route for the user to depart.

[1215] 4. Sending and displaying simulation results

[1216] The simulation results are sent from the server to the user's device, which then displays the results in a visually easy-to-understand manner, providing a notification function such as, "Tomorrow's best departure time is 7:45, and the proposed route is XX."

[1217] 5. Collecting user feedback

[1218] Feedback is collected from the user on whether they accepted the suggestions and is reflected in the next simulation. This feedback is used to continuously improve the accuracy of the system.

[1219] Specific examples

[1220] Suppose a user leaves home every morning at 8:00 and commutes to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. The server passes this information to the AI ​​generator, which runs a simulation. The AI ​​generator generates a suggestion, such as "It would be better to leave at 7:45 and use a different route." The result is sent to the device and notified to the user. The user accepts this suggestion, avoiding the congestion and enjoying a smoother commute than usual.

[1221] The above is an embodiment of the present invention. This system continuously learns the behavioral patterns of the user and is able to make more accurate suggestions.

[1222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1223] Step 1:

[1224] Users use their smartphones or in-car smart displays to input their location information, schedule information, and health data. This data is recorded and collected in real time by an application on the device. The collected data is encrypted with the user's consent and securely sent to a server. A specific example of input data is schedule information such as "Leave home at 8:00 every morning." The output is encrypted behavioral data.

[1225] Step 2:

[1226] The server uses weather forecast APIs, traffic information APIs, and news information APIs to collect real-time situation information. This includes weather forecasts, traffic congestion information, and emergency alerts. The server also encrypts this data and stores it in a database. Specific examples of input data include "tomorrow's weather forecast is heavy rain" and "traffic congestion on major transportation routes." The output is encrypted situation information.

[1227] Step 3:

[1228] The server integrates the collected user behavioral data and situational information and inputs it into a generative AI model (for example, OpenAI's GPT-4). At this time, an appropriate prompt sentence is passed to the generative AI model. An example of a specific prompt sentence is, "User's schedule information: Leave home at 8:00 every morning and commute to work. Weather forecast information: Heavy rain is predicted for tomorrow. Traffic information: There is traffic congestion on the main commuting route. Based on this information, please suggest the optimal time and route for the user to depart." The input data is the integrated behavioral data and situational information, and the output is a simulation result.

[1229] Step 4:

[1230] The simulation results generated by the generative AI model are sent by the server to the user's device. The device receives these results and displays them in a visually easy-to-understand format. Specifically, the user is notified via a notification function or interface in the form of, "The best departure time tomorrow is 7:45, and the proposed route is XX." The input data are the simulation results from the generative AI, and the output is a notification sent to the user's device.

[1231] Step 5:

[1232] The user decides what to do based on the notification from the device. If the user accepts the suggestion, the feedback is sent from the device to the server. The server stores this feedback in a database and uses it in the next simulation. The input data is the user's feedback, and the output is updated learning data. For example, the user may provide feedback such as "I left at 7:45 as suggested and took a different route."

[1233] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1234] The present invention relates to a system that performs a future simulation based on user behavior data and situation information, and adjusts the simulation results by recognizing the user's emotions.

[1235] First, users provide their daily schedule, location, health, and emotional data using devices such as smartphones or tablets. This behavioral data is recorded and collected in real time through an application on the device. For example, if a user inputs, "I leave home at 8:00 every morning," this information is recorded. The emotion engine also uses the smartphone's camera and voice input to collect emotional data from the user's facial expressions and tone of voice.

[1236] The collected behavioral and emotional data is sent to a server at regular intervals. The server receives the encrypted data and stores it in a database. At the same time, the server also collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This information includes weather forecasts, traffic congestion information, and emergency alerts. This information is also stored on the server and linked to the behavioral and emotional data.

[1237] Next, the server combines the behavioral data, emotional data, and situational information and passes it to the generative AI. The generative AI analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00. Furthermore, the generative AI dynamically adjusts the simulation results based on the user's emotional data. For example, if the user is feeling stressed, it might suggest leaving a little earlier and taking some time to relax.

[1238] The simulation results are then sent back to the device from the server. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[1239] The user decides their own actions based on this feedback. If the user accepts the suggestion, that feedback is also collected and used to make the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[1240] As a concrete example, suppose a user leaves home every morning at 8:00 to commute to work. One day, the weather forecast predicts heavy rain, and traffic information indicates congestion on a major commute route. Furthermore, the emotion engine recognizes that the user's stress level is high. The server passes this information to the generation AI, which performs a simulation. The generation AI generates specific suggestions, such as "leave at 7:45, take a different route, and do some stretching to relax before leaving." The results are sent to the device and the user is notified. The user accepts the suggestions, avoiding the congestion and reducing stress, resulting in a smoother and more secure commute than usual.

[1241] In this way, the present invention is a system that integrates user behavioral data, situational information, and emotional data, and uses generative AI to simulate future situations, thereby providing users with appropriate guidelines for action and supporting a safe and efficient lifestyle.

[1242] The processing flow will be explained below.

[1243] Step 1:

[1244] The device collects user behavior data, including location information using the GPS sensor, schedule information from the calendar app, and health data from wearable devices. For example, if a user inputs "I leave home at 8:00 every morning," this information will be recorded.

[1245] Step 2:

[1246] The device uses an emotion engine to collect user emotion data. This includes facial expression recognition and voice analysis using the smartphone's camera and microphone. For example, if the user is feeling stressed, the emotion engine will recognize that state and record it as data.

[1247] Step 3:

[1248] The device encrypts the collected behavioral and emotional data and sends it to the server at regular intervals. For example, by setting the device to send data every hour, the latest user behavioral and emotional data can be stored on the server.

[1249] Step 4:

[1250] The server receives the user's behavioral and emotional data and stores it in a database. This information is organized and managed for each user. For example, the behavioral and emotional data is stored using the user ID as a key.

[1251] Step 5:

[1252] The server sends API requests to gather situational information from external sources, including news, weather, and traffic APIs, to retrieve real-time data, such as the weather API to get the next day's weather forecast and the traffic API to get the latest traffic congestion information.

[1253] Step 6:

[1254] The server stores the collected situation information in a database and prepares it for integration with behavioral and emotional data. The situation information is normalized and formatted into a unified format, allowing for smooth integration with behavioral and emotional data.

[1255] Step 7:

[1256] The server combines behavioral data, emotional data, and situational information and passes it to the generative AI, which analyzes this data and performs future simulations. For example, it predicts the commute time and optimal route if the user leaves at 8:00.

[1257] Step 8:

[1258] The generative AI calculates the simulation results and generates multiple scenarios. This includes optimal action suggestions that take into account weather, traffic conditions, and the user's emotional state. For example, if the user is feeling highly stressed, the AI ​​will suggest leaving a little earlier and doing some stretching to relax.

[1259] Step 9:

[1260] The server then formats the simulation results received from the generative AI into a user-friendly format, including graphs, maps, and text explanations. For example, it could display the optimal route on a map and explain the departure time in text.

[1261] Step 10:

[1262] The server encrypts the formatted simulation results and sends them to the device, thereby protecting the user's privacy.

[1263] Step 11:

[1264] The device then notifies the user of the received simulation results. For example, the app's notification function might display a message such as, "Tomorrow's best commute time is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[1265] Step 12:

[1266] The user then checks the notification and decides whether to adopt the proposed course of action. If the user adopts the suggestion, that decision is also recorded on the device.

[1267] Step 13:

[1268] The device then sends the data on the user's accepted suggestions back to the server, which uses it for the next prediction. This allows the system to continuously learn the user's behavioral patterns and emotional state, enabling it to make more accurate suggestions.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1271] In modern society, users' daily lives are hectic, and they need to quickly process a large amount of information to determine appropriate guidelines for action. However, conventional systems have difficulty comprehensively analyzing user behavioral data and situational information to simulate future situations. Furthermore, they lack the ability to adjust simulations to take into account the user's emotional state, making it impossible to provide users with more appropriate guidelines for action. This can lead to unnecessary stress and difficulties.

[1272] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1273] In this invention, the server includes means for collecting user behavioral data and emotional data, means for collecting situation information from external sources, generation AI means for integrating the behavioral data, emotional data, and situation information to perform a future simulation, means for transmitting the simulation results to the user's terminal, and means for displaying the simulation results to the user on the terminal. This makes it possible to comprehensively analyze the user's behavioral data and situation information and perform a future simulation that takes the user's emotional state into consideration, thereby providing the user with more appropriate and useful guidelines for action.

[1274] "Behavioral data" refers to data including the user's daily schedule information, location information, health data, etc., and indicates the user's behavioral patterns.

[1275] "Emotional data" is data obtained by analyzing a user's facial expressions and tone of voice, and indicates the user's emotional state.

[1276] "Situation information" refers to real-time information collected from external sources such as news, weather, and traffic information, and indicates changes in the external environment.

[1277] "Generative AI methods" are methods that use AI models to analyze collected data and simulate future situations.

[1278] "Server" means a computer system that receives, stores, and analyzes data and operates the Generating AI Means.

[1279] A "terminal" is a device used by a user, such as a smartphone or tablet, that collects data and displays simulation results via communication with the server.

[1280] "Simulation results" are the predicted results of future situations and courses of action analyzed by the generative AI means.

[1281] This invention relates to a system that integrates user behavioral data, emotional data, and situational information collected from external sources, and uses a generative AI model to simulate the future. The system is mainly composed of a server and a terminal. Specific embodiments of the system are described below.

[1282] Behavioral data collection

[1283] Users use devices such as smartphones or tablets to provide their daily schedules, location information, and health data. This behavioral data is recorded in real time via an application on the device. For example, if a user inputs "I leave home at 8:00 every morning," this information is recorded on the device. The device's emotion engine also uses the smartphone's camera and voice input to analyze the user's facial expressions and tone of voice to collect emotional data.

[1284] Sending and storing data on the server

[1285] The collected behavioral and emotional data is sent from the device to a server at regular intervals. The server encrypts the data before receiving it and stores it securely in a database. Database management systems such as MySQL and PostgreSQL are used.

[1286] Gathering situational information

[1287] The server collects real-time situation information from external sources such as news APIs, weather APIs, and traffic information APIs. This situation information includes weather forecasts, traffic congestion information, and emergency alerts. This information is stored on the server and managed in conjunction with behavioral and emotional data.

[1288] Data integration and analysis

[1289] The server integrates behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). This generative AI model performs future simulations based on the integrated data. For example, based on behavioral data such as a user's decision to leave at 8:00, emotional data such as high stress levels, and situational information such as heavy rain and traffic congestion in the weather forecast, the model can suggest the optimal departure time, route, and recommended activities.

[1290] Sending and viewing simulation results

[1291] The simulation results are sent from the server to the device. The device then displays the results in an easy-to-understand format for the user. Specifically, the optimal course of action is presented through the app's notification function and widgets. For example, information is provided in the form of, "Tomorrow's optimal commute time is 7:45, and the suggested route is XX. We also recommend doing a little stretching before you leave."

[1292] Gathering feedback and learning

[1293] Feedback on whether the user accepted the suggestions is also collected and used for the next prediction. The server continuously trains the generative AI model based on this feedback data, enabling more accurate predictions and suggestions.

[1294] Prompt Sentence Examples

[1295] Let's say a user leaves home at 8:00 every morning to commute to work. The weather forecast for that day predicts heavy rain, and traffic information indicates congestion on major commuter routes. The emotion engine also recognizes that the user's stress level is high. Based on this information, the generative AI model generates the following simulation results:

[1296] For example, specific suggestions such as "Leave at 7:45, take an alternative route, and do some relaxing stretches before leaving" are generated and notified to the user.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Step 1:

[1299] Users use a smartphone or tablet application to input their daily schedule, health data, and location information. The input data is recorded in real time by the application on the device. For example, a user might input a schedule such as "Leave home at 8:00 every morning."

[1300] Input: User schedule information, health data, location information

[1301] Output: Behavioral data recorded in the device

[1302] Step 2:

[1303] The device automatically records the user's real-time location using the built-in GPS sensor. It also uses the device's camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. For example, if the user is smiling while speaking, it is considered to have a "positive emotion."

[1304] Input: Location information from GPS sensor, emotion analysis data from camera and microphone

[1305] Output: Location information and emotion data recorded on the device

[1306] Step 3:

[1307] The device encrypts and transmits the collected behavioral and emotional data to the server at regular intervals, for example, every hour. The transmitted data is then encrypted and received by the server.

[1308] Input: Collected behavioral and emotional data

[1309] Output: Encrypted data is sent to and received from the server

[1310] Step 4:

[1311] The server decrypts the received encrypted data and stores it in a database using a relational database management system such as MySQL or PostgreSQL.

[1312] Input: Encrypted data sent to the server

[1313] Output: Behavioral and emotional data stored in a database

[1314] Step 5:

[1315] The server periodically collects situation information from external sources such as news APIs, weather APIs, and traffic information APIs. The collected information is analyzed within the server and stored in a database. For example, an API request is sent every morning at 6:00.

[1316] Input: Situation information obtained from the news API, weather API, and traffic information API

[1317] Output: Situation information stored in the database

[1318] Step 6:

[1319] The server combines behavioral data, emotional data, and situational information and passes it to a generative AI model (e.g., GPT-4). A prompt is generated based on the combined data. For example, the prompt might read, "The user's departure time is 8:00, the emotional data shows a high stress level, and the traffic information shows congestion."

[1320] Input: Integrated behavioral data, emotional data, and situational information

[1321] Output: A prompt to be input to the generative AI model

[1322] Step 7:

[1323] The generative AI model simulates the future based on the prompt text. For example, it simulates the optimal departure time, route, and recommended activities (such as stretching). The generated simulation results are output to the server.

[1324] Input: Prompt sentence for generative AI model

[1325] Output: Simulation results

[1326] Step 8:

[1327] The server sends the generated simulation results to the terminal in real time.

[1328] Input: Generated simulation results

[1329] Output: Simulation results sent to the terminal

[1330] Step 9:

[1331] The device application displays the simulation results in an easy-to-understand format for the user. Specifically, it suggests optimal courses of action through the app's notification function and widgets. For example, it might say, "Tomorrow's best commute time is 7:45, and the suggested route is ____. We also recommend doing a little stretching before you leave."

[1332] Input: Simulation results sent to the terminal

[1333] Output: The course of action displayed to the user

[1334] Step 10:

[1335] The device collects feedback on whether the user accepted the proposed offer. The device stores this feedback data and transmits it to the server at regular intervals.

[1336] Input: User feedback

[1337] Output: Feedback data is stored on the device and sent to the server.

[1338] Step 11:

[1339] The server continuously trains the generative AI model based on the collected feedback data, allowing the system to make more accurate predictions and suggestions.

[1340] Input: Feedback data

[1341] Output: An improved generative AI model

[1342] (Application example 2)

[1343] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1344] In today's world, it is important to improve the convenience and safety of autonomous vehicles. However, current autonomous vehicle systems are inadequate at proposing optimal driving plans that take into account the individual behavioral patterns and emotional state of the user. As a result, it is difficult to reduce user stress and provide a comfortable driving experience. Another challenge is generating driving plans that appropriately reflect changing situational information in real time.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1346] In this invention, the server includes means for collecting user behavioral data, means for collecting situational information from external sources, generation AI means for integrating the behavioral data and situational information to perform future simulations, means for transmitting the simulation results to the user's terminal, means for displaying the simulation results to the user on the terminal, means for collecting user emotion data, means for adjusting the simulation results using the emotion data, and means for making suggestions to optimize a driving plan for an autonomous vehicle. This makes it possible to generate an optimal driving plan that reflects the user's emotional state and real-time situational information.

[1347] "Means for collecting user behavioral data" refers to devices and applications that collect users' daily schedule information, location information, health data, etc.

[1348] "Means for collecting situational information from external sources" refers to APIs or interfaces for collecting real-time data from external sources such as news, weather, and traffic information.

[1349] The "generative AI means for integrating the behavioral data and situational information to perform future simulations" is a generative AI model for performing future simulations based on user behavioral data and external situational information.

[1350] The "means for transmitting the simulation results to the user's terminal" refers to a communication means for transmitting the simulation results in real time to the user's terminal such as a smartphone or tablet.

[1351] The "means for displaying the simulation results to the user on the terminal" refers to a user interface for displaying the simulation results on the user's terminal in an easy-to-understand manner.

[1352] The "means for collecting user emotional data" refers to a system that uses a smartphone's camera and microphone to analyze the user's emotional state from their facial expressions and tone of voice, and collects this data.

[1353] The "means for adjusting the simulation results using said emotional data" is an algorithm for dynamically adjusting the simulation results taking into account the collected emotional data.

[1354] The "means for making proposals to optimize driving plans for autonomous vehicles" is a system that provides specific proposals for optimizing the driving time and route of autonomous vehicles based on the generated simulation results.

[1355] This invention is a system that collects user behavior data, emotion data, and situation information and performs future simulations using a generative AI model to optimize driving plans for autonomous vehicles. The following describes in detail the embodiments of the invention.

[1356] Overall system flow

[1357] The server collects user behavioral data (schedule information, location information, health data) and emotional data. It also obtains real-time situational information (news information, weather information, traffic information) from external sources. This data is integrated and a generative AI model is used to perform a simulation. The server then sends the simulation results to the user's device and displays an optimized driving plan on the device.

[1358] Hardware and software used

[1359] Hardware:

[1360] Smartphone (camera, microphone, GPS, internet connection)

[1361] Server (database, generative AI model)

[1362] software:

[1363] EmotionRecognizer (emotion recognition engine)

[1364] API (weather and traffic information acquisition)

[1365] Generative AI Models

[1366] Data collection and integration

[1367] The device collects the user's schedule, location, and health data in real time. Emotion data is captured using the smartphone's camera and microphone, and analyzed by EmotionRecognizer. This data is then sent to a server at regular intervals.

[1368] The server obtains situation information from external sources using weather APIs, traffic information APIs, etc. All data is stored and integrated in a database within the server.

[1369] Simulation using generative AI models

[1370] Based on the integrated data, the server uses a generative AI model to simulate future driving plans. This simulation also takes into account the user's emotional data and dynamically adjusts suggestions to reduce stress. For example, if the user is in a high stress state, the plan can suggest an earlier departure or incorporate breaks for relaxation.

[1371] Presenting simulation results to users

[1372] The generated simulation results are sent from the server to the device. The device displays the results in an easy-to-understand format for the user. Specifically, the app provides users with guidelines for their actions through notifications and widgets. For example, it provides information such as, "The best time to leave tomorrow is 7:45, and the suggested route is XX. We also recommend doing some stretching before you leave."

[1373] Examples of concrete examples and prompts

[1374] For example, consider a situation where a user plans to leave at 8:00, but heavy rain is forecast and traffic congestion is occurring along a major commute route. Furthermore, analysis of the user's facial expression using a smartphone camera reveals high stress levels. The system suggests leaving at 7:45, taking a different route, and doing some stretching to relax before leaving.

[1375] Example prompt sentence:

[1376] Use the user's schedule, location, health data, emotional data, and real-time weather and traffic information to simulate optimal driving plans and generate suggestions to reduce stress.

[1377] As described above, by implementing the present invention, the user can obtain an optimal driving plan that reflects the user's emotional state and real-time situation information, enabling comfortable and efficient driving.

[1378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1379] Step 1:

[1380] The device collects user behavior data. The input is schedule information, location information, and health data, and the output is a collection of these data. The device acquires this data and periodically collects it in the background.

[1381] Step 2:

[1382] The device uses a camera and microphone to collect user emotion data. The input is camera video and audio data, and the output is emotion data (e.g., stress level). The device analyzes the emotion data using EmotionRecognizer.

[1383] Step 3:

[1384] The device sends the collected behavioral and emotional data to the server. The input is behavioral and emotional data, and the output is a transmission packet containing those data. The data is periodically uploaded to the server.

[1385] Step 4:

[1386] The server uses the weather API and traffic information API to collect real-time situation information. The input is the URL of the weather API and traffic information API, and the output is weather data and traffic data. The server sends requests to obtain various situation information.

[1387] Step 5:

[1388] The server integrates the collected behavioral data, emotional data, and situational information. The inputs are behavioral data, emotional data, weather data, and traffic data, and the output is an integrated dataset. The server stores these data in a database and integrates them into a single dataset.

[1389] Step 6:

[1390] The server passes the integrated dataset to the generative AI model to simulate future driving plans. The input is the integrated dataset, and the output is the driving simulation results. The generative AI model analyzes the data and generates an optimal driving plan.

[1391] Step 7:

[1392] The server sends the generated driving simulation results to the user's terminal. The input is the driving simulation results and the output is the transmission packets. The server sends the simulation results in packet format to the terminal.

[1393] Step 8:

[1394] The device displays the received driving simulation results to the user. The input is the driving simulation results, and the output is the display information on the user interface. The device presents the results to the user using notifications and widgets.

[1395] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1397] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1398] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1399] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1400] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1401] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1402] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1403] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1404] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1405] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1406] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1407] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1408] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1409] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1410] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1411] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1412] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1413] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1414] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1415] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1416] The following is further disclosed regarding the above embodiment.

[1417] (Claim 1)

[1418] means of collecting user behavior data;

[1419] means of gathering situational information from external sources; and

[1420] A generation AI means for integrating the behavioral data and situation information to perform a future simulation;

[1421] means for transmitting the simulation results to a user terminal;

[1422] means for displaying the simulation results to a user on said terminal;

[1423] A system including:

[1424] (Claim 2)

[1425] 2. The system of claim 1, wherein the behavioral data includes user location information, schedule information, and health data.

[1426] (Claim 3)

[1427] 2. The system of claim 1, wherein the situation information includes news information, weather information, and traffic information.

[1428] (Claim 4)

[1429] 2. The system of claim 1, wherein the generating AI means integrates the behavioral data and the situation information and uses an algorithm to perform future simulations.

[1430] (Claim 5)

[1431] 10. The system of claim 1, wherein the simulation results include suggestions for a course of action for the user.

[1432] "Example 1"

[1433] (Claim 1)

[1434] means of collecting user behavior data;

[1435] means of gathering situational information from external sources; and

[1436] A generation AI means for integrating the behavioral data and situation information to perform a future simulation;

[1437] means for transmitting the simulation results to a user's information processing device;

[1438] means for displaying a simulation result to a user in the information processing device;

[1439] A means of collecting feedback from users and reflecting it in the next prediction,

[1440] A system including:

[1441] (Claim 2)

[1442] 2. The system of claim 1, wherein the behavioral data includes the user's location information, schedule information, and health data.

[1443] (Claim 3)

[1444] 2. The system according to claim 1, wherein the situation information includes news information, weather information, and traffic information.

[1445] "Application Example 1"

[1446] (Claim 1)

[1447] means of collecting user behavior data;

[1448] means of gathering situational information from external sources; and

[1449] A generation AI means for integrating the behavioral data and situation information to perform a future simulation;

[1450] means for transmitting the simulation results to a user terminal;

[1451] means for displaying the simulation results to a user on said terminal;

[1452] A way to collect user feedback and incorporate it into the next prediction,

[1453] A system including:

[1454] (Claim 2)

[1455] 2. The system of claim 1, wherein the behavioral data includes user location information, schedule information, and health data.

[1456] (Claim 3)

[1457] 2. The system of claim 1, wherein the situation information includes news information, weather information, and traffic information.

[1458] "Example 2: Combining Emotion Engines"

[1459] (Claim 1)

[1460] a means for collecting user behavioral and emotional data;

[1461] means of gathering situational information from external sources; and

[1462] a generation AI means for integrating the behavioral data, the emotional data, and the situation information to perform a future simulation;

[1463] means for transmitting the simulation results to a user terminal;

[1464] means for displaying the simulation results to a user on said terminal;

[1465] A system including:

[1466] (Claim 2)

[1467] 2. The system of claim 1, wherein the behavioral data includes user location information, schedule information, and health data.

[1468] (Claim 3)

[1469] 2. The system of claim 1, wherein the situation information includes news information, weather information, and traffic information.

[1470] "Application example 2 when combining emotion engines"

[1471] (Claim 1)

[1472] means of collecting user behavior data;

[1473] means of gathering situational information from external sources; and

[1474] A generation AI means for integrating the behavioral data and situation information to perform a future simulation;

[1475] means for transmitting the simulation results to a user terminal;

[1476] means for displaying the simulation results to a user on said terminal;

[1477] a means for collecting user emotional data;

[1478] means for adjusting a simulation result using said emotion data;

[1479] A means for making suggestions to optimize a driving plan for an autonomous vehicle;

[1480] A system including:

[1481] (Claim 2)

[1482] 2. The system of claim 1, wherein the behavioral data includes user location information, schedule information, and health data.

[1483] (Claim 3)

[1484] 2. The system of claim 1, wherein the situation information includes news information, weather information, and traffic information. [Explanation of symbols]

[1485] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means of collecting user behavior data; means of gathering situational information from external sources; and A generation AI means for integrating the behavioral data and situation information to perform a future simulation; means for transmitting the simulation results to a user terminal; means for displaying the simulation results to a user on said terminal; A system including:

2. The system of claim 1 , wherein the behavioral data includes user location information, schedule information, and health data.

3. 2. The system of claim 1, wherein the situation information includes news information, weather information, and traffic information.

4. 2. The system according to claim 1, wherein the generating AI means integrates the behavioral data and the situation information and uses an algorithm to perform a future simulation.

5. 2. The system of claim 1, wherein the simulation results include suggestions for a course of action for the user.

Images