System

Abstract

A system is provided.SOLUTION: A system including means for collecting behavior data by obtaining access permission from a user, means for storing the collected behavior data in a cloud database, means for training a deep learning model using the preprocessed behavior data, means for analyzing 5W1H information input from the user, means for generating an optimal behavior plan by referring to the past behavior data and the deep learning model, and means for transmitting and displaying the generated behavior plan to a terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people struggle to decide what to do on their days off. Choosing the best option from multiple events and activities requires a lot of time and effort, which can result in people not getting the full value of their holidays. Another issue is the lack of personalized suggestions for indoor and outdoor types. This makes it difficult for users to spend their days off in a way that satisfies them. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for collecting behavioral data with permission from a user and storing it in a cloud database. It also includes a means for training a deep learning model using the preprocessed behavioral data and a means for analyzing the 5W1H information entered by the user. Furthermore, the system provides a means for generating an optimal behavior plan by referencing past behavioral data and the deep learning model, and includes a means for transmitting the generated behavior plan to the user's device and displaying it. This allows users to easily find the best way to spend their holidays without any hassle.

[0006] "Behavioral Data" refers to data related to a user's behavior or activities, such as a user's location, calendar information, or search history.

[0007] A "cloud database" is a database stored in a location accessible via the Internet, and refers to a system for collecting, storing, and analyzing information from multiple users.

[0008] "Preprocessing" refers to the process of converting data into a format suitable for training a deep learning model, such as cleaning the data, removing noise, and imputing missing values.

[0009] A "deep learning model" is a machine learning model based on an artificial neural network, and refers to a technology for learning, analyzing, and predicting complex data patterns.

[0010] "5W1H information" is a general term for information on who, what, where, when, why, and how.

[0011] "Analysis" refers to the process of interpreting user-entered information and converting it into meaningful data.

[0012] An "action plan" refers to a plan that includes specific schedules and suggestions generated to optimize a user's holiday.

[0013] "Generation" refers to the process of creating new data or information using specific algorithms.

[0014] "Verification" refers to the process of evaluating whether the generated action plan is feasible and amending it if necessary. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented in the following steps.

[0037] Data collection and preprocessing

[0038] 1. Login and Permissions

[0039] User: After installing the app and logging in for the first time, the user grants permission for the app to access their behavioral data (location, calendar information, search history, etc.).

[0040] Terminal: Prepares to periodically collect various data permitted by the user and sends that data to the server.

[0041] 2. Data storage and preprocessing

[0042] Server: Collected behavioral data is stored in a cloud database, where it is preprocessed by noise removal and missing value imputation.

[0043] Training a deep learning model

[0044] 3. Training the model

[0045] Server: Trains a deep learning model using the preprocessed data. This model learns the unique behavioral patterns and preferences of each user. This training is performed periodically, and the accuracy of the model improves as the amount of data increases.

[0046] User requests and analysis

[0047] 4. Request acceptance and analysis

[0048] User: Enter specific 5W1H information into the app, such as "Date in Tachikawa from 10am to 9pm on January 20th."

[0049] Terminal: Sends the input request information to the server.

[0050] Server: Analyzes the request information and generates an optimal action plan by referring to user behavior data and deep learning models.

[0051] Generate and present an action plan

[0052] 5. Plan Generation and Verification

[0053] Server: Based on the user's preferences and past behavioral data, the server generates a specific action plan by combining event information, restaurants, transportation options, etc. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll around the park at 15:00 → Dinner at Restaurant B at 18:00" may be created.

[0054] Server: Validates whether the generated plan is feasible, adjusts the plan if necessary, and finalizes the action plan.

[0055] 6. Plan presentation and selection

[0056] Server: Sends the determined action plan to the user's device.

[0057] On the device: The proposed action plan is displayed to the user, who can review the details of the proposed plan and make changes if necessary.

[0058] Users can select their favorite action plans and add them to their calendar within the app. They can also set reminders to ensure they don't forget to follow through with their plans.

[0059] Specific examples

[0060] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[0061] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[0062] 2. Terminal: Sends information to the server.

[0063] 3. Server: Refers to the deep learning model and generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa.

[0064] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[0065] 5. Server: Sends the finalized action plan to the device.

[0066] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[0067] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[0068] These steps of implementing the present invention allow users to easily find the perfect holiday experience and maximize its value.

[0069] The processing flow will be explained below.

[0070] Specific processing steps of the system

[0071] Step 1: Log in and get permissions

[0072] User: After installing the app for the first time, launch the app and access the login screen. Log in with your social media account or email address.

[0073] Device: When logging in for the first time, a popup will appear asking the user for permission to collect behavioral data.

[0074] Users: Grant data permissions such as location, calendar information, and search history.

[0075] Step 2: Collecting and sending behavioral data

[0076] On the device: Permitted data (GPS information, calendar events, search history) is periodically retrieved and sent to the server in batch or real-time.

[0077] Server: Stores the received data in a cloud database.

[0078] Step 3: Preprocessing the data

[0079] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0080] Server: Prepares the preprocessed data as input for deep learning models.

[0081] Step 4: Train the deep learning model

[0082] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, which then acts as a custom model for each user.

[0083] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0084] Step 5: User Request Input

[0085] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0086] Terminal: Sends the input request information to the server.

[0087] Step 6: Parsing the request information

[0088] Server: Receives user request information and analyzes it using a deep learning model, while also referencing past behavioral history and preference data.

[0089] Step 7: Generate an action plan

[0090] Server: Generates an optimal action plan based on the request and learned behavioral pattern data, for example, by combining information on date spots, restaurants, and current events.

[0091] Server: Verifies the feasibility of the generated plan (e.g., whether reservations are possible or the event is being held).

[0092] Server: Readjust the plan as needed and prepare a finalized plan.

[0093] Step 8: Submit and view your plan

[0094] Server: Sends the determined action plan to the user's device.

[0095] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0096] Step 9: Review and select your plan

[0097] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0098] Users: Add the selected plan to their calendar and set a reminder.

[0099] The above are the specific processing steps in the system of the present invention, which is designed to maximize the value of a user's holiday and propose an optimal activity plan.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] In modern society, it is important to provide optimal activity plans for individual users so that they can make effective use of their limited holidays. However, conventional systems have had difficulty effectively collecting and analyzing user behavioral data to generate highly accurate activity plans. Furthermore, issues remain, such as verifying the feasibility of generated plans and providing a reminder function to ensure users do not forget to execute them.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes a means for collecting behavioral data with permission from the user, a means for storing the collected behavioral data in a cloud database, and a means for performing preprocessing on the stored behavioral data, such as noise removal and missing value imputation. This enables the generation of highly accurate behavior plans. The system also includes a means for training a deep learning model using the preprocessed behavioral data, a means for analyzing 5W1H information entered by the user, and a means for generating an optimal behavior plan by referencing past behavioral data and the deep learning model. The system also includes a means for verifying whether the generated behavior plan is feasible and for making adjustments as necessary, and a means for sending and displaying the finalized behavior plan to the user's device. This allows users to easily plan their optimal holiday and maximize its value. The system also includes a means for setting a reminder function, helping users remember to carry out their behavior plan.

[0105] "Access permission" means that a user grants an application or system access to their data.

[0106] "Behavioral data" refers to data related to a user's daily behavior and activities, such as location information, calendar information, and search history.

[0107] A "cloud database" is a remote database management system that can be accessed via the Internet, and is a means of efficiently storing and managing large amounts of data.

[0108] "Preprocessing" refers to the process of processing raw data into a form suitable for data analysis and training machine learning models, such as removing noise and filling in missing values.

[0109] A "deep learning model" refers to an algorithm that uses a multi-layer neural network to learn the characteristics of data and perform advanced analysis and predictions.

[0110] "5W1H information" refers to information entered by the user regarding "when, where, who, what, why, and how."

[0111] An "action plan" refers to a schedule that specifically plans activities or events that should be carried out on a user's holidays or at specific times.

[0112] The "reminder function" refers to a function that notifies users so that they do not forget their schedules or plans.

[0113] "User device" means an electronic device that is directly operated by a user, such as a smartphone or tablet.

[0114] This invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented by the following detailed process.

[0115] First, the user installs the application and grants permission to access behavioral data such as location information, calendar information, and search history when logging in for the first time. The device periodically collects the data permitted by the user and sends it to the server.

[0116] The server stores the received behavioral data in a cloud database. This can be a remote database management system such as Amazon RDS or Google Cloud SQL. The collected data is then preprocessed, including noise removal and missing value imputation. This improves the data quality and makes it suitable for training deep learning models.

[0117] The server trains a deep learning model based on the preprocessed data. Machine learning frameworks such as TENSORFLOW (registered trademark) and PyTorch are used for training. This deep learning model learns user behavior patterns and preferences and incorporates a mechanism to periodically update its accuracy.

[0118] When a user enters specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th," the device sends this information to the server. The server analyzes the information and generates an optimal action plan by referencing a pre-trained deep learning model and past behavioral data.

[0119] The server verifies various conditions (e.g., business hours and traffic information) to ensure that the generated plan is feasible. If necessary, it readjusts the plan and sends the finalized plan to the device. The device then displays this plan to the user, who can review the proposed plan and make changes if necessary.

[0120] Additionally, the finalized action plan is added to the calendar and a reminder function is set, so users can remember to carry out their plan.

[0121] As a concrete example, if a user inputs a request for "a date in Tachikawa from 10:00 to 21:00 on January 20th," the system generates the following prompt:

[0122] Example prompt sentence:

[0123] "Please create a specific action plan for a date in Tachikawa from 10:00 to 21:00 on January 20th."

[0124] As described above, this invention provides an optimal action plan tailored to individual needs based on the user's behavioral data, enabling the user to spend their holidays efficiently and productively.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] The user installs the application and grants permission to access behavioral data such as location, calendar information, and search history when logging in for the first time. Based on this input data, the device authenticates the user and prepares access to various data. The output is an authentication token and authorized data access.

[0128] Step 2:

[0129] The device periodically collects user behavioral data such as location information, calendar information, and search history. Specifically, it obtains location information using GPS, schedule information via a calendar API, and browser history data. The collected data is sent to a server. The input in this step is the user's behavioral data, and the output is the collected behavioral dataset.

[0130] Step 3:

[0131] The server stores the received behavioral data in a cloud database, typically using Amazon RDS or Google Cloud SQL. A timestamp is added when the data is saved, and checks are performed to ensure data consistency and integrity. This input data is the collected behavioral data, and is saved as output in the cloud database.

[0132] Step 4:

[0133] The server performs preprocessing on the stored behavioral data, such as noise removal and missing value imputation. For example, it runs filtering algorithms to remove outliers and applies imputation algorithms to fill in missing data. The input data is the stored behavioral data, and the output is a preprocessed, clean dataset.

[0134] Step 5:

[0135] The server uses the preprocessed data to train a deep learning model. For training, the model is implemented using TensorFlow or PyTorch to learn each user's behavioral patterns and preferences. The input data is the preprocessed data, and the output is a trained deep learning model.

[0136] Step 6:

[0137] The user inputs specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th." The input information is retrieved through the application interface. This input information is then sent to the server for further analysis, becoming a specific user request.

[0138] Step 7:

[0139] The server analyzes the input 5W1H information and generates an optimal action plan by referencing a deep learning model and past behavioral data. For example, it combines information on date spots around Tachikawa, good restaurants, and ongoing events. The input data is the 5W1H information and the user's behavioral data, and the output is the generated action plan.

[0140] Step 8:

[0141] The server verifies the generated action plan based on business hours and traffic information to confirm whether it is feasible. If necessary, the action plan is readjusted and finally confirmed. The input data is the generated action plan and various verification data, and the output is a feasible and confirmed action plan.

[0142] Step 9:

[0143] The server sends the finalized action plan to the user's device. The device displays the received action plan to the user, who then checks the proposed plan and makes changes if necessary. The input in this step is the finalized action plan, and the output is the action plan displayed on the user's device.

[0144] Step 10:

[0145] The user adds the finalized action plan to the calendar in the app. The device sets a reminder for the event added to the calendar. The reminder notifies the user so that they can remember to carry out the event. The input data is the finalized action plan, and the output is the event added to the calendar and the reminder setting.

[0146] (Application example 1)

[0147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] Previously, systems existed that generated personalized action plans using user preferences and past behavioral data. However, in the case of food delivery services, these systems were unable to effectively incorporate a user's specific preferences and past ordering history into their proposals with sufficient accuracy. Furthermore, in the case of food delivery, the system was unable to provide sufficient recommendations for optimal restaurants and delivery times, making it difficult to provide a satisfying service. Therefore, it is necessary to improve the accuracy and personalization of systems to improve user convenience.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0150] In this invention, the server includes means for collecting behavioral data with permission from the user, means for storing the collected behavioral data in a cloud database, means for training a deep learning model using the preprocessed behavioral data, means for analyzing 5W1H information entered by the user, means for generating an optimal action plan by referring to past behavioral data and the deep learning model, means for transmitting and displaying the generated action plan to the user's terminal, and means for suggesting the optimal food delivery based on the user's preferences and past order history, thereby enabling users to easily enjoy highly satisfying food delivery.

[0151] - "Permission" means that the user gives the application permission to collect and use behavioral data.

[0152] "Behavioral data" refers to data that indicates a user's behavior, such as the user's location information, calendar information, search history, and order history.

[0153] "Cloud database" means a database accessible via the internet that is used to store behavioral data.

[0154] "Preprocessing" refers to the process of performing processes such as noise removal and missing value completion on collected data to prepare it in a format suitable for analysis and model learning.

[0155] A "deep learning model" is a machine learning model that uses a multi-layer neural network to learn complex data patterns and perform predictions and classifications.

[0156] "5W1H information" refers to the basic elements of information such as Who, What, When, Where, Why, and How.

[0157] "Analysis" refers to the process of processing and calculating collected data and input information for a specific purpose to derive meaningful results.

[0158] An "action plan" is a specific schedule or plan suggested based on the user's actions.

[0159] A "terminal" is a device that can be directly operated by a user, such as a smartphone or tablet.

[0160] "Preferences" refer to a user's preferences, interests, and concerns.

[0161] "Order History" is a record of orders placed by a user in the past.

[0162] "Food delivery" is a service that delivers meals from restaurants or food services to a location specified by the user.

[0163] System Overview

[0164] This invention is a system that collects and analyzes user behavior data and suggests optimal food delivery options based on the user's preferences and past order history. This system is primarily configured using a cloud-based database and deep learning model.

[0165] Hardware and software used

[0166] Hardware

[0167] Smartphone: The user installs and uses the application.

[0168] Cloud server (e.g., AWS (registered trademark) EC2 or Google Cloud Platform): Analyzes data and trains models.

[0169] software

[0170] Database (e.g., Firebase, AWS DynamoDB): Stores collected behavioral data.

[0171] Deep learning frameworks (e.g., TensorFlow, PyTorch): Model user behavior patterns and preferences and make predictions.

[0172] Detailed processing flow

[0173] 1. User Permissions and Data Collection

[0174] Users: Install applications and grant permissions.

[0175] Smartphones: Collect user behavioral data such as location information, calendar information, search history, and order history.

[0176] Cloud Server: Collected data is stored in a cloud database in real time.

[0177] 2. Data storage and preprocessing

[0178] Cloud server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data.

[0179] 3. Training the deep learning model

[0180] Cloud server: Using the pre-processed data, the deep learning model is periodically trained and updated to learn each user's preferences and behavioral patterns.

[0181] 4. User Requests and Analysis

[0182] User: Enters a specific request into the app, such as "I'd like a healthy lunch delivered on January 20th at 12 noon."

[0183] Smartphone: Sends request information to the cloud server.

[0184] Cloud server: Analyzes the request information and generates optimal food delivery suggestions by referencing the user's preferences, past order history, and deep learning models.

[0185] 5. Generate and present food delivery offers

[0186] Cloud server: Generates and validates an action plan including recommended restaurants, dishes, and delivery times.

[0187] Smartphone: Displays suggestions sent from the cloud server to the user.

[0188] Specific examples

[0189] User input: "I'd like a healthy lunch delivered on January 20th at 12 noon."

[0190] Cloud server response: "Recommended plans: Organic salad bowl (Restaurant A), sushi set (Restaurant B), Japanese set meal (Restaurant C)"

[0191] Prompt Sentence Examples

[0192] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[0193] In this way, the present invention allows users to easily find the best food delivery options, greatly improving convenience.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] User Permissions and Data Collection

[0197] User: Installs the application and grants permissions to access behavioral data (input: operations; output: permissions).

[0198] Device: Collects behavioral data such as location information, calendar information, search history, and order history (input: user behavior. output: behavioral data).

[0199] Terminal: Sends collected behavioral data to the cloud server in real time (input: behavioral data; output: sending data to the server).

[0200] Step 2:

[0201] Data storage and preprocessing

[0202] Server: Stores the behavioral data sent to the cloud database (input: behavioral data; output: data in the cloud database).

[0203] Server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data (input: data in the cloud database; output: preprocessed data).

[0204] Step 3:

[0205] Training a deep learning model

[0206] Server: Trains a deep learning model using the preprocessed data (Input: preprocessed data. Output: trained model).

[0207] Server: This model is periodically updated to maintain predictive accuracy based on the latest data (input: new behavioral data. output: updated model).

[0208] Step 4:

[0209] User requests and analysis

[0210] User: Enters specific 5W1H information such as "I would like a healthy delivery lunch at 12 o'clock on January 20th" (Input: 5W1H information. Output: Request data).

[0211] Terminal: Sends request information to the cloud server (Input: Request data. Output: Sends request data to the server).

[0212] Server: Analyzes the request information and processes (analyzes) the data by referring to the deep learning model (input: request data & model. output: analysis results).

[0213] Step 5:

[0214] Generate and present food delivery offers

[0215] Server: Generates and validates food delivery plans including recommended restaurants, dishes, and optimal delivery times (Input: Analysis results. Output: Generated food delivery plans).

[0216] Server: Readjusts the plan if necessary and confirms the final proposal (Input: Generated food delivery plan. Output: Confirmed plan).

[0217] Terminal: Shows the proposal sent from the cloud server to the user (Input: Confirmed plan. Output: Display to user).

[0218] Examples:

[0219] Example prompt sentence:

[0220] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[0221] Step 6:

[0222] Selecting and implementing an action plan

[0223] User: Review the proposed food delivery plans in detail and select the appropriate plan (Input: Proposed plans. Output: Selected plan).

[0224] Terminal: Add selected action plan to calendar and set reminder (Input: selected plan. Output: calendar and reminder settings).

[0225] Through this process, users can receive personalized food delivery suggestions efficiently.

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] This invention relates to a system that collects and analyzes user behavioral and emotional data to propose optimal holiday plans. By combining this system with an emotion engine, it is possible to make more personalized proposals that take into account the user's emotions.

[0228] Data collection and preprocessing

[0229] 1. Login and Permissions

[0230] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[0231] Device: When logging in for the first time, a pop-up will appear asking the user to grant permission to access their behavioral data (location, calendar information, search history) and emotional data (facial expressions, voice, text).

[0232] User: Grants access permissions to various data.

[0233] 2. Collecting behavioral and emotional data

[0234] Device: Permitted data (GPS information, calendar events, search history, facial expression data, voice data, text data) is periodically collected and sent to the server in batch processing or real time.

[0235] Server: Stores the received data in a cloud database.

[0236] 3. Data Preprocessing

[0237] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0238] Server: Prepares the preprocessed data as input for the deep learning model and emotion engine.

[0239] Deep learning model and emotion engine training

[0240] 4. Training the model

[0241] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences, and also works with sentiment data to improve the model's accuracy.

[0242] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0243] User requests and analysis

[0244] 5. Request acceptance and analysis

[0245] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0246] Terminal: Sends the input request information to the server.

[0247] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[0248] Generate and present an action plan

[0249] 6. Generate an action plan

[0250] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and ongoing events.

[0251] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[0252] 7. Submitting and Viewing Plans

[0253] Server: Sends the determined action plan to the user's device.

[0254] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0255] 8. Check and select your plan

[0256] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0257] Users: Add the selected plan to their calendar and set a reminder.

[0258] Specific examples

[0259] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[0260] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[0261] 2. Terminal: Sends information to the server.

[0262] 3. Server: Refers to the deep learning model to generate an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes into account the user's real-time emotional data.

[0263] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[0264] 5. Server: Sends the finalized action plan to the device.

[0265] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[0266] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[0267] These steps for implementing the present invention create a system that maximizes the value of a user's holiday and suggests an action plan. The introduction of an emotion engine enables highly personalized suggestions that take into account the user's emotional state.

[0268] The processing flow will be explained below.

[0269] Specific processing steps of the system

[0270] Step 1:

[0271] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[0272] Step 2:

[0273] On your device: When you first log in, a pop-up will appear asking you to grant permission to access your behavioral and emotional data. Behavioral data includes location, calendar information, and search history, while emotional data includes facial expressions, voice, and text.

[0274] Step 3:

[0275] User: Grants access permissions to various data.

[0276] Step 4:

[0277] Device: Acquires authorized data and sends it to the server in batch or real-time, such as GPS information, calendar events, search history, facial expression data, voice data, and text data.

[0278] Step 5:

[0279] Server: Stores the received data in a cloud database.

[0280] Step 6:

[0281] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0282] Step 7:

[0283] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, including incorporating sentiment data to improve the model's accuracy.

[0284] Step 8:

[0285] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0286] Step 9:

[0287] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0288] Step 10:

[0289] Terminal: Sends the input request information to the server.

[0290] Step 11:

[0291] Server: Receives the request information and analyzes it using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[0292] Step 12:

[0293] Server: Generates an optimal action plan based on the user's request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and current events.

[0294] Step 13:

[0295] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[0296] Step 14:

[0297] Server: Sends the determined action plan to the user's device.

[0298] Step 15:

[0299] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0300] Step 16:

[0301] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0302] Step 17:

[0303] Users: Add the selected plan to their calendar and set a reminder.

[0304] In this way, we can realize a system that makes maximum use of user behavioral and emotional data to generate and propose optimal action plans. The introduction of an emotion engine also takes into account the user's psychological state, enabling more personalized proposals.

[0305] Example 2

[0306] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] Existing action plan suggestion systems only consider user behavioral data, which means they lack advanced personalization that reflects the user's emotional state. Furthermore, they lack feasibility verification and refinement of the generated action plans, resulting in a poor user experience.

[0308] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data with permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for performing noise removal, missing value completion, and data standardization on the stored raw data, means for training a deep learning model using the preprocessed behavioral data and emotional data, means for analyzing 5W1H information entered by the user and generating an optimal action plan by referring to past behavioral data and emotional data, means for verifying whether the generated action plan is feasible and readjusting it as necessary, and means for transmitting the generated action plan to the user's terminal and displaying it in a visually easy-to-understand manner. This makes it possible to propose a highly personalized action plan that also takes into account the user's emotional state, and to verify and adjust its feasibility.

[0309] "Access permission" refers to the user's permission for an application to access personal data such as location information, calendar information, search history, facial expression data, voice data, and text data.

[0310] A "cloud database" is a type of data storage service that stores and manages data over the Internet, and can handle large amounts of data flexibly and efficiently.

[0311] "Denoising" refers to the process of removing unnecessary or inaccurate data from a dataset.

[0312] "Missing value imputation" refers to the process of filling in missing values ​​in a dataset with an appropriate value (e.g., the mean or median).

[0313] "Data standardization" refers to the process of scaling data so that each data point in a data set is on a consistent scale.

[0314] A "deep learning model" refers to a computational model that uses a multi-layer neural network to learn the characteristics of data and perform tasks such as prediction and classification.

[0315] "5W1H information" typically refers to information that consists of the elements of "When," "Where," "Who," "What," "Why," and "How."

[0316] "Emotion data" is information that indicates the emotional state of the user, and includes facial expression data, voice data, text data, and the like.

[0317] "Behavioral data" refers to information about a user's behavior, including location information, calendar information, search history, etc.

[0318] An "action plan" refers to a schedule or proposal of specific activities to be carried out within a time period specified by the user.

[0319] "Feasibility" refers to the process of verifying whether the generated action plan can be carried out in reality (for example, whether reservations can be made or whether events will be held).

[0320] An "emotion engine" refers to an algorithm or system that analyzes a user's emotional data and generates an appropriate action plan based on that emotional state.

[0321] "Visually easy-to-understand display" refers to displaying information in a format that is easy for users to understand, including timeline displays and map displays.

[0322] This invention is a system that collects and analyzes user behavioral and emotional data and proposes optimal action plans. This system uses a cloud database, deep learning models, and an emotion engine to enable advanced personalization that reflects the user's emotional state.

[0323] Data collection and preprocessing

[0324] 1. Login and Permissions

[0325] User: Installs the app and logs in with a social media account or email address.

[0326] Device: After logging in, a pop-up will appear requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expression data, voice data, text data).

[0327] User: Grants access permissions to various data.

[0328] 2. Collecting behavioral and emotional data

[0329] Device: Periodically collects and transmits permitted data to a server, including GPS information, calendar events, search history, facial expression data, voice data, and text data.

[0330] Server: Stores the received data in a cloud database (e.g., AWS DynamoDB).

[0331] 3. Data Preprocessing

[0332] Server: Performs noise removal (e.g., removing inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling).

[0333] Deep learning model and emotion engine training

[0334] 4. Training the model

[0335] Server: Trains deep learning models using preprocessed data. Builds models using TensorFlow or PyTorch.

[0336] Server: Split the dataset into training and validation datasets to improve the accuracy of the model.

[0337] Server: Retrain the model periodically to incorporate new data, for example, on the first day of every month.

[0338] User requests and analysis

[0339] 5. Request acceptance and analysis

[0340] User: Enter a specific request for how to spend the day off into the app. Example: "Date in Tachikawa from 10am to 9pm on January 20th."

[0341] Terminal: Sends the input request information to the server.

[0342] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[0343] Generate and present an action plan

[0344] 6. Generate an action plan

[0345] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events."

[0346] Server: Validates the feasibility of the generated plan (e.g., whether reservations are available or events are running) and adjusts the plan if necessary. Retrieves reservation and event information using the OpenTable API and Eventbrite API.

[0347] 7. Submitting and Viewing Plans

[0348] Server: Sends the determined action plan to the user's device.

[0349] Device: Display the proposed action plan visually, for example as a timeline or points on a map.

[0350] 8. Check and select your plan

[0351] User: Review the suggested action plans and adjust accordingly. Select the plan you like best and add it to your calendar.

[0352] Users: Set reminders for selected plans so you can receive timely notifications.

[0353] Specific examples

[0354] Example 1: "Date in Tachikawa from 10am to 9pm on January 20th"

[0355] 1. User: Enters a request into the app: "Date in Tachikawa from 10am to 9pm on January 20th."

[0356] 2. Terminal: Sends the request to the server.

[0357] 3. Server: Using a deep learning model, it generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes real-time emotion data into account.

[0358] 4. Server: Validates the feasibility of the generated plan and adjusts it if necessary.

[0359] 5. Server: Sends the finalized action plan to the device.

[0360] 6. Terminal: Display the proposed plan as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00."

[0361] 7. User: Review the plans, adopt the one they like, add it to their calendar, and set a reminder.

[0362] Examples of prompt statements

[0363] Example 1: "Please suggest a date plan in Tachikawa from 10:00 to 21:00 on January 20th."

[0364] Example 2: "What is your plan of action for relaxing on the weekend?"

[0365] This allows users to get a highly personalized holiday action plan that takes into account their emotional state.

[0366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0367] Step 1: Login and Permissions

[0368] User: Installs the app and logs in using a social media account or email address. The input is account information and the output is a login session.

[0369] Terminal: After logging in, a popup is displayed requesting permission to access behavioral and emotional data. The input is the login session, and the output is the popup screen.

[0370] User: Set the access permission for various data to "Allow". The input is a popup screen, and the output is the permission status.

[0371] Step 2: Collect behavioral and emotional data

[0372] Terminal: Periodically collects authorized data (GPS information, calendar events, search history, facial expression data, voice data, text data). The input is the authorization status, and the output is the collected data.

[0373] Terminal: Sends collected data to the server in batch processing or real time. For example, when the amount of data exceeds a certain threshold, the data is sent. The input is the collected data, and the output is the data sent to the server.

[0374] Server: Stores the received data in a cloud database. The input is the transmitted data, and the output is the data stored in the database.

[0375] Step 3: Preprocessing the data

[0376] Server: Performs noise removal (removal of inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling). Input is raw data, and output is preprocessed data.

[0377] Server: Prepares the preprocessed data as input data for the deep learning model and emotion engine. The input is the preprocessed data, and the output is the input data for the model.

[0378] Step 4: Training the model

[0379] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. The input is the preprocessed data, and the output is the trained model.

[0380] Server: Splits the dataset into training and validation datasets to improve the accuracy of the model. The input is the dataset, and the output is the evaluation of the model accuracy.

[0381] Server: Periodically ingests new data and retrains the model. For example, retraining occurs on the first day of each month. The input is the new data, and the output is the updated model.

[0382] Step 5: Accepting and parsing the request

[0383] User: Enters a specific request for how to spend the day off into the app. For example, a request might be "Date in Tachikawa from 10:00 to 21:00 on January 20th." The input is the request information, and the output is sending the request to the device.

[0384] Terminal: Sends the input request information to the server. The input is the request information, and the output is the data to be sent to the server.

[0385] Server: Analyzes the request information and performs analysis using a deep learning model and emotion engine. This also refers to past behavioral history, preference data, and real-time emotion data. The input is the request and reference data, and the output is the analysis results.

[0386] Step 6: Generate an action plan

[0387] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotion data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events." The input is the request and model data, and the output is an action plan.

[0388] Server: Validates whether the generated plan is feasible (reservation availability and event status), and adjusts the plan as necessary. For example, it obtains reservation and event information using the OpenTable API or Eventbrite API. The input is the action plan, and the output is the finalized plan.

[0389] Step 7: Submit and view your plan

[0390] Server: Sends the finalized action plan to the user's terminal. The input is the finalized plan, and the output is the transmitted data.

[0391] Terminal: A device that visually displays the proposed action plan, for example, in a timeline format or as points on a map. The input is the transmitted data, and the output is the displayed information.

[0392] Step 8: Review and select your plan

[0393] User: Review the proposed action plans and adjust accordingly. Select the plan they like and add it to their calendar. Input is display information, output is calendar event.

[0394] User: Set a reminder for the selected plan and receive notifications. Input is a calendar event, output is a reminder setting.

[0395] (Application example 2)

[0396] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] Conventional itinerary suggestion systems are based solely on user behavioral data, making it difficult to provide highly personalized suggestions that take into account the user's emotions and moods. Furthermore, because these systems are primarily focused on travel and daily activity planning, they do not offer functionality to link with autonomous driving systems to suggest optimal driving routes and stopover spots.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data after obtaining access permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for training a deep learning model using preprocessed behavioral data and emotional data, means for analyzing specific request information entered by the user, means for generating an optimal action plan by referencing past behavioral data and emotional data and the deep learning model, and means for cooperating with an autonomous driving system to suggest optimal driving routes and stopover spots. This makes it possible to propose highly personalized action plans that take the user's emotional state into consideration, and also makes it possible to provide optimal driving plans using autonomous driving vehicles.

[0399] "User Permission" is the consent an application requests from a user to allow the application to access their behavioral and emotional data.

[0400] "Behavioral Data" refers to information related to a user's behavior, such as the user's location, calendar information, and search history.

[0401] "Emotional data" refers to information about a user's emotional state, such as their facial expressions, voice, or text.

[0402] "Cloud Database" means a database accessible over the internet that is used to store and manage behavioral and emotional data.

[0403] "Preprocessing" refers to the process of removing noise, filling in missing values, standardizing, and otherwise processing the collected raw data to prepare it in a format suitable for training a deep learning model.

[0404] A "deep learning model" is a machine learning model that uses deep learning technology to analyze a user's behavioral patterns and emotional state and generate an action plan.

[0405] "Specific request information" refers to the specific action plan or request information that the user enters into the application, including 5W1H information such as date, time, location, purpose, and desired activity.

[0406] The "optimal action plan" is a plan that proposes the most suitable action plan for the user based on the user's behavioral and emotional data.

[0407] An "autonomous driving system" is a system that uses artificial intelligence and sensor technology to automate driving and operate vehicles without a human driver.

[0408] A "driving route" is the route an autonomous vehicle will travel, and is part of a plan that includes destinations and stopovers requested by the user.

[0409] "Stopover spots" are places where users are encouraged to stop during their drive, such as scenic spots, restaurants, cafes, and event venues.

[0410] This invention relates to a system that collects and analyzes user behavioral and emotional data, and then works in conjunction with an autonomous driving system to suggest optimal driving routes and stopover spots. This system uses a wide range of devices, including smartphones and autonomous vehicles.

[0411] System configuration and operation overview

[0412] User Device

[0413] 1. Login and Permissions:

[0414] Users install the application on their smartphone and log in the first time they start it, using their social media account or email address.

[0415] When logging in for the first time, the device will display a pop-up requesting the user to grant permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text).

[0416] Users grant access permissions for various data.

[0417] 2. Data Collection:

[0418] The device periodically acquires permitted data (GPS information, calendar events, search history, facial expression data, voice data, and text data) and transmits it to the server in real time or in batches.

[0419] server

[0420] 3. Data preprocessing:

[0421] The server performs preprocessing such as noise removal, missing value completion, and data standardization on the raw data stored in the cloud database.

[0422] The preprocessed data is prepared as input for the deep learning model and emotion engine. The data is managed by a deep learning framework (e.g., TensorFlow or PyTorch).

[0423] 4. Train the model:

[0424] The deep learning model uses pre-processed data to learn user behavior patterns and preferences, and also incorporates sentiment data to improve the model's accuracy.

[0425] The model is periodically retrained to incorporate new data and improve its accuracy.

[0426] 5. Driving route generation and suggestions:

[0427] When a user enters a specific driving plan request into the app (for example, "I want to go on a four-hour drive with a friend next Sunday"), the device sends that information to the server.

[0428] The server performs analysis using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[0429] Based on the request, the server works with the autonomous driving system to generate the optimal driving route and stopover spots.

[0430] The generated plan is verified for feasibility and adjusted if necessary, and this information is sent to the user's device for presentation.

[0431] 6. Displaying Proposals:

[0432] The device will clearly display the suggested driving route and stops to the user, who can then review the proposed plan, adjust accordingly, and select. The selected plan will be added to the calendar and a reminder will be set.

[0433] Specific examples

[0434] Consider a case where a user requests, "I want to enjoy a four-hour drive with a friend next Sunday." In this case, the server generates the following driving plan based on the user's past behavioral data, emotional data, and real-time data.

[0435] Driving route: "Leave home at 10:00 → Visit a scenic park at 10:30 → Have lunch at Restaurant A at 12:00 → Have lunch at Event B at 14:00 → Arrive home at 16:00"

[0436] Users can review the suggestions and, if they like them, add them to their calendar and set a reminder.

[0437] An example prompt is:

[0438] I'd like to go on a 4-hour drive with a friend next Sunday. Can you suggest a good route and places to stop?

[0439] In this way, a system will be realized that can provide highly personalized action plans based on user behavioral and emotional data, and propose optimal driving plans using autonomous vehicles.

[0440] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0441] Step 1:

[0442] Logging in a user and getting permissions

[0443] A user installs a smartphone application and logs in the first time they start it. They log in using a social media account or email address. When the user logs in for the first time, the device displays a pop-up requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text). If the user agrees, access to various data is granted. The input to this step is the user's login information and consent information, and the output is the status of data access permission.

[0444] Step 2:

[0445] Data collection

[0446] The device periodically acquires authorized behavioral data (GPS information, calendar events, search history) and emotional data (facial expression data, voice data, text data). These data are sent to the server in real time or batch processing. The input of this step is the user's behavioral data and emotional data, and the output is the raw data sent to the server.

[0447] Step 3:

[0448] Data Preprocessing

[0449] The server performs preprocessing on the raw data stored in the cloud database, such as noise removal, missing value imputation, and data standardization, which makes the data suitable for training deep learning models and emotion engines. The input of this step is raw data, and the output is preprocessed data.

[0450] Step 4:

[0451] Training a deep learning model

[0452] The server uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. It also incorporates sentiment data to improve the model's accuracy. The model is periodically retrained by incorporating new data to improve its accuracy. The input of this step is the preprocessed data, and the output is the trained deep learning model.

[0453] Step 5:

[0454] Request input and analysis

[0455] The user inputs a specific driving plan request (for example, "I want to enjoy a four-hour drive with a friend next Sunday") into the app. The device sends the input request information to the server. The server analyzes this request information using a deep learning model and emotion engine. This analysis also references past behavioral history, preference data, and real-time emotion data. The input of this step is the user's request, and the output is the analysis result.

[0456] Step 6:

[0457] Generate driving routes and stopovers

[0458] The server generates an optimal driving route and stopovers based on the request, learned behavioral pattern data, and emotion data. This includes driving routes and stopovers (scenic spots, restaurants, cafes, event venues, etc.). The server then verifies the feasibility of this generated plan (checking event availability and reservations), adjusts the plan as necessary, and prepares a finalized plan. The input of this step is the analysis result of the deep learning model, and the output is the optimal driving plan.

[0459] Step 7:

[0460] Submitting and Viewing Plans

[0461] The server sends the confirmed driving plan to the user's device. The device displays the proposed driving plan in an easy-to-understand manner to the user. For example, a schedule such as "Leave home at 10:00 → Visit a scenic park at 10:30 → Lunch (Restaurant A) at 12:00 → Lunch event (Event B) at 14:00 → Arrive home at 16:00" is displayed. The user reviews this proposed plan, and if they like it, it is added to their calendar and a reminder is set. The input of this step is the confirmed driving plan, and the output is the plan information displayed on the user's device.

[0462] The above steps realize a system that provides highly personalized, optimal driving plans using autonomous vehicles based on user behavioral and emotional data.

[0463] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0464] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0465] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0466] [Second embodiment]

[0467] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0468] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0470] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0472] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0474] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0475] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0476] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0477] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0478] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0479] The present invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented in the following steps.

[0480] Data collection and preprocessing

[0481] 1. Login and Permissions

[0482] User: After installing the app and logging in for the first time, the user grants permission for the app to access their behavioral data (location, calendar information, search history, etc.).

[0483] Terminal: Prepares to periodically collect various data permitted by the user and sends that data to the server.

[0484] 2. Data storage and preprocessing

[0485] Server: Collected behavioral data is stored in a cloud database, where it is preprocessed by noise removal and missing value imputation.

[0486] Training a deep learning model

[0487] 3. Training the model

[0488] Server: Trains a deep learning model using the preprocessed data. This model learns the unique behavioral patterns and preferences of each user. This training is performed periodically, and the accuracy of the model improves as the amount of data increases.

[0489] User requests and analysis

[0490] 4. Request acceptance and analysis

[0491] User: Enter specific 5W1H information into the app, such as "Date in Tachikawa from 10am to 9pm on January 20th."

[0492] Terminal: Sends the input request information to the server.

[0493] Server: Analyzes the request information and generates an optimal action plan by referring to user behavior data and deep learning models.

[0494] Generate and present an action plan

[0495] 5. Plan Generation and Verification

[0496] Server: Based on the user's preferences and past behavioral data, the server generates a specific action plan by combining event information, restaurants, transportation options, etc. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll around the park at 15:00 → Dinner at Restaurant B at 18:00" may be created.

[0497] Server: Validates whether the generated plan is feasible, adjusts the plan if necessary, and finalizes the action plan.

[0498] 6. Plan presentation and selection

[0499] Server: Sends the determined action plan to the user's device.

[0500] On the device: The proposed action plan is displayed to the user, who can review the details of the proposed plan and make changes if necessary.

[0501] Users can select their favorite action plans and add them to their calendar within the app. They can also set reminders to ensure they don't forget to follow through with their plans.

[0502] Specific examples

[0503] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[0504] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[0505] 2. Terminal: Sends information to the server.

[0506] 3. Server: Refers to the deep learning model and generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa.

[0507] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[0508] 5. Server: Sends the finalized action plan to the device.

[0509] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[0510] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[0511] These steps of implementing the present invention allow users to easily find the perfect holiday experience and maximize its value.

[0512] The processing flow will be explained below.

[0513] Specific processing steps of the system

[0514] Step 1: Log in and get permissions

[0515] User: After installing the app for the first time, launch the app and access the login screen. Log in with your social media account or email address.

[0516] Device: When logging in for the first time, a popup will appear asking the user for permission to collect behavioral data.

[0517] Users: Grant data permissions such as location, calendar information, and search history.

[0518] Step 2: Collecting and sending behavioral data

[0519] On the device: Permitted data (GPS information, calendar events, search history) is periodically retrieved and sent to the server in batch or real-time.

[0520] Server: Stores the received data in a cloud database.

[0521] Step 3: Preprocessing the data

[0522] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0523] Server: Prepares the preprocessed data as input for deep learning models.

[0524] Step 4: Train the deep learning model

[0525] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, which then acts as a custom model for each user.

[0526] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0527] Step 5: User Request Input

[0528] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0529] Terminal: Sends the input request information to the server.

[0530] Step 6: Parsing the request information

[0531] Server: Receives user request information and analyzes it using a deep learning model, while also referencing past behavioral history and preference data.

[0532] Step 7: Generate an action plan

[0533] Server: Generates an optimal action plan based on the request and learned behavioral pattern data, for example, by combining information on date spots, restaurants, and current events.

[0534] Server: Verifies the feasibility of the generated plan (e.g., whether reservations are possible or the event is being held).

[0535] Server: Readjust the plan as needed and prepare a finalized plan.

[0536] Step 8: Submit and view your plan

[0537] Server: Sends the determined action plan to the user's device.

[0538] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0539] Step 9: Review and select your plan

[0540] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0541] Users: Add the selected plan to their calendar and set a reminder.

[0542] The above are the specific processing steps in the system of the present invention, which is designed to maximize the value of a user's holiday and propose an optimal activity plan.

[0543] Example 1

[0544] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0545] In modern society, it is important to provide optimal activity plans for individual users so that they can make effective use of their limited holidays. However, conventional systems have had difficulty effectively collecting and analyzing user behavioral data to generate highly accurate activity plans. Furthermore, issues remain, such as verifying the feasibility of generated plans and providing a reminder function to ensure users do not forget to execute them.

[0546] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0547] In this invention, the server includes a means for collecting behavioral data with permission from the user, a means for storing the collected behavioral data in a cloud database, and a means for performing preprocessing on the stored behavioral data, such as noise removal and missing value imputation. This enables the generation of highly accurate behavior plans. The system also includes a means for training a deep learning model using the preprocessed behavioral data, a means for analyzing 5W1H information entered by the user, and a means for generating an optimal behavior plan by referencing past behavioral data and the deep learning model. The system also includes a means for verifying whether the generated behavior plan is feasible and for making adjustments as necessary, and a means for sending and displaying the finalized behavior plan to the user's device. This allows users to easily plan their optimal holiday and maximize its value. The system also includes a means for setting a reminder function, helping users remember to carry out their behavior plan.

[0548] "Access permission" means that a user grants an application or system access to their data.

[0549] "Behavioral data" refers to data related to a user's daily behavior and activities, such as location information, calendar information, and search history.

[0550] A "cloud database" is a remote database management system that can be accessed via the Internet, and is a means of efficiently storing and managing large amounts of data.

[0551] "Preprocessing" refers to the process of processing raw data into a form suitable for data analysis and training machine learning models, such as removing noise and filling in missing values.

[0552] A "deep learning model" refers to an algorithm that uses a multi-layer neural network to learn the characteristics of data and perform advanced analysis and predictions.

[0553] "5W1H information" refers to information entered by the user regarding "when, where, who, what, why, and how."

[0554] An "action plan" refers to a schedule that specifically plans activities or events that should be carried out on a user's holidays or at specific times.

[0555] The "reminder function" refers to a function that notifies users so that they do not forget their schedules or plans.

[0556] "User device" means an electronic device that is directly operated by a user, such as a smartphone or tablet.

[0557] This invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented by the following detailed process.

[0558] First, the user installs the application and grants permission to access behavioral data such as location information, calendar information, and search history when logging in for the first time. The device periodically collects the data permitted by the user and sends it to the server.

[0559] The server then stores the received behavioral data in a cloud database, using a remote database management system such as Amazon RDS or Google Cloud SQL. The collected data is then preprocessed, including noise removal and missing value imputation, to improve data quality and make it suitable for training deep learning models.

[0560] The server uses the preprocessed data to train a deep learning model using machine learning frameworks such as TensorFlow and PyTorch. This deep learning model learns user behavior patterns and preferences and periodically updates its accuracy.

[0561] When a user enters specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th," the device sends this information to the server. The server analyzes the information and generates an optimal action plan by referencing a pre-trained deep learning model and past behavioral data.

[0562] The server verifies various conditions (e.g., business hours and traffic information) to ensure that the generated plan is feasible. If necessary, it readjusts the plan and sends the finalized plan to the device. The device then displays this plan to the user, who can review the proposed plan and make changes if necessary.

[0563] Additionally, the finalized action plan is added to the calendar and a reminder function is set, so users can remember to carry out their plan.

[0564] As a concrete example, if a user inputs a request for "a date in Tachikawa from 10:00 to 21:00 on January 20th," the system generates the following prompt:

[0565] Example prompt sentence:

[0566] "Please create a specific action plan for a date in Tachikawa from 10:00 to 21:00 on January 20th."

[0567] As described above, this invention provides an optimal action plan tailored to individual needs based on the user's behavioral data, enabling the user to spend their holidays efficiently and productively.

[0568] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0569] Step 1:

[0570] The user installs the application and grants permission to access behavioral data such as location, calendar information, and search history when logging in for the first time. Based on this input data, the device authenticates the user and prepares access to various data. The output is an authentication token and authorized data access.

[0571] Step 2:

[0572] The device periodically collects user behavioral data such as location information, calendar information, and search history. Specifically, it obtains location information using GPS, schedule information via a calendar API, and browser history data. The collected data is sent to a server. The input in this step is the user's behavioral data, and the output is the collected behavioral dataset.

[0573] Step 3:

[0574] The server stores the received behavioral data in a cloud database, typically using Amazon RDS or Google Cloud SQL. A timestamp is added when the data is saved, and checks are performed to ensure data consistency and integrity. This input data is the collected behavioral data, and is saved as output in the cloud database.

[0575] Step 4:

[0576] The server performs preprocessing on the stored behavioral data, such as noise removal and missing value imputation. For example, it runs filtering algorithms to remove outliers and applies imputation algorithms to fill in missing data. The input data is the stored behavioral data, and the output is a preprocessed, clean dataset.

[0577] Step 5:

[0578] The server uses the preprocessed data to train a deep learning model. For training, the model is implemented using TensorFlow or PyTorch to learn each user's behavioral patterns and preferences. The input data is the preprocessed data, and the output is a trained deep learning model.

[0579] Step 6:

[0580] The user inputs specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th." The input information is retrieved through the application interface. This input information is then sent to the server for further analysis, becoming a specific user request.

[0581] Step 7:

[0582] The server analyzes the input 5W1H information and generates an optimal action plan by referencing a deep learning model and past behavioral data. For example, it combines information on date spots around Tachikawa, good restaurants, and ongoing events. The input data is the 5W1H information and the user's behavioral data, and the output is the generated action plan.

[0583] Step 8:

[0584] The server verifies the generated action plan based on business hours and traffic information to confirm whether it is feasible. If necessary, the action plan is readjusted and finally confirmed. The input data is the generated action plan and various verification data, and the output is a feasible and confirmed action plan.

[0585] Step 9:

[0586] The server sends the finalized action plan to the user's device. The device displays the received action plan to the user, who then checks the proposed plan and makes changes if necessary. The input in this step is the finalized action plan, and the output is the action plan displayed on the user's device.

[0587] Step 10:

[0588] The user adds the finalized action plan to the calendar in the app. The device sets a reminder for the event added to the calendar. The reminder notifies the user so that they can remember to carry out the event. The input data is the finalized action plan, and the output is the event added to the calendar and the reminder setting.

[0589] (Application example 1)

[0590] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] Previously, systems existed that generated personalized action plans using user preferences and past behavioral data. However, in the case of food delivery services, these systems were unable to effectively incorporate a user's specific preferences and past ordering history into their proposals with sufficient accuracy. Furthermore, in the case of food delivery, the system was unable to provide sufficient recommendations for optimal restaurants and delivery times, making it difficult to provide a satisfying service. Therefore, it is necessary to improve the accuracy and personalization of systems to improve user convenience.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0593] In this invention, the server includes means for collecting behavioral data with permission from the user, means for storing the collected behavioral data in a cloud database, means for training a deep learning model using the preprocessed behavioral data, means for analyzing 5W1H information entered by the user, means for generating an optimal action plan by referring to past behavioral data and the deep learning model, means for transmitting and displaying the generated action plan to the user's terminal, and means for suggesting the optimal food delivery based on the user's preferences and past order history, thereby enabling users to easily enjoy highly satisfying food delivery.

[0594] - "Permission" means that the user gives the application permission to collect and use behavioral data.

[0595] "Behavioral data" refers to data that indicates a user's behavior, such as the user's location information, calendar information, search history, and order history.

[0596] "Cloud database" means a database accessible via the internet that is used to store behavioral data.

[0597] "Preprocessing" refers to the process of performing processes such as noise removal and missing value completion on collected data to prepare it in a format suitable for analysis and model learning.

[0598] A "deep learning model" is a machine learning model that uses a multi-layer neural network to learn complex data patterns and perform predictions and classifications.

[0599] "5W1H information" refers to the basic elements of information such as Who, What, When, Where, Why, and How.

[0600] "Analysis" refers to the process of processing and calculating collected data and input information for a specific purpose to derive meaningful results.

[0601] An "action plan" is a specific schedule or plan suggested based on the user's actions.

[0602] A "terminal" is a device that can be directly operated by a user, such as a smartphone or tablet.

[0603] "Preferences" refer to a user's preferences, interests, and concerns.

[0604] "Order History" is a record of orders placed by a user in the past.

[0605] "Food delivery" is a service that delivers meals from restaurants or food services to a location specified by the user.

[0606] System Overview

[0607] This invention is a system that collects and analyzes user behavior data and suggests optimal food delivery options based on the user's preferences and past order history. This system is primarily configured using a cloud-based database and deep learning model.

[0608] Hardware and software used

[0609] Hardware

[0610] Smartphone: The user installs and uses the application.

[0611] Cloud servers (e.g., AWS EC2 and Google Cloud Platform): Analyze data and train models.

[0612] software

[0613] Database (e.g., Firebase, AWS DynamoDB): Stores collected behavioral data.

[0614] Deep learning frameworks (e.g., TensorFlow, PyTorch): Model user behavior patterns and preferences and make predictions.

[0615] Detailed processing flow

[0616] 1. User Permissions and Data Collection

[0617] Users: Install applications and grant permissions.

[0618] Smartphones: Collect user behavioral data such as location information, calendar information, search history, and order history.

[0619] Cloud Server: Collected data is stored in a cloud database in real time.

[0620] 2. Data storage and preprocessing

[0621] Cloud server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data.

[0622] 3. Training the deep learning model

[0623] Cloud server: Using the pre-processed data, the deep learning model is periodically trained and updated to learn each user's preferences and behavioral patterns.

[0624] 4. User Requests and Analysis

[0625] User: Enters a specific request into the app, such as "I'd like a healthy lunch delivered on January 20th at 12 noon."

[0626] Smartphone: Sends request information to the cloud server.

[0627] Cloud server: Analyzes the request information and generates optimal food delivery suggestions by referencing the user's preferences, past order history, and deep learning models.

[0628] 5. Generate and present food delivery offers

[0629] Cloud server: Generates and validates an action plan including recommended restaurants, dishes, and delivery times.

[0630] Smartphone: Displays suggestions sent from the cloud server to the user.

[0631] Specific examples

[0632] User input: "I'd like a healthy lunch delivered on January 20th at 12 noon."

[0633] Cloud server response: "Recommended plans: Organic salad bowl (Restaurant A), sushi set (Restaurant B), Japanese set meal (Restaurant C)"

[0634] Prompt Sentence Examples

[0635] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[0636] In this way, the present invention allows users to easily find the best food delivery options, greatly improving convenience.

[0637] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0638] Step 1:

[0639] User Permissions and Data Collection

[0640] User: Installs the application and grants permissions to access behavioral data (input: operations; output: permissions).

[0641] Device: Collects behavioral data such as location information, calendar information, search history, and order history (input: user behavior. output: behavioral data).

[0642] Terminal: Sends collected behavioral data to the cloud server in real time (input: behavioral data; output: sending data to the server).

[0643] Step 2:

[0644] Data storage and preprocessing

[0645] Server: Stores the behavioral data sent to the cloud database (input: behavioral data; output: data in the cloud database).

[0646] Server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data (input: data in the cloud database; output: preprocessed data).

[0647] Step 3:

[0648] Training a deep learning model

[0649] Server: Trains a deep learning model using the preprocessed data (Input: preprocessed data. Output: trained model).

[0650] Server: This model is periodically updated to maintain predictive accuracy based on the latest data (input: new behavioral data. output: updated model).

[0651] Step 4:

[0652] User requests and analysis

[0653] User: Enters specific 5W1H information such as "I would like a healthy delivery lunch at 12 o'clock on January 20th" (Input: 5W1H information. Output: Request data).

[0654] Terminal: Sends request information to the cloud server (Input: Request data. Output: Sends request data to the server).

[0655] Server: Analyzes the request information and processes (analyzes) the data by referring to the deep learning model (input: request data & model. output: analysis results).

[0656] Step 5:

[0657] Generate and present food delivery offers

[0658] Server: Generates and validates food delivery plans including recommended restaurants, dishes, and optimal delivery times (Input: Analysis results. Output: Generated food delivery plans).

[0659] Server: Readjusts the plan if necessary and confirms the final proposal (Input: Generated food delivery plan. Output: Confirmed plan).

[0660] Terminal: Shows the proposal sent from the cloud server to the user (Input: Confirmed plan. Output: Display to user).

[0661] Examples:

[0662] Example prompt sentence:

[0663] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[0664] Step 6:

[0665] Selecting and implementing an action plan

[0666] User: Review the proposed food delivery plans in detail and select the appropriate plan (Input: Proposed plans. Output: Selected plan).

[0667] Terminal: Add selected action plan to calendar and set reminder (Input: selected plan. Output: calendar and reminder settings).

[0668] Through this process, users can receive personalized food delivery suggestions efficiently.

[0669] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0670] This invention relates to a system that collects and analyzes user behavioral and emotional data to propose optimal holiday plans. By combining this system with an emotion engine, it is possible to make more personalized proposals that take into account the user's emotions.

[0671] Data collection and preprocessing

[0672] 1. Login and Permissions

[0673] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[0674] Device: When logging in for the first time, a pop-up will appear asking the user to grant permission to access their behavioral data (location, calendar information, search history) and emotional data (facial expressions, voice, text).

[0675] User: Grants access permissions to various data.

[0676] 2. Collecting behavioral and emotional data

[0677] Device: Permitted data (GPS information, calendar events, search history, facial expression data, voice data, text data) is periodically collected and sent to the server in batch processing or real time.

[0678] Server: Stores the received data in a cloud database.

[0679] 3. Data Preprocessing

[0680] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0681] Server: Prepares the preprocessed data as input for the deep learning model and emotion engine.

[0682] Deep learning model and emotion engine training

[0683] 4. Training the model

[0684] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences, and also works with sentiment data to improve the model's accuracy.

[0685] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0686] User requests and analysis

[0687] 5. Request acceptance and analysis

[0688] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0689] Terminal: Sends the input request information to the server.

[0690] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[0691] Generate and present an action plan

[0692] 6. Generate an action plan

[0693] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and ongoing events.

[0694] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[0695] 7. Submitting and Viewing Plans

[0696] Server: Sends the determined action plan to the user's device.

[0697] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0698] 8. Check and select your plan

[0699] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0700] Users: Add the selected plan to their calendar and set a reminder.

[0701] Specific examples

[0702] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[0703] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[0704] 2. Terminal: Sends information to the server.

[0705] 3. Server: Refers to the deep learning model to generate an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes into account the user's real-time emotional data.

[0706] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[0707] 5. Server: Sends the finalized action plan to the device.

[0708] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[0709] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[0710] These steps for implementing the present invention create a system that maximizes the value of a user's holiday and suggests an action plan. The introduction of an emotion engine enables highly personalized suggestions that take into account the user's emotional state.

[0711] The processing flow will be explained below.

[0712] Specific processing steps of the system

[0713] Step 1:

[0714] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[0715] Step 2:

[0716] On your device: When you first log in, a pop-up will appear asking you to grant permission to access your behavioral and emotional data. Behavioral data includes location, calendar information, and search history, while emotional data includes facial expressions, voice, and text.

[0717] Step 3:

[0718] User: Grants access permissions to various data.

[0719] Step 4:

[0720] Device: Acquires authorized data and sends it to the server in batch or real-time, such as GPS information, calendar events, search history, facial expression data, voice data, and text data.

[0721] Step 5:

[0722] Server: Stores the received data in a cloud database.

[0723] Step 6:

[0724] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0725] Step 7:

[0726] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, including incorporating sentiment data to improve the model's accuracy.

[0727] Step 8:

[0728] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0729] Step 9:

[0730] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0731] Step 10:

[0732] Terminal: Sends the input request information to the server.

[0733] Step 11:

[0734] Server: Receives the request information and analyzes it using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[0735] Step 12:

[0736] Server: Generates an optimal action plan based on the user's request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and current events.

[0737] Step 13:

[0738] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[0739] Step 14:

[0740] Server: Sends the determined action plan to the user's device.

[0741] Step 15:

[0742] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0743] Step 16:

[0744] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0745] Step 17:

[0746] Users: Add the selected plan to their calendar and set a reminder.

[0747] In this way, we can realize a system that makes maximum use of user behavioral and emotional data to generate and propose optimal action plans. The introduction of an emotion engine also takes into account the user's psychological state, enabling more personalized proposals.

[0748] Example 2

[0749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] Existing action plan suggestion systems only consider user behavioral data, which means they lack advanced personalization that reflects the user's emotional state. Furthermore, they lack feasibility verification and refinement of the generated action plans, resulting in a poor user experience.

[0751] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data with permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for performing noise removal, missing value completion, and data standardization on the stored raw data, means for training a deep learning model using the preprocessed behavioral data and emotional data, means for analyzing 5W1H information entered by the user and generating an optimal action plan by referring to past behavioral data and emotional data, means for verifying whether the generated action plan is feasible and readjusting it as necessary, and means for transmitting the generated action plan to the user's terminal and displaying it in a visually easy-to-understand manner. This makes it possible to propose a highly personalized action plan that also takes into account the user's emotional state, and to verify and adjust its feasibility.

[0752] "Access permission" refers to the user's permission for an application to access personal data such as location information, calendar information, search history, facial expression data, voice data, and text data.

[0753] A "cloud database" is a type of data storage service that stores and manages data over the Internet, and can handle large amounts of data flexibly and efficiently.

[0754] "Denoising" refers to the process of removing unnecessary or inaccurate data from a dataset.

[0755] "Missing value imputation" refers to the process of filling in missing values ​​in a dataset with an appropriate value (e.g., the mean or median).

[0756] "Data standardization" refers to the process of scaling data so that each data point in a data set is on a consistent scale.

[0757] A "deep learning model" refers to a computational model that uses a multi-layer neural network to learn the characteristics of data and perform tasks such as prediction and classification.

[0758] "5W1H information" typically refers to information that consists of the elements of "When," "Where," "Who," "What," "Why," and "How."

[0759] "Emotion data" is information that indicates the emotional state of the user, and includes facial expression data, voice data, text data, and the like.

[0760] "Behavioral data" refers to information about a user's behavior, including location information, calendar information, search history, etc.

[0761] An "action plan" refers to a schedule or proposal of specific activities to be carried out within a time period specified by the user.

[0762] "Feasibility" refers to the process of verifying whether the generated action plan can be carried out in reality (for example, whether reservations can be made or whether events will be held).

[0763] An "emotion engine" refers to an algorithm or system that analyzes a user's emotional data and generates an appropriate action plan based on that emotional state.

[0764] "Visually easy-to-understand display" refers to displaying information in a format that is easy for users to understand, including timeline displays and map displays.

[0765] This invention is a system that collects and analyzes user behavioral and emotional data and proposes optimal action plans. This system uses a cloud database, deep learning models, and an emotion engine to enable advanced personalization that reflects the user's emotional state.

[0766] Data collection and preprocessing

[0767] 1. Login and Permissions

[0768] User: Installs the app and logs in with a social media account or email address.

[0769] Device: After logging in, a pop-up will appear requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expression data, voice data, text data).

[0770] User: Grants access permissions to various data.

[0771] 2. Collecting behavioral and emotional data

[0772] Device: Periodically collects and transmits permitted data to a server, including GPS information, calendar events, search history, facial expression data, voice data, and text data.

[0773] Server: Stores the received data in a cloud database (e.g., AWS DynamoDB).

[0774] 3. Data Preprocessing

[0775] Server: Performs noise removal (e.g., removing inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling).

[0776] Deep learning model and emotion engine training

[0777] 4. Training the model

[0778] Server: Trains deep learning models using preprocessed data. Builds models using TensorFlow or PyTorch.

[0779] Server: Split the dataset into training and validation datasets to improve the accuracy of the model.

[0780] Server: Retrain the model periodically to incorporate new data, for example, on the first day of every month.

[0781] User requests and analysis

[0782] 5. Request acceptance and analysis

[0783] User: Enter a specific request for how to spend the day off into the app. Example: "Date in Tachikawa from 10am to 9pm on January 20th."

[0784] Terminal: Sends the input request information to the server.

[0785] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[0786] Generate and present an action plan

[0787] 6. Generate an action plan

[0788] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events."

[0789] Server: Validates the feasibility of the generated plan (e.g., whether reservations are available or events are running) and adjusts the plan if necessary. Retrieves reservation and event information using the OpenTable API and Eventbrite API.

[0790] 7. Submitting and Viewing Plans

[0791] Server: Sends the determined action plan to the user's device.

[0792] Device: Display the proposed action plan visually, for example as a timeline or points on a map.

[0793] 8. Check and select your plan

[0794] User: Review the suggested action plans and adjust accordingly. Select the plan you like best and add it to your calendar.

[0795] Users: Set reminders for selected plans so you can receive timely notifications.

[0796] Specific examples

[0797] Example 1: "Date in Tachikawa from 10am to 9pm on January 20th"

[0798] 1. User: Enters a request into the app: "Date in Tachikawa from 10am to 9pm on January 20th."

[0799] 2. Terminal: Sends the request to the server.

[0800] 3. Server: Using a deep learning model, it generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes real-time emotion data into account.

[0801] 4. Server: Validates the feasibility of the generated plan and adjusts it if necessary.

[0802] 5. Server: Sends the finalized action plan to the device.

[0803] 6. Terminal: Display the proposed plan as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00."

[0804] 7. User: Review the plans, adopt the one they like, add it to their calendar, and set a reminder.

[0805] Examples of prompt statements

[0806] Example 1: "Please suggest a date plan in Tachikawa from 10:00 to 21:00 on January 20th."

[0807] Example 2: "What is your plan of action for relaxing on the weekend?"

[0808] This allows users to get a highly personalized holiday action plan that takes into account their emotional state.

[0809] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0810] Step 1: Login and Permissions

[0811] User: Installs the app and logs in using a social media account or email address. The input is account information and the output is a login session.

[0812] Terminal: After logging in, a popup is displayed requesting permission to access behavioral and emotional data. The input is the login session, and the output is the popup screen.

[0813] User: Set the access permission for various data to "Allow". The input is a popup screen, and the output is the permission status.

[0814] Step 2: Collect behavioral and emotional data

[0815] Terminal: Periodically collects authorized data (GPS information, calendar events, search history, facial expression data, voice data, text data). The input is the authorization status, and the output is the collected data.

[0816] Terminal: Sends collected data to the server in batch processing or real time. For example, when the amount of data exceeds a certain threshold, the data is sent. The input is the collected data, and the output is the data sent to the server.

[0817] Server: Stores the received data in a cloud database. The input is the transmitted data, and the output is the data stored in the database.

[0818] Step 3: Preprocessing the data

[0819] Server: Performs noise removal (removal of inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling). Input is raw data, and output is preprocessed data.

[0820] Server: Prepares the preprocessed data as input data for the deep learning model and emotion engine. The input is the preprocessed data, and the output is the input data for the model.

[0821] Step 4: Training the model

[0822] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. The input is the preprocessed data, and the output is the trained model.

[0823] Server: Splits the dataset into training and validation datasets to improve the accuracy of the model. The input is the dataset, and the output is the evaluation of the model accuracy.

[0824] Server: Periodically ingests new data and retrains the model. For example, retraining occurs on the first day of each month. The input is the new data, and the output is the updated model.

[0825] Step 5: Accepting and parsing the request

[0826] User: Enters a specific request for how to spend the day off into the app. For example, a request might be "Date in Tachikawa from 10:00 to 21:00 on January 20th." The input is the request information, and the output is sending the request to the device.

[0827] Terminal: Sends the input request information to the server. The input is the request information, and the output is the data to be sent to the server.

[0828] Server: Analyzes the request information and performs analysis using a deep learning model and emotion engine. This also refers to past behavioral history, preference data, and real-time emotion data. The input is the request and reference data, and the output is the analysis results.

[0829] Step 6: Generate an action plan

[0830] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotion data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events." The input is the request and model data, and the output is an action plan.

[0831] Server: Validates whether the generated plan is feasible (reservation availability and event status), and adjusts the plan as necessary. For example, it obtains reservation and event information using the OpenTable API or Eventbrite API. The input is the action plan, and the output is the finalized plan.

[0832] Step 7: Submit and view your plan

[0833] Server: Sends the finalized action plan to the user's terminal. The input is the finalized plan, and the output is the transmitted data.

[0834] Terminal: A device that visually displays the proposed action plan, for example, in a timeline format or as points on a map. The input is the transmitted data, and the output is the displayed information.

[0835] Step 8: Review and select your plan

[0836] User: Review the proposed action plans and adjust accordingly. Select the plan they like and add it to their calendar. Input is display information, output is calendar event.

[0837] User: Set a reminder for the selected plan and receive notifications. Input is a calendar event, output is a reminder setting.

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Conventional itinerary suggestion systems are based solely on user behavioral data, making it difficult to provide highly personalized suggestions that take into account the user's emotions and moods. Furthermore, because these systems are primarily focused on travel and daily activity planning, they do not offer functionality to link with autonomous driving systems to suggest optimal driving routes and stopover spots.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data after obtaining access permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for training a deep learning model using preprocessed behavioral data and emotional data, means for analyzing specific request information entered by the user, means for generating an optimal action plan by referencing past behavioral data and emotional data and the deep learning model, and means for cooperating with an autonomous driving system to suggest optimal driving routes and stopover spots. This makes it possible to propose highly personalized action plans that take the user's emotional state into consideration, and also makes it possible to provide optimal driving plans using autonomous driving vehicles.

[0842] "User Permission" is the consent an application requests from a user to allow the application to access their behavioral and emotional data.

[0843] "Behavioral Data" refers to information related to a user's behavior, such as the user's location, calendar information, and search history.

[0844] "Emotional data" refers to information about a user's emotional state, such as their facial expressions, voice, or text.

[0845] "Cloud Database" means a database accessible over the internet that is used to store and manage behavioral and emotional data.

[0846] "Preprocessing" refers to the process of removing noise, filling in missing values, standardizing, and otherwise processing the collected raw data to prepare it in a format suitable for training a deep learning model.

[0847] A "deep learning model" is a machine learning model that uses deep learning technology to analyze a user's behavioral patterns and emotional state and generate an action plan.

[0848] "Specific request information" refers to the specific action plan or request information that the user enters into the application, including 5W1H information such as date, time, location, purpose, and desired activity.

[0849] The "optimal action plan" is a plan that proposes the most suitable action plan for the user based on the user's behavioral and emotional data.

[0850] An "autonomous driving system" is a system that uses artificial intelligence and sensor technology to automate driving and operate vehicles without a human driver.

[0851] A "driving route" is the route an autonomous vehicle will travel, and is part of a plan that includes destinations and stopovers requested by the user.

[0852] "Stopover spots" are places where users are encouraged to stop during their drive, such as scenic spots, restaurants, cafes, and event venues.

[0853] This invention relates to a system that collects and analyzes user behavioral and emotional data, and then works in conjunction with an autonomous driving system to suggest optimal driving routes and stopover spots. This system uses a wide range of devices, including smartphones and autonomous vehicles.

[0854] System configuration and operation overview

[0855] User Device

[0856] 1. Login and Permissions:

[0857] Users install the application on their smartphone and log in the first time they start it, using their social media account or email address.

[0858] When logging in for the first time, the device will display a pop-up requesting the user to grant permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text).

[0859] Users grant access permissions for various data.

[0860] 2. Data Collection:

[0861] The device periodically acquires permitted data (GPS information, calendar events, search history, facial expression data, voice data, and text data) and transmits it to the server in real time or in batches.

[0862] server

[0863] 3. Data preprocessing:

[0864] The server performs preprocessing such as noise removal, missing value completion, and data standardization on the raw data stored in the cloud database.

[0865] The preprocessed data is prepared as input for the deep learning model and emotion engine. The data is managed by a deep learning framework (e.g., TensorFlow or PyTorch).

[0866] 4. Train the model:

[0867] The deep learning model uses pre-processed data to learn user behavior patterns and preferences, and also incorporates sentiment data to improve the model's accuracy.

[0868] The model is periodically retrained to incorporate new data and improve its accuracy.

[0869] 5. Driving route generation and suggestions:

[0870] When a user enters a specific driving plan request into the app (for example, "I want to go on a four-hour drive with a friend next Sunday"), the device sends that information to the server.

[0871] The server performs analysis using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[0872] Based on the request, the server works with the autonomous driving system to generate the optimal driving route and stopover spots.

[0873] The generated plan is verified for feasibility and adjusted if necessary, and this information is sent to the user's device for presentation.

[0874] 6. Displaying Proposals:

[0875] The device will clearly display the suggested driving route and stops to the user, who can then review the proposed plan, adjust accordingly, and select. The selected plan will be added to the calendar and a reminder will be set.

[0876] Specific examples

[0877] Consider a case where a user requests, "I want to enjoy a four-hour drive with a friend next Sunday." In this case, the server generates the following driving plan based on the user's past behavioral data, emotional data, and real-time data.

[0878] Driving route: "Leave home at 10:00 → Visit a scenic park at 10:30 → Have lunch at Restaurant A at 12:00 → Have lunch at Event B at 14:00 → Arrive home at 16:00"

[0879] Users can review the suggestions and, if they like them, add them to their calendar and set a reminder.

[0880] An example prompt is:

[0881] I'd like to go on a 4-hour drive with a friend next Sunday. Can you suggest a good route and places to stop?

[0882] In this way, a system will be realized that can provide highly personalized action plans based on user behavioral and emotional data, and propose optimal driving plans using autonomous vehicles.

[0883] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0884] Step 1:

[0885] Logging in a user and getting permissions

[0886] A user installs a smartphone application and logs in the first time they start it. They log in using a social media account or email address. When the user logs in for the first time, the device displays a pop-up requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text). If the user agrees, access to various data is granted. The input to this step is the user's login information and consent information, and the output is the status of data access permission.

[0887] Step 2:

[0888] Data collection

[0889] The device periodically acquires authorized behavioral data (GPS information, calendar events, search history) and emotional data (facial expression data, voice data, text data). These data are sent to the server in real time or batch processing. The input of this step is the user's behavioral data and emotional data, and the output is the raw data sent to the server.

[0890] Step 3:

[0891] Data Preprocessing

[0892] The server performs preprocessing on the raw data stored in the cloud database, such as noise removal, missing value imputation, and data standardization, which makes the data suitable for training deep learning models and emotion engines. The input of this step is raw data, and the output is preprocessed data.

[0893] Step 4:

[0894] Training a deep learning model

[0895] The server uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. It also incorporates sentiment data to improve the model's accuracy. The model is periodically retrained by incorporating new data to improve its accuracy. The input of this step is the preprocessed data, and the output is the trained deep learning model.

[0896] Step 5:

[0897] Request input and analysis

[0898] The user inputs a specific driving plan request (for example, "I want to enjoy a four-hour drive with a friend next Sunday") into the app. The device sends the input request information to the server. The server analyzes this request information using a deep learning model and emotion engine. This analysis also references past behavioral history, preference data, and real-time emotion data. The input of this step is the user's request, and the output is the analysis result.

[0899] Step 6:

[0900] Generate driving routes and stopovers

[0901] The server generates an optimal driving route and stopovers based on the request, learned behavioral pattern data, and emotion data. This includes driving routes and stopovers (scenic spots, restaurants, cafes, event venues, etc.). The server then verifies the feasibility of this generated plan (checking event availability and reservations), adjusts the plan as necessary, and prepares a finalized plan. The input of this step is the analysis result of the deep learning model, and the output is the optimal driving plan.

[0902] Step 7:

[0903] Submitting and Viewing Plans

[0904] The server sends the confirmed driving plan to the user's device. The device displays the proposed driving plan in an easy-to-understand manner to the user. For example, a schedule such as "Leave home at 10:00 → Visit a scenic park at 10:30 → Lunch (Restaurant A) at 12:00 → Lunch event (Event B) at 14:00 → Arrive home at 16:00" is displayed. The user reviews this proposed plan, and if they like it, it is added to their calendar and a reminder is set. The input of this step is the confirmed driving plan, and the output is the plan information displayed on the user's device.

[0905] The above steps realize a system that provides highly personalized, optimal driving plans using autonomous vehicles based on user behavioral and emotional data.

[0906] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0907] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0908] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0909] [Third embodiment]

[0910] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0911] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0913] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0917] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0918] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0919] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0920] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0921] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0922] The present invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented in the following steps.

[0923] Data collection and preprocessing

[0924] 1. Login and Permissions

[0925] User: After installing the app and logging in for the first time, the user grants permission for the app to access their behavioral data (location, calendar information, search history, etc.).

[0926] Terminal: Prepares to periodically collect various data permitted by the user and sends that data to the server.

[0927] 2. Data storage and preprocessing

[0928] Server: Collected behavioral data is stored in a cloud database, where it is preprocessed by noise removal and missing value imputation.

[0929] Training a deep learning model

[0930] 3. Training the model

[0931] Server: Trains a deep learning model using the preprocessed data. This model learns the unique behavioral patterns and preferences of each user. This training is performed periodically, and the accuracy of the model improves as the amount of data increases.

[0932] User requests and analysis

[0933] 4. Request acceptance and analysis

[0934] User: Enter specific 5W1H information into the app, such as "Date in Tachikawa from 10am to 9pm on January 20th."

[0935] Terminal: Sends the input request information to the server.

[0936] Server: Analyzes the request information and generates an optimal action plan by referring to user behavior data and deep learning models.

[0937] Generate and present an action plan

[0938] 5. Plan Generation and Verification

[0939] Server: Based on the user's preferences and past behavioral data, the server generates a specific action plan by combining event information, restaurants, transportation options, etc. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll around the park at 15:00 → Dinner at Restaurant B at 18:00" may be created.

[0940] Server: Validates whether the generated plan is feasible, adjusts the plan if necessary, and finalizes the action plan.

[0941] 6. Plan presentation and selection

[0942] Server: Sends the determined action plan to the user's device.

[0943] On the device: The proposed action plan is displayed to the user, who can review the details of the proposed plan and make changes if necessary.

[0944] Users can select their favorite action plans and add them to their calendar within the app. They can also set reminders to ensure they don't forget to follow through with their plans.

[0945] Specific examples

[0946] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[0947] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[0948] 2. Terminal: Sends information to the server.

[0949] 3. Server: Refers to the deep learning model and generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa.

[0950] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[0951] 5. Server: Sends the finalized action plan to the device.

[0952] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[0953] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[0954] These steps of implementing the present invention allow users to easily find the perfect holiday experience and maximize its value.

[0955] The processing flow will be explained below.

[0956] Specific processing steps of the system

[0957] Step 1: Log in and get permissions

[0958] User: After installing the app for the first time, launch the app and access the login screen. Log in with your social media account or email address.

[0959] Device: When logging in for the first time, a popup will appear asking the user for permission to collect behavioral data.

[0960] Users: Grant data permissions such as location, calendar information, and search history.

[0961] Step 2: Collecting and sending behavioral data

[0962] On the device: Permitted data (GPS information, calendar events, search history) is periodically retrieved and sent to the server in batch or real-time.

[0963] Server: Stores the received data in a cloud database.

[0964] Step 3: Preprocessing the data

[0965] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[0966] Server: Prepares the preprocessed data as input for deep learning models.

[0967] Step 4: Train the deep learning model

[0968] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, which then acts as a custom model for each user.

[0969] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[0970] Step 5: User Request Input

[0971] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[0972] Terminal: Sends the input request information to the server.

[0973] Step 6: Parsing the request information

[0974] Server: Receives user request information and analyzes it using a deep learning model, while also referencing past behavioral history and preference data.

[0975] Step 7: Generate an action plan

[0976] Server: Generates an optimal action plan based on the request and learned behavioral pattern data, for example, by combining information on date spots, restaurants, and current events.

[0977] Server: Verifies the feasibility of the generated plan (e.g., whether reservations are possible or the event is being held).

[0978] Server: Readjust the plan as needed and prepare a finalized plan.

[0979] Step 8: Submit and view your plan

[0980] Server: Sends the determined action plan to the user's device.

[0981] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[0982] Step 9: Review and select your plan

[0983] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[0984] Users: Add the selected plan to their calendar and set a reminder.

[0985] The above are the specific processing steps in the system of the present invention, which is designed to maximize the value of a user's holiday and propose an optimal activity plan.

[0986] Example 1

[0987] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0988] In modern society, it is important to provide optimal activity plans for individual users so that they can make effective use of their limited holidays. However, conventional systems have had difficulty effectively collecting and analyzing user behavioral data to generate highly accurate activity plans. Furthermore, issues remain, such as verifying the feasibility of generated plans and providing a reminder function to ensure users do not forget to execute them.

[0989] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0990] In this invention, the server includes a means for collecting behavioral data with permission from the user, a means for storing the collected behavioral data in a cloud database, and a means for performing preprocessing on the stored behavioral data, such as noise removal and missing value imputation. This enables the generation of highly accurate behavior plans. The system also includes a means for training a deep learning model using the preprocessed behavioral data, a means for analyzing 5W1H information entered by the user, and a means for generating an optimal behavior plan by referencing past behavioral data and the deep learning model. The system also includes a means for verifying whether the generated behavior plan is feasible and for making adjustments as necessary, and a means for sending and displaying the finalized behavior plan to the user's device. This allows users to easily plan their optimal holiday and maximize its value. The system also includes a means for setting a reminder function, helping users remember to carry out their behavior plan.

[0991] "Access permission" means that a user grants an application or system access to their data.

[0992] "Behavioral data" refers to data related to a user's daily behavior and activities, such as location information, calendar information, and search history.

[0993] A "cloud database" is a remote database management system that can be accessed via the Internet, and is a means of efficiently storing and managing large amounts of data.

[0994] "Preprocessing" refers to the process of processing raw data into a form suitable for data analysis and training machine learning models, such as removing noise and filling in missing values.

[0995] A "deep learning model" refers to an algorithm that uses a multi-layer neural network to learn the characteristics of data and perform advanced analysis and predictions.

[0996] "5W1H information" refers to information entered by the user regarding "when, where, who, what, why, and how."

[0997] An "action plan" refers to a schedule that specifically plans activities or events that should be carried out on a user's holidays or at specific times.

[0998] The "reminder function" refers to a function that notifies users so that they do not forget their schedules or plans.

[0999] "User device" means an electronic device that is directly operated by a user, such as a smartphone or tablet.

[1000] This invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented by the following detailed process.

[1001] First, the user installs the application and grants permission to access behavioral data such as location information, calendar information, and search history when logging in for the first time. The device periodically collects the data permitted by the user and sends it to the server.

[1002] The server then stores the received behavioral data in a cloud database, using a remote database management system such as Amazon RDS or Google Cloud SQL. The collected data is then preprocessed, including noise removal and missing value imputation, to improve data quality and make it suitable for training deep learning models.

[1003] The server uses the preprocessed data to train a deep learning model using machine learning frameworks such as TensorFlow and PyTorch. This deep learning model learns user behavior patterns and preferences and periodically updates its accuracy.

[1004] When a user enters specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th," the device sends this information to the server. The server analyzes the information and generates an optimal action plan by referencing a pre-trained deep learning model and past behavioral data.

[1005] The server verifies various conditions (e.g., business hours and traffic information) to ensure that the generated plan is feasible. If necessary, it readjusts the plan and sends the finalized plan to the device. The device then displays this plan to the user, who can review the proposed plan and make changes if necessary.

[1006] Additionally, the finalized action plan is added to the calendar and a reminder function is set, so users can remember to carry out their plan.

[1007] As a concrete example, if a user inputs a request for "a date in Tachikawa from 10:00 to 21:00 on January 20th," the system generates the following prompt:

[1008] Example prompt sentence:

[1009] "Please create a specific action plan for a date in Tachikawa from 10:00 to 21:00 on January 20th."

[1010] As described above, this invention provides an optimal action plan tailored to individual needs based on the user's behavioral data, enabling the user to spend their holidays efficiently and productively.

[1011] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1012] Step 1:

[1013] The user installs the application and grants permission to access behavioral data such as location, calendar information, and search history when logging in for the first time. Based on this input data, the device authenticates the user and prepares access to various data. The output is an authentication token and authorized data access.

[1014] Step 2:

[1015] The device periodically collects user behavioral data such as location information, calendar information, and search history. Specifically, it obtains location information using GPS, schedule information via a calendar API, and browser history data. The collected data is sent to a server. The input in this step is the user's behavioral data, and the output is the collected behavioral dataset.

[1016] Step 3:

[1017] The server stores the received behavioral data in a cloud database, typically using Amazon RDS or Google Cloud SQL. A timestamp is added when the data is saved, and checks are performed to ensure data consistency and integrity. This input data is the collected behavioral data, and is saved as output in the cloud database.

[1018] Step 4:

[1019] The server performs preprocessing on the stored behavioral data, such as noise removal and missing value imputation. For example, it runs filtering algorithms to remove outliers and applies imputation algorithms to fill in missing data. The input data is the stored behavioral data, and the output is a preprocessed, clean dataset.

[1020] Step 5:

[1021] The server uses the preprocessed data to train a deep learning model. For training, the model is implemented using TensorFlow or PyTorch to learn each user's behavioral patterns and preferences. The input data is the preprocessed data, and the output is a trained deep learning model.

[1022] Step 6:

[1023] The user inputs specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th." The input information is retrieved through the application interface. This input information is then sent to the server for further analysis, becoming a specific user request.

[1024] Step 7:

[1025] The server analyzes the input 5W1H information and generates an optimal action plan by referencing a deep learning model and past behavioral data. For example, it combines information on date spots around Tachikawa, good restaurants, and ongoing events. The input data is the 5W1H information and the user's behavioral data, and the output is the generated action plan.

[1026] Step 8:

[1027] The server verifies the generated action plan based on business hours and traffic information to confirm whether it is feasible. If necessary, the action plan is readjusted and finally confirmed. The input data is the generated action plan and various verification data, and the output is a feasible and confirmed action plan.

[1028] Step 9:

[1029] The server sends the finalized action plan to the user's device. The device displays the received action plan to the user, who then checks the proposed plan and makes changes if necessary. The input in this step is the finalized action plan, and the output is the action plan displayed on the user's device.

[1030] Step 10:

[1031] The user adds the finalized action plan to the calendar in the app. The device sets a reminder for the event added to the calendar. The reminder notifies the user so that they can remember to carry out the event. The input data is the finalized action plan, and the output is the event added to the calendar and the reminder setting.

[1032] (Application example 1)

[1033] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1034] Previously, systems existed that generated personalized action plans using user preferences and past behavioral data. However, in the case of food delivery services, these systems were unable to effectively incorporate a user's specific preferences and past ordering history into their proposals with sufficient accuracy. Furthermore, in the case of food delivery, the system was unable to provide sufficient recommendations for optimal restaurants and delivery times, making it difficult to provide a satisfying service. Therefore, it is necessary to improve the accuracy and personalization of systems to improve user convenience.

[1035] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1036] In this invention, the server includes means for collecting behavioral data with permission from the user, means for storing the collected behavioral data in a cloud database, means for training a deep learning model using the preprocessed behavioral data, means for analyzing 5W1H information entered by the user, means for generating an optimal action plan by referring to past behavioral data and the deep learning model, means for transmitting and displaying the generated action plan to the user's terminal, and means for suggesting the optimal food delivery based on the user's preferences and past order history, thereby enabling users to easily enjoy highly satisfying food delivery.

[1037] - "Permission" means that the user gives the application permission to collect and use behavioral data.

[1038] "Behavioral data" refers to data that indicates a user's behavior, such as the user's location information, calendar information, search history, and order history.

[1039] "Cloud database" means a database accessible via the internet that is used to store behavioral data.

[1040] "Preprocessing" refers to the process of performing processes such as noise removal and missing value completion on collected data to prepare it in a format suitable for analysis and model learning.

[1041] A "deep learning model" is a machine learning model that uses a multi-layer neural network to learn complex data patterns and perform predictions and classifications.

[1042] "5W1H information" refers to the basic elements of information such as Who, What, When, Where, Why, and How.

[1043] "Analysis" refers to the process of processing and calculating collected data and input information for a specific purpose to derive meaningful results.

[1044] An "action plan" is a specific schedule or plan suggested based on the user's actions.

[1045] A "terminal" is a device that can be directly operated by a user, such as a smartphone or tablet.

[1046] "Preferences" refer to a user's preferences, interests, and concerns.

[1047] "Order History" is a record of orders placed by a user in the past.

[1048] "Food delivery" is a service that delivers meals from restaurants or food services to a location specified by the user.

[1049] System Overview

[1050] This invention is a system that collects and analyzes user behavior data and suggests optimal food delivery options based on the user's preferences and past order history. This system is primarily configured using a cloud-based database and deep learning model.

[1051] Hardware and software used

[1052] Hardware

[1053] Smartphone: The user installs and uses the application.

[1054] Cloud servers (e.g., AWS EC2 and Google Cloud Platform): Analyze data and train models.

[1055] software

[1056] Database (e.g., Firebase, AWS DynamoDB): Stores collected behavioral data.

[1057] Deep learning frameworks (e.g., TensorFlow, PyTorch): Model user behavior patterns and preferences and make predictions.

[1058] Detailed processing flow

[1059] 1. User Permissions and Data Collection

[1060] Users: Install applications and grant permissions.

[1061] Smartphones: Collect user behavioral data such as location information, calendar information, search history, and order history.

[1062] Cloud Server: Collected data is stored in a cloud database in real time.

[1063] 2. Data storage and preprocessing

[1064] Cloud server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data.

[1065] 3. Training the deep learning model

[1066] Cloud server: Using the pre-processed data, the deep learning model is periodically trained and updated to learn each user's preferences and behavioral patterns.

[1067] 4. User Requests and Analysis

[1068] User: Enters a specific request into the app, such as "I'd like a healthy lunch delivered on January 20th at 12 noon."

[1069] Smartphone: Sends request information to the cloud server.

[1070] Cloud server: Analyzes the request information and generates optimal food delivery suggestions by referencing the user's preferences, past order history, and deep learning models.

[1071] 5. Generate and present food delivery offers

[1072] Cloud server: Generates and validates an action plan including recommended restaurants, dishes, and delivery times.

[1073] Smartphone: Displays suggestions sent from the cloud server to the user.

[1074] Specific examples

[1075] User input: "I'd like a healthy lunch delivered on January 20th at 12 noon."

[1076] Cloud server response: "Recommended plans: Organic salad bowl (Restaurant A), sushi set (Restaurant B), Japanese set meal (Restaurant C)"

[1077] Prompt Sentence Examples

[1078] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[1079] In this way, the present invention allows users to easily find the best food delivery options, greatly improving convenience.

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] User Permissions and Data Collection

[1083] User: Installs the application and grants permissions to access behavioral data (input: operations; output: permissions).

[1084] Device: Collects behavioral data such as location information, calendar information, search history, and order history (input: user behavior. output: behavioral data).

[1085] Terminal: Sends collected behavioral data to the cloud server in real time (input: behavioral data; output: sending data to the server).

[1086] Step 2:

[1087] Data storage and preprocessing

[1088] Server: Stores the behavioral data sent to the cloud database (input: behavioral data; output: data in the cloud database).

[1089] Server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data (input: data in the cloud database; output: preprocessed data).

[1090] Step 3:

[1091] Training a deep learning model

[1092] Server: Trains a deep learning model using the preprocessed data (Input: preprocessed data. Output: trained model).

[1093] Server: This model is periodically updated to maintain predictive accuracy based on the latest data (input: new behavioral data. output: updated model).

[1094] Step 4:

[1095] User requests and analysis

[1096] User: Enters specific 5W1H information such as "I would like a healthy delivery lunch at 12 o'clock on January 20th" (Input: 5W1H information. Output: Request data).

[1097] Terminal: Sends request information to the cloud server (Input: Request data. Output: Sends request data to the server).

[1098] Server: Analyzes the request information and processes (analyzes) the data by referring to the deep learning model (input: request data & model. output: analysis results).

[1099] Step 5:

[1100] Generate and present food delivery offers

[1101] Server: Generates and validates food delivery plans including recommended restaurants, dishes, and optimal delivery times (Input: Analysis results. Output: Generated food delivery plans).

[1102] Server: Readjusts the plan if necessary and confirms the final proposal (Input: Generated food delivery plan. Output: Confirmed plan).

[1103] Terminal: Shows the proposal sent from the cloud server to the user (Input: Confirmed plan. Output: Display to user).

[1104] Examples:

[1105] Example prompt sentence:

[1106] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[1107] Step 6:

[1108] Selecting and implementing an action plan

[1109] User: Review the proposed food delivery plans in detail and select the appropriate plan (Input: Proposed plans. Output: Selected plan).

[1110] Terminal: Add selected action plan to calendar and set reminder (Input: selected plan. Output: calendar and reminder settings).

[1111] Through this process, users can receive personalized food delivery suggestions efficiently.

[1112] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1113] This invention relates to a system that collects and analyzes user behavioral and emotional data to propose optimal holiday plans. By combining this system with an emotion engine, it is possible to make more personalized proposals that take into account the user's emotions.

[1114] Data collection and preprocessing

[1115] 1. Login and Permissions

[1116] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[1117] Device: When logging in for the first time, a pop-up will appear asking the user to grant permission to access their behavioral data (location, calendar information, search history) and emotional data (facial expressions, voice, text).

[1118] User: Grants access permissions to various data.

[1119] 2. Collecting behavioral and emotional data

[1120] Device: Permitted data (GPS information, calendar events, search history, facial expression data, voice data, text data) is periodically collected and sent to the server in batch processing or real time.

[1121] Server: Stores the received data in a cloud database.

[1122] 3. Data Preprocessing

[1123] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[1124] Server: Prepares the preprocessed data as input for the deep learning model and emotion engine.

[1125] Deep learning model and emotion engine training

[1126] 4. Training the model

[1127] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences, and also works with sentiment data to improve the model's accuracy.

[1128] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[1129] User requests and analysis

[1130] 5. Request acceptance and analysis

[1131] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[1132] Terminal: Sends the input request information to the server.

[1133] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[1134] Generate and present an action plan

[1135] 6. Generate an action plan

[1136] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and ongoing events.

[1137] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[1138] 7. Submitting and Viewing Plans

[1139] Server: Sends the determined action plan to the user's device.

[1140] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[1141] 8. Check and select your plan

[1142] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[1143] Users: Add the selected plan to their calendar and set a reminder.

[1144] Specific examples

[1145] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[1146] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[1147] 2. Terminal: Sends information to the server.

[1148] 3. Server: Refers to the deep learning model to generate an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes into account the user's real-time emotional data.

[1149] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[1150] 5. Server: Sends the finalized action plan to the device.

[1151] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[1152] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[1153] These steps for implementing the present invention create a system that maximizes the value of a user's holiday and suggests an action plan. The introduction of an emotion engine enables highly personalized suggestions that take into account the user's emotional state.

[1154] The processing flow will be explained below.

[1155] Specific processing steps of the system

[1156] Step 1:

[1157] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[1158] Step 2:

[1159] On your device: When you first log in, a pop-up will appear asking you to grant permission to access your behavioral and emotional data. Behavioral data includes location, calendar information, and search history, while emotional data includes facial expressions, voice, and text.

[1160] Step 3:

[1161] User: Grants access permissions to various data.

[1162] Step 4:

[1163] Device: Acquires authorized data and sends it to the server in batch or real-time, such as GPS information, calendar events, search history, facial expression data, voice data, and text data.

[1164] Step 5:

[1165] Server: Stores the received data in a cloud database.

[1166] Step 6:

[1167] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[1168] Step 7:

[1169] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, including incorporating sentiment data to improve the model's accuracy.

[1170] Step 8:

[1171] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[1172] Step 9:

[1173] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[1174] Step 10:

[1175] Terminal: Sends the input request information to the server.

[1176] Step 11:

[1177] Server: Receives the request information and analyzes it using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[1178] Step 12:

[1179] Server: Generates an optimal action plan based on the user's request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and current events.

[1180] Step 13:

[1181] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[1182] Step 14:

[1183] Server: Sends the determined action plan to the user's device.

[1184] Step 15:

[1185] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[1186] Step 16:

[1187] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[1188] Step 17:

[1189] Users: Add the selected plan to their calendar and set a reminder.

[1190] In this way, we can realize a system that makes maximum use of user behavioral and emotional data to generate and propose optimal action plans. The introduction of an emotion engine also takes into account the user's psychological state, enabling more personalized proposals.

[1191] Example 2

[1192] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1193] Existing action plan suggestion systems only consider user behavioral data, which means they lack advanced personalization that reflects the user's emotional state. Furthermore, they lack feasibility verification and refinement of the generated action plans, resulting in a poor user experience.

[1194] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data with permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for performing noise removal, missing value completion, and data standardization on the stored raw data, means for training a deep learning model using the preprocessed behavioral data and emotional data, means for analyzing 5W1H information entered by the user and generating an optimal action plan by referring to past behavioral data and emotional data, means for verifying whether the generated action plan is feasible and readjusting it as necessary, and means for transmitting the generated action plan to the user's terminal and displaying it in a visually easy-to-understand manner. This makes it possible to propose a highly personalized action plan that also takes into account the user's emotional state, and to verify and adjust its feasibility.

[1195] "Access permission" refers to the user's permission for an application to access personal data such as location information, calendar information, search history, facial expression data, voice data, and text data.

[1196] A "cloud database" is a type of data storage service that stores and manages data over the Internet, and can handle large amounts of data flexibly and efficiently.

[1197] "Denoising" refers to the process of removing unnecessary or inaccurate data from a dataset.

[1198] "Missing value imputation" refers to the process of filling in missing values ​​in a dataset with an appropriate value (e.g., the mean or median).

[1199] "Data standardization" refers to the process of scaling data so that each data point in a data set is on a consistent scale.

[1200] A "deep learning model" refers to a computational model that uses a multi-layer neural network to learn the characteristics of data and perform tasks such as prediction and classification.

[1201] "5W1H information" typically refers to information that consists of the elements of "When," "Where," "Who," "What," "Why," and "How."

[1202] "Emotion data" is information that indicates the emotional state of the user, and includes facial expression data, voice data, text data, and the like.

[1203] "Behavioral data" refers to information about a user's behavior, including location information, calendar information, search history, etc.

[1204] An "action plan" refers to a schedule or proposal of specific activities to be carried out within a time period specified by the user.

[1205] "Feasibility" refers to the process of verifying whether the generated action plan can be carried out in reality (for example, whether reservations can be made or whether events will be held).

[1206] An "emotion engine" refers to an algorithm or system that analyzes a user's emotional data and generates an appropriate action plan based on that emotional state.

[1207] "Visually easy-to-understand display" refers to displaying information in a format that is easy for users to understand, including timeline displays and map displays.

[1208] This invention is a system that collects and analyzes user behavioral and emotional data and proposes optimal action plans. This system uses a cloud database, deep learning models, and an emotion engine to enable advanced personalization that reflects the user's emotional state.

[1209] Data collection and preprocessing

[1210] 1. Login and Permissions

[1211] User: Installs the app and logs in with a social media account or email address.

[1212] Device: After logging in, a pop-up will appear requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expression data, voice data, text data).

[1213] User: Grants access permissions to various data.

[1214] 2. Collecting behavioral and emotional data

[1215] Device: Periodically collects and transmits permitted data to a server, including GPS information, calendar events, search history, facial expression data, voice data, and text data.

[1216] Server: Stores the received data in a cloud database (e.g., AWS DynamoDB).

[1217] 3. Data Preprocessing

[1218] Server: Performs noise removal (e.g., removing inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling).

[1219] Deep learning model and emotion engine training

[1220] 4. Training the model

[1221] Server: Trains deep learning models using preprocessed data. Builds models using TensorFlow or PyTorch.

[1222] Server: Split the dataset into training and validation datasets to improve the accuracy of the model.

[1223] Server: Retrain the model periodically to incorporate new data, for example, on the first day of every month.

[1224] User requests and analysis

[1225] 5. Request acceptance and analysis

[1226] User: Enter a specific request for how to spend the day off into the app. Example: "Date in Tachikawa from 10am to 9pm on January 20th."

[1227] Terminal: Sends the input request information to the server.

[1228] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[1229] Generate and present an action plan

[1230] 6. Generate an action plan

[1231] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events."

[1232] Server: Validates the feasibility of the generated plan (e.g., whether reservations are available or events are running) and adjusts the plan if necessary. Retrieves reservation and event information using the OpenTable API and Eventbrite API.

[1233] 7. Submitting and Viewing Plans

[1234] Server: Sends the determined action plan to the user's device.

[1235] Device: Display the proposed action plan visually, for example as a timeline or points on a map.

[1236] 8. Check and select your plan

[1237] User: Review the suggested action plans and adjust accordingly. Select the plan you like best and add it to your calendar.

[1238] Users: Set reminders for selected plans so you can receive timely notifications.

[1239] Specific examples

[1240] Example 1: "Date in Tachikawa from 10am to 9pm on January 20th"

[1241] 1. User: Enters a request into the app: "Date in Tachikawa from 10am to 9pm on January 20th."

[1242] 2. Terminal: Sends the request to the server.

[1243] 3. Server: Using a deep learning model, it generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes real-time emotion data into account.

[1244] 4. Server: Validates the feasibility of the generated plan and adjusts it if necessary.

[1245] 5. Server: Sends the finalized action plan to the device.

[1246] 6. Terminal: Display the proposed plan as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00."

[1247] 7. User: Review the plans, adopt the one they like, add it to their calendar, and set a reminder.

[1248] Examples of prompt statements

[1249] Example 1: "Please suggest a date plan in Tachikawa from 10:00 to 21:00 on January 20th."

[1250] Example 2: "What is your plan of action for relaxing on the weekend?"

[1251] This allows users to get a highly personalized holiday action plan that takes into account their emotional state.

[1252] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1253] Step 1: Login and Permissions

[1254] User: Installs the app and logs in using a social media account or email address. The input is account information and the output is a login session.

[1255] Terminal: After logging in, a popup is displayed requesting permission to access behavioral and emotional data. The input is the login session, and the output is the popup screen.

[1256] User: Set the access permission for various data to "Allow". The input is a popup screen, and the output is the permission status.

[1257] Step 2: Collect behavioral and emotional data

[1258] Terminal: Periodically collects authorized data (GPS information, calendar events, search history, facial expression data, voice data, text data). The input is the authorization status, and the output is the collected data.

[1259] Terminal: Sends collected data to the server in batch processing or real time. For example, when the amount of data exceeds a certain threshold, the data is sent. The input is the collected data, and the output is the data sent to the server.

[1260] Server: Stores the received data in a cloud database. The input is the transmitted data, and the output is the data stored in the database.

[1261] Step 3: Preprocessing the data

[1262] Server: Performs noise removal (removal of inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling). Input is raw data, and output is preprocessed data.

[1263] Server: Prepares the preprocessed data as input data for the deep learning model and emotion engine. The input is the preprocessed data, and the output is the input data for the model.

[1264] Step 4: Training the model

[1265] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. The input is the preprocessed data, and the output is the trained model.

[1266] Server: Splits the dataset into training and validation datasets to improve the accuracy of the model. The input is the dataset, and the output is the evaluation of the model accuracy.

[1267] Server: Periodically ingests new data and retrains the model. For example, retraining occurs on the first day of each month. The input is the new data, and the output is the updated model.

[1268] Step 5: Accepting and parsing the request

[1269] User: Enters a specific request for how to spend the day off into the app. For example, a request might be "Date in Tachikawa from 10:00 to 21:00 on January 20th." The input is the request information, and the output is sending the request to the device.

[1270] Terminal: Sends the input request information to the server. The input is the request information, and the output is the data to be sent to the server.

[1271] Server: Analyzes the request information and performs analysis using a deep learning model and emotion engine. This also refers to past behavioral history, preference data, and real-time emotion data. The input is the request and reference data, and the output is the analysis results.

[1272] Step 6: Generate an action plan

[1273] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotion data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events." The input is the request and model data, and the output is an action plan.

[1274] Server: Validates whether the generated plan is feasible (reservation availability and event status), and adjusts the plan as necessary. For example, it obtains reservation and event information using the OpenTable API or Eventbrite API. The input is the action plan, and the output is the finalized plan.

[1275] Step 7: Submit and view your plan

[1276] Server: Sends the finalized action plan to the user's terminal. The input is the finalized plan, and the output is the transmitted data.

[1277] Terminal: A device that visually displays the proposed action plan, for example, in a timeline format or as points on a map. The input is the transmitted data, and the output is the displayed information.

[1278] Step 8: Review and select your plan

[1279] User: Review the proposed action plans and adjust accordingly. Select the plan they like and add it to their calendar. Input is display information, output is calendar event.

[1280] User: Set a reminder for the selected plan and receive notifications. Input is a calendar event, output is a reminder setting.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] Conventional itinerary suggestion systems are based solely on user behavioral data, making it difficult to provide highly personalized suggestions that take into account the user's emotions and moods. Furthermore, because these systems are primarily focused on travel and daily activity planning, they do not offer functionality to link with autonomous driving systems to suggest optimal driving routes and stopover spots.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data after obtaining access permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for training a deep learning model using preprocessed behavioral data and emotional data, means for analyzing specific request information entered by the user, means for generating an optimal action plan by referencing past behavioral data and emotional data and the deep learning model, and means for cooperating with an autonomous driving system to suggest optimal driving routes and stopover spots. This makes it possible to propose highly personalized action plans that take the user's emotional state into consideration, and also makes it possible to provide optimal driving plans using autonomous driving vehicles.

[1285] "User Permission" is the consent an application requests from a user to allow the application to access their behavioral and emotional data.

[1286] "Behavioral Data" refers to information related to a user's behavior, such as the user's location, calendar information, and search history.

[1287] "Emotional data" refers to information about a user's emotional state, such as their facial expressions, voice, or text.

[1288] "Cloud Database" means a database accessible over the internet that is used to store and manage behavioral and emotional data.

[1289] "Preprocessing" refers to the process of removing noise, filling in missing values, standardizing, and otherwise processing the collected raw data to prepare it in a format suitable for training a deep learning model.

[1290] A "deep learning model" is a machine learning model that uses deep learning technology to analyze a user's behavioral patterns and emotional state and generate an action plan.

[1291] "Specific request information" refers to the specific action plan or request information that the user enters into the application, including 5W1H information such as date, time, location, purpose, and desired activity.

[1292] The "optimal action plan" is a plan that proposes the most suitable action plan for the user based on the user's behavioral and emotional data.

[1293] An "autonomous driving system" is a system that uses artificial intelligence and sensor technology to automate driving and operate vehicles without a human driver.

[1294] A "driving route" is the route an autonomous vehicle will travel, and is part of a plan that includes destinations and stopovers requested by the user.

[1295] "Stopover spots" are places where users are encouraged to stop during their drive, such as scenic spots, restaurants, cafes, and event venues.

[1296] This invention relates to a system that collects and analyzes user behavioral and emotional data, and then works in conjunction with an autonomous driving system to suggest optimal driving routes and stopover spots. This system uses a wide range of devices, including smartphones and autonomous vehicles.

[1297] System configuration and operation overview

[1298] User Device

[1299] 1. Login and Permissions:

[1300] Users install the application on their smartphone and log in the first time they start it, using their social media account or email address.

[1301] When logging in for the first time, the device will display a pop-up requesting the user to grant permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text).

[1302] Users grant access permissions for various data.

[1303] 2. Data Collection:

[1304] The device periodically acquires permitted data (GPS information, calendar events, search history, facial expression data, voice data, and text data) and transmits it to the server in real time or in batches.

[1305] server

[1306] 3. Data preprocessing:

[1307] The server performs preprocessing such as noise removal, missing value completion, and data standardization on the raw data stored in the cloud database.

[1308] The preprocessed data is prepared as input for the deep learning model and emotion engine. The data is managed by a deep learning framework (e.g., TensorFlow or PyTorch).

[1309] 4. Train the model:

[1310] The deep learning model uses pre-processed data to learn user behavior patterns and preferences, and also incorporates sentiment data to improve the model's accuracy.

[1311] The model is periodically retrained to incorporate new data and improve its accuracy.

[1312] 5. Driving route generation and suggestions:

[1313] When a user enters a specific driving plan request into the app (for example, "I want to go on a four-hour drive with a friend next Sunday"), the device sends that information to the server.

[1314] The server performs analysis using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[1315] Based on the request, the server works with the autonomous driving system to generate the optimal driving route and stopover spots.

[1316] The generated plan is verified for feasibility and adjusted if necessary, and this information is sent to the user's device for presentation.

[1317] 6. Displaying Proposals:

[1318] The device will clearly display the suggested driving route and stops to the user, who can then review the proposed plan, adjust accordingly, and select. The selected plan will be added to the calendar and a reminder will be set.

[1319] Specific examples

[1320] Consider a case where a user requests, "I want to enjoy a four-hour drive with a friend next Sunday." In this case, the server generates the following driving plan based on the user's past behavioral data, emotional data, and real-time data.

[1321] Driving route: "Leave home at 10:00 → Visit a scenic park at 10:30 → Have lunch at Restaurant A at 12:00 → Have lunch at Event B at 14:00 → Arrive home at 16:00"

[1322] Users can review the suggestions and, if they like them, add them to their calendar and set a reminder.

[1323] An example prompt is:

[1324] I'd like to go on a 4-hour drive with a friend next Sunday. Can you suggest a good route and places to stop?

[1325] In this way, a system will be realized that can provide highly personalized action plans based on user behavioral and emotional data, and propose optimal driving plans using autonomous vehicles.

[1326] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1327] Step 1:

[1328] Logging in a user and getting permissions

[1329] A user installs a smartphone application and logs in the first time they start it. They log in using a social media account or email address. When the user logs in for the first time, the device displays a pop-up requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text). If the user agrees, access to various data is granted. The input to this step is the user's login information and consent information, and the output is the status of data access permission.

[1330] Step 2:

[1331] Data collection

[1332] The device periodically acquires authorized behavioral data (GPS information, calendar events, search history) and emotional data (facial expression data, voice data, text data). These data are sent to the server in real time or batch processing. The input of this step is the user's behavioral data and emotional data, and the output is the raw data sent to the server.

[1333] Step 3:

[1334] Data Preprocessing

[1335] The server performs preprocessing on the raw data stored in the cloud database, such as noise removal, missing value imputation, and data standardization, which makes the data suitable for training deep learning models and emotion engines. The input of this step is raw data, and the output is preprocessed data.

[1336] Step 4:

[1337] Training a deep learning model

[1338] The server uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. It also incorporates sentiment data to improve the model's accuracy. The model is periodically retrained by incorporating new data to improve its accuracy. The input of this step is the preprocessed data, and the output is the trained deep learning model.

[1339] Step 5:

[1340] Request input and analysis

[1341] The user inputs a specific driving plan request (for example, "I want to enjoy a four-hour drive with a friend next Sunday") into the app. The device sends the input request information to the server. The server analyzes this request information using a deep learning model and emotion engine. This analysis also references past behavioral history, preference data, and real-time emotion data. The input of this step is the user's request, and the output is the analysis result.

[1342] Step 6:

[1343] Generate driving routes and stopovers

[1344] The server generates an optimal driving route and stopovers based on the request, learned behavioral pattern data, and emotion data. This includes driving routes and stopovers (scenic spots, restaurants, cafes, event venues, etc.). The server then verifies the feasibility of this generated plan (checking event availability and reservations), adjusts the plan as necessary, and prepares a finalized plan. The input of this step is the analysis result of the deep learning model, and the output is the optimal driving plan.

[1345] Step 7:

[1346] Submitting and Viewing Plans

[1347] The server sends the confirmed driving plan to the user's device. The device displays the proposed driving plan in an easy-to-understand manner to the user. For example, a schedule such as "Leave home at 10:00 → Visit a scenic park at 10:30 → Lunch (Restaurant A) at 12:00 → Lunch event (Event B) at 14:00 → Arrive home at 16:00" is displayed. The user reviews this proposed plan, and if they like it, it is added to their calendar and a reminder is set. The input of this step is the confirmed driving plan, and the output is the plan information displayed on the user's device.

[1348] The above steps realize a system that provides highly personalized, optimal driving plans using autonomous vehicles based on user behavioral and emotional data.

[1349] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1350] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1351] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1352] [Fourth embodiment]

[1353] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1354] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1355] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1356] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1357] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1358] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1359] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1360] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1361] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1362] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1363] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1364] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1365] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1366] The present invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented in the following steps.

[1367] Data collection and preprocessing

[1368] 1. Login and Permissions

[1369] User: After installing the app and logging in for the first time, the user grants permission for the app to access their behavioral data (location, calendar information, search history, etc.).

[1370] Terminal: Prepares to periodically collect various data permitted by the user and sends that data to the server.

[1371] 2. Data storage and preprocessing

[1372] Server: Collected behavioral data is stored in a cloud database, where it is preprocessed by noise removal and missing value imputation.

[1373] Training a deep learning model

[1374] 3. Training the model

[1375] Server: Trains a deep learning model using the preprocessed data. This model learns the unique behavioral patterns and preferences of each user. This training is performed periodically, and the accuracy of the model improves as the amount of data increases.

[1376] User requests and analysis

[1377] 4. Request acceptance and analysis

[1378] User: Enter specific 5W1H information into the app, such as "Date in Tachikawa from 10am to 9pm on January 20th."

[1379] Terminal: Sends the input request information to the server.

[1380] Server: Analyzes the request information and generates an optimal action plan by referring to user behavior data and deep learning models.

[1381] Generate and present an action plan

[1382] 5. Plan Generation and Verification

[1383] Server: Based on the user's preferences and past behavioral data, the server generates a specific action plan by combining event information, restaurants, transportation options, etc. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll around the park at 15:00 → Dinner at Restaurant B at 18:00" may be created.

[1384] Server: Validates whether the generated plan is feasible, adjusts the plan if necessary, and finalizes the action plan.

[1385] 6. Plan presentation and selection

[1386] Server: Sends the determined action plan to the user's device.

[1387] On the device: The proposed action plan is displayed to the user, who can review the details of the proposed plan and make changes if necessary.

[1388] Users can select their favorite action plans and add them to their calendar within the app. They can also set reminders to ensure they don't forget to follow through with their plans.

[1389] Specific examples

[1390] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[1391] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[1392] 2. Terminal: Sends information to the server.

[1393] 3. Server: Refers to the deep learning model and generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa.

[1394] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[1395] 5. Server: Sends the finalized action plan to the device.

[1396] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[1397] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[1398] These steps of implementing the present invention allow users to easily find the perfect holiday experience and maximize its value.

[1399] The processing flow will be explained below.

[1400] Specific processing steps of the system

[1401] Step 1: Log in and get permissions

[1402] User: After installing the app for the first time, launch the app and access the login screen. Log in with your social media account or email address.

[1403] Device: When logging in for the first time, a popup will appear asking the user for permission to collect behavioral data.

[1404] Users: Grant data permissions such as location, calendar information, and search history.

[1405] Step 2: Collecting and sending behavioral data

[1406] On the device: Permitted data (GPS information, calendar events, search history) is periodically retrieved and sent to the server in batch or real-time.

[1407] Server: Stores the received data in a cloud database.

[1408] Step 3: Preprocessing the data

[1409] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[1410] Server: Prepares the preprocessed data as input for deep learning models.

[1411] Step 4: Train the deep learning model

[1412] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, which then acts as a custom model for each user.

[1413] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[1414] Step 5: User Request Input

[1415] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[1416] Terminal: Sends the input request information to the server.

[1417] Step 6: Parsing the request information

[1418] Server: Receives user request information and analyzes it using a deep learning model, while also referencing past behavioral history and preference data.

[1419] Step 7: Generate an action plan

[1420] Server: Generates an optimal action plan based on the request and learned behavioral pattern data, for example, by combining information on date spots, restaurants, and current events.

[1421] Server: Verifies the feasibility of the generated plan (e.g., whether reservations are possible or the event is being held).

[1422] Server: Readjust the plan as needed and prepare a finalized plan.

[1423] Step 8: Submit and view your plan

[1424] Server: Sends the determined action plan to the user's device.

[1425] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[1426] Step 9: Review and select your plan

[1427] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[1428] Users: Add the selected plan to their calendar and set a reminder.

[1429] The above are the specific processing steps in the system of the present invention, which is designed to maximize the value of a user's holiday and propose an optimal activity plan.

[1430] Example 1

[1431] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] In modern society, it is important to provide optimal activity plans for individual users so that they can make effective use of their limited holidays. However, conventional systems have had difficulty effectively collecting and analyzing user behavioral data to generate highly accurate activity plans. Furthermore, issues remain, such as verifying the feasibility of generated plans and providing a reminder function to ensure users do not forget to execute them.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1434] In this invention, the server includes a means for collecting behavioral data with permission from the user, a means for storing the collected behavioral data in a cloud database, and a means for performing preprocessing on the stored behavioral data, such as noise removal and missing value imputation. This enables the generation of highly accurate behavior plans. The system also includes a means for training a deep learning model using the preprocessed behavioral data, a means for analyzing 5W1H information entered by the user, and a means for generating an optimal behavior plan by referencing past behavioral data and the deep learning model. The system also includes a means for verifying whether the generated behavior plan is feasible and for making adjustments as necessary, and a means for sending and displaying the finalized behavior plan to the user's device. This allows users to easily plan their optimal holiday and maximize its value. The system also includes a means for setting a reminder function, helping users remember to carry out their behavior plan.

[1435] "Access permission" means that a user grants an application or system access to their data.

[1436] "Behavioral data" refers to data related to a user's daily behavior and activities, such as location information, calendar information, and search history.

[1437] A "cloud database" is a remote database management system that can be accessed via the Internet, and is a means of efficiently storing and managing large amounts of data.

[1438] "Preprocessing" refers to the process of processing raw data into a form suitable for data analysis and training machine learning models, such as removing noise and filling in missing values.

[1439] A "deep learning model" refers to an algorithm that uses a multi-layer neural network to learn the characteristics of data and perform advanced analysis and predictions.

[1440] "5W1H information" refers to information entered by the user regarding "when, where, who, what, why, and how."

[1441] An "action plan" refers to a schedule that specifically plans activities or events that should be carried out on a user's holidays or at specific times.

[1442] The "reminder function" refers to a function that notifies users so that they do not forget their schedules or plans.

[1443] "User device" means an electronic device that is directly operated by a user, such as a smartphone or tablet.

[1444] This invention relates to a system that collects and analyzes user behavior data and proposes optimal holiday activity plans. This system is implemented by the following detailed process.

[1445] First, the user installs the application and grants permission to access behavioral data such as location information, calendar information, and search history when logging in for the first time. The device periodically collects the data permitted by the user and sends it to the server.

[1446] The server then stores the received behavioral data in a cloud database, using a remote database management system such as Amazon RDS or Google Cloud SQL. The collected data is then preprocessed, including noise removal and missing value imputation, to improve data quality and make it suitable for training deep learning models.

[1447] The server uses the preprocessed data to train a deep learning model using machine learning frameworks such as TensorFlow and PyTorch. This deep learning model learns user behavior patterns and preferences and periodically updates its accuracy.

[1448] When a user enters specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th," the device sends this information to the server. The server analyzes the information and generates an optimal action plan by referencing a pre-trained deep learning model and past behavioral data.

[1449] The server verifies various conditions (e.g., business hours and traffic information) to ensure that the generated plan is feasible. If necessary, it readjusts the plan and sends the finalized plan to the device. The device then displays this plan to the user, who can review the proposed plan and make changes if necessary.

[1450] Additionally, the finalized action plan is added to the calendar and a reminder function is set, so users can remember to carry out their plan.

[1451] As a concrete example, if a user inputs a request for "a date in Tachikawa from 10:00 to 21:00 on January 20th," the system generates the following prompt:

[1452] Example prompt sentence:

[1453] "Please create a specific action plan for a date in Tachikawa from 10:00 to 21:00 on January 20th."

[1454] As described above, this invention provides an optimal action plan tailored to individual needs based on the user's behavioral data, enabling the user to spend their holidays efficiently and productively.

[1455] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1456] Step 1:

[1457] The user installs the application and grants permission to access behavioral data such as location, calendar information, and search history when logging in for the first time. Based on this input data, the device authenticates the user and prepares access to various data. The output is an authentication token and authorized data access.

[1458] Step 2:

[1459] The device periodically collects user behavioral data such as location information, calendar information, and search history. Specifically, it obtains location information using GPS, schedule information via a calendar API, and browser history data. The collected data is sent to a server. The input in this step is the user's behavioral data, and the output is the collected behavioral dataset.

[1460] Step 3:

[1461] The server stores the received behavioral data in a cloud database, typically using Amazon RDS or Google Cloud SQL. A timestamp is added when the data is saved, and checks are performed to ensure data consistency and integrity. This input data is the collected behavioral data, and is saved as output in the cloud database.

[1462] Step 4:

[1463] The server performs preprocessing on the stored behavioral data, such as noise removal and missing value imputation. For example, it runs filtering algorithms to remove outliers and applies imputation algorithms to fill in missing data. The input data is the stored behavioral data, and the output is a preprocessed, clean dataset.

[1464] Step 5:

[1465] The server uses the preprocessed data to train a deep learning model. For training, the model is implemented using TensorFlow or PyTorch to learn each user's behavioral patterns and preferences. The input data is the preprocessed data, and the output is a trained deep learning model.

[1466] Step 6:

[1467] The user inputs specific 5W1H information into the app, such as "A date in Tachikawa from 10am to 9pm on January 20th." The input information is retrieved through the application interface. This input information is then sent to the server for further analysis, becoming a specific user request.

[1468] Step 7:

[1469] The server analyzes the input 5W1H information and generates an optimal action plan by referencing a deep learning model and past behavioral data. For example, it combines information on date spots around Tachikawa, good restaurants, and ongoing events. The input data is the 5W1H information and the user's behavioral data, and the output is the generated action plan.

[1470] Step 8:

[1471] The server verifies the generated action plan based on business hours and traffic information to confirm whether it is feasible. If necessary, the action plan is readjusted and finally confirmed. The input data is the generated action plan and various verification data, and the output is a feasible and confirmed action plan.

[1472] Step 9:

[1473] The server sends the finalized action plan to the user's device. The device displays the received action plan to the user, who then checks the proposed plan and makes changes if necessary. The input in this step is the finalized action plan, and the output is the action plan displayed on the user's device.

[1474] Step 10:

[1475] The user adds the finalized action plan to the calendar in the app. The device sets a reminder for the event added to the calendar. The reminder notifies the user so that they can remember to carry out the event. The input data is the finalized action plan, and the output is the event added to the calendar and the reminder setting.

[1476] (Application example 1)

[1477] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1478] Previously, systems existed that generated personalized action plans using user preferences and past behavioral data. However, in the case of food delivery services, these systems were unable to effectively incorporate a user's specific preferences and past ordering history into their proposals with sufficient accuracy. Furthermore, in the case of food delivery, the system was unable to provide sufficient recommendations for optimal restaurants and delivery times, making it difficult to provide a satisfying service. Therefore, it is necessary to improve the accuracy and personalization of systems to improve user convenience.

[1479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1480] In this invention, the server includes means for collecting behavioral data with permission from the user, means for storing the collected behavioral data in a cloud database, means for training a deep learning model using the preprocessed behavioral data, means for analyzing 5W1H information entered by the user, means for generating an optimal action plan by referring to past behavioral data and the deep learning model, means for transmitting and displaying the generated action plan to the user's terminal, and means for suggesting the optimal food delivery based on the user's preferences and past order history, thereby enabling users to easily enjoy highly satisfying food delivery.

[1481] - "Permission" means that the user gives the application permission to collect and use behavioral data.

[1482] "Behavioral data" refers to data that indicates a user's behavior, such as the user's location information, calendar information, search history, and order history.

[1483] "Cloud database" means a database accessible via the internet that is used to store behavioral data.

[1484] "Preprocessing" refers to the process of performing processes such as noise removal and missing value completion on collected data to prepare it in a format suitable for analysis and model learning.

[1485] A "deep learning model" is a machine learning model that uses a multi-layer neural network to learn complex data patterns and perform predictions and classifications.

[1486] "5W1H information" refers to the basic elements of information such as Who, What, When, Where, Why, and How.

[1487] "Analysis" refers to the process of processing and calculating collected data and input information for a specific purpose to derive meaningful results.

[1488] An "action plan" is a specific schedule or plan suggested based on the user's actions.

[1489] A "terminal" is a device that can be directly operated by a user, such as a smartphone or tablet.

[1490] "Preferences" refer to a user's preferences, interests, and concerns.

[1491] "Order History" is a record of orders placed by a user in the past.

[1492] "Food delivery" is a service that delivers meals from restaurants or food services to a location specified by the user.

[1493] System Overview

[1494] This invention is a system that collects and analyzes user behavior data and suggests optimal food delivery options based on the user's preferences and past order history. This system is primarily configured using a cloud-based database and deep learning model.

[1495] Hardware and software used

[1496] Hardware

[1497] Smartphone: The user installs and uses the application.

[1498] Cloud servers (e.g., AWS EC2 and Google Cloud Platform): Analyze data and train models.

[1499] software

[1500] Database (e.g., Firebase, AWS DynamoDB): Stores collected behavioral data.

[1501] Deep learning frameworks (e.g., TensorFlow, PyTorch): Model user behavior patterns and preferences and make predictions.

[1502] Detailed processing flow

[1503] 1. User Permissions and Data Collection

[1504] Users: Install applications and grant permissions.

[1505] Smartphones: Collect user behavioral data such as location information, calendar information, search history, and order history.

[1506] Cloud Server: Collected data is stored in a cloud database in real time.

[1507] 2. Data storage and preprocessing

[1508] Cloud server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data.

[1509] 3. Training the deep learning model

[1510] Cloud server: Using the pre-processed data, the deep learning model is periodically trained and updated to learn each user's preferences and behavioral patterns.

[1511] 4. User Requests and Analysis

[1512] User: Enters a specific request into the app, such as "I'd like a healthy lunch delivered on January 20th at 12 noon."

[1513] Smartphone: Sends request information to the cloud server.

[1514] Cloud server: Analyzes the request information and generates optimal food delivery suggestions by referencing the user's preferences, past order history, and deep learning models.

[1515] 5. Generate and present food delivery offers

[1516] Cloud server: Generates and validates an action plan including recommended restaurants, dishes, and delivery times.

[1517] Smartphone: Displays suggestions sent from the cloud server to the user.

[1518] Specific examples

[1519] User input: "I'd like a healthy lunch delivered on January 20th at 12 noon."

[1520] Cloud server response: "Recommended plans: Organic salad bowl (Restaurant A), sushi set (Restaurant B), Japanese set meal (Restaurant C)"

[1521] Prompt Sentence Examples

[1522] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[1523] In this way, the present invention allows users to easily find the best food delivery options, greatly improving convenience.

[1524] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1525] Step 1:

[1526] User Permissions and Data Collection

[1527] User: Installs the application and grants permissions to access behavioral data (input: operations; output: permissions).

[1528] Device: Collects behavioral data such as location information, calendar information, search history, and order history (input: user behavior. output: behavioral data).

[1529] Terminal: Sends collected behavioral data to the cloud server in real time (input: behavioral data; output: sending data to the server).

[1530] Step 2:

[1531] Data storage and preprocessing

[1532] Server: Stores the behavioral data sent to the cloud database (input: behavioral data; output: data in the cloud database).

[1533] Server: Performs preprocessing such as noise removal and missing value imputation on the stored behavioral data (input: data in the cloud database; output: preprocessed data).

[1534] Step 3:

[1535] Training a deep learning model

[1536] Server: Trains a deep learning model using the preprocessed data (Input: preprocessed data. Output: trained model).

[1537] Server: This model is periodically updated to maintain predictive accuracy based on the latest data (input: new behavioral data. output: updated model).

[1538] Step 4:

[1539] User requests and analysis

[1540] User: Enters specific 5W1H information such as "I would like a healthy delivery lunch at 12 o'clock on January 20th" (Input: 5W1H information. Output: Request data).

[1541] Terminal: Sends request information to the cloud server (Input: Request data. Output: Sends request data to the server).

[1542] Server: Analyzes the request information and processes (analyzes) the data by referring to the deep learning model (input: request data & model. output: analysis results).

[1543] Step 5:

[1544] Generate and present food delivery offers

[1545] Server: Generates and validates food delivery plans including recommended restaurants, dishes, and optimal delivery times (Input: Analysis results. Output: Generated food delivery plans).

[1546] Server: Readjusts the plan if necessary and confirms the final proposal (Input: Generated food delivery plan. Output: Confirmed plan).

[1547] Terminal: Shows the proposal sent from the cloud server to the user (Input: Confirmed plan. Output: Display to user).

[1548] Examples:

[1549] Example prompt sentence:

[1550] Example prompt: "Based on the user's past ordering history and preferences, suggest healthy delivery lunch options for January 20, 2023 at 12:00 PM."

[1551] Step 6:

[1552] Selecting and implementing an action plan

[1553] User: Review the proposed food delivery plans in detail and select the appropriate plan (Input: Proposed plans. Output: Selected plan).

[1554] Terminal: Add selected action plan to calendar and set reminder (Input: selected plan. Output: calendar and reminder settings).

[1555] Through this process, users can receive personalized food delivery suggestions efficiently.

[1556] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1557] This invention relates to a system that collects and analyzes user behavioral and emotional data to propose optimal holiday plans. By combining this system with an emotion engine, it is possible to make more personalized proposals that take into account the user's emotions.

[1558] Data collection and preprocessing

[1559] 1. Login and Permissions

[1560] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[1561] Device: When logging in for the first time, a pop-up will appear asking the user to grant permission to access their behavioral data (location, calendar information, search history) and emotional data (facial expressions, voice, text).

[1562] User: Grants access permissions to various data.

[1563] 2. Collecting behavioral and emotional data

[1564] Device: Permitted data (GPS information, calendar events, search history, facial expression data, voice data, text data) is periodically collected and sent to the server in batch processing or real time.

[1565] Server: Stores the received data in a cloud database.

[1566] 3. Data Preprocessing

[1567] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[1568] Server: Prepares the preprocessed data as input for the deep learning model and emotion engine.

[1569] Deep learning model and emotion engine training

[1570] 4. Training the model

[1571] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences, and also works with sentiment data to improve the model's accuracy.

[1572] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[1573] User requests and analysis

[1574] 5. Request acceptance and analysis

[1575] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[1576] Terminal: Sends the input request information to the server.

[1577] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[1578] Generate and present an action plan

[1579] 6. Generate an action plan

[1580] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and ongoing events.

[1581] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[1582] 7. Submitting and Viewing Plans

[1583] Server: Sends the determined action plan to the user's device.

[1584] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[1585] 8. Check and select your plan

[1586] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[1587] Users: Add the selected plan to their calendar and set a reminder.

[1588] Specific examples

[1589] Example 1: Request for a date in Tachikawa from 10:00 to 21:00 on January 20th

[1590] 1. User: Enters the 5W1H information: "Date in Tachikawa from 10am to 9pm on January 20th."

[1591] 2. Terminal: Sends information to the server.

[1592] 3. Server: Refers to the deep learning model to generate an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes into account the user's real-time emotional data.

[1593] 4. Server: Validates the generated plan for feasibility and makes adjustments as necessary.

[1594] 5. Server: Sends the finalized action plan to the device.

[1595] 6. Device: The proposed plan is displayed to the user. For example, it might look like this: "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Walk around the park at 15:00 → Dinner at Restaurant B at 18:00."

[1596] 7. User: Review the proposed plan and, if they agree, add it to their calendar and set a reminder.

[1597] These steps for implementing the present invention create a system that maximizes the value of a user's holiday and suggests an action plan. The introduction of an emotion engine enables highly personalized suggestions that take into account the user's emotional state.

[1598] The processing flow will be explained below.

[1599] Specific processing steps of the system

[1600] Step 1:

[1601] User: Install the app and log in the first time you launch it. Log in using your social media account or email address.

[1602] Step 2:

[1603] On your device: When you first log in, a pop-up will appear asking you to grant permission to access your behavioral and emotional data. Behavioral data includes location, calendar information, and search history, while emotional data includes facial expressions, voice, and text.

[1604] Step 3:

[1605] User: Grants access permissions to various data.

[1606] Step 4:

[1607] Device: Acquires authorized data and sends it to the server in batch or real-time, such as GPS information, calendar events, search history, facial expression data, voice data, and text data.

[1608] Step 5:

[1609] Server: Stores the received data in a cloud database.

[1610] Step 6:

[1611] Server: Performs preprocessing such as noise removal, missing value imputation, and data standardization on the raw data stored in the cloud database.

[1612] Step 7:

[1613] Server: The preprocessed data is used to train a deep learning model to learn user behavior patterns and preferences, including incorporating sentiment data to improve the model's accuracy.

[1614] Step 8:

[1615] Server: Regularly incorporates new data to retrain the model and improve its accuracy.

[1616] Step 9:

[1617] User: Enter a specific request for how to spend their holiday (for example, "A date in Tachikawa from 10am to 9pm on January 20th") into the app.

[1618] Step 10:

[1619] Terminal: Sends the input request information to the server.

[1620] Step 11:

[1621] Server: Receives the request information and analyzes it using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[1622] Step 12:

[1623] Server: Generates an optimal action plan based on the user's request, learned behavioral pattern data, and emotional data. For example, it combines information on date spots, restaurants, and current events.

[1624] Step 13:

[1625] Server: Verifies the feasibility of the generated plan (e.g., availability of reservations and event status), adjusts the plan as necessary, and prepares the finalized plan.

[1626] Step 14:

[1627] Server: Sends the determined action plan to the user's device.

[1628] Step 15:

[1629] Device: The proposed action plan is displayed to the user in an easy-to-understand manner. For example, a schedule such as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00" is displayed.

[1630] Step 16:

[1631] User: Review the proposed action plans and adjust accordingly. Select the plan you like best.

[1632] Step 17:

[1633] Users: Add the selected plan to their calendar and set a reminder.

[1634] In this way, we can realize a system that makes maximum use of user behavioral and emotional data to generate and propose optimal action plans. The introduction of an emotion engine also takes into account the user's psychological state, enabling more personalized proposals.

[1635] Example 2

[1636] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1637] Existing action plan suggestion systems only consider user behavioral data, which means they lack advanced personalization that reflects the user's emotional state. Furthermore, they lack feasibility verification and refinement of the generated action plans, resulting in a poor user experience.

[1638] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data with permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for performing noise removal, missing value completion, and data standardization on the stored raw data, means for training a deep learning model using the preprocessed behavioral data and emotional data, means for analyzing 5W1H information entered by the user and generating an optimal action plan by referring to past behavioral data and emotional data, means for verifying whether the generated action plan is feasible and readjusting it as necessary, and means for transmitting the generated action plan to the user's terminal and displaying it in a visually easy-to-understand manner. This makes it possible to propose a highly personalized action plan that also takes into account the user's emotional state, and to verify and adjust its feasibility.

[1639] "Access permission" refers to the user's permission for an application to access personal data such as location information, calendar information, search history, facial expression data, voice data, and text data.

[1640] A "cloud database" is a type of data storage service that stores and manages data over the Internet, and can handle large amounts of data flexibly and efficiently.

[1641] "Denoising" refers to the process of removing unnecessary or inaccurate data from a dataset.

[1642] "Missing value imputation" refers to the process of filling in missing values ​​in a dataset with an appropriate value (e.g., the mean or median).

[1643] "Data standardization" refers to the process of scaling data so that each data point in a data set is on a consistent scale.

[1644] A "deep learning model" refers to a computational model that uses a multi-layer neural network to learn the characteristics of data and perform tasks such as prediction and classification.

[1645] "5W1H information" typically refers to information that consists of the elements of "When," "Where," "Who," "What," "Why," and "How."

[1646] "Emotion data" is information that indicates the emotional state of the user, and includes facial expression data, voice data, text data, and the like.

[1647] "Behavioral data" refers to information about a user's behavior, including location information, calendar information, search history, etc.

[1648] An "action plan" refers to a schedule or proposal of specific activities to be carried out within a time period specified by the user.

[1649] "Feasibility" refers to the process of verifying whether the generated action plan can be carried out in reality (for example, whether reservations can be made or whether events will be held).

[1650] An "emotion engine" refers to an algorithm or system that analyzes a user's emotional data and generates an appropriate action plan based on that emotional state.

[1651] "Visually easy-to-understand display" refers to displaying information in a format that is easy for users to understand, including timeline displays and map displays.

[1652] This invention is a system that collects and analyzes user behavioral and emotional data and proposes optimal action plans. This system uses a cloud database, deep learning models, and an emotion engine to enable advanced personalization that reflects the user's emotional state.

[1653] Data collection and preprocessing

[1654] 1. Login and Permissions

[1655] User: Installs the app and logs in with a social media account or email address.

[1656] Device: After logging in, a pop-up will appear requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expression data, voice data, text data).

[1657] User: Grants access permissions to various data.

[1658] 2. Collecting behavioral and emotional data

[1659] Device: Periodically collects and transmits permitted data to a server, including GPS information, calendar events, search history, facial expression data, voice data, and text data.

[1660] Server: Stores the received data in a cloud database (e.g., AWS DynamoDB).

[1661] 3. Data Preprocessing

[1662] Server: Performs noise removal (e.g., removing inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling).

[1663] Deep learning model and emotion engine training

[1664] 4. Training the model

[1665] Server: Trains deep learning models using preprocessed data. Builds models using TensorFlow or PyTorch.

[1666] Server: Split the dataset into training and validation datasets to improve the accuracy of the model.

[1667] Server: Retrain the model periodically to incorporate new data, for example, on the first day of every month.

[1668] User requests and analysis

[1669] 5. Request acceptance and analysis

[1670] User: Enter a specific request for how to spend the day off into the app. Example: "Date in Tachikawa from 10am to 9pm on January 20th."

[1671] Terminal: Sends the input request information to the server.

[1672] Server: Analyzes the request information and performs analysis using deep learning models and emotion engines, taking into account past behavioral history, preference data, and real-time emotion data.

[1673] Generate and present an action plan

[1674] 6. Generate an action plan

[1675] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotional data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events."

[1676] Server: Validates the feasibility of the generated plan (e.g., whether reservations are available or events are running) and adjusts the plan if necessary. Retrieves reservation and event information using the OpenTable API and Eventbrite API.

[1677] 7. Submitting and Viewing Plans

[1678] Server: Sends the determined action plan to the user's device.

[1679] Device: Display the proposed action plan visually, for example as a timeline or points on a map.

[1680] 8. Check and select your plan

[1681] User: Review the suggested action plans and adjust accordingly. Select the plan you like best and add it to your calendar.

[1682] Users: Set reminders for selected plans so you can receive timely notifications.

[1683] Specific examples

[1684] Example 1: "Date in Tachikawa from 10am to 9pm on January 20th"

[1685] 1. User: Enters a request into the app: "Date in Tachikawa from 10am to 9pm on January 20th."

[1686] 2. Terminal: Sends the request to the server.

[1687] 3. Server: Using a deep learning model, it generates an optimal action plan that combines date spots, good restaurants, ongoing events, etc. in Tachikawa. The emotion engine also takes real-time emotion data into account.

[1688] 4. Server: Validates the feasibility of the generated plan and adjusts it if necessary.

[1689] 5. Server: Sends the finalized action plan to the device.

[1690] 6. Terminal: Display the proposed plan as "Meet at Tachikawa Station at 10:00 → Watch a movie at 11:00 → Lunch at Restaurant A at 13:30 → Stroll in the park at 15:00 → Dinner at Restaurant B at 18:00."

[1691] 7. User: Review the plans, adopt the one they like, add it to their calendar, and set a reminder.

[1692] Examples of prompt statements

[1693] Example 1: "Please suggest a date plan in Tachikawa from 10:00 to 21:00 on January 20th."

[1694] Example 2: "What is your plan of action for relaxing on the weekend?"

[1695] This allows users to get a highly personalized holiday action plan that takes into account their emotional state.

[1696] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1697] Step 1: Login and Permissions

[1698] User: Installs the app and logs in using a social media account or email address. The input is account information and the output is a login session.

[1699] Terminal: After logging in, a popup is displayed requesting permission to access behavioral and emotional data. The input is the login session, and the output is the popup screen.

[1700] User: Set the access permission for various data to "Allow". The input is a popup screen, and the output is the permission status.

[1701] Step 2: Collect behavioral and emotional data

[1702] Terminal: Periodically collects authorized data (GPS information, calendar events, search history, facial expression data, voice data, text data). The input is the authorization status, and the output is the collected data.

[1703] Terminal: Sends collected data to the server in batch processing or real time. For example, when the amount of data exceeds a certain threshold, the data is sent. The input is the collected data, and the output is the data sent to the server.

[1704] Server: Stores the received data in a cloud database. The input is the transmitted data, and the output is the data stored in the database.

[1705] Step 3: Preprocessing the data

[1706] Server: Performs noise removal (removal of inaccurate data), missing value imputation (e.g., filling with mean or median), and data standardization (e.g., scaling). Input is raw data, and output is preprocessed data.

[1707] Server: Prepares the preprocessed data as input data for the deep learning model and emotion engine. The input is the preprocessed data, and the output is the input data for the model.

[1708] Step 4: Training the model

[1709] Server: Uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. The input is the preprocessed data, and the output is the trained model.

[1710] Server: Splits the dataset into training and validation datasets to improve the accuracy of the model. The input is the dataset, and the output is the evaluation of the model accuracy.

[1711] Server: Periodically ingests new data and retrains the model. For example, retraining occurs on the first day of each month. The input is the new data, and the output is the updated model.

[1712] Step 5: Accepting and parsing the request

[1713] User: Enters a specific request for how to spend the day off into the app. For example, a request might be "Date in Tachikawa from 10:00 to 21:00 on January 20th." The input is the request information, and the output is sending the request to the device.

[1714] Terminal: Sends the input request information to the server. The input is the request information, and the output is the data to be sent to the server.

[1715] Server: Analyzes the request information and performs analysis using a deep learning model and emotion engine. This also refers to past behavioral history, preference data, and real-time emotion data. The input is the request and reference data, and the output is the analysis results.

[1716] Step 6: Generate an action plan

[1717] Server: Generates an optimal action plan based on the request, learned behavioral pattern data, and emotion data. For example, it combines information on "date spots in Tachikawa," "recommended restaurants," and "ongoing events." The input is the request and model data, and the output is an action plan.

[1718] Server: Validates whether the generated plan is feasible (reservation availability and event status), and adjusts the plan as necessary. For example, it obtains reservation and event information using the OpenTable API or Eventbrite API. The input is the action plan, and the output is the finalized plan.

[1719] Step 7: Submit and view your plan

[1720] Server: Sends the finalized action plan to the user's terminal. The input is the finalized plan, and the output is the transmitted data.

[1721] Terminal: A device that visually displays the proposed action plan, for example, in a timeline format or as points on a map. The input is the transmitted data, and the output is the displayed information.

[1722] Step 8: Review and select your plan

[1723] User: Review the proposed action plans and adjust accordingly. Select the plan they like and add it to their calendar. Input is display information, output is calendar event.

[1724] User: Set a reminder for the selected plan and receive notifications. Input is a calendar event, output is a reminder setting.

[1725] (Application example 2)

[1726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1727] Conventional itinerary suggestion systems are based solely on user behavioral data, making it difficult to provide highly personalized suggestions that take into account the user's emotions and moods. Furthermore, because these systems are primarily focused on travel and daily activity planning, they do not offer functionality to link with autonomous driving systems to suggest optimal driving routes and stopover spots.

[1728] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting behavioral data and emotional data after obtaining access permission from the user, means for storing the collected behavioral data and emotional data in a cloud database, means for training a deep learning model using preprocessed behavioral data and emotional data, means for analyzing specific request information entered by the user, means for generating an optimal action plan by referencing past behavioral data and emotional data and the deep learning model, and means for cooperating with an autonomous driving system to suggest optimal driving routes and stopover spots. This makes it possible to propose highly personalized action plans that take the user's emotional state into consideration, and also makes it possible to provide optimal driving plans using autonomous driving vehicles.

[1729] "User Permission" is the consent an application requests from a user to allow the application to access their behavioral and emotional data.

[1730] "Behavioral Data" refers to information related to a user's behavior, such as the user's location, calendar information, and search history.

[1731] "Emotional data" refers to information about a user's emotional state, such as their facial expressions, voice, or text.

[1732] "Cloud Database" means a database accessible over the internet that is used to store and manage behavioral and emotional data.

[1733] "Preprocessing" refers to the process of removing noise, filling in missing values, standardizing, and otherwise processing the collected raw data to prepare it in a format suitable for training a deep learning model.

[1734] A "deep learning model" is a machine learning model that uses deep learning technology to analyze a user's behavioral patterns and emotional state and generate an action plan.

[1735] "Specific request information" refers to the specific action plan or request information that the user enters into the application, including 5W1H information such as date, time, location, purpose, and desired activity.

[1736] The "optimal action plan" is a plan that proposes the most suitable action plan for the user based on the user's behavioral and emotional data.

[1737] An "autonomous driving system" is a system that uses artificial intelligence and sensor technology to automate driving and operate vehicles without a human driver.

[1738] A "driving route" is the route an autonomous vehicle will travel, and is part of a plan that includes destinations and stopovers requested by the user.

[1739] "Stopover spots" are places where users are encouraged to stop during their drive, such as scenic spots, restaurants, cafes, and event venues.

[1740] This invention relates to a system that collects and analyzes user behavioral and emotional data, and then works in conjunction with an autonomous driving system to suggest optimal driving routes and stopover spots. This system uses a wide range of devices, including smartphones and autonomous vehicles.

[1741] System configuration and operation overview

[1742] User Device

[1743] 1. Login and Permissions:

[1744] Users install the application on their smartphone and log in the first time they start it, using their social media account or email address.

[1745] When logging in for the first time, the device will display a pop-up requesting the user to grant permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text).

[1746] Users grant access permissions for various data.

[1747] 2. Data Collection:

[1748] The device periodically acquires permitted data (GPS information, calendar events, search history, facial expression data, voice data, and text data) and transmits it to the server in real time or in batches.

[1749] server

[1750] 3. Data preprocessing:

[1751] The server performs preprocessing such as noise removal, missing value completion, and data standardization on the raw data stored in the cloud database.

[1752] The preprocessed data is prepared as input for the deep learning model and emotion engine. The data is managed by a deep learning framework (e.g., TensorFlow or PyTorch).

[1753] 4. Train the model:

[1754] The deep learning model uses pre-processed data to learn user behavior patterns and preferences, and also incorporates sentiment data to improve the model's accuracy.

[1755] The model is periodically retrained to incorporate new data and improve its accuracy.

[1756] 5. Driving route generation and suggestions:

[1757] When a user enters a specific driving plan request into the app (for example, "I want to go on a four-hour drive with a friend next Sunday"), the device sends that information to the server.

[1758] The server performs analysis using a deep learning model and emotion engine, taking into account past behavioral history, preference data, and real-time emotion data.

[1759] Based on the request, the server works with the autonomous driving system to generate the optimal driving route and stopover spots.

[1760] The generated plan is verified for feasibility and adjusted if necessary, and this information is sent to the user's device for presentation.

[1761] 6. Displaying Proposals:

[1762] The device will clearly display the suggested driving route and stops to the user, who can then review the proposed plan, adjust accordingly, and select. The selected plan will be added to the calendar and a reminder will be set.

[1763] Specific examples

[1764] Consider a case where a user requests, "I want to enjoy a four-hour drive with a friend next Sunday." In this case, the server generates the following driving plan based on the user's past behavioral data, emotional data, and real-time data.

[1765] Driving route: "Leave home at 10:00 → Visit a scenic park at 10:30 → Have lunch at Restaurant A at 12:00 → Have lunch at Event B at 14:00 → Arrive home at 16:00"

[1766] Users can review the suggestions and, if they like them, add them to their calendar and set a reminder.

[1767] An example prompt is:

[1768] I'd like to go on a 4-hour drive with a friend next Sunday. Can you suggest a good route and places to stop?

[1769] In this way, a system will be realized that can provide highly personalized action plans based on user behavioral and emotional data, and propose optimal driving plans using autonomous vehicles.

[1770] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1771] Step 1:

[1772] Logging in a user and getting permissions

[1773] A user installs a smartphone application and logs in the first time they start it. They log in using a social media account or email address. When the user logs in for the first time, the device displays a pop-up requesting permission to access behavioral data (location information, calendar information, search history) and emotional data (facial expressions, voice, text). If the user agrees, access to various data is granted. The input to this step is the user's login information and consent information, and the output is the status of data access permission.

[1774] Step 2:

[1775] Data collection

[1776] The device periodically acquires authorized behavioral data (GPS information, calendar events, search history) and emotional data (facial expression data, voice data, text data). These data are sent to the server in real time or batch processing. The input of this step is the user's behavioral data and emotional data, and the output is the raw data sent to the server.

[1777] Step 3:

[1778] Data Preprocessing

[1779] The server performs preprocessing on the raw data stored in the cloud database, such as noise removal, missing value imputation, and data standardization, which makes the data suitable for training deep learning models and emotion engines. The input of this step is raw data, and the output is preprocessed data.

[1780] Step 4:

[1781] Training a deep learning model

[1782] The server uses the preprocessed data to train a deep learning model to learn user behavior patterns and preferences. It also incorporates sentiment data to improve the model's accuracy. The model is periodically retrained by incorporating new data to improve its accuracy. The input of this step is the preprocessed data, and the output is the trained deep learning model.

[1783] Step 5:

[1784] Request input and analysis

[1785] The user inputs a specific driving plan request (for example, "I want to enjoy a four-hour drive with a friend next Sunday") into the app. The device sends the input request information to the server. The server analyzes this request information using a deep learning model and emotion engine. This analysis also references past behavioral history, preference data, and real-time emotion data. The input of this step is the user's request, and the output is the analysis result.

[1786] Step 6:

[1787] Generate driving routes and stopovers

[1788] The server generates an optimal driving route and stopovers based on the request, learned behavioral pattern data, and emotion data. This includes driving routes and stopovers (scenic spots, restaurants, cafes, event venues, etc.). The server then verifies the feasibility of this generated plan (checking event availability and reservations), adjusts the plan as necessary, and prepares a finalized plan. The input of this step is the analysis result of the deep learning model, and the output is the optimal driving plan.

[1789] Step 7:

[1790] Submitting and Viewing Plans

[1791] The server sends the confirmed driving plan to the user's device. The device displays the proposed driving plan in an easy-to-understand manner to the user. For example, a schedule such as "Leave home at 10:00 → Visit a scenic park at 10:30 → Lunch (Restaurant A) at 12:00 → Lunch event (Event B) at 14:00 → Arrive home at 16:00" is displayed. The user reviews this proposed plan, and if they like it, it is added to their calendar and a reminder is set. The input of this step is the confirmed driving plan, and the output is the plan information displayed on the user's device.

[1792] The above steps realize a system that provides highly personalized, optimal driving plans using autonomous vehicles based on user behavioral and emotional data.

[1793] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1794] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1795] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1796] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1797] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1798] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1799] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1800] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1801] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1802] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1803] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1804] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1805] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1806] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1807] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1808] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1809] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1810] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1811] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1812] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1813] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1814] The following is further disclosed regarding the above embodiment.

[1815] (Claim 1)

[1816] A means of collecting behavioral data with permission from users;

[1817] a means for storing the collected behavioral data in a cloud database;

[1818] means for training a deep learning model using the preprocessed behavioral data;

[1819] A means of analyzing the 5W1H information entered by the user,

[1820] A means of generating an optimal action plan by referencing past behavioral data and deep learning models; and

[1821] The system includes a means for transmitting and displaying the generated action plan on the user's terminal.

[1822] (Claim 2)

[1823] 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

[1824] (Claim 3)

[1825] 2. The system according to claim 1, further comprising means for verifying whether the generated action plan is feasible and for making readjustments if necessary.

[1826] "Example 1"

[1827] (Claim 1)

[1828] A means of collecting behavioral data with permission from users;

[1829] a means for storing the collected behavioral data in a cloud database;

[1830] A means of preprocessing the saved behavioral data, such as noise removal and missing value imputation,

[1831] means for training a deep learning model using the preprocessed behavioral data;

[1832] A means of analyzing the 5W1H information entered by the user,

[1833] A means of generating an optimal action plan by referencing past behavioral data and deep learning models; and

[1834] A means to verify the feasibility of the generated action plan and make adjustments if necessary;

[1835] The system includes a means for transmitting and displaying the finalized action plan on the user's terminal.

[1836] (Claim 2)

[1837] 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

[1838] (Claim 3)

[1839] 10. The system according to claim 1, further comprising means for setting a reminder function to assist the user in remembering to carry out the action plan.

[1840] "Application Example 1"

[1841] (Claim 1)

[1842] A means of collecting behavioral data with permission from users;

[1843] a means for storing the collected behavioral data in a cloud database;

[1844] means for training a deep learning model using the preprocessed behavioral data;

[1845] A means of analyzing the 5W1H information entered by the user,

[1846] A means of generating an optimal action plan by referencing past behavioral data and deep learning models; and

[1847] means for transmitting and displaying the generated action plan to a user's terminal;

[1848] The system includes a means to suggest optimal food delivery options based on the user's preferences and past ordering history.

[1849] (Claim 2)

[1850] 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

[1851] (Claim 3)

[1852] 2. The system according to claim 1, further comprising means for verifying whether the generated action plan is feasible and for making readjustments if necessary.

[1853] "Example 2: Combining Emotion Engines"

[1854] (Claim 1)

[1855] A means of collecting behavioral and emotional data with permission from users;

[1856] a means for storing the collected behavioral and emotional data in a cloud database;

[1857] A means of denoising, imputing missing values ​​and standardising the stored raw data;

[1858] means for training a deep learning model using the preprocessed behavioral and emotional data;

[1859] A means for analyzing the 5W1H information entered by the user and generating an optimal action plan by referring to past behavioral data and emotional data;

[1860] A means to verify the feasibility of the action plan generated and to make adjustments if necessary;

[1861] The system includes a means for transmitting the generated action plan to the user's terminal and displaying it in a visually easy-to-understand manner.

[1862] (Claim 2)

[1863] 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

[1864] (Claim 3)

[1865] 2. The system of claim 1, further comprising means for generating an action plan with reference to emotion data collected in real time.

[1866] "Application example 2 when combining emotion engines"

[1867] (Claim 1)

[1868] A means of collecting behavioral and emotional data with permission from users;

[1869] a means for storing the collected behavioral and emotional data in a cloud database;

[1870] means for training a deep learning model using the preprocessed behavioral and emotional data;

[1871] A means for analyzing specific request information input by a user;

[1872] A means for generating an optimal action plan by referencing past behavioral data, emotional data, and a deep learning model;

[1873] means for transmitting and displaying the generated action plan to a user's terminal;

[1874] A system that works in conjunction with the autonomous driving system and includes a means to suggest optimal driving routes and stopover spots.

[1875] (Claim 2)

[1876] 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

[1877] (Claim 3)

[1878] 2. The system according to claim 1, further comprising means for verifying whether the generated action plan is feasible and for making readjustments if necessary. [Explanation of symbols]

[1879] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting behavioral data with permission from users; a means for storing the collected behavioral data in a cloud database; means for training a deep learning model using the preprocessed behavioral data; A means of analyzing the 5W1H information entered by the user, A means of generating an optimal action plan by referencing past behavioral data and deep learning models; and The system includes a means for transmitting and displaying the generated action plan on the user's terminal.

2. 10. The system of claim 1, further comprising means for periodically updating the accuracy of the deep learning model based on the preprocessed data.

3. 2. The system according to claim 1, further comprising means for verifying whether the generated action plan is feasible and for making readjustments if necessary.

Images