System

Abstract

A system is provided.SOLUTION: Generating a new plan in accordance with an unplanned situation by a means for inputting a user's desire or condition, a means for analyzing the input desire or condition and extracting main information, a means for generating a plurality of plan candidates on the basis of the analyzed information, a means for evaluating the generated plan candidates and selecting an optimal plan, a means for calculating a moving route on the basis of the optimal plan, a means for transmitting the calculated route information and plan information to a user's terminal, and a means for optimizing a route on the basis of a user's current position and traffic information; A system comprising: means for transmitting to a user terminal; and means for making a reservation to an external service when a reservation is required.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people face challenges such as not knowing where to go on their days off, not knowing the most efficient route, and not being able to deal with sudden troubles or changes. These challenges call for optimal solutions that reduce the time and effort required for planning, while enabling people to spend stress-free, fulfilling holidays. In particular, flexible responses to situations such as family outings and dates are required, and the reality is that traditional manual planning is not only inefficient, but also makes it difficult to quickly obtain the right information. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides a system including means for inputting a user's vague wishes and conditions, means for analyzing the input wishes and conditions and extracting key information, means for generating multiple candidate plans based on this information, means for evaluating the generated candidate plans and selecting an optimal plan, means for calculating a travel route based on the selected plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, and means for making a reservation with an external service when a reservation is necessary. This allows the user to turn vague wishes into a specific plan, quickly create a plan, and flexibly respond to problems and changes in real time.

[0006] "User" refers to a person who uses the system to input their wishes and requirements and receive the optimal plan and navigation.

[0007] "Desires and conditions" refers to information related to the places the user wants to go and the goals they want to achieve, such as the type of tourist spot, desired meal times, and means of transportation.

[0008] "Input means" refers to the interface that a user uses to provide their preferences and requirements to the system, such as a smartphone or tablet application or website.

[0009] "Means for analysis" refers to technology that processes text information entered by the user, understands its meaning, and extracts the necessary information, such as a natural language processing engine.

[0010] "Key information" or "requirements" are specific keywords or propositions derived from desires and conditions, and refer to the basic information the system uses to generate a plan.

[0011] "Plan candidates" refer to multiple proposed travel and activity schedules generated based on the user's preferences and conditions.

[0012] "Means for generating" refers to an algorithm and database for automatically generating plan candidates based on input of the user's wishes and conditions.

[0013] The "means of evaluation" refers to the criteria and algorithms for comparing the generated multiple plan candidates and selecting the optimal plan.

[0014] A "travel route" refers to the optimal route for a user to travel around designated locations.

[0015] "Means of calculation" refers to an algorithm that uses map information and traffic information to calculate the optimal travel route.

[0016] "Transmitting means" refers to the communications technology used to transfer the calculated plan and route information to the user's terminal.

[0017] "Current location" refers to GPS information obtained from the user's terminal.

[0018] "Traffic information" refers to real-time data related to travel, such as road congestion and public transportation operation status.

[0019] "Optimization means" refers to an algorithm that uses the user's current location and traffic information to recalculate the route and modify it as necessary.

[0020] "Unplanned circumstances" refers to unexpected changes or disruptions to the plan.

[0021] "New Plan" refers to a proposed travel or activity schedule that is regenerated to accommodate unplanned circumstances.

[0022] "Means of making reservations" refers to API calls and communication technologies used to make reservations for required facilities and activities in cooperation with external reservation services. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention is a concierge system that helps users to enjoy their holidays and leisure time in an optimal and enjoyable way. This system starts by inputting the user's wishes and requirements, then generates an optimal plan, calculates the corresponding travel route, and makes reservations, thereby reducing the user's stress and helping them to spend their time in a fulfilling way.

[0045] Program processing

[0046] User Input Processing

[0047] Users connect to the device and input their preferences and requirements into the system in natural language. For example, they could input a request such as, "I'm looking for an indoor facility that I can enjoy with my children. I'd also like to enjoy a delicious lunch." The device then sends this information to the server, which begins analyzing it.

[0048] Hope Data Analysis

[0049] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch," etc.). This extracted information is used in the subsequent plan generation process.

[0050] Generation and evaluation of plan candidates

[0051] Based on the analyzed requirements, the server searches for corresponding spots and activities from an internal database and external APIs. For example, nearby indoor facilities for children or popular lunch spots are potential candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[0052] Calculating travel routes

[0053] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, etc.

[0054] Plan and route distribution

[0055] The server sends the calculated plan and travel route information to the terminal, which receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0056] Real-time support

[0057] The device periodically sends the user's current location and traffic information to the server. The server uses this information to optimize the route accordingly. For example, if an unexpected traffic jam occurs, the server calculates a detour route and notifies the user by sending it to the device.

[0058] Emergency troubleshooting

[0059] If a user encounters an unexpected situation, for example by entering information such as "I'm going to be late for my appointment," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changes in plans.

[0060] Specific examples

[0061] Example 1: One-day trip for families

[0062] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0063] Example 2: Weekend date plan for couples

[0064] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0065] This invention allows users to significantly reduce the effort required for planning and to respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user operates the terminal and inputs their requests and requirements into the system. For example, they might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy lunch." The terminal receives this information and sends it to the server.

[0069] Step 2:

[0070] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "kids," "indoor," "lunch," etc.).

[0071] Step 3:

[0072] The server generates multiple search queries based on the extracted requirements and sends them to an internal database and an external API, which returns a list of corresponding indoor facilities for children and lunch spots, while the external API returns reviews, ratings, and current occupancy status of the spots.

[0073] Step 4:

[0074] The server then aggregates the returned data and generates multiple itinerary options based on the information it has gathered, such as multiple itinerary options combining several facilities and lunch spots.

[0075] Step 5:

[0076] The server evaluates the generated plan candidates and selects the best one. The evaluation algorithm takes into account the weighting of the user's requirements and the local conditions (weather, congestion, etc.).

[0077] Step 6:

[0078] The server calculates the route based on the optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, etc.

[0079] Step 7:

[0080] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[0081] Step 8:

[0082] The user checks the plan displayed on the device and presses a button to start navigation. The device acquires its current location via GPS and begins navigation to the first destination.

[0083] Step 9:

[0084] The device periodically sends the user's current location and traffic information to the server, which then recalculates the route as needed and sends the new route to the device.

[0085] Step 10:

[0086] When a user encounters an unexpected situation, for example, "I'm late for my appointment," the user inputs this information into the terminal, which then transmits it to the server.

[0087] Step 11:

[0088] The server analyzes the situation and generates a new plan and route, which may change the next destination or rearrange the schedule to accommodate delays.

[0089] Step 12:

[0090] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and displays new instructions to the user.

[0091] Step 13:

[0092] When a user needs to make a reservation at a restaurant, for example, they send a request through their terminal, which then forwards the reservation request to the server.

[0093] Step 14:

[0094] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[0095] The above are the specific processing steps and detailed operations of the program based on the claims, which allow users to spend their time stress-free and productively planning and troubleshooting.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional planning systems for users' holidays and leisure time have had problems such as difficulty reflecting users' detailed wishes and requirements and difficulty in dealing with problems in real time. Furthermore, users had to manually make reservations and check traffic conditions, which made planning inefficient. The present invention aims to solve these problems and provide a system that allows users to spend their time stress-free and fulfilling.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions and extracting key information, means for generating a plurality of plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for extracting key information using natural language processing, means for calculating a travel route taking weather information and traffic congestion into consideration, and means for optimizing the route in real time based on the user's current location and traffic information. This not only enables the user to automatically and efficiently obtain an optimal plan, but also enables flexible response to unexpected problems.

[0101] The "means for inputting user's wishes and conditions" is a system component that provides an interface that allows the user to input his or her wishes and conditions in natural language.

[0102] The "means for analyzing input preferences and conditions and extracting key information" is a system component that uses natural language processing to extract and analyze important keywords and phrases from the text entered by the user.

[0103] A "means for generating multiple candidate plans" is a system component that designs and generates multiple possible action plans based on the analyzed information.

[0104] The "means for evaluating the generated plan candidates and selecting the optimal plan" is a system component that compares and evaluates the multiple generated plan candidates based on predetermined evaluation criteria and selects the optimal plan.

[0105] The "means for calculating a travel route" is a system component that efficiently calculates a travel route between each spot based on the selected plan.

[0106] "Means for transmitting calculated route information and plan information to a user's terminal" refers to a system component that transmits travel route and plan information calculated and generated by the server to a user's terminal.

[0107] The "means for optimizing a route based on the user's current location and traffic information" is a system component that dynamically optimizes a travel route using the user's real-time location information and traffic information.

[0108] The "means for generating a new plan in response to an unexpected situation and transmitting it to the user terminal" is a system component that quickly generates a new plan when the user encounters an unexpected situation and transmits it to the user's terminal.

[0109] The "means for making a reservation with an external service when a reservation is necessary" is a system component that makes a reservation for a facility or service for the user using an external reservation service as needed.

[0110] "Means for extracting key information using natural language processing" refers to a system component that utilizes natural language processing technology to analyze and extract key information from text entered by a user.

[0111] The "means for calculating a travel route taking into consideration weather information and congestion status" is a system component that calculates an optimal travel route taking into consideration external information such as weather information and congestion status of a location.

[0112] "Means for optimizing routes in real time based on the user's current location and traffic information" refers to a system component that dynamically changes and optimizes travel routes in real time based on the user's current location information and traffic conditions.

[0113] This invention is a concierge system that provides optimal holiday and leisure plans based on the user's wishes and requirements, helping the user reduce stress and spend their time in a fulfilling manner. This system analyzes the wishes and requirements entered by the user, and generates and provides the optimal plan based on them.

[0114] Specifically, a user first uses a device (smartphone or PC) to input their wishes and requirements into the system in natural language. For example, a user might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy a delicious lunch." This input information is then sent from the device to the server.

[0115] The server uses a natural language processing engine, such as the Google Cloud Natural Language API, to analyze the received data. The NLP engine analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch"), which is then stored in the server's database for further processing.

[0116] Next, the server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for spots and activities that match the user's criteria. For example, nearby indoor facilities for children or popular restaurants may be suggested as candidates. From these search results, multiple plan candidates are generated and each plan is evaluated. Evaluation criteria include weather information (weather information API), congestion level (traffic information API), and review information (review API).

[0117] After the optimal plan is selected, the server calculates the travel route between each spot based on the plan using Google Maps API, Mapbox, etc. This travel route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, and weather information.

[0118] The calculated plan and travel route are sent from the server to the user's device. The device receives it and displays the entire plan and travel route to each spot on the user interface. The user can confirm the presented plan and start navigation.

[0119] In addition, the device periodically obtains the user's current location using its GPS function and sends it to the server. The server dynamically optimizes the travel route as needed based on real-time location and traffic information. For example, if an unexpected traffic jam occurs, the server quickly calculates a detour route, sends it to the device, and notifies the user.

[0120] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment" into their device, the device will send this information to the server. The server will then regenerate a plan based on the new information, adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changing situations.

[0121] Finally, if a reservation is required, the server will automatically make the necessary reservation by connecting with an external reservation service, such as a restaurant reservation site or facility reservation system.

[0122] Specific examples

[0123] Example 1: One-day trip for families

[0124] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0125] Example 2: Weekend date plan for couples

[0126] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of the information.

[0127] Examples of prompt statements

[0128] 1. "I'm looking for an indoor venue that's fun for the kids and I'd also like to enjoy a delicious lunch."

[0129] 2. "Can you recommend a romantic restaurant and a quiet park I can go to before going there?"

[0130] As described above, by using the system of the present invention, users can significantly reduce the effort required for planning and can respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1: User Input Processing

[0133] The user uses a terminal to input their preferences and requirements in natural language. For example, they might input, "I'm looking for an indoor facility that I can enjoy with my kids. I'd also like to enjoy a delicious lunch." This input information is sent to the server as an HTTP request.

[0134] Input: User's wishes and requirements (natural language text)

[0135] Output: Input sent to the server

[0136] Step 2: Data Analysis

[0137] The server sends the received requests and conditions to a natural language processing engine such as Google Cloud Natural Language API for analysis, specifically extracting key information (keywords and phrases).

[0138] The server passes the received text data to the NLP engine and obtains the analysis results.

[0139] Input: Natural language text (user wishes and conditions)

[0140] Output: Key information (keywords: "kids," "indoor," "lunch," etc.)

[0141] Step 3: Generate candidate plans

[0142] The server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for relevant spots and activities based on key information, and generates multiple plan suggestions from the search results.

[0143] Specifically, the server calls each API to build a plan candidate, and analyzes and integrates the response data.

[0144] Input: Key information (keywords)

[0145] Output: A list of possible plans (e.g., "10:00 at XX indoor amusement park, 12:30 at XX restaurant for lunch")

[0146] Step 4: Plan Evaluation and Selection

[0147] The server evaluates the generated plan candidates based on evaluation criteria, including weather information, congestion level, and review information, and selects the optimal plan.

[0148] The server uses the evaluation criteria API to collect the necessary data and uses it to score each plan candidate, and the plan with the highest score is selected.

[0149] Input: List of plan candidates

[0150] Output: Optimal plan

[0151] Step 5: Calculating the travel route

[0152] Based on the optimal plan, the server calculates the travel route between each spot using Google Maps API or Mapbox.

[0153] The server calculates an efficient route taking into account the user's mode of transportation and current traffic conditions.

[0154] Input: Best plan, transportation method, traffic information

[0155] Output: Calculated travel route

[0156] Step 6: Distribute your plan and route

[0157] The server sends the calculated plan and travel route as an HTTP response to the user's device, which receives it and displays it on the user interface.

[0158] Specifically, the terminal analyzes the data received from the server and formats it for display on the screen.

[0159] Input: Calculated travel route, optimal plan

[0160] Output: Plan and route displayed on the user interface

[0161] Step 7: Sending real-time location and traffic information

[0162] The device periodically acquires the user's current location using the GPS function and sends it to the server. It also acquires current traffic information and sends it to the server.

[0163] Input: User's current location, traffic information

[0164] Output: Real-time location and traffic information sent to the server

[0165] Step 8: Optimize travel routes

[0166] The server dynamically optimizes the travel route based on the received real-time information, for example, calculating a new route taking into account traffic congestion and accident information and sending it to the device.

[0167] Input: User's current location, traffic information

[0168] Output: New optimized route information

[0169] Step 9: Dealing with emergency situations

[0170] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment," the device sends this information to the server. The server then generates a new plan, adjusts the schedule, and sends the new plan information to the user's device.

[0171] Input: Trouble information from the user

[0172] Output: New plan information

[0173] At each step, the program efficiently processes data and provides the most appropriate information for the user.

[0174] (Application example 1)

[0175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] In today's world, planning is crucial to making users' holidays and leisure time more efficient and enjoyable. However, in many cases, gathering the information users want and creating optimal plans is time-consuming and requires real-time troubleshooting. This situation increases users' stress and makes it difficult for them to spend their time productively. Furthermore, existing planning systems lack the ability to provide user experiences using virtual reality devices, limiting the means by which users can intuitively review and select plans.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0178] In this invention, the server includes means for inputting a user's wishes and requirements, means for analyzing the input wishes and requirements and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for inputting a user's request by voice using a virtual reality device, means for converting the voice input into text using natural language processing technology, and means for visualizing plans in a virtual space and supporting transitions and exploration. This reduces the user's planning efforts, allows the user to intuitively check and select plans through the virtual reality device, and enables flexible responses to real-time situations.

[0179] The "means for inputting the user's wishes and conditions" is an interface that allows the user to input their wishes and conditions using a terminal or virtual reality device.

[0180] The "means for analyzing input desires and conditions and extracting key information" is a technique for analyzing the data input by the user and extracting key information necessary for generating a plan.

[0181] The "means for generating multiple plan candidates" is an algorithm for generating multiple activity and spot candidates based on the extracted key information.

[0182] The "means for evaluating the generated plan candidates and selecting the optimum plan" is a process for evaluating the generated plan candidates and selecting the optimum plan based on the evaluation results.

[0183] The "means for calculating a travel route" is a method for calculating a route for efficiently traveling between each spot based on the selected plan.

[0184] The "means for transmitting route information and plan information to the user's terminal" refers to a communication technique for sending the calculated travel route and plan information to the user's terminal.

[0185] The "means for optimizing a route based on the user's current location and traffic information" is a system for recalculating and optimizing a travel route based on the user's real-time location information and traffic conditions.

[0186] The "means for generating a new plan and transmitting it to the user terminal" is a function for generating a new plan in response to an unexpected situation and transmitting the information about the plan to the user terminal.

[0187] The "means for making a reservation with an external service when a reservation is necessary" is a system for making a necessary reservation based on the user's wishes and conditions in cooperation with an external reservation service.

[0188] The "means for inputting user requests by voice using a virtual reality device" is an interface that allows the user to wear the virtual reality device and input their wishes and conditions by voice.

[0189] "Means for converting voice input into text using natural language processing technology" refers to a mechanism for converting data input by voice by a user into text data using natural language processing technology.

[0190] "Means for visualizing plans in a virtual space and supporting transitions and exploration" refers to technology that visually displays plans generated using virtual reality technology and supports the process of users intuitively confirming and selecting plans.

[0191] This invention is a concierge system that makes users' holidays and leisure time optimal and enjoyable. This system uses virtual reality (VR) to analyze users' wishes and requirements using natural language processing technology, and proposes and visualizes appropriate plans, thereby improving the user experience.

[0192] To achieve this, users first put on the virtual reality device and input their wishes and requirements through voice. This voice input is converted into text data using the Google Cloud Speech-to-Text API. This text data is then analyzed by the server to extract the key information needed. The NLP (Natural Language Processing) engine used is the Python spaCy library.

[0193] Next, based on the extracted information, the server retrieves data on the relevant spots and activities from an internal database and external APIs, and generates multiple plan candidates. These candidates are evaluated based on evaluation criteria (e.g., weather, congestion, user reviews, etc.), and the most suitable plan is selected. Based on this selected plan, the server calculates the user's most efficient route. The networkx library is used for this calculation.

[0194] The optimal plan and calculated route information are then sent to the user's virtual reality device, where the user can visually check and select a plan within the virtual space. Specifically, the virtual space is created using a 3D engine such as Unity, allowing users to intuitively operate the interface. This allows users to get a visually realistic experience, allowing them to check the details and atmosphere of each spot before visiting it.

[0195] For example, suppose a user voice-inputs into a virtual reality device, "I want to know about indoor attractions to enjoy with my kids and places to have lunch." This input is instantly converted into text data and analyzed by the server. The server searches for related facilities and proposes the optimal plan: "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." These facilities can be visually confirmed in the virtual space along with this plan, allowing the user to understand the environment of the places they want to go in advance. In this way, users can significantly reduce the effort of planning and spend their time more fulfillingly.

[0196] Real-time response is also an important feature. If a user encounters an unexpected situation while out and about, such as traffic congestion or a need to change a reservation time, the server will constantly optimize the route based on the user's current location and traffic information, and send the user new plans and route information. In this way, users can flexibly respond to changes in their plans.

[0197] An example prompt might look like this:

[0198] "I want to know about indoor attractions and places to have lunch that I can enjoy with my kids."

[0199] "I'm looking for a romantic restaurant and a quiet park to go to in front of it."

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] The user wears the virtual reality device and inputs their wishes and requirements by voice. This voice data is captured by a microphone built into the virtual reality device. Input: User's voice input. Output: Voice data.

[0203] Step 2:

[0204] The device sends the captured voice data to the Google Cloud Speech-to-Text API and converts it into text. Input: Voice data. Output: Text data.

[0205] How it works: Sends audio data to an API over the internet and receives the converted text data.

[0206] Step 3:

[0207] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy) and extracts key information. Input: Text data. Output: Extracted key information.

[0208] How it works: Feeds text data into an NLP engine to extract keywords and phrases.

[0209] Step 4:

[0210] The server searches and generates multiple plan candidates from an internal database and external API based on the extracted key information. Input: Extracted key information. Output: Multiple plan candidates.

[0211] What it does: Executes database queries and API requests, and collects and organizes the results.

[0212] Step 5:

[0213] The server evaluates the generated plan candidates and selects the optimal plan. The evaluation takes into account weather information, congestion, reviews, etc. Input: Multiple plan candidates. Output: The optimal plan.

[0214] How it works: Each plan is scored according to the evaluation criteria, and the plan with the highest score is selected.

[0215] Step 6:

[0216] The server uses the networkx library to calculate the most efficient route for the user based on the optimal plan. Input: Optimal plan. Output: Travel route.

[0217] Operation: Calculates travel costs between each spot and finds the shortest route.

[0218] Step 7:

[0219] The server sends the calculated travel route and plan information to the terminal. Input: Travel route and plan information. Output: Information displayed on the user's terminal.

[0220] Operation: Data is delivered to the terminal using a communication protocol.

[0221] Step 8:

[0222] The user confirms and selects the received plan and travel route in the virtual space. The virtual reality device visually displays the plan using a 3D engine such as Unity. Input: Travel route and plan information. Output: User selection.

[0223] Action: Check and select a plan through interface operations within the virtual space.

[0224] Step 9:

[0225] Based on the user's current location and traffic information, the server optimizes the route. Input: User's current location and real-time traffic information. Output: Optimized route.

[0226] How it works: Periodically collects location and traffic information and recalculates routes as needed.

[0227] Step 10:

[0228] When the server encounters an unexpected situation, it generates a new plan and sends it to the user terminal. Input: Information about the unexpected situation. Output: New plan information.

[0229] Operation: Generates a plan according to the situation and notifies the user.

[0230] Step 11:

[0231] If a reservation is required, the server makes the reservation to an external service. Input: Information about the reservation. Output: Reservation confirmation information.

[0232] What it does: Calls the API of an external booking service, makes the necessary reservations, and receives confirmation.

[0233] In this way, the system of the present invention starts with user input and, through a series of steps including analysis, plan generation, evaluation, route calculation, information distribution, user selection, real-time response, and reservation, proposes and realizes the optimal way for users to spend their holidays and leisure time.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] The present invention is a system that generates plans based on the user's wishes and conditions, and suggests optimal categories and routes in real time to make the user's holidays and leisure time optimal and enjoyable. It also uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that.

[0236] Program processing

[0237] User Input Processing

[0238] The user connects to the device and inputs their wishes and requirements into the system in natural language. For example, they can input wishes such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0239] Hope Data Analysis

[0240] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation," "fun places," etc.). This extracted information is used in the subsequent plan generation process.

[0241] Recognition of emotional states

[0242] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stress, relaxation, excitement, etc.).

[0243] Generation and evaluation of plan candidates

[0244] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external API. For example, relaxation facilities and fun activity spots are considered as candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[0245] Calculating travel routes

[0246] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0247] Plan and route distribution

[0248] The server sends the calculated plan and route information in JSON format to the device. The device receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0249] Real-time support

[0250] The device periodically transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[0251] Emergency troubleshooting

[0252] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0253] Specific examples

[0254] Example 1: A plan for families who want to relax

[0255] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0256] Example 2: A weekend date plan for couples who want to have fun

[0257] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0258] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The user operates the device and inputs their wishes and requirements to the system in natural language. For example, they might input a wish such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0262] Step 2:

[0263] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation" or "fun place").

[0264] Step 3:

[0265] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[0266] Step 4:

[0267] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external APIs, extracting candidates such as relaxation facilities and fun activities.

[0268] Step 5:

[0269] The server generates multiple plan candidates based on the search results, each based on improved data (e.g., facility reviews, ratings, current occupancy status, etc.).

[0270] Step 6:

[0271] The server evaluates the generated plan candidates using an evaluation algorithm to select the optimal plan, taking into account the weighting of the user's requirements and the on-site conditions (weather, congestion, the user's emotional state, etc.).

[0272] Step 7:

[0273] The server then calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, and the user's emotional state.

[0274] Step 8:

[0275] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[0276] Step 9:

[0277] The user checks the plan displayed on the device and presses the button to start navigation. The device acquires the current location using GPS and starts navigation to the first destination.

[0278] Step 10:

[0279] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route based on this information and sends the new route information to the device.

[0280] Step 11:

[0281] When a user encounters an unexpected situation, for example, "I'm going to be late for my appointment," the user inputs information into the terminal, which then transmits this information to the server.

[0282] Step 12:

[0283] The server then generates an optimal plan based on the relevant information and adjusts the schedule, for example, by changing the next destination or adding new spots.

[0284] Step 13:

[0285] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and presents the new instructions to the user.

[0286] Step 14:

[0287] When a user needs to make a reservation at a restaurant, for example, the user sends a request through the terminal, which then forwards the reservation request to the server.

[0288] Step 15:

[0289] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[0290] Through this series of processes, users receive support from the system throughout the entire process, from planning to execution, and by using an emotion engine, the system provides an optimal experience tailored to the user's emotional state.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] Conventional holiday and leisure planning systems only consider the user's preferences and conditions, failing to fully reflect the user's emotional state and making it difficult to respond flexibly when the user is unable to carry out their planned activities or when trouble occurs. Furthermore, there is a lack of a way to recognize the user's emotional state in real time and optimize the plan based on that. Furthermore, it is difficult to optimize travel routes based on traffic and weather information in real time, making it difficult to improve the user experience.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0295] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions using natural language processing and extracting key information, means for generating multiple plan candidates based on the analyzed information and an emotion engine that recognizes the user's emotional state, means for evaluating the generated plan candidates based on weather information, congestion status, user reviews, etc. and selecting an optimal plan, means for calculating a travel route based on the optimal plan while taking current traffic conditions and weather information into consideration, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route in real time based on the user's current location, traffic information, and emotional state, means for generating a new plan in response to an unplanned situation and transmitting it to the user's terminal, and means for making a reservation with an external service if a reservation is necessary. This makes it possible to provide an optimal plan that takes the user's emotional state into consideration, optimize the travel route in real time, and flexibly respond to unplanned situations.

[0296] "Means for inputting user wishes and conditions" refers to an interface or device that allows a user to input their own objectives or requests.

[0297] "Means of analyzing and extracting key information using natural language processing" refers to software or algorithms that analyze text entered by a user in natural language and automatically extract important keywords and related information.

[0298] An "emotional state recognition engine" refers to a system or engine that analyzes user input data and behavioral data to determine the user's current emotional state.

[0299] "Means for generating candidate plans" refers to functions or algorithms for creating a plan of multiple activities and places to visit based on the analyzed information and emotional state.

[0300] "Means of evaluating and selecting the optimal plan based on weather information, congestion status, user reviews, etc." refers to a function that evaluates generated plan candidates based on various external data and allows the user to select the optimal plan.

[0301] "Means for calculating travel routes taking into account current traffic and weather information" refers to systems or algorithms that take into account current traffic and weather information when calculating efficient travel routes based on the selected plan.

[0302] "Means for transmitting route information and plan information to a user's device" refers to the communications technology and infrastructure for delivering real-time calculated travel routes and activity plans to a user's device.

[0303] "Means of optimizing routes in real time based on the user's current location, traffic information, and emotional state" refers to a function that acquires and analyzes data such as the user's location, traffic conditions, and emotional state in real time, and optimizes routes as necessary.

[0304] "Means for generating a new plan in response to an unexpected situation and transmitting it to a user terminal" refers to a system that quickly creates a new plan and delivers it to the user's device when the user faces an unexpected situation.

[0305] "Means for making reservations with external services when reservations are required" refers to a function that automatically makes reservations using external reservation services when reservations are required for activities or spots within a plan.

[0306] The present invention is a system that generates a plan for a user to spend their holidays or leisure time optimally and enjoyably, and suggests optimal categories and routes in real time. It is particularly distinctive in that it uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that. Specific embodiments of the present invention are described below.

[0307] User Input Processing

[0308] Users can connect to the device and input their wishes and requirements in natural language. For example, they can input content such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0309] Hope Data Analysis

[0310] The server analyzes the data received from the user using a natural language processing (NLP) engine. The NLP engine analyzes the text and extracts key keywords and requirements. For example, keywords such as "relaxation" and "fun places" may be extracted. This information is used in the subsequent plan generation process.

[0311] Recognition of emotional states

[0312] The server passes the user's input data and behavior data (e.g., input speed, facial expression recognition data, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[0313] Generation and evaluation of plan candidates

[0314] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested as candidates. Multiple plan candidates are generated from the search results, and the optimal plan is selected by evaluating them based on conditions such as weather information, congestion level, and user reviews.

[0315] Calculating travel routes

[0316] The server calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0317] Plan and route distribution

[0318] The server sends the calculated plan and route information in JSON format to the device. The device receives this information and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0319] Real-time support

[0320] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[0321] Emergency troubleshooting

[0322] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0323] Specific examples

[0324] Example 1: A plan for families who want to relax

[0325] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0326] Example 2: A weekend date plan for couples who want to have fun

[0327] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0328] Prompt Sentence Examples

[0329] "I want to relax today"

[0330] "I want to go somewhere that makes me feel happy."

[0331] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1:

[0334] The user inputs their wishes and conditions into the device. For example, they input text in natural language such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this input and temporarily stores the data.

[0335] Input: User's wishes and requirements (natural language text)

[0336] Output: Temporarily saved user preferences and conditions (natural language text)

[0337] Step 2:

[0338] The device sends the temporarily saved text data to the server, where it becomes the raw material for analysis.

[0339] Input: Temporarily saved user preferences and conditions (natural language text)

[0340] Output: The desired and required data sent to the server (natural language text)

[0341] Step 3:

[0342] The server receives the data and passes it to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements, such as "relaxing" and "fun places."

[0343] Input: Desired and condition data sent to the server (natural language text)

[0344] Output: Extracted keywords and requirements (list format)

[0345] Step 4:

[0346] The server sends the user's input data and movement data (such as input speed and facial expression recognition data) to the emotion engine, which analyzes this data and recognizes the user's current emotional state (such as stress, relaxation, or excitement).

[0347] Input: User input and action data

[0348] Output: Recognized emotional state (list format)

[0349] Step 5:

[0350] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested.

[0351] Input: extracted keywords and requirements, recognized emotional states

[0352] Output: Plan candidates (list format)

[0353] Step 6:

[0354] The server evaluates the generated plan candidates based on weather information, congestion levels, user reviews, etc., and selects the optimal plan. A scoring algorithm is used for the evaluation.

[0355] Input: Plan options, weather information, congestion level, user reviews

[0356] Output: Best plan (list format)

[0357] Step 7:

[0358] The server calculates an efficient route based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0359] Input: Best plan, transportation method, traffic conditions, weather information, emotional state

[0360] Output: Optimal travel route (route information)

[0361] Step 8:

[0362] The server sends the calculated route and plan information in JSON format to the device, which displays the received information on the user interface. The user confirms this and starts navigation.

[0363] Input: Optimal travel route, plan information

[0364] Output: Navigation information displayed on the user's device

[0365] Step 9:

[0366] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[0367] Input: User's current location, traffic information, emotional state

[0368] Output: Recalculated route information

[0369] Step 10:

[0370] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0371] Input: User input for unplanned situations

[0372] Output: New plan information

[0373] (Application example 2)

[0374] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Conventional plan generation systems only considered the user's preferences and conditions, making it difficult to provide optimal plans that reflected the user's emotional state. Food delivery services also faced the problem of being unable to provide suggestions tailored to the user's emotions and mood. The present invention aims to solve this problem by providing optimal plans and food delivery plans that reflect the user's emotional state.

[0376] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting a user's preferences and conditions, means for analyzing the input preferences and conditions and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for recognizing the user's emotional state, and means for generating and notifying a food delivery plan based on the user's emotional state. This makes it possible to provide an optimal plan and food delivery proposal that takes the user's emotional state into consideration.

[0377] "User's wishes and conditions" refer to the specific objectives and needs that the user wants to achieve with the system, as well as the constraints imposed on them.

[0378] "Key information" refers to keywords and requirements that are important for generating plans and routes, extracted from the analyzed user's wishes and conditions.

[0379] "Plan candidates" refer to multiple suggested activities or actions that are generated based on the user's wishes, conditions, and emotional state.

[0380] The "optimal plan" refers to the plan that best suits the user's wishes, conditions, and emotional state from among the generated plan candidates.

[0381] A "travel route" refers to a route that a user takes to travel efficiently to a destination or activity location based on the selected optimal plan.

[0382] "Calculated route information and plan information" refers to information on specific routes and activity details required when a user moves or performs an action.

[0383] "Current location and traffic information" refers to data relating to the user's current location and data relating to the traffic conditions in the vicinity thereof.

[0384] "Unplanned circumstances" refers to unexpected events or changes that a user may encounter while executing a plan.

[0385] "External services" refers to external service providers or platforms with which the system works in order to make reservations, provide information, etc.

[0386] "User's emotional state" refers to the user's mental state, such as stress, relaxation, excitement, etc., as recognized by the emotion engine.

[0387] "Food Delivery Plan" refers to an optimal food delivery proposal generated based on the user's preferences, conditions and emotional state.

[0388] The present invention is a system that uses an emotion engine to recognize a user's emotional state and generate and provide an optimal plan based on the user's wishes and conditions, in order to make the user's holidays and leisure time optimal and enjoyable. A specific embodiment of this system is described below.

[0389] Hardware and software used

[0390] 1. Hardware:

[0391] Smartphone (including camera and microphone)

[0392] 2. Software:

[0393] Python

[0394] Emotion recognition module (e.g., Face++, OpenCV, etc.)

[0395] Natural language processing modules (e.g., spaCy, Google Natural Language API, etc.)

[0396] Program processing

[0397] User Input Processing

[0398] Through a smartphone application, users input their wishes and conditions in natural language, such as "I want to relax today" or "I want to go to a place that makes me feel happy." The system receives this information and sends it to the server.

[0399] Hope Data Analysis

[0400] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements. This extracted information is used in the subsequent plan generation process.

[0401] Recognition of emotional states

[0402] The server passes data such as user input and facial expression recognition (using a camera) to the emotion engine, which analyzes this data and recognizes the user's emotional state. For example, it determines whether the user is stressed, relaxed, excited, etc.

[0403] Generation and evaluation of plan candidates

[0404] The server searches for multiple options, such as food delivery services and tourist attractions, based on the analyzed preferences and emotional state. For example, options could include a relaxing cafe or a fun restaurant. From the search results, multiple plan options are generated, and the optimal plan is selected by evaluating the conditions of each plan (reviews, congestion, etc.).

[0405] Route calculation and notifications

[0406] The server calculates an efficient route to visit each spot based on the selected optimal plan. This calculation takes into account current traffic conditions and weather information. The calculated route and plan information are sent to the user's device and displayed on the user interface.

[0407] Real-time response and troubleshooting

[0408] The device periodically sends the user's current location, traffic information, and emotional state to the server. The server uses this information to recalculate the route as needed and sends a new route to the device. For example, if the user is running late for an appointment, the server generates a new plan and adjusts the schedule.

[0409] Specific examples

[0410] Example 1: A plan for families looking to relax

[0411] The user inputs "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. After that, it proposes the optimal schedule and route for the day, such as "hot spring at 10:00, lunch at a restaurant at 12:30." If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0412] Example 2: A weekend date plan for couples who want to have fun

[0413] The user inputs "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotions are indicative of a desire for fun, suggests a plan of "Amusement park at 15:00, dinner at a restaurant at 18:00." If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0414] Prompt Sentence Examples

[0415] "I want to feel refreshed today"

[0416] "I want the perfect food to relieve stress."

[0417] This allows users to enjoy greater convenience and flexibility in planning and troubleshooting, and provides an optimal experience tailored to their emotional state.

[0418] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0419] Step 1:

[0420] User Input

[0421] The user starts the smartphone application and inputs their wishes and conditions in natural language. For example, they input prompts such as "I want to feel refreshed today" or "I want to go to a place that makes me feel happy." This generates user input data, which the device then sends to the server.

[0422] Input: User's wishes and conditions (e.g., I want to feel refreshed today)

[0423] Output: User input data

[0424] Step 2:

[0425] Natural Language Processing (NLP)

[0426] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the input data and extracts key keywords and requirements. The extracted information is used in the subsequent plan generation process.

[0427] Input: User-entered data

[0428] Data processing / calculation: natural language analysis, keyword extraction

[0429] Output: Key words (e.g., refresh)

[0430] Step 3:

[0431] Recognition of emotional states

[0432] The server passes the user's facial expressions and input speed to the emotion engine via the smartphone's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited).

[0433] Input: facial expression data, input speed data

[0434] Data processing / calculation: Emotion analysis

[0435] Output: User's emotional state (e.g., relaxed state)

[0436] Step 4:

[0437] Generate candidate plans

[0438] The server generates multiple plan candidates based on keywords extracted by the NLP engine and emotional states recognized by the emotion engine. For example, hot springs and relaxing cafes are suggested as plan candidates for refreshment. Related external APIs are also used to obtain information.

[0439] Input: Key keywords, user's emotional state

[0440] Data processing / calculation: Plan generation, external API calls

[0441] Output: Plan candidates (e.g., hot spring facilities, relaxation cafes)

[0442] Step 5:

[0443] Evaluation and selection of plan candidates

[0444] The server evaluates the generated plan candidates and selects the optimal plan that best suits the user's desires and emotional state, taking into account criteria such as reviews, congestion, and ease of access.

[0445] Input: Plan candidate

[0446] Data processing / calculation: Plan evaluation, condition comparison

[0447] Output: Optimal plan (e.g., hot spring facility)

[0448] Step 6:

[0449] Calculating travel routes

[0450] Based on the selected optimal plan, the server calculates an efficient route to visit each spot, taking into account current traffic and weather conditions.

[0451] Input: Optimal plan

[0452] Data processing / calculation: Route calculation, traffic and weather information taken into account

[0453] Output: Travel route information

[0454] Step 7:

[0455] Plan and route distribution

[0456] The server sends the calculated route and plan information in JSON format to the user's device, which receives it and displays it on the user interface.

[0457] Input: Travel route information, plan information

[0458] Data processing / calculation: Data format conversion (JSON)

[0459] Output: Data delivered to the user's device

[0460] Step 8:

[0461] Real-time support

[0462] The device continuously transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[0463] Input: User's current location, traffic information, emotional state

[0464] Data processing / calculation: Real-time data analysis, route recalculation

[0465] Output: New route information

[0466] Step 9:

[0467] Emergency troubleshooting

[0468] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate a new plan or adjust the schedule, and send the new plan information to the user.

[0469] Input: Trouble information (e.g., being late for an appointment)

[0470] Data processing / calculation: Plan regeneration, schedule adjustment

[0471] Output: New plan information

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] This invention is a concierge system that helps users to enjoy their holidays and leisure time in an optimal and enjoyable way. This system starts by inputting the user's wishes and requirements, then generates an optimal plan, calculates the corresponding travel route, and makes reservations, thereby reducing the user's stress and helping them to spend their time in a fulfilling way.

[0489] Program processing

[0490] User Input Processing

[0491] Users connect to the device and input their preferences and requirements into the system in natural language. For example, they could input a request such as, "I'm looking for an indoor facility that I can enjoy with my children. I'd also like to enjoy a delicious lunch." The device then sends this information to the server, which begins analyzing it.

[0492] Hope Data Analysis

[0493] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch," etc.). This extracted information is used in the subsequent plan generation process.

[0494] Generation and evaluation of plan candidates

[0495] Based on the analyzed requirements, the server searches for corresponding spots and activities from an internal database and external APIs. For example, nearby indoor facilities for children or popular lunch spots are potential candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[0496] Calculating travel routes

[0497] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, etc.

[0498] Plan and route distribution

[0499] The server sends the calculated plan and travel route information to the terminal, which receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0500] Real-time support

[0501] The device periodically sends the user's current location and traffic information to the server. The server uses this information to optimize the route accordingly. For example, if an unexpected traffic jam occurs, the server calculates a detour route and notifies the user by sending it to the device.

[0502] Emergency troubleshooting

[0503] If a user encounters an unexpected situation, for example by entering information such as "I'm going to be late for my appointment," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changes in plans.

[0504] Specific examples

[0505] Example 1: One-day trip for families

[0506] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0507] Example 2: Weekend date plan for couples

[0508] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0509] This invention allows users to significantly reduce the effort required for planning and to respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[0510] The processing flow will be explained below.

[0511] Step 1:

[0512] The user operates the terminal and inputs their requests and requirements into the system. For example, they might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy lunch." The terminal receives this information and sends it to the server.

[0513] Step 2:

[0514] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "kids," "indoor," "lunch," etc.).

[0515] Step 3:

[0516] The server generates multiple search queries based on the extracted requirements and sends them to an internal database and an external API, which returns a list of corresponding indoor facilities for children and lunch spots, while the external API returns reviews, ratings, and current occupancy status of the spots.

[0517] Step 4:

[0518] The server then aggregates the returned data and generates multiple itinerary options based on the information it has gathered, such as multiple itinerary options combining several facilities and lunch spots.

[0519] Step 5:

[0520] The server evaluates the generated plan candidates and selects the best one. The evaluation algorithm takes into account the weighting of the user's requirements and the local conditions (weather, congestion, etc.).

[0521] Step 6:

[0522] The server calculates the route based on the optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, etc.

[0523] Step 7:

[0524] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[0525] Step 8:

[0526] The user checks the plan displayed on the device and presses a button to start navigation. The device acquires its current location via GPS and begins navigation to the first destination.

[0527] Step 9:

[0528] The device periodically sends the user's current location and traffic information to the server, which then recalculates the route as needed and sends the new route to the device.

[0529] Step 10:

[0530] When a user encounters an unexpected situation, for example, "I'm late for my appointment," the user inputs this information into the terminal, which then transmits it to the server.

[0531] Step 11:

[0532] The server analyzes the situation and generates a new plan and route, which may change the next destination or rearrange the schedule to accommodate delays.

[0533] Step 12:

[0534] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and displays new instructions to the user.

[0535] Step 13:

[0536] When a user needs to make a reservation at a restaurant, for example, they send a request through their terminal, which then forwards the reservation request to the server.

[0537] Step 14:

[0538] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[0539] The above are the specific processing steps and detailed operations of the program based on the claims, which allow users to spend their time stress-free and productively planning and troubleshooting.

[0540] Example 1

[0541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0542] Conventional planning systems for users' holidays and leisure time have had problems such as difficulty reflecting users' detailed wishes and requirements and difficulty in dealing with problems in real time. Furthermore, users had to manually make reservations and check traffic conditions, which made planning inefficient. The present invention aims to solve these problems and provide a system that allows users to spend their time stress-free and fulfilling.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0544] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions and extracting key information, means for generating a plurality of plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for extracting key information using natural language processing, means for calculating a travel route taking weather information and traffic congestion into consideration, and means for optimizing the route in real time based on the user's current location and traffic information. This not only enables the user to automatically and efficiently obtain an optimal plan, but also enables flexible response to unexpected problems.

[0545] The "means for inputting user's wishes and conditions" is a system component that provides an interface that allows the user to input his or her wishes and conditions in natural language.

[0546] The "means for analyzing input preferences and conditions and extracting key information" is a system component that uses natural language processing to extract and analyze important keywords and phrases from the text entered by the user.

[0547] A "means for generating multiple candidate plans" is a system component that designs and generates multiple possible action plans based on the analyzed information.

[0548] The "means for evaluating the generated plan candidates and selecting the optimal plan" is a system component that compares and evaluates the multiple generated plan candidates based on predetermined evaluation criteria and selects the optimal plan.

[0549] The "means for calculating a travel route" is a system component that efficiently calculates a travel route between each spot based on the selected plan.

[0550] "Means for transmitting calculated route information and plan information to a user's terminal" refers to a system component that transmits travel route and plan information calculated and generated by the server to a user's terminal.

[0551] The "means for optimizing a route based on the user's current location and traffic information" is a system component that dynamically optimizes a travel route using the user's real-time location information and traffic information.

[0552] The "means for generating a new plan in response to an unexpected situation and transmitting it to the user terminal" is a system component that quickly generates a new plan when the user encounters an unexpected situation and transmits it to the user's terminal.

[0553] The "means for making a reservation with an external service when a reservation is necessary" is a system component that makes a reservation for a facility or service for the user using an external reservation service as needed.

[0554] "Means for extracting key information using natural language processing" refers to a system component that utilizes natural language processing technology to analyze and extract key information from text entered by a user.

[0555] The "means for calculating a travel route taking into consideration weather information and congestion status" is a system component that calculates an optimal travel route taking into consideration external information such as weather information and congestion status of a location.

[0556] "Means for optimizing routes in real time based on the user's current location and traffic information" refers to a system component that dynamically changes and optimizes travel routes in real time based on the user's current location information and traffic conditions.

[0557] This invention is a concierge system that provides optimal holiday and leisure plans based on the user's wishes and requirements, helping the user reduce stress and spend their time in a fulfilling manner. This system analyzes the wishes and requirements entered by the user, and generates and provides the optimal plan based on them.

[0558] Specifically, a user first uses a device (smartphone or PC) to input their wishes and requirements into the system in natural language. For example, a user might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy a delicious lunch." This input information is then sent from the device to the server.

[0559] The server uses a natural language processing engine, such as the Google Cloud Natural Language API, to analyze the received data. The NLP engine analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch"), which is then stored in the server's database for further processing.

[0560] Next, the server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for spots and activities that match the user's criteria. For example, nearby indoor facilities for children or popular restaurants may be suggested as candidates. From these search results, multiple plan candidates are generated and each plan is evaluated. Evaluation criteria include weather information (weather information API), congestion level (traffic information API), and review information (review API).

[0561] After the optimal plan is selected, the server calculates the travel route between each spot based on the plan using Google Maps API, Mapbox, etc. This travel route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, and weather information.

[0562] The calculated plan and travel route are sent from the server to the user's device. The device receives it and displays the entire plan and travel route to each spot on the user interface. The user can confirm the presented plan and start navigation.

[0563] In addition, the device periodically obtains the user's current location using its GPS function and sends it to the server. The server dynamically optimizes the travel route as needed based on real-time location and traffic information. For example, if an unexpected traffic jam occurs, the server quickly calculates a detour route, sends it to the device, and notifies the user.

[0564] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment" into their device, the device will send this information to the server. The server will then regenerate a plan based on the new information, adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changing situations.

[0565] Finally, if a reservation is required, the server will automatically make the necessary reservation by connecting with an external reservation service, such as a restaurant reservation site or facility reservation system.

[0566] Specific examples

[0567] Example 1: One-day trip for families

[0568] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0569] Example 2: Weekend date plan for couples

[0570] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of the information.

[0571] Examples of prompt statements

[0572] 1. "I'm looking for an indoor venue that's fun for the kids and I'd also like to enjoy a delicious lunch."

[0573] 2. "Can you recommend a romantic restaurant and a quiet park I can go to before going there?"

[0574] As described above, by using the system of the present invention, users can significantly reduce the effort required for planning and can respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[0575] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0576] Step 1: User Input Processing

[0577] The user uses a terminal to input their preferences and requirements in natural language. For example, they might input, "I'm looking for an indoor facility that I can enjoy with my kids. I'd also like to enjoy a delicious lunch." This input information is sent to the server as an HTTP request.

[0578] Input: User's wishes and requirements (natural language text)

[0579] Output: Input sent to the server

[0580] Step 2: Data Analysis

[0581] The server sends the received requests and conditions to a natural language processing engine such as Google Cloud Natural Language API for analysis, specifically extracting key information (keywords and phrases).

[0582] The server passes the received text data to the NLP engine and obtains the analysis results.

[0583] Input: Natural language text (user wishes and conditions)

[0584] Output: Key information (keywords: "kids," "indoor," "lunch," etc.)

[0585] Step 3: Generate candidate plans

[0586] The server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for relevant spots and activities based on key information, and generates multiple plan suggestions from the search results.

[0587] Specifically, the server calls each API to build a plan candidate, and analyzes and integrates the response data.

[0588] Input: Key information (keywords)

[0589] Output: A list of possible plans (e.g., "10:00 at XX indoor amusement park, 12:30 at XX restaurant for lunch")

[0590] Step 4: Plan Evaluation and Selection

[0591] The server evaluates the generated plan candidates based on evaluation criteria, including weather information, congestion level, and review information, and selects the optimal plan.

[0592] The server uses the evaluation criteria API to collect the necessary data and uses it to score each plan candidate, and the plan with the highest score is selected.

[0593] Input: List of plan candidates

[0594] Output: Optimal plan

[0595] Step 5: Calculating the travel route

[0596] Based on the optimal plan, the server calculates the travel route between each spot using Google Maps API or Mapbox.

[0597] The server calculates an efficient route taking into account the user's mode of transportation and current traffic conditions.

[0598] Input: Best plan, transportation method, traffic information

[0599] Output: Calculated travel route

[0600] Step 6: Distribute your plan and route

[0601] The server sends the calculated plan and travel route as an HTTP response to the user's device, which receives it and displays it on the user interface.

[0602] Specifically, the terminal analyzes the data received from the server and formats it for display on the screen.

[0603] Input: Calculated travel route, optimal plan

[0604] Output: Plan and route displayed on the user interface

[0605] Step 7: Sending real-time location and traffic information

[0606] The device periodically acquires the user's current location using the GPS function and sends it to the server. It also acquires current traffic information and sends it to the server.

[0607] Input: User's current location, traffic information

[0608] Output: Real-time location and traffic information sent to the server

[0609] Step 8: Optimize travel routes

[0610] The server dynamically optimizes the travel route based on the received real-time information, for example, calculating a new route taking into account traffic congestion and accident information and sending it to the device.

[0611] Input: User's current location, traffic information

[0612] Output: New optimized route information

[0613] Step 9: Dealing with emergency situations

[0614] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment," the device sends this information to the server. The server then generates a new plan, adjusts the schedule, and sends the new plan information to the user's device.

[0615] Input: Trouble information from the user

[0616] Output: New plan information

[0617] At each step, the program efficiently processes data and provides the most appropriate information for the user.

[0618] (Application example 1)

[0619] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] In today's world, planning is crucial to making users' holidays and leisure time more efficient and enjoyable. However, in many cases, gathering the information users want and creating optimal plans is time-consuming and requires real-time troubleshooting. This situation increases users' stress and makes it difficult for them to spend their time productively. Furthermore, existing planning systems lack the ability to provide user experiences using virtual reality devices, limiting the means by which users can intuitively review and select plans.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0622] In this invention, the server includes means for inputting a user's wishes and requirements, means for analyzing the input wishes and requirements and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for inputting a user's request by voice using a virtual reality device, means for converting the voice input into text using natural language processing technology, and means for visualizing plans in a virtual space and supporting transitions and exploration. This reduces the user's planning efforts, allows the user to intuitively check and select plans through the virtual reality device, and enables flexible responses to real-time situations.

[0623] The "means for inputting the user's wishes and conditions" is an interface that allows the user to input their wishes and conditions using a terminal or virtual reality device.

[0624] The "means for analyzing input desires and conditions and extracting key information" is a technique for analyzing the data input by the user and extracting key information necessary for generating a plan.

[0625] The "means for generating multiple plan candidates" is an algorithm for generating multiple activity and spot candidates based on the extracted key information.

[0626] The "means for evaluating the generated plan candidates and selecting the optimum plan" is a process for evaluating the generated plan candidates and selecting the optimum plan based on the evaluation results.

[0627] The "means for calculating a travel route" is a method for calculating a route for efficiently traveling between each spot based on the selected plan.

[0628] The "means for transmitting route information and plan information to the user's terminal" refers to a communication technique for sending the calculated travel route and plan information to the user's terminal.

[0629] The "means for optimizing a route based on the user's current location and traffic information" is a system for recalculating and optimizing a travel route based on the user's real-time location information and traffic conditions.

[0630] The "means for generating a new plan and transmitting it to the user terminal" is a function for generating a new plan in response to an unexpected situation and transmitting the information about the plan to the user terminal.

[0631] The "means for making a reservation with an external service when a reservation is necessary" is a system for making a necessary reservation based on the user's wishes and conditions in cooperation with an external reservation service.

[0632] The "means for inputting user requests by voice using a virtual reality device" is an interface that allows the user to wear the virtual reality device and input their wishes and conditions by voice.

[0633] "Means for converting voice input into text using natural language processing technology" refers to a mechanism for converting data input by voice by a user into text data using natural language processing technology.

[0634] "Means for visualizing plans in a virtual space and supporting transitions and exploration" refers to technology that visually displays plans generated using virtual reality technology and supports the process of users intuitively confirming and selecting plans.

[0635] This invention is a concierge system that makes users' holidays and leisure time optimal and enjoyable. This system uses virtual reality (VR) to analyze users' wishes and requirements using natural language processing technology, and proposes and visualizes appropriate plans, thereby improving the user experience.

[0636] To achieve this, users first put on the virtual reality device and input their wishes and requirements through voice. This voice input is converted into text data using the Google Cloud Speech-to-Text API. This text data is then analyzed by the server to extract the key information needed. The NLP (Natural Language Processing) engine used is the Python spaCy library.

[0637] Next, based on the extracted information, the server retrieves data on the relevant spots and activities from an internal database and external APIs, and generates multiple plan candidates. These candidates are evaluated based on evaluation criteria (e.g., weather, congestion, user reviews, etc.), and the most suitable plan is selected. Based on this selected plan, the server calculates the user's most efficient route. The networkx library is used for this calculation.

[0638] The optimal plan and calculated route information are then sent to the user's virtual reality device, where the user can visually check and select a plan within the virtual space. Specifically, the virtual space is created using a 3D engine such as Unity, allowing users to intuitively operate the interface. This allows users to get a visually realistic experience, allowing them to check the details and atmosphere of each spot before visiting it.

[0639] For example, suppose a user voice-inputs into a virtual reality device, "I want to know about indoor attractions to enjoy with my kids and places to have lunch." This input is instantly converted into text data and analyzed by the server. The server searches for related facilities and proposes the optimal plan: "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." These facilities can be visually confirmed in the virtual space along with this plan, allowing the user to understand the environment of the places they want to go in advance. In this way, users can significantly reduce the effort of planning and spend their time more fulfillingly.

[0640] Real-time response is also an important feature. If a user encounters an unexpected situation while out and about, such as traffic congestion or a need to change a reservation time, the server will constantly optimize the route based on the user's current location and traffic information, and send the user new plans and route information. In this way, users can flexibly respond to changes in their plans.

[0641] An example prompt might look like this:

[0642] "I want to know about indoor attractions and places to have lunch that I can enjoy with my kids."

[0643] "I'm looking for a romantic restaurant and a quiet park to go to in front of it."

[0644] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0645] Step 1:

[0646] The user wears the virtual reality device and inputs their wishes and requirements by voice. This voice data is captured by a microphone built into the virtual reality device. Input: User's voice input. Output: Voice data.

[0647] Step 2:

[0648] The device sends the captured voice data to the Google Cloud Speech-to-Text API and converts it into text. Input: Voice data. Output: Text data.

[0649] How it works: Sends audio data to an API over the internet and receives the converted text data.

[0650] Step 3:

[0651] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy) and extracts key information. Input: Text data. Output: Extracted key information.

[0652] How it works: Feeds text data into an NLP engine to extract keywords and phrases.

[0653] Step 4:

[0654] The server searches and generates multiple plan candidates from an internal database and external API based on the extracted key information. Input: Extracted key information. Output: Multiple plan candidates.

[0655] What it does: Executes database queries and API requests, and collects and organizes the results.

[0656] Step 5:

[0657] The server evaluates the generated plan candidates and selects the optimal plan. The evaluation takes into account weather information, congestion, reviews, etc. Input: Multiple plan candidates. Output: The optimal plan.

[0658] How it works: Each plan is scored according to the evaluation criteria, and the plan with the highest score is selected.

[0659] Step 6:

[0660] The server uses the networkx library to calculate the most efficient route for the user based on the optimal plan. Input: Optimal plan. Output: Travel route.

[0661] Operation: Calculates travel costs between each spot and finds the shortest route.

[0662] Step 7:

[0663] The server sends the calculated travel route and plan information to the terminal. Input: Travel route and plan information. Output: Information displayed on the user's terminal.

[0664] Operation: Data is delivered to the terminal using a communication protocol.

[0665] Step 8:

[0666] The user confirms and selects the received plan and travel route in the virtual space. The virtual reality device visually displays the plan using a 3D engine such as Unity. Input: Travel route and plan information. Output: User selection.

[0667] Action: Check and select a plan through interface operations within the virtual space.

[0668] Step 9:

[0669] Based on the user's current location and traffic information, the server optimizes the route. Input: User's current location and real-time traffic information. Output: Optimized route.

[0670] How it works: Periodically collects location and traffic information and recalculates routes as needed.

[0671] Step 10:

[0672] When the server encounters an unexpected situation, it generates a new plan and sends it to the user terminal. Input: Information about the unexpected situation. Output: New plan information.

[0673] Operation: Generates a plan according to the situation and notifies the user.

[0674] Step 11:

[0675] If a reservation is required, the server makes the reservation to an external service. Input: Information about the reservation. Output: Reservation confirmation information.

[0676] What it does: Calls the API of an external booking service, makes the necessary reservations, and receives confirmation.

[0677] In this way, the system of the present invention starts with user input and, through a series of steps including analysis, plan generation, evaluation, route calculation, information distribution, user selection, real-time response, and reservation, proposes and realizes the optimal way for users to spend their holidays and leisure time.

[0678] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0679] The present invention is a system that generates plans based on the user's wishes and conditions, and suggests optimal categories and routes in real time to make the user's holidays and leisure time optimal and enjoyable. It also uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that.

[0680] Program processing

[0681] User Input Processing

[0682] The user connects to the device and inputs their wishes and requirements into the system in natural language. For example, they can input wishes such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0683] Hope Data Analysis

[0684] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation," "fun places," etc.). This extracted information is used in the subsequent plan generation process.

[0685] Recognition of emotional states

[0686] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stress, relaxation, excitement, etc.).

[0687] Generation and evaluation of plan candidates

[0688] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external API. For example, relaxation facilities and fun activity spots are considered as candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[0689] Calculating travel routes

[0690] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0691] Plan and route distribution

[0692] The server sends the calculated plan and route information in JSON format to the device. The device receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0693] Real-time support

[0694] The device periodically transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[0695] Emergency troubleshooting

[0696] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0697] Specific examples

[0698] Example 1: A plan for families who want to relax

[0699] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0700] Example 2: A weekend date plan for couples who want to have fun

[0701] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0702] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The user operates the device and inputs their wishes and requirements to the system in natural language. For example, they might input a wish such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0706] Step 2:

[0707] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation" or "fun place").

[0708] Step 3:

[0709] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[0710] Step 4:

[0711] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external APIs, extracting candidates such as relaxation facilities and fun activities.

[0712] Step 5:

[0713] The server generates multiple plan candidates based on the search results, each based on improved data (e.g., facility reviews, ratings, current occupancy status, etc.).

[0714] Step 6:

[0715] The server evaluates the generated plan candidates using an evaluation algorithm to select the optimal plan, taking into account the weighting of the user's requirements and the on-site conditions (weather, congestion, the user's emotional state, etc.).

[0716] Step 7:

[0717] The server then calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, and the user's emotional state.

[0718] Step 8:

[0719] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[0720] Step 9:

[0721] The user checks the plan displayed on the device and presses the button to start navigation. The device acquires the current location using GPS and starts navigation to the first destination.

[0722] Step 10:

[0723] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route based on this information and sends the new route information to the device.

[0724] Step 11:

[0725] When a user encounters an unexpected situation, for example, "I'm going to be late for my appointment," the user inputs information into the terminal, which then transmits this information to the server.

[0726] Step 12:

[0727] The server then generates an optimal plan based on the relevant information and adjusts the schedule, for example, by changing the next destination or adding new spots.

[0728] Step 13:

[0729] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and presents the new instructions to the user.

[0730] Step 14:

[0731] When a user needs to make a reservation at a restaurant, for example, the user sends a request through the terminal, which then forwards the reservation request to the server.

[0732] Step 15:

[0733] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[0734] Through this series of processes, users receive support from the system throughout the entire process, from planning to execution, and by using an emotion engine, the system provides an optimal experience tailored to the user's emotional state.

[0735] Example 2

[0736] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Conventional holiday and leisure planning systems only consider the user's preferences and conditions, failing to fully reflect the user's emotional state and making it difficult to respond flexibly when the user is unable to carry out their planned activities or when trouble occurs. Furthermore, there is a lack of a way to recognize the user's emotional state in real time and optimize the plan based on that. Furthermore, it is difficult to optimize travel routes based on traffic and weather information in real time, making it difficult to improve the user experience.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0739] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions using natural language processing and extracting key information, means for generating multiple plan candidates based on the analyzed information and an emotion engine that recognizes the user's emotional state, means for evaluating the generated plan candidates based on weather information, congestion status, user reviews, etc. and selecting an optimal plan, means for calculating a travel route based on the optimal plan while taking current traffic conditions and weather information into consideration, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route in real time based on the user's current location, traffic information, and emotional state, means for generating a new plan in response to an unplanned situation and transmitting it to the user's terminal, and means for making a reservation with an external service if a reservation is necessary. This makes it possible to provide an optimal plan that takes the user's emotional state into consideration, optimize the travel route in real time, and flexibly respond to unplanned situations.

[0740] "Means for inputting user wishes and conditions" refers to an interface or device that allows a user to input their own objectives or requests.

[0741] "Means of analyzing and extracting key information using natural language processing" refers to software or algorithms that analyze text entered by a user in natural language and automatically extract important keywords and related information.

[0742] An "emotional state recognition engine" refers to a system or engine that analyzes user input data and behavioral data to determine the user's current emotional state.

[0743] "Means for generating candidate plans" refers to functions or algorithms for creating a plan of multiple activities and places to visit based on the analyzed information and emotional state.

[0744] "Means of evaluating and selecting the optimal plan based on weather information, congestion status, user reviews, etc." refers to a function that evaluates generated plan candidates based on various external data and allows the user to select the optimal plan.

[0745] "Means for calculating travel routes taking into account current traffic and weather information" refers to systems or algorithms that take into account current traffic and weather information when calculating efficient travel routes based on the selected plan.

[0746] "Means for transmitting route information and plan information to a user's device" refers to the communications technology and infrastructure for delivering real-time calculated travel routes and activity plans to a user's device.

[0747] "Means of optimizing routes in real time based on the user's current location, traffic information, and emotional state" refers to a function that acquires and analyzes data such as the user's location, traffic conditions, and emotional state in real time, and optimizes routes as necessary.

[0748] "Means for generating a new plan in response to an unexpected situation and transmitting it to a user terminal" refers to a system that quickly creates a new plan and delivers it to the user's device when the user faces an unexpected situation.

[0749] "Means for making reservations with external services when reservations are required" refers to a function that automatically makes reservations using external reservation services when reservations are required for activities or spots within a plan.

[0750] The present invention is a system that generates a plan for a user to spend their holidays or leisure time optimally and enjoyably, and suggests optimal categories and routes in real time. It is particularly distinctive in that it uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that. Specific embodiments of the present invention are described below.

[0751] User Input Processing

[0752] Users can connect to the device and input their wishes and requirements in natural language. For example, they can input content such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[0753] Hope Data Analysis

[0754] The server analyzes the data received from the user using a natural language processing (NLP) engine. The NLP engine analyzes the text and extracts key keywords and requirements. For example, keywords such as "relaxation" and "fun places" may be extracted. This information is used in the subsequent plan generation process.

[0755] Recognition of emotional states

[0756] The server passes the user's input data and behavior data (e.g., input speed, facial expression recognition data, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[0757] Generation and evaluation of plan candidates

[0758] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested as candidates. Multiple plan candidates are generated from the search results, and the optimal plan is selected by evaluating them based on conditions such as weather information, congestion level, and user reviews.

[0759] Calculating travel routes

[0760] The server calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0761] Plan and route distribution

[0762] The server sends the calculated plan and route information in JSON format to the device. The device receives this information and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0763] Real-time support

[0764] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[0765] Emergency troubleshooting

[0766] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0767] Specific examples

[0768] Example 1: A plan for families who want to relax

[0769] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0770] Example 2: A weekend date plan for couples who want to have fun

[0771] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0772] Prompt Sentence Examples

[0773] "I want to relax today"

[0774] "I want to go somewhere that makes me feel happy."

[0775] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[0776] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0777] Step 1:

[0778] The user inputs their wishes and conditions into the device. For example, they input text in natural language such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this input and temporarily stores the data.

[0779] Input: User's wishes and requirements (natural language text)

[0780] Output: Temporarily saved user preferences and conditions (natural language text)

[0781] Step 2:

[0782] The device sends the temporarily saved text data to the server, where it becomes the raw material for analysis.

[0783] Input: Temporarily saved user preferences and conditions (natural language text)

[0784] Output: The desired and required data sent to the server (natural language text)

[0785] Step 3:

[0786] The server receives the data and passes it to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements, such as "relaxing" and "fun places."

[0787] Input: Desired and condition data sent to the server (natural language text)

[0788] Output: Extracted keywords and requirements (list format)

[0789] Step 4:

[0790] The server sends the user's input data and movement data (such as input speed and facial expression recognition data) to the emotion engine, which analyzes this data and recognizes the user's current emotional state (such as stress, relaxation, or excitement).

[0791] Input: User input and action data

[0792] Output: Recognized emotional state (list format)

[0793] Step 5:

[0794] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested.

[0795] Input: extracted keywords and requirements, recognized emotional states

[0796] Output: Plan candidates (list format)

[0797] Step 6:

[0798] The server evaluates the generated plan candidates based on weather information, congestion levels, user reviews, etc., and selects the optimal plan. A scoring algorithm is used for the evaluation.

[0799] Input: Plan options, weather information, congestion level, user reviews

[0800] Output: Best plan (list format)

[0801] Step 7:

[0802] The server calculates an efficient route based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[0803] Input: Best plan, transportation method, traffic conditions, weather information, emotional state

[0804] Output: Optimal travel route (route information)

[0805] Step 8:

[0806] The server sends the calculated route and plan information in JSON format to the device, which displays the received information on the user interface. The user confirms this and starts navigation.

[0807] Input: Optimal travel route, plan information

[0808] Output: Navigation information displayed on the user's device

[0809] Step 9:

[0810] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[0811] Input: User's current location, traffic information, emotional state

[0812] Output: Recalculated route information

[0813] Step 10:

[0814] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[0815] Input: User input for unplanned situations

[0816] Output: New plan information

[0817] (Application example 2)

[0818] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Conventional plan generation systems only considered the user's preferences and conditions, making it difficult to provide optimal plans that reflected the user's emotional state. Food delivery services also faced the problem of being unable to provide suggestions tailored to the user's emotions and mood. The present invention aims to solve this problem by providing optimal plans and food delivery plans that reflect the user's emotional state.

[0820] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting a user's preferences and conditions, means for analyzing the input preferences and conditions and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for recognizing the user's emotional state, and means for generating and notifying a food delivery plan based on the user's emotional state. This makes it possible to provide an optimal plan and food delivery proposal that takes the user's emotional state into consideration.

[0821] "User's wishes and conditions" refer to the specific objectives and needs that the user wants to achieve with the system, as well as the constraints imposed on them.

[0822] "Key information" refers to keywords and requirements that are important for generating plans and routes, extracted from the analyzed user's wishes and conditions.

[0823] "Plan candidates" refer to multiple suggested activities or actions that are generated based on the user's wishes, conditions, and emotional state.

[0824] The "optimal plan" refers to the plan that best suits the user's wishes, conditions, and emotional state from among the generated plan candidates.

[0825] A "travel route" refers to a route that a user takes to travel efficiently to a destination or activity location based on the selected optimal plan.

[0826] "Calculated route information and plan information" refers to information on specific routes and activity details required when a user moves or performs an action.

[0827] "Current location and traffic information" refers to data relating to the user's current location and data relating to the traffic conditions in the vicinity thereof.

[0828] "Unplanned circumstances" refers to unexpected events or changes that a user may encounter while executing a plan.

[0829] "External services" refers to external service providers or platforms with which the system works in order to make reservations, provide information, etc.

[0830] "User's emotional state" refers to the user's mental state, such as stress, relaxation, excitement, etc., as recognized by the emotion engine.

[0831] "Food Delivery Plan" refers to an optimal food delivery proposal generated based on the user's preferences, conditions and emotional state.

[0832] The present invention is a system that uses an emotion engine to recognize a user's emotional state and generate and provide an optimal plan based on the user's wishes and conditions, in order to make the user's holidays and leisure time optimal and enjoyable. A specific embodiment of this system is described below.

[0833] Hardware and software used

[0834] 1. Hardware:

[0835] Smartphone (including camera and microphone)

[0836] 2. Software:

[0837] Python

[0838] Emotion recognition module (e.g., Face++, OpenCV, etc.)

[0839] Natural language processing modules (e.g., spaCy, Google Natural Language API, etc.)

[0840] Program processing

[0841] User Input Processing

[0842] Through a smartphone application, users input their wishes and conditions in natural language, such as "I want to relax today" or "I want to go to a place that makes me feel happy." The system receives this information and sends it to the server.

[0843] Hope Data Analysis

[0844] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements. This extracted information is used in the subsequent plan generation process.

[0845] Recognition of emotional states

[0846] The server passes data such as user input and facial expression recognition (using a camera) to the emotion engine, which analyzes this data and recognizes the user's emotional state. For example, it determines whether the user is stressed, relaxed, excited, etc.

[0847] Generation and evaluation of plan candidates

[0848] The server searches for multiple options, such as food delivery services and tourist attractions, based on the analyzed preferences and emotional state. For example, options could include a relaxing cafe or a fun restaurant. From the search results, multiple plan options are generated, and the optimal plan is selected by evaluating the conditions of each plan (reviews, congestion, etc.).

[0849] Route calculation and notifications

[0850] The server calculates an efficient route to visit each spot based on the selected optimal plan. This calculation takes into account current traffic conditions and weather information. The calculated route and plan information are sent to the user's device and displayed on the user interface.

[0851] Real-time response and troubleshooting

[0852] The device periodically sends the user's current location, traffic information, and emotional state to the server. The server uses this information to recalculate the route as needed and sends a new route to the device. For example, if the user is running late for an appointment, the server generates a new plan and adjusts the schedule.

[0853] Specific examples

[0854] Example 1: A plan for families looking to relax

[0855] The user inputs "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. After that, it proposes the optimal schedule and route for the day, such as "hot spring at 10:00, lunch at a restaurant at 12:30." If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0856] Example 2: A weekend date plan for couples who want to have fun

[0857] The user inputs "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotions are indicative of a desire for fun, suggests a plan of "Amusement park at 15:00, dinner at a restaurant at 18:00." If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0858] Prompt Sentence Examples

[0859] "I want to feel refreshed today"

[0860] "I want the perfect food to relieve stress."

[0861] This allows users to enjoy greater convenience and flexibility in planning and troubleshooting, and provides an optimal experience tailored to their emotional state.

[0862] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0863] Step 1:

[0864] User Input

[0865] The user starts the smartphone application and inputs their wishes and conditions in natural language. For example, they input prompts such as "I want to feel refreshed today" or "I want to go to a place that makes me feel happy." This generates user input data, which the device then sends to the server.

[0866] Input: User's wishes and conditions (e.g., I want to feel refreshed today)

[0867] Output: User input data

[0868] Step 2:

[0869] Natural Language Processing (NLP)

[0870] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the input data and extracts key keywords and requirements. The extracted information is used in the subsequent plan generation process.

[0871] Input: User-entered data

[0872] Data processing / calculation: natural language analysis, keyword extraction

[0873] Output: Key words (e.g., refresh)

[0874] Step 3:

[0875] Recognition of emotional states

[0876] The server passes the user's facial expressions and input speed to the emotion engine via the smartphone's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited).

[0877] Input: facial expression data, input speed data

[0878] Data processing / calculation: Emotion analysis

[0879] Output: User's emotional state (e.g., relaxed state)

[0880] Step 4:

[0881] Generate candidate plans

[0882] The server generates multiple plan candidates based on keywords extracted by the NLP engine and emotional states recognized by the emotion engine. For example, hot springs and relaxing cafes are suggested as plan candidates for refreshment. Related external APIs are also used to obtain information.

[0883] Input: Key keywords, user's emotional state

[0884] Data processing / calculation: Plan generation, external API calls

[0885] Output: Plan candidates (e.g., hot spring facilities, relaxation cafes)

[0886] Step 5:

[0887] Evaluation and selection of plan candidates

[0888] The server evaluates the generated plan candidates and selects the optimal plan that best suits the user's desires and emotional state, taking into account criteria such as reviews, congestion, and ease of access.

[0889] Input: Plan candidate

[0890] Data processing / calculation: Plan evaluation, condition comparison

[0891] Output: Optimal plan (e.g., hot spring facility)

[0892] Step 6:

[0893] Calculating travel routes

[0894] Based on the selected optimal plan, the server calculates an efficient route to visit each spot, taking into account current traffic and weather conditions.

[0895] Input: Optimal plan

[0896] Data processing / calculation: Route calculation, traffic and weather information taken into account

[0897] Output: Travel route information

[0898] Step 7:

[0899] Plan and route distribution

[0900] The server sends the calculated route and plan information in JSON format to the user's device, which receives it and displays it on the user interface.

[0901] Input: Travel route information, plan information

[0902] Data processing / calculation: Data format conversion (JSON)

[0903] Output: Data delivered to the user's device

[0904] Step 8:

[0905] Real-time support

[0906] The device continuously transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[0907] Input: User's current location, traffic information, emotional state

[0908] Data processing / calculation: Real-time data analysis, route recalculation

[0909] Output: New route information

[0910] Step 9:

[0911] Emergency troubleshooting

[0912] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate a new plan or adjust the schedule, and send the new plan information to the user.

[0913] Input: Trouble information (e.g., being late for an appointment)

[0914] Data processing / calculation: Plan regeneration, schedule adjustment

[0915] Output: New plan information

[0916] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0918] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0919] [Third embodiment]

[0920] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0921] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0922] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0923] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0924] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0925] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0926] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0927] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0932] This invention is a concierge system that helps users to enjoy their holidays and leisure time in an optimal and enjoyable way. This system starts by inputting the user's wishes and requirements, then generates an optimal plan, calculates the corresponding travel route, and makes reservations, thereby reducing the user's stress and helping them to spend their time in a fulfilling way.

[0933] Program processing

[0934] User Input Processing

[0935] Users connect to the device and input their preferences and requirements into the system in natural language. For example, they could input a request such as, "I'm looking for an indoor facility that I can enjoy with my children. I'd also like to enjoy a delicious lunch." The device then sends this information to the server, which begins analyzing it.

[0936] Hope Data Analysis

[0937] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch," etc.). This extracted information is used in the subsequent plan generation process.

[0938] Generation and evaluation of plan candidates

[0939] Based on the analyzed requirements, the server searches for corresponding spots and activities from an internal database and external APIs. For example, nearby indoor facilities for children or popular lunch spots are potential candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[0940] Calculating travel routes

[0941] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, etc.

[0942] Plan and route distribution

[0943] The server sends the calculated plan and travel route information to the terminal, which receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[0944] Real-time support

[0945] The device periodically sends the user's current location and traffic information to the server. The server uses this information to optimize the route accordingly. For example, if an unexpected traffic jam occurs, the server calculates a detour route and notifies the user by sending it to the device.

[0946] Emergency troubleshooting

[0947] If a user encounters an unexpected situation, for example by entering information such as "I'm going to be late for my appointment," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changes in plans.

[0948] Specific examples

[0949] Example 1: One-day trip for families

[0950] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[0951] Example 2: Weekend date plan for couples

[0952] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[0953] This invention allows users to significantly reduce the effort required for planning and to respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[0954] The processing flow will be explained below.

[0955] Step 1:

[0956] The user operates the terminal and inputs their requests and requirements into the system. For example, they might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy lunch." The terminal receives this information and sends it to the server.

[0957] Step 2:

[0958] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "kids," "indoor," "lunch," etc.).

[0959] Step 3:

[0960] The server generates multiple search queries based on the extracted requirements and sends them to an internal database and an external API, which returns a list of corresponding indoor facilities for children and lunch spots, while the external API returns reviews, ratings, and current occupancy status of the spots.

[0961] Step 4:

[0962] The server then aggregates the returned data and generates multiple itinerary options based on the information it has gathered, such as multiple itinerary options combining several facilities and lunch spots.

[0963] Step 5:

[0964] The server evaluates the generated plan candidates and selects the best one. The evaluation algorithm takes into account the weighting of the user's requirements and the local conditions (weather, congestion, etc.).

[0965] Step 6:

[0966] The server calculates the route based on the optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, etc.

[0967] Step 7:

[0968] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[0969] Step 8:

[0970] The user checks the plan displayed on the device and presses a button to start navigation. The device acquires its current location via GPS and begins navigation to the first destination.

[0971] Step 9:

[0972] The device periodically sends the user's current location and traffic information to the server, which then recalculates the route as needed and sends the new route to the device.

[0973] Step 10:

[0974] When a user encounters an unexpected situation, for example, "I'm late for my appointment," the user inputs this information into the terminal, which then transmits it to the server.

[0975] Step 11:

[0976] The server analyzes the situation and generates a new plan and route, which may change the next destination or rearrange the schedule to accommodate delays.

[0977] Step 12:

[0978] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and displays new instructions to the user.

[0979] Step 13:

[0980] When a user needs to make a reservation at a restaurant, for example, they send a request through their terminal, which then forwards the reservation request to the server.

[0981] Step 14:

[0982] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[0983] The above are the specific processing steps and detailed operations of the program based on the claims, which allow users to spend their time stress-free and productively planning and troubleshooting.

[0984] Example 1

[0985] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0986] Conventional planning systems for users' holidays and leisure time have had problems such as difficulty reflecting users' detailed wishes and requirements and difficulty in dealing with problems in real time. Furthermore, users had to manually make reservations and check traffic conditions, which made planning inefficient. The present invention aims to solve these problems and provide a system that allows users to spend their time stress-free and fulfilling.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0988] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions and extracting key information, means for generating a plurality of plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for extracting key information using natural language processing, means for calculating a travel route taking weather information and traffic congestion into consideration, and means for optimizing the route in real time based on the user's current location and traffic information. This not only enables the user to automatically and efficiently obtain an optimal plan, but also enables flexible response to unexpected problems.

[0989] The "means for inputting user's wishes and conditions" is a system component that provides an interface that allows the user to input his or her wishes and conditions in natural language.

[0990] The "means for analyzing input preferences and conditions and extracting key information" is a system component that uses natural language processing to extract and analyze important keywords and phrases from the text entered by the user.

[0991] A "means for generating multiple candidate plans" is a system component that designs and generates multiple possible action plans based on the analyzed information.

[0992] The "means for evaluating the generated plan candidates and selecting the optimal plan" is a system component that compares and evaluates the multiple generated plan candidates based on predetermined evaluation criteria and selects the optimal plan.

[0993] The "means for calculating a travel route" is a system component that efficiently calculates a travel route between each spot based on the selected plan.

[0994] "Means for transmitting calculated route information and plan information to a user's terminal" refers to a system component that transmits travel route and plan information calculated and generated by the server to a user's terminal.

[0995] The "means for optimizing a route based on the user's current location and traffic information" is a system component that dynamically optimizes a travel route using the user's real-time location information and traffic information.

[0996] The "means for generating a new plan in response to an unexpected situation and transmitting it to the user terminal" is a system component that quickly generates a new plan when the user encounters an unexpected situation and transmits it to the user's terminal.

[0997] The "means for making a reservation with an external service when a reservation is necessary" is a system component that makes a reservation for a facility or service for the user using an external reservation service as needed.

[0998] "Means for extracting key information using natural language processing" refers to a system component that utilizes natural language processing technology to analyze and extract key information from text entered by a user.

[0999] The "means for calculating a travel route taking into consideration weather information and congestion status" is a system component that calculates an optimal travel route taking into consideration external information such as weather information and congestion status of a location.

[1000] "Means for optimizing routes in real time based on the user's current location and traffic information" refers to a system component that dynamically changes and optimizes travel routes in real time based on the user's current location information and traffic conditions.

[1001] This invention is a concierge system that provides optimal holiday and leisure plans based on the user's wishes and requirements, helping the user reduce stress and spend their time in a fulfilling manner. This system analyzes the wishes and requirements entered by the user, and generates and provides the optimal plan based on them.

[1002] Specifically, a user first uses a device (smartphone or PC) to input their wishes and requirements into the system in natural language. For example, a user might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy a delicious lunch." This input information is then sent from the device to the server.

[1003] The server uses a natural language processing engine, such as the Google Cloud Natural Language API, to analyze the received data. The NLP engine analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch"), which is then stored in the server's database for further processing.

[1004] Next, the server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for spots and activities that match the user's criteria. For example, nearby indoor facilities for children or popular restaurants may be suggested as candidates. From these search results, multiple plan candidates are generated and each plan is evaluated. Evaluation criteria include weather information (weather information API), congestion level (traffic information API), and review information (review API).

[1005] After the optimal plan is selected, the server calculates the travel route between each spot based on the plan using Google Maps API, Mapbox, etc. This travel route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, and weather information.

[1006] The calculated plan and travel route are sent from the server to the user's device. The device receives it and displays the entire plan and travel route to each spot on the user interface. The user can confirm the presented plan and start navigation.

[1007] In addition, the device periodically obtains the user's current location using its GPS function and sends it to the server. The server dynamically optimizes the travel route as needed based on real-time location and traffic information. For example, if an unexpected traffic jam occurs, the server quickly calculates a detour route, sends it to the device, and notifies the user.

[1008] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment" into their device, the device will send this information to the server. The server will then regenerate a plan based on the new information, adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changing situations.

[1009] Finally, if a reservation is required, the server will automatically make the necessary reservation by connecting with an external reservation service, such as a restaurant reservation site or facility reservation system.

[1010] Specific examples

[1011] Example 1: One-day trip for families

[1012] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1013] Example 2: Weekend date plan for couples

[1014] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of the information.

[1015] Examples of prompt statements

[1016] 1. "I'm looking for an indoor venue that's fun for the kids and I'd also like to enjoy a delicious lunch."

[1017] 2. "Can you recommend a romantic restaurant and a quiet park I can go to before going there?"

[1018] As described above, by using the system of the present invention, users can significantly reduce the effort required for planning and can respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[1019] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1020] Step 1: User Input Processing

[1021] The user uses a terminal to input their preferences and requirements in natural language. For example, they might input, "I'm looking for an indoor facility that I can enjoy with my kids. I'd also like to enjoy a delicious lunch." This input information is sent to the server as an HTTP request.

[1022] Input: User's wishes and requirements (natural language text)

[1023] Output: Input sent to the server

[1024] Step 2: Data Analysis

[1025] The server sends the received requests and conditions to a natural language processing engine such as Google Cloud Natural Language API for analysis, specifically extracting key information (keywords and phrases).

[1026] The server passes the received text data to the NLP engine and obtains the analysis results.

[1027] Input: Natural language text (user wishes and conditions)

[1028] Output: Key information (keywords: "kids," "indoor," "lunch," etc.)

[1029] Step 3: Generate candidate plans

[1030] The server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for relevant spots and activities based on key information, and generates multiple plan suggestions from the search results.

[1031] Specifically, the server calls each API to build a plan candidate, and analyzes and integrates the response data.

[1032] Input: Key information (keywords)

[1033] Output: A list of possible plans (e.g., "10:00 at XX indoor amusement park, 12:30 at XX restaurant for lunch")

[1034] Step 4: Plan Evaluation and Selection

[1035] The server evaluates the generated plan candidates based on evaluation criteria, including weather information, congestion level, and review information, and selects the optimal plan.

[1036] The server uses the evaluation criteria API to collect the necessary data and uses it to score each plan candidate, and the plan with the highest score is selected.

[1037] Input: List of plan candidates

[1038] Output: Optimal plan

[1039] Step 5: Calculating the travel route

[1040] Based on the optimal plan, the server calculates the travel route between each spot using Google Maps API or Mapbox.

[1041] The server calculates an efficient route taking into account the user's mode of transportation and current traffic conditions.

[1042] Input: Best plan, transportation method, traffic information

[1043] Output: Calculated travel route

[1044] Step 6: Distribute your plan and route

[1045] The server sends the calculated plan and travel route as an HTTP response to the user's device, which receives it and displays it on the user interface.

[1046] Specifically, the terminal analyzes the data received from the server and formats it for display on the screen.

[1047] Input: Calculated travel route, optimal plan

[1048] Output: Plan and route displayed on the user interface

[1049] Step 7: Sending real-time location and traffic information

[1050] The device periodically acquires the user's current location using the GPS function and sends it to the server. It also acquires current traffic information and sends it to the server.

[1051] Input: User's current location, traffic information

[1052] Output: Real-time location and traffic information sent to the server

[1053] Step 8: Optimize travel routes

[1054] The server dynamically optimizes the travel route based on the received real-time information, for example, calculating a new route taking into account traffic congestion and accident information and sending it to the device.

[1055] Input: User's current location, traffic information

[1056] Output: New optimized route information

[1057] Step 9: Dealing with emergency situations

[1058] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment," the device sends this information to the server. The server then generates a new plan, adjusts the schedule, and sends the new plan information to the user's device.

[1059] Input: Trouble information from the user

[1060] Output: New plan information

[1061] At each step, the program efficiently processes data and provides the most appropriate information for the user.

[1062] (Application example 1)

[1063] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1064] In today's world, planning is crucial to making users' holidays and leisure time more efficient and enjoyable. However, in many cases, gathering the information users want and creating optimal plans is time-consuming and requires real-time troubleshooting. This situation increases users' stress and makes it difficult for them to spend their time productively. Furthermore, existing planning systems lack the ability to provide user experiences using virtual reality devices, limiting the means by which users can intuitively review and select plans.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1066] In this invention, the server includes means for inputting a user's wishes and requirements, means for analyzing the input wishes and requirements and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for inputting a user's request by voice using a virtual reality device, means for converting the voice input into text using natural language processing technology, and means for visualizing plans in a virtual space and supporting transitions and exploration. This reduces the user's planning efforts, allows the user to intuitively check and select plans through the virtual reality device, and enables flexible responses to real-time situations.

[1067] The "means for inputting the user's wishes and conditions" is an interface that allows the user to input their wishes and conditions using a terminal or virtual reality device.

[1068] The "means for analyzing input desires and conditions and extracting key information" is a technique for analyzing the data input by the user and extracting key information necessary for generating a plan.

[1069] The "means for generating multiple plan candidates" is an algorithm for generating multiple activity and spot candidates based on the extracted key information.

[1070] The "means for evaluating the generated plan candidates and selecting the optimum plan" is a process for evaluating the generated plan candidates and selecting the optimum plan based on the evaluation results.

[1071] The "means for calculating a travel route" is a method for calculating a route for efficiently traveling between each spot based on the selected plan.

[1072] The "means for transmitting route information and plan information to the user's terminal" refers to a communication technique for sending the calculated travel route and plan information to the user's terminal.

[1073] The "means for optimizing a route based on the user's current location and traffic information" is a system for recalculating and optimizing a travel route based on the user's real-time location information and traffic conditions.

[1074] The "means for generating a new plan and transmitting it to the user terminal" is a function for generating a new plan in response to an unexpected situation and transmitting the information about the plan to the user terminal.

[1075] The "means for making a reservation with an external service when a reservation is necessary" is a system for making a necessary reservation based on the user's wishes and conditions in cooperation with an external reservation service.

[1076] The "means for inputting user requests by voice using a virtual reality device" is an interface that allows the user to wear the virtual reality device and input their wishes and conditions by voice.

[1077] "Means for converting voice input into text using natural language processing technology" refers to a mechanism for converting data input by voice by a user into text data using natural language processing technology.

[1078] "Means for visualizing plans in a virtual space and supporting transitions and exploration" refers to technology that visually displays plans generated using virtual reality technology and supports the process of users intuitively confirming and selecting plans.

[1079] This invention is a concierge system that makes users' holidays and leisure time optimal and enjoyable. This system uses virtual reality (VR) to analyze users' wishes and requirements using natural language processing technology, and proposes and visualizes appropriate plans, thereby improving the user experience.

[1080] To achieve this, users first put on the virtual reality device and input their wishes and requirements through voice. This voice input is converted into text data using the Google Cloud Speech-to-Text API. This text data is then analyzed by the server to extract the key information needed. The NLP (Natural Language Processing) engine used is the Python spaCy library.

[1081] Next, based on the extracted information, the server retrieves data on the relevant spots and activities from an internal database and external APIs, and generates multiple plan candidates. These candidates are evaluated based on evaluation criteria (e.g., weather, congestion, user reviews, etc.), and the most suitable plan is selected. Based on this selected plan, the server calculates the user's most efficient route. The networkx library is used for this calculation.

[1082] The optimal plan and calculated route information are then sent to the user's virtual reality device, where the user can visually check and select a plan within the virtual space. Specifically, the virtual space is created using a 3D engine such as Unity, allowing users to intuitively operate the interface. This allows users to get a visually realistic experience, allowing them to check the details and atmosphere of each spot before visiting it.

[1083] For example, suppose a user voice-inputs into a virtual reality device, "I want to know about indoor attractions to enjoy with my kids and places to have lunch." This input is instantly converted into text data and analyzed by the server. The server searches for related facilities and proposes the optimal plan: "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." These facilities can be visually confirmed in the virtual space along with this plan, allowing the user to understand the environment of the places they want to go in advance. In this way, users can significantly reduce the effort of planning and spend their time more fulfillingly.

[1084] Real-time response is also an important feature. If a user encounters an unexpected situation while out and about, such as traffic congestion or a need to change a reservation time, the server will constantly optimize the route based on the user's current location and traffic information, and send the user new plans and route information. In this way, users can flexibly respond to changes in their plans.

[1085] An example prompt might look like this:

[1086] "I want to know about indoor attractions and places to have lunch that I can enjoy with my kids."

[1087] "I'm looking for a romantic restaurant and a quiet park to go to in front of it."

[1088] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1089] Step 1:

[1090] The user wears the virtual reality device and inputs their wishes and requirements by voice. This voice data is captured by a microphone built into the virtual reality device. Input: User's voice input. Output: Voice data.

[1091] Step 2:

[1092] The device sends the captured voice data to the Google Cloud Speech-to-Text API and converts it into text. Input: Voice data. Output: Text data.

[1093] How it works: Sends audio data to an API over the internet and receives the converted text data.

[1094] Step 3:

[1095] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy) and extracts key information. Input: Text data. Output: Extracted key information.

[1096] How it works: Feeds text data into an NLP engine to extract keywords and phrases.

[1097] Step 4:

[1098] The server searches and generates multiple plan candidates from an internal database and external API based on the extracted key information. Input: Extracted key information. Output: Multiple plan candidates.

[1099] What it does: Executes database queries and API requests, and collects and organizes the results.

[1100] Step 5:

[1101] The server evaluates the generated plan candidates and selects the optimal plan. The evaluation takes into account weather information, congestion, reviews, etc. Input: Multiple plan candidates. Output: The optimal plan.

[1102] How it works: Each plan is scored according to the evaluation criteria, and the plan with the highest score is selected.

[1103] Step 6:

[1104] The server uses the networkx library to calculate the most efficient route for the user based on the optimal plan. Input: Optimal plan. Output: Travel route.

[1105] Operation: Calculates travel costs between each spot and finds the shortest route.

[1106] Step 7:

[1107] The server sends the calculated travel route and plan information to the terminal. Input: Travel route and plan information. Output: Information displayed on the user's terminal.

[1108] Operation: Data is delivered to the terminal using a communication protocol.

[1109] Step 8:

[1110] The user confirms and selects the received plan and travel route in the virtual space. The virtual reality device visually displays the plan using a 3D engine such as Unity. Input: Travel route and plan information. Output: User selection.

[1111] Action: Check and select a plan through interface operations within the virtual space.

[1112] Step 9:

[1113] Based on the user's current location and traffic information, the server optimizes the route. Input: User's current location and real-time traffic information. Output: Optimized route.

[1114] How it works: Periodically collects location and traffic information and recalculates routes as needed.

[1115] Step 10:

[1116] When the server encounters an unexpected situation, it generates a new plan and sends it to the user terminal. Input: Information about the unexpected situation. Output: New plan information.

[1117] Operation: Generates a plan according to the situation and notifies the user.

[1118] Step 11:

[1119] If a reservation is required, the server makes the reservation to an external service. Input: Information about the reservation. Output: Reservation confirmation information.

[1120] What it does: Calls the API of an external booking service, makes the necessary reservations, and receives confirmation.

[1121] In this way, the system of the present invention starts with user input and, through a series of steps including analysis, plan generation, evaluation, route calculation, information distribution, user selection, real-time response, and reservation, proposes and realizes the optimal way for users to spend their holidays and leisure time.

[1122] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1123] The present invention is a system that generates plans based on the user's wishes and conditions, and suggests optimal categories and routes in real time to make the user's holidays and leisure time optimal and enjoyable. It also uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that.

[1124] Program processing

[1125] User Input Processing

[1126] The user connects to the device and inputs their wishes and requirements into the system in natural language. For example, they can input wishes such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1127] Hope Data Analysis

[1128] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation," "fun places," etc.). This extracted information is used in the subsequent plan generation process.

[1129] Recognition of emotional states

[1130] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stress, relaxation, excitement, etc.).

[1131] Generation and evaluation of plan candidates

[1132] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external API. For example, relaxation facilities and fun activity spots are considered as candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[1133] Calculating travel routes

[1134] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1135] Plan and route distribution

[1136] The server sends the calculated plan and route information in JSON format to the device. The device receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[1137] Real-time support

[1138] The device periodically transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[1139] Emergency troubleshooting

[1140] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1141] Specific examples

[1142] Example 1: A plan for families who want to relax

[1143] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1144] Example 2: A weekend date plan for couples who want to have fun

[1145] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1146] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[1147] The processing flow will be explained below.

[1148] Step 1:

[1149] The user operates the device and inputs their wishes and requirements to the system in natural language. For example, they might input a wish such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1150] Step 2:

[1151] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation" or "fun place").

[1152] Step 3:

[1153] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[1154] Step 4:

[1155] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external APIs, extracting candidates such as relaxation facilities and fun activities.

[1156] Step 5:

[1157] The server generates multiple plan candidates based on the search results, each based on improved data (e.g., facility reviews, ratings, current occupancy status, etc.).

[1158] Step 6:

[1159] The server evaluates the generated plan candidates using an evaluation algorithm to select the optimal plan, taking into account the weighting of the user's requirements and the on-site conditions (weather, congestion, the user's emotional state, etc.).

[1160] Step 7:

[1161] The server then calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, and the user's emotional state.

[1162] Step 8:

[1163] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[1164] Step 9:

[1165] The user checks the plan displayed on the device and presses the button to start navigation. The device acquires the current location using GPS and starts navigation to the first destination.

[1166] Step 10:

[1167] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route based on this information and sends the new route information to the device.

[1168] Step 11:

[1169] When a user encounters an unexpected situation, for example, "I'm going to be late for my appointment," the user inputs information into the terminal, which then transmits this information to the server.

[1170] Step 12:

[1171] The server then generates an optimal plan based on the relevant information and adjusts the schedule, for example, by changing the next destination or adding new spots.

[1172] Step 13:

[1173] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and presents the new instructions to the user.

[1174] Step 14:

[1175] When a user needs to make a reservation at a restaurant, for example, the user sends a request through the terminal, which then forwards the reservation request to the server.

[1176] Step 15:

[1177] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[1178] Through this series of processes, users receive support from the system throughout the entire process, from planning to execution, and by using an emotion engine, the system provides an optimal experience tailored to the user's emotional state.

[1179] Example 2

[1180] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional holiday and leisure planning systems only consider the user's preferences and conditions, failing to fully reflect the user's emotional state and making it difficult to respond flexibly when the user is unable to carry out their planned activities or when trouble occurs. Furthermore, there is a lack of a way to recognize the user's emotional state in real time and optimize the plan based on that. Furthermore, it is difficult to optimize travel routes based on traffic and weather information in real time, making it difficult to improve the user experience.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1183] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions using natural language processing and extracting key information, means for generating multiple plan candidates based on the analyzed information and an emotion engine that recognizes the user's emotional state, means for evaluating the generated plan candidates based on weather information, congestion status, user reviews, etc. and selecting an optimal plan, means for calculating a travel route based on the optimal plan while taking current traffic conditions and weather information into consideration, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route in real time based on the user's current location, traffic information, and emotional state, means for generating a new plan in response to an unplanned situation and transmitting it to the user's terminal, and means for making a reservation with an external service if a reservation is necessary. This makes it possible to provide an optimal plan that takes the user's emotional state into consideration, optimize the travel route in real time, and flexibly respond to unplanned situations.

[1184] "Means for inputting user wishes and conditions" refers to an interface or device that allows a user to input their own objectives or requests.

[1185] "Means of analyzing and extracting key information using natural language processing" refers to software or algorithms that analyze text entered by a user in natural language and automatically extract important keywords and related information.

[1186] An "emotional state recognition engine" refers to a system or engine that analyzes user input data and behavioral data to determine the user's current emotional state.

[1187] "Means for generating candidate plans" refers to functions or algorithms for creating a plan of multiple activities and places to visit based on the analyzed information and emotional state.

[1188] "Means of evaluating and selecting the optimal plan based on weather information, congestion status, user reviews, etc." refers to a function that evaluates generated plan candidates based on various external data and allows the user to select the optimal plan.

[1189] "Means for calculating travel routes taking into account current traffic and weather information" refers to systems or algorithms that take into account current traffic and weather information when calculating efficient travel routes based on the selected plan.

[1190] "Means for transmitting route information and plan information to a user's device" refers to the communications technology and infrastructure for delivering real-time calculated travel routes and activity plans to a user's device.

[1191] "Means of optimizing routes in real time based on the user's current location, traffic information, and emotional state" refers to a function that acquires and analyzes data such as the user's location, traffic conditions, and emotional state in real time, and optimizes routes as necessary.

[1192] "Means for generating a new plan in response to an unexpected situation and transmitting it to a user terminal" refers to a system that quickly creates a new plan and delivers it to the user's device when the user faces an unexpected situation.

[1193] "Means for making reservations with external services when reservations are required" refers to a function that automatically makes reservations using external reservation services when reservations are required for activities or spots within a plan.

[1194] The present invention is a system that generates a plan for a user to spend their holidays or leisure time optimally and enjoyably, and suggests optimal categories and routes in real time. It is particularly distinctive in that it uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that. Specific embodiments of the present invention are described below.

[1195] User Input Processing

[1196] Users can connect to the device and input their wishes and requirements in natural language. For example, they can input content such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1197] Hope Data Analysis

[1198] The server analyzes the data received from the user using a natural language processing (NLP) engine. The NLP engine analyzes the text and extracts key keywords and requirements. For example, keywords such as "relaxation" and "fun places" may be extracted. This information is used in the subsequent plan generation process.

[1199] Recognition of emotional states

[1200] The server passes the user's input data and behavior data (e.g., input speed, facial expression recognition data, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[1201] Generation and evaluation of plan candidates

[1202] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested as candidates. Multiple plan candidates are generated from the search results, and the optimal plan is selected by evaluating them based on conditions such as weather information, congestion level, and user reviews.

[1203] Calculating travel routes

[1204] The server calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1205] Plan and route distribution

[1206] The server sends the calculated plan and route information in JSON format to the device. The device receives this information and displays it on the user interface. The user confirms the presented plan and starts navigation.

[1207] Real-time support

[1208] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[1209] Emergency troubleshooting

[1210] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1211] Specific examples

[1212] Example 1: A plan for families who want to relax

[1213] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1214] Example 2: A weekend date plan for couples who want to have fun

[1215] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1216] Prompt Sentence Examples

[1217] "I want to relax today"

[1218] "I want to go somewhere that makes me feel happy."

[1219] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[1220] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1221] Step 1:

[1222] The user inputs their wishes and conditions into the device. For example, they input text in natural language such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this input and temporarily stores the data.

[1223] Input: User's wishes and requirements (natural language text)

[1224] Output: Temporarily saved user preferences and conditions (natural language text)

[1225] Step 2:

[1226] The device sends the temporarily saved text data to the server, where it becomes the raw material for analysis.

[1227] Input: Temporarily saved user preferences and conditions (natural language text)

[1228] Output: The desired and required data sent to the server (natural language text)

[1229] Step 3:

[1230] The server receives the data and passes it to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements, such as "relaxing" and "fun places."

[1231] Input: Desired and condition data sent to the server (natural language text)

[1232] Output: Extracted keywords and requirements (list format)

[1233] Step 4:

[1234] The server sends the user's input data and movement data (such as input speed and facial expression recognition data) to the emotion engine, which analyzes this data and recognizes the user's current emotional state (such as stress, relaxation, or excitement).

[1235] Input: User input and action data

[1236] Output: Recognized emotional state (list format)

[1237] Step 5:

[1238] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested.

[1239] Input: extracted keywords and requirements, recognized emotional states

[1240] Output: Plan candidates (list format)

[1241] Step 6:

[1242] The server evaluates the generated plan candidates based on weather information, congestion levels, user reviews, etc., and selects the optimal plan. A scoring algorithm is used for the evaluation.

[1243] Input: Plan options, weather information, congestion level, user reviews

[1244] Output: Best plan (list format)

[1245] Step 7:

[1246] The server calculates an efficient route based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1247] Input: Best plan, transportation method, traffic conditions, weather information, emotional state

[1248] Output: Optimal travel route (route information)

[1249] Step 8:

[1250] The server sends the calculated route and plan information in JSON format to the device, which displays the received information on the user interface. The user confirms this and starts navigation.

[1251] Input: Optimal travel route, plan information

[1252] Output: Navigation information displayed on the user's device

[1253] Step 9:

[1254] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[1255] Input: User's current location, traffic information, emotional state

[1256] Output: Recalculated route information

[1257] Step 10:

[1258] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1259] Input: User input for unplanned situations

[1260] Output: New plan information

[1261] (Application example 2)

[1262] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1263] Conventional plan generation systems only considered the user's preferences and conditions, making it difficult to provide optimal plans that reflected the user's emotional state. Food delivery services also faced the problem of being unable to provide suggestions tailored to the user's emotions and mood. The present invention aims to solve this problem by providing optimal plans and food delivery plans that reflect the user's emotional state.

[1264] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting a user's preferences and conditions, means for analyzing the input preferences and conditions and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for recognizing the user's emotional state, and means for generating and notifying a food delivery plan based on the user's emotional state. This makes it possible to provide an optimal plan and food delivery proposal that takes the user's emotional state into consideration.

[1265] "User's wishes and conditions" refer to the specific objectives and needs that the user wants to achieve with the system, as well as the constraints imposed on them.

[1266] "Key information" refers to keywords and requirements that are important for generating plans and routes, extracted from the analyzed user's wishes and conditions.

[1267] "Plan candidates" refer to multiple suggested activities or actions that are generated based on the user's wishes, conditions, and emotional state.

[1268] The "optimal plan" refers to the plan that best suits the user's wishes, conditions, and emotional state from among the generated plan candidates.

[1269] A "travel route" refers to a route that a user takes to travel efficiently to a destination or activity location based on the selected optimal plan.

[1270] "Calculated route information and plan information" refers to information on specific routes and activity details required when a user moves or performs an action.

[1271] "Current location and traffic information" refers to data relating to the user's current location and data relating to the traffic conditions in the vicinity thereof.

[1272] "Unplanned circumstances" refers to unexpected events or changes that a user may encounter while executing a plan.

[1273] "External services" refers to external service providers or platforms with which the system works in order to make reservations, provide information, etc.

[1274] "User's emotional state" refers to the user's mental state, such as stress, relaxation, excitement, etc., as recognized by the emotion engine.

[1275] "Food Delivery Plan" refers to an optimal food delivery proposal generated based on the user's preferences, conditions and emotional state.

[1276] The present invention is a system that uses an emotion engine to recognize a user's emotional state and generate and provide an optimal plan based on the user's wishes and conditions, in order to make the user's holidays and leisure time optimal and enjoyable. A specific embodiment of this system is described below.

[1277] Hardware and software used

[1278] 1. Hardware:

[1279] Smartphone (including camera and microphone)

[1280] 2. Software:

[1281] Python

[1282] Emotion recognition module (e.g., Face++, OpenCV, etc.)

[1283] Natural language processing modules (e.g., spaCy, Google Natural Language API, etc.)

[1284] Program processing

[1285] User Input Processing

[1286] Through a smartphone application, users input their wishes and conditions in natural language, such as "I want to relax today" or "I want to go to a place that makes me feel happy." The system receives this information and sends it to the server.

[1287] Hope Data Analysis

[1288] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements. This extracted information is used in the subsequent plan generation process.

[1289] Recognition of emotional states

[1290] The server passes data such as user input and facial expression recognition (using a camera) to the emotion engine, which analyzes this data and recognizes the user's emotional state. For example, it determines whether the user is stressed, relaxed, excited, etc.

[1291] Generation and evaluation of plan candidates

[1292] The server searches for multiple options, such as food delivery services and tourist attractions, based on the analyzed preferences and emotional state. For example, options could include a relaxing cafe or a fun restaurant. From the search results, multiple plan options are generated, and the optimal plan is selected by evaluating the conditions of each plan (reviews, congestion, etc.).

[1293] Route calculation and notifications

[1294] The server calculates an efficient route to visit each spot based on the selected optimal plan. This calculation takes into account current traffic conditions and weather information. The calculated route and plan information are sent to the user's device and displayed on the user interface.

[1295] Real-time response and troubleshooting

[1296] The device periodically sends the user's current location, traffic information, and emotional state to the server. The server uses this information to recalculate the route as needed and sends a new route to the device. For example, if the user is running late for an appointment, the server generates a new plan and adjusts the schedule.

[1297] Specific examples

[1298] Example 1: A plan for families looking to relax

[1299] The user inputs "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. After that, it proposes the optimal schedule and route for the day, such as "hot spring at 10:00, lunch at a restaurant at 12:30." If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1300] Example 2: A weekend date plan for couples who want to have fun

[1301] The user inputs "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotions are indicative of a desire for fun, suggests a plan of "Amusement park at 15:00, dinner at a restaurant at 18:00." If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1302] Prompt Sentence Examples

[1303] "I want to feel refreshed today"

[1304] "I want the perfect food to relieve stress."

[1305] This allows users to enjoy greater convenience and flexibility in planning and troubleshooting, and provides an optimal experience tailored to their emotional state.

[1306] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1307] Step 1:

[1308] User Input

[1309] The user starts the smartphone application and inputs their wishes and conditions in natural language. For example, they input prompts such as "I want to feel refreshed today" or "I want to go to a place that makes me feel happy." This generates user input data, which the device then sends to the server.

[1310] Input: User's wishes and conditions (e.g., I want to feel refreshed today)

[1311] Output: User input data

[1312] Step 2:

[1313] Natural Language Processing (NLP)

[1314] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the input data and extracts key keywords and requirements. The extracted information is used in the subsequent plan generation process.

[1315] Input: User-entered data

[1316] Data processing / calculation: natural language analysis, keyword extraction

[1317] Output: Key words (e.g., refresh)

[1318] Step 3:

[1319] Recognition of emotional states

[1320] The server passes the user's facial expressions and input speed to the emotion engine via the smartphone's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited).

[1321] Input: facial expression data, input speed data

[1322] Data processing / calculation: Emotion analysis

[1323] Output: User's emotional state (e.g., relaxed state)

[1324] Step 4:

[1325] Generate candidate plans

[1326] The server generates multiple plan candidates based on keywords extracted by the NLP engine and emotional states recognized by the emotion engine. For example, hot springs and relaxing cafes are suggested as plan candidates for refreshment. Related external APIs are also used to obtain information.

[1327] Input: Key keywords, user's emotional state

[1328] Data processing / calculation: Plan generation, external API calls

[1329] Output: Plan candidates (e.g., hot spring facilities, relaxation cafes)

[1330] Step 5:

[1331] Evaluation and selection of plan candidates

[1332] The server evaluates the generated plan candidates and selects the optimal plan that best suits the user's desires and emotional state, taking into account criteria such as reviews, congestion, and ease of access.

[1333] Input: Plan candidate

[1334] Data processing / calculation: Plan evaluation, condition comparison

[1335] Output: Optimal plan (e.g., hot spring facility)

[1336] Step 6:

[1337] Calculating travel routes

[1338] Based on the selected optimal plan, the server calculates an efficient route to visit each spot, taking into account current traffic and weather conditions.

[1339] Input: Optimal plan

[1340] Data processing / calculation: Route calculation, traffic and weather information taken into account

[1341] Output: Travel route information

[1342] Step 7:

[1343] Plan and route distribution

[1344] The server sends the calculated route and plan information in JSON format to the user's device, which receives it and displays it on the user interface.

[1345] Input: Travel route information, plan information

[1346] Data processing / calculation: Data format conversion (JSON)

[1347] Output: Data delivered to the user's device

[1348] Step 8:

[1349] Real-time support

[1350] The device continuously transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[1351] Input: User's current location, traffic information, emotional state

[1352] Data processing / calculation: Real-time data analysis, route recalculation

[1353] Output: New route information

[1354] Step 9:

[1355] Emergency troubleshooting

[1356] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate a new plan or adjust the schedule, and send the new plan information to the user.

[1357] Input: Trouble information (e.g., being late for an appointment)

[1358] Data processing / calculation: Plan regeneration, schedule adjustment

[1359] Output: New plan information

[1360] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1361] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1362] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1363] [Fourth embodiment]

[1364] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1365] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1366] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1367] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1368] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1369] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1370] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1371] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1372] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1373] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1374] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1375] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1376] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1377] This invention is a concierge system that helps users to enjoy their holidays and leisure time in an optimal and enjoyable way. This system starts by inputting the user's wishes and requirements, then generates an optimal plan, calculates the corresponding travel route, and makes reservations, thereby reducing the user's stress and helping them to spend their time in a fulfilling way.

[1378] Program processing

[1379] User Input Processing

[1380] Users connect to the device and input their preferences and requirements into the system in natural language. For example, they could input a request such as, "I'm looking for an indoor facility that I can enjoy with my children. I'd also like to enjoy a delicious lunch." The device then sends this information to the server, which begins analyzing it.

[1381] Hope Data Analysis

[1382] The server passes the received user input data to a natural language processing (NLP) engine, which analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch," etc.). This extracted information is used in the subsequent plan generation process.

[1383] Generation and evaluation of plan candidates

[1384] Based on the analyzed requirements, the server searches for corresponding spots and activities from an internal database and external APIs. For example, nearby indoor facilities for children or popular lunch spots are potential candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[1385] Calculating travel routes

[1386] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, etc.

[1387] Plan and route distribution

[1388] The server sends the calculated plan and travel route information to the terminal, which receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[1389] Real-time support

[1390] The device periodically sends the user's current location and traffic information to the server. The server uses this information to optimize the route accordingly. For example, if an unexpected traffic jam occurs, the server calculates a detour route and notifies the user by sending it to the device.

[1391] Emergency troubleshooting

[1392] If a user encounters an unexpected situation, for example by entering information such as "I'm going to be late for my appointment," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changes in plans.

[1393] Specific examples

[1394] Example 1: One-day trip for families

[1395] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1396] Example 2: Weekend date plan for couples

[1397] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1398] This invention allows users to significantly reduce the effort required for planning and to respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[1399] The processing flow will be explained below.

[1400] Step 1:

[1401] The user operates the terminal and inputs their requests and requirements into the system. For example, they might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy lunch." The terminal receives this information and sends it to the server.

[1402] Step 2:

[1403] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "kids," "indoor," "lunch," etc.).

[1404] Step 3:

[1405] The server generates multiple search queries based on the extracted requirements and sends them to an internal database and an external API, which returns a list of corresponding indoor facilities for children and lunch spots, while the external API returns reviews, ratings, and current occupancy status of the spots.

[1406] Step 4:

[1407] The server then aggregates the returned data and generates multiple itinerary options based on the information it has gathered, such as multiple itinerary options combining several facilities and lunch spots.

[1408] Step 5:

[1409] The server evaluates the generated plan candidates and selects the best one. The evaluation algorithm takes into account the weighting of the user's requirements and the local conditions (weather, congestion, etc.).

[1410] Step 6:

[1411] The server calculates the route based on the optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, etc.

[1412] Step 7:

[1413] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[1414] Step 8:

[1415] The user checks the plan displayed on the device and presses a button to start navigation. The device acquires its current location via GPS and begins navigation to the first destination.

[1416] Step 9:

[1417] The device periodically sends the user's current location and traffic information to the server, which then recalculates the route as needed and sends the new route to the device.

[1418] Step 10:

[1419] When a user encounters an unexpected situation, for example, "I'm late for my appointment," the user inputs this information into the terminal, which then transmits it to the server.

[1420] Step 11:

[1421] The server analyzes the situation and generates a new plan and route, which may change the next destination or rearrange the schedule to accommodate delays.

[1422] Step 12:

[1423] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and displays new instructions to the user.

[1424] Step 13:

[1425] When a user needs to make a reservation at a restaurant, for example, they send a request through their terminal, which then forwards the reservation request to the server.

[1426] Step 14:

[1427] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[1428] The above are the specific processing steps and detailed operations of the program based on the claims, which allow users to spend their time stress-free and productively planning and troubleshooting.

[1429] Example 1

[1430] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] Conventional planning systems for users' holidays and leisure time have had problems such as difficulty reflecting users' detailed wishes and requirements and difficulty in dealing with problems in real time. Furthermore, users had to manually make reservations and check traffic conditions, which made planning inefficient. The present invention aims to solve these problems and provide a system that allows users to spend their time stress-free and fulfilling.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1433] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions and extracting key information, means for generating a plurality of plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for extracting key information using natural language processing, means for calculating a travel route taking weather information and traffic congestion into consideration, and means for optimizing the route in real time based on the user's current location and traffic information. This not only enables the user to automatically and efficiently obtain an optimal plan, but also enables flexible response to unexpected problems.

[1434] The "means for inputting user's wishes and conditions" is a system component that provides an interface that allows the user to input his or her wishes and conditions in natural language.

[1435] The "means for analyzing input preferences and conditions and extracting key information" is a system component that uses natural language processing to extract and analyze important keywords and phrases from the text entered by the user.

[1436] A "means for generating multiple candidate plans" is a system component that designs and generates multiple possible action plans based on the analyzed information.

[1437] The "means for evaluating the generated plan candidates and selecting the optimal plan" is a system component that compares and evaluates the multiple generated plan candidates based on predetermined evaluation criteria and selects the optimal plan.

[1438] The "means for calculating a travel route" is a system component that efficiently calculates a travel route between each spot based on the selected plan.

[1439] "Means for transmitting calculated route information and plan information to a user's terminal" refers to a system component that transmits travel route and plan information calculated and generated by the server to a user's terminal.

[1440] The "means for optimizing a route based on the user's current location and traffic information" is a system component that dynamically optimizes a travel route using the user's real-time location information and traffic information.

[1441] The "means for generating a new plan in response to an unexpected situation and transmitting it to the user terminal" is a system component that quickly generates a new plan when the user encounters an unexpected situation and transmits it to the user's terminal.

[1442] The "means for making a reservation with an external service when a reservation is necessary" is a system component that makes a reservation for a facility or service for the user using an external reservation service as needed.

[1443] "Means for extracting key information using natural language processing" refers to a system component that utilizes natural language processing technology to analyze and extract key information from text entered by a user.

[1444] The "means for calculating a travel route taking into consideration weather information and congestion status" is a system component that calculates an optimal travel route taking into consideration external information such as weather information and congestion status of a location.

[1445] "Means for optimizing routes in real time based on the user's current location and traffic information" refers to a system component that dynamically changes and optimizes travel routes in real time based on the user's current location information and traffic conditions.

[1446] This invention is a concierge system that provides optimal holiday and leisure plans based on the user's wishes and requirements, helping the user reduce stress and spend their time in a fulfilling manner. This system analyzes the wishes and requirements entered by the user, and generates and provides the optimal plan based on them.

[1447] Specifically, a user first uses a device (smartphone or PC) to input their wishes and requirements into the system in natural language. For example, a user might input a request such as, "I'm looking for an indoor facility where I can enjoy my children. I'd also like to enjoy a delicious lunch." This input information is then sent from the device to the server.

[1448] The server uses a natural language processing engine, such as the Google Cloud Natural Language API, to analyze the received data. The NLP engine analyzes the text and extracts key information (e.g., "for kids," "indoors," "lunch"), which is then stored in the server's database for further processing.

[1449] Next, the server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for spots and activities that match the user's criteria. For example, nearby indoor facilities for children or popular restaurants may be suggested as candidates. From these search results, multiple plan candidates are generated and each plan is evaluated. Evaluation criteria include weather information (weather information API), congestion level (traffic information API), and review information (review API).

[1450] After the optimal plan is selected, the server calculates the travel route between each spot based on the plan using Google Maps API, Mapbox, etc. This travel route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, and weather information.

[1451] The calculated plan and travel route are sent from the server to the user's device. The device receives it and displays the entire plan and travel route to each spot on the user interface. The user can confirm the presented plan and start navigation.

[1452] In addition, the device periodically obtains the user's current location using its GPS function and sends it to the server. The server dynamically optimizes the travel route as needed based on real-time location and traffic information. For example, if an unexpected traffic jam occurs, the server quickly calculates a detour route, sends it to the device, and notifies the user.

[1453] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment" into their device, the device will send this information to the server. The server will then regenerate a plan based on the new information, adjust the schedule, and send the new plan information to the user's device. This allows the user to flexibly respond to changing situations.

[1454] Finally, if a reservation is required, the server will automatically make the necessary reservation by connecting with an external reservation service, such as a restaurant reservation site or facility reservation system.

[1455] Specific examples

[1456] Example 1: One-day trip for families

[1457] The user inputs, "I want to know indoor attractions and places for lunch that I can go to with my kids." The server searches for related facilities and proposes the optimal schedule and route for the day, such as "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1458] Example 2: Weekend date plan for couples

[1459] The user inputs, "I'm looking for a romantic restaurant and a quiet park to go to before it." The server searches for spots that meet the criteria and proposes a plan such as "A stroll around X park at 15:00, followed by dinner at a specified restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of the information.

[1460] Examples of prompt statements

[1461] 1. "I'm looking for an indoor venue that's fun for the kids and I'd also like to enjoy a delicious lunch."

[1462] 2. "Can you recommend a romantic restaurant and a quiet park I can go to before going there?"

[1463] As described above, by using the system of the present invention, users can significantly reduce the effort required for planning and can respond flexibly to unexpected problems, enabling them to spend their holidays and leisure time stress-free and in a fulfilling manner.

[1464] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1465] Step 1: User Input Processing

[1466] The user uses a terminal to input their preferences and requirements in natural language. For example, they might input, "I'm looking for an indoor facility that I can enjoy with my kids. I'd also like to enjoy a delicious lunch." This input information is sent to the server as an HTTP request.

[1467] Input: User's wishes and requirements (natural language text)

[1468] Output: Input sent to the server

[1469] Step 2: Data Analysis

[1470] The server sends the received requests and conditions to a natural language processing engine such as Google Cloud Natural Language API for analysis, specifically extracting key information (keywords and phrases).

[1471] The server passes the received text data to the NLP engine and obtains the analysis results.

[1472] Input: Natural language text (user wishes and conditions)

[1473] Output: Key information (keywords: "kids," "indoor," "lunch," etc.)

[1474] Step 3: Generate candidate plans

[1475] The server uses an internal database and external APIs (e.g., Google Places API and Yelp API) to search for relevant spots and activities based on key information, and generates multiple plan suggestions from the search results.

[1476] Specifically, the server calls each API to build a plan candidate, and analyzes and integrates the response data.

[1477] Input: Key information (keywords)

[1478] Output: A list of possible plans (e.g., "10:00 at XX indoor amusement park, 12:30 at XX restaurant for lunch")

[1479] Step 4: Plan Evaluation and Selection

[1480] The server evaluates the generated plan candidates based on evaluation criteria, including weather information, congestion level, and review information, and selects the optimal plan.

[1481] The server uses the evaluation criteria API to collect the necessary data and uses it to score each plan candidate, and the plan with the highest score is selected.

[1482] Input: List of plan candidates

[1483] Output: Optimal plan

[1484] Step 5: Calculating the travel route

[1485] Based on the optimal plan, the server calculates the travel route between each spot using Google Maps API or Mapbox.

[1486] The server calculates an efficient route taking into account the user's mode of transportation and current traffic conditions.

[1487] Input: Best plan, transportation method, traffic information

[1488] Output: Calculated travel route

[1489] Step 6: Distribute your plan and route

[1490] The server sends the calculated plan and travel route as an HTTP response to the user's device, which receives it and displays it on the user interface.

[1491] Specifically, the terminal analyzes the data received from the server and formats it for display on the screen.

[1492] Input: Calculated travel route, optimal plan

[1493] Output: Plan and route displayed on the user interface

[1494] Step 7: Sending real-time location and traffic information

[1495] The device periodically acquires the user's current location using the GPS function and sends it to the server. It also acquires current traffic information and sends it to the server.

[1496] Input: User's current location, traffic information

[1497] Output: Real-time location and traffic information sent to the server

[1498] Step 8: Optimize travel routes

[1499] The server dynamically optimizes the travel route based on the received real-time information, for example, calculating a new route taking into account traffic congestion and accident information and sending it to the device.

[1500] Input: User's current location, traffic information

[1501] Output: New optimized route information

[1502] Step 9: Dealing with emergency situations

[1503] If a user encounters an unexpected situation, for example, by inputting information such as "I'm going to be late for my appointment," the device sends this information to the server. The server then generates a new plan, adjusts the schedule, and sends the new plan information to the user's device.

[1504] Input: Trouble information from the user

[1505] Output: New plan information

[1506] At each step, the program efficiently processes data and provides the most appropriate information for the user.

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] In today's world, planning is crucial to making users' holidays and leisure time more efficient and enjoyable. However, in many cases, gathering the information users want and creating optimal plans is time-consuming and requires real-time troubleshooting. This situation increases users' stress and makes it difficult for them to spend their time productively. Furthermore, existing planning systems lack the ability to provide user experiences using virtual reality devices, limiting the means by which users can intuitively review and select plans.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes means for inputting a user's wishes and requirements, means for analyzing the input wishes and requirements and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal plan, means for transmitting calculated route information and plan information to the user's terminal, means for optimizing the route based on the user's current location and traffic information, means for generating a new plan in response to an unexpected situation and transmitting it to the user's terminal, means for making a reservation with an external service if a reservation is necessary, means for inputting a user's request by voice using a virtual reality device, means for converting the voice input into text using natural language processing technology, and means for visualizing plans in a virtual space and supporting transitions and exploration. This reduces the user's planning efforts, allows the user to intuitively check and select plans through the virtual reality device, and enables flexible responses to real-time situations.

[1512] The "means for inputting the user's wishes and conditions" is an interface that allows the user to input their wishes and conditions using a terminal or virtual reality device.

[1513] The "means for analyzing input desires and conditions and extracting key information" is a technique for analyzing the data input by the user and extracting key information necessary for generating a plan.

[1514] The "means for generating multiple plan candidates" is an algorithm for generating multiple activity and spot candidates based on the extracted key information.

[1515] The "means for evaluating the generated plan candidates and selecting the optimum plan" is a process for evaluating the generated plan candidates and selecting the optimum plan based on the evaluation results.

[1516] The "means for calculating a travel route" is a method for calculating a route for efficiently traveling between each spot based on the selected plan.

[1517] The "means for transmitting route information and plan information to the user's terminal" refers to a communication technique for sending the calculated travel route and plan information to the user's terminal.

[1518] The "means for optimizing a route based on the user's current location and traffic information" is a system for recalculating and optimizing a travel route based on the user's real-time location information and traffic conditions.

[1519] The "means for generating a new plan and transmitting it to the user terminal" is a function for generating a new plan in response to an unexpected situation and transmitting the information about the plan to the user terminal.

[1520] The "means for making a reservation with an external service when a reservation is necessary" is a system for making a necessary reservation based on the user's wishes and conditions in cooperation with an external reservation service.

[1521] The "means for inputting user requests by voice using a virtual reality device" is an interface that allows the user to wear the virtual reality device and input their wishes and conditions by voice.

[1522] "Means for converting voice input into text using natural language processing technology" refers to a mechanism for converting data input by voice by a user into text data using natural language processing technology.

[1523] "Means for visualizing plans in a virtual space and supporting transitions and exploration" refers to technology that visually displays plans generated using virtual reality technology and supports the process of users intuitively confirming and selecting plans.

[1524] This invention is a concierge system that makes users' holidays and leisure time optimal and enjoyable. This system uses virtual reality (VR) to analyze users' wishes and requirements using natural language processing technology, and proposes and visualizes appropriate plans, thereby improving the user experience.

[1525] To achieve this, users first put on the virtual reality device and input their wishes and requirements through voice. This voice input is converted into text data using the Google Cloud Speech-to-Text API. This text data is then analyzed by the server to extract the key information needed. The NLP (Natural Language Processing) engine used is the Python spaCy library.

[1526] Next, based on the extracted information, the server retrieves data on the relevant spots and activities from an internal database and external APIs, and generates multiple plan candidates. These candidates are evaluated based on evaluation criteria (e.g., weather, congestion, user reviews, etc.), and the most suitable plan is selected. Based on this selected plan, the server calculates the user's most efficient route. The networkx library is used for this calculation.

[1527] The optimal plan and calculated route information are then sent to the user's virtual reality device, where the user can visually check and select a plan within the virtual space. Specifically, the virtual space is created using a 3D engine such as Unity, allowing users to intuitively operate the interface. This allows users to get a visually realistic experience, allowing them to check the details and atmosphere of each spot before visiting it.

[1528] For example, suppose a user voice-inputs into a virtual reality device, "I want to know about indoor attractions to enjoy with my kids and places to have lunch." This input is instantly converted into text data and analyzed by the server. The server searches for related facilities and proposes the optimal plan: "X indoor amusement park at 10:00, lunch at XX restaurant at 12:30." These facilities can be visually confirmed in the virtual space along with this plan, allowing the user to understand the environment of the places they want to go in advance. In this way, users can significantly reduce the effort of planning and spend their time more fulfillingly.

[1529] Real-time response is also an important feature. If a user encounters an unexpected situation while out and about, such as traffic congestion or a need to change a reservation time, the server will constantly optimize the route based on the user's current location and traffic information, and send the user new plans and route information. In this way, users can flexibly respond to changes in their plans.

[1530] An example prompt might look like this:

[1531] "I want to know about indoor attractions and places to have lunch that I can enjoy with my kids."

[1532] "I'm looking for a romantic restaurant and a quiet park to go to in front of it."

[1533] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1534] Step 1:

[1535] The user wears the virtual reality device and inputs their wishes and requirements by voice. This voice data is captured by a microphone built into the virtual reality device. Input: User's voice input. Output: Voice data.

[1536] Step 2:

[1537] The device sends the captured voice data to the Google Cloud Speech-to-Text API and converts it into text. Input: Voice data. Output: Text data.

[1538] How it works: Sends audio data to an API over the internet and receives the converted text data.

[1539] Step 3:

[1540] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy) and extracts key information. Input: Text data. Output: Extracted key information.

[1541] How it works: Feeds text data into an NLP engine to extract keywords and phrases.

[1542] Step 4:

[1543] The server searches and generates multiple plan candidates from an internal database and external API based on the extracted key information. Input: Extracted key information. Output: Multiple plan candidates.

[1544] What it does: Executes database queries and API requests, and collects and organizes the results.

[1545] Step 5:

[1546] The server evaluates the generated plan candidates and selects the optimal plan. The evaluation takes into account weather information, congestion, reviews, etc. Input: Multiple plan candidates. Output: The optimal plan.

[1547] How it works: Each plan is scored according to the evaluation criteria, and the plan with the highest score is selected.

[1548] Step 6:

[1549] The server uses the networkx library to calculate the most efficient route for the user based on the optimal plan. Input: Optimal plan. Output: Travel route.

[1550] Operation: Calculates travel costs between each spot and finds the shortest route.

[1551] Step 7:

[1552] The server sends the calculated travel route and plan information to the terminal. Input: Travel route and plan information. Output: Information displayed on the user's terminal.

[1553] Operation: Data is delivered to the terminal using a communication protocol.

[1554] Step 8:

[1555] The user confirms and selects the received plan and travel route in the virtual space. The virtual reality device visually displays the plan using a 3D engine such as Unity. Input: Travel route and plan information. Output: User selection.

[1556] Action: Check and select a plan through interface operations within the virtual space.

[1557] Step 9:

[1558] Based on the user's current location and traffic information, the server optimizes the route. Input: User's current location and real-time traffic information. Output: Optimized route.

[1559] How it works: Periodically collects location and traffic information and recalculates routes as needed.

[1560] Step 10:

[1561] When the server encounters an unexpected situation, it generates a new plan and sends it to the user terminal. Input: Information about the unexpected situation. Output: New plan information.

[1562] Operation: Generates a plan according to the situation and notifies the user.

[1563] Step 11:

[1564] If a reservation is required, the server makes the reservation to an external service. Input: Information about the reservation. Output: Reservation confirmation information.

[1565] What it does: Calls the API of an external booking service, makes the necessary reservations, and receives confirmation.

[1566] In this way, the system of the present invention starts with user input and, through a series of steps including analysis, plan generation, evaluation, route calculation, information distribution, user selection, real-time response, and reservation, proposes and realizes the optimal way for users to spend their holidays and leisure time.

[1567] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1568] The present invention is a system that generates plans based on the user's wishes and conditions, and suggests optimal categories and routes in real time to make the user's holidays and leisure time optimal and enjoyable. It also uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that.

[1569] Program processing

[1570] User Input Processing

[1571] The user connects to the device and inputs their wishes and requirements into the system in natural language. For example, they can input wishes such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1572] Hope Data Analysis

[1573] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation," "fun places," etc.). This extracted information is used in the subsequent plan generation process.

[1574] Recognition of emotional states

[1575] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stress, relaxation, excitement, etc.).

[1576] Generation and evaluation of plan candidates

[1577] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external API. For example, relaxation facilities and fun activity spots are considered as candidates. From the search results, multiple plan candidates are generated, and the optimal plan is selected by evaluating the conditions of each plan (weather, congestion, reviews, etc.).

[1578] Calculating travel routes

[1579] The server calculates an efficient route to visit each spot based on the selected optimal plan. This route is optimized taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1580] Plan and route distribution

[1581] The server sends the calculated plan and route information in JSON format to the device. The device receives it and displays it on the user interface. The user confirms the presented plan and starts navigation.

[1582] Real-time support

[1583] The device periodically transmits the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and transmits the new route to the device.

[1584] Emergency troubleshooting

[1585] If a user encounters an unexpected situation, for example by entering "I'm going to be late for my reservation," the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1586] Specific examples

[1587] Example 1: A plan for families who want to relax

[1588] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1589] Example 2: A weekend date plan for couples who want to have fun

[1590] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1591] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[1592] The processing flow will be explained below.

[1593] Step 1:

[1594] The user operates the device and inputs their wishes and requirements to the system in natural language. For example, they might input a wish such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1595] Step 2:

[1596] The server passes the input data received from the user to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements (e.g., "relaxation" or "fun place").

[1597] Step 3:

[1598] The server passes data on the user's input and behavior (e.g., input speed, facial expression recognition, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[1599] Step 4:

[1600] Based on the analyzed requirements and emotional state, the server searches for corresponding spots and activities from an internal database and external APIs, extracting candidates such as relaxation facilities and fun activities.

[1601] Step 5:

[1602] The server generates multiple plan candidates based on the search results, each based on improved data (e.g., facility reviews, ratings, current occupancy status, etc.).

[1603] Step 6:

[1604] The server evaluates the generated plan candidates using an evaluation algorithm to select the optimal plan, taking into account the weighting of the user's requirements and the on-site conditions (weather, congestion, the user's emotional state, etc.).

[1605] Step 7:

[1606] The server then calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation, current traffic conditions, weather information, and the user's emotional state.

[1607] Step 8:

[1608] The server sends the calculated plan and route information in JSON format to the device, which then analyzes the information and displays it on the user interface.

[1609] Step 9:

[1610] The user checks the plan displayed on the device and presses the button to start navigation. The device acquires the current location using GPS and starts navigation to the first destination.

[1611] Step 10:

[1612] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route based on this information and sends the new route information to the device.

[1613] Step 11:

[1614] When a user encounters an unexpected situation, for example, "I'm going to be late for my appointment," the user inputs information into the terminal, which then transmits this information to the server.

[1615] Step 12:

[1616] The server then generates an optimal plan based on the relevant information and adjusts the schedule, for example, by changing the next destination or adding new spots.

[1617] Step 13:

[1618] The server sends the new plan and route information to the terminal, which updates its user interface with the received information and presents the new instructions to the user.

[1619] Step 14:

[1620] When a user needs to make a reservation at a restaurant, for example, the user sends a request through the terminal, which then forwards the reservation request to the server.

[1621] Step 15:

[1622] The server calls the external reservation service API to complete the reservation. The reservation completion information is sent from the server to the terminal and notified to the user.

[1623] Through this series of processes, users receive support from the system throughout the entire process, from planning to execution, and by using an emotion engine, the system provides an optimal experience tailored to the user's emotional state.

[1624] Example 2

[1625] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1626] Conventional holiday and leisure planning systems only consider the user's preferences and conditions, failing to fully reflect the user's emotional state and making it difficult to respond flexibly when the user is unable to carry out their planned activities or when trouble occurs. Furthermore, there is a lack of a way to recognize the user's emotional state in real time and optimize the plan based on that. Furthermore, it is difficult to optimize travel routes based on traffic and weather information in real time, making it difficult to improve the user experience.

[1627] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1628] In this invention, the server includes means for inputting a user's wishes and conditions, means for analyzing the input wishes and conditions using natural language processing and extracting key information, means for generating multiple plan candidates based on the analyzed information and an emotion engine that recognizes the user's emotional state, means for evaluating the generated plan candidates based on weather information, congestion status, user reviews, etc. and selecting an optimal plan, means for calculating a travel route based on the optimal plan while taking current traffic conditions and weather information into consideration, means for transmitting the calculated route information and plan information to the user's terminal, means for optimizing the route in real time based on the user's current location, traffic information, and emotional state, means for generating a new plan in response to an unplanned situation and transmitting it to the user's terminal, and means for making a reservation with an external service if a reservation is necessary. This makes it possible to provide an optimal plan that takes the user's emotional state into consideration, optimize the travel route in real time, and flexibly respond to unplanned situations.

[1629] "Means for inputting user wishes and conditions" refers to an interface or device that allows a user to input their own objectives or requests.

[1630] "Means of analyzing and extracting key information using natural language processing" refers to software or algorithms that analyze text entered by a user in natural language and automatically extract important keywords and related information.

[1631] An "emotional state recognition engine" refers to a system or engine that analyzes user input data and behavioral data to determine the user's current emotional state.

[1632] "Means for generating candidate plans" refers to functions or algorithms for creating a plan of multiple activities and places to visit based on the analyzed information and emotional state.

[1633] "Means of evaluating and selecting the optimal plan based on weather information, congestion status, user reviews, etc." refers to a function that evaluates generated plan candidates based on various external data and allows the user to select the optimal plan.

[1634] "Means for calculating travel routes taking into account current traffic and weather information" refers to systems or algorithms that take into account current traffic and weather information when calculating efficient travel routes based on the selected plan.

[1635] "Means for transmitting route information and plan information to a user's device" refers to the communications technology and infrastructure for delivering real-time calculated travel routes and activity plans to a user's device.

[1636] "Means of optimizing routes in real time based on the user's current location, traffic information, and emotional state" refers to a function that acquires and analyzes data such as the user's location, traffic conditions, and emotional state in real time, and optimizes routes as necessary.

[1637] "Means for generating a new plan in response to an unexpected situation and transmitting it to a user terminal" refers to a system that quickly creates a new plan and delivers it to the user's device when the user faces an unexpected situation.

[1638] "Means for making reservations with external services when reservations are required" refers to a function that automatically makes reservations using external reservation services when reservations are required for activities or spots within a plan.

[1639] The present invention is a system that generates a plan for a user to spend their holidays or leisure time optimally and enjoyably, and suggests optimal categories and routes in real time. It is particularly distinctive in that it uses an emotion engine to recognize the user's emotional state and provides an optimized plan based on that. Specific embodiments of the present invention are described below.

[1640] User Input Processing

[1641] Users can connect to the device and input their wishes and requirements in natural language. For example, they can input content such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this information and sends it to the server.

[1642] Hope Data Analysis

[1643] The server analyzes the data received from the user using a natural language processing (NLP) engine. The NLP engine analyzes the text and extracts key keywords and requirements. For example, keywords such as "relaxation" and "fun places" may be extracted. This information is used in the subsequent plan generation process.

[1644] Recognition of emotional states

[1645] The server passes the user's input data and behavior data (e.g., input speed, facial expression recognition data, etc.) to the emotion engine, which analyzes this data and recognizes the user's emotional state (e.g., stressed, relaxed, excited, etc.).

[1646] Generation and evaluation of plan candidates

[1647] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested as candidates. Multiple plan candidates are generated from the search results, and the optimal plan is selected by evaluating them based on conditions such as weather information, congestion level, and user reviews.

[1648] Calculating travel routes

[1649] The server calculates an efficient route to visit each spot based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1650] Plan and route distribution

[1651] The server sends the calculated plan and route information in JSON format to the device. The device receives this information and displays it on the user interface. The user confirms the presented plan and starts navigation.

[1652] Real-time support

[1653] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[1654] Emergency troubleshooting

[1655] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1656] Specific examples

[1657] Example 1: A plan for families who want to relax

[1658] The user inputs, "I want to relax today." The server searches for related facilities and checks the user's state of relaxation using an emotion engine. Then, it proposes the optimal schedule and route for the day, such as "X hot spring at 10:00, lunch at ✓✓ restaurant at 12:30." The user follows this plan and visits each spot. If traffic congestion occurs along the way, the server calculates new route information and notifies the user.

[1659] Example 2: A weekend date plan for couples who want to have fun

[1660] The user inputs, "I want to go to a place that will make me feel happy." The server searches for spots that meet the criteria, and after confirming using its emotion engine that the user's current emotion is one of wanting fun, it proposes a plan of "△△ amusement park at 15:00, dinner at that restaurant at 18:00." The user follows this plan and visits each spot. If a reservation is required, the server makes the reservation with an external service and notifies the user's device of that information.

[1661] Prompt Sentence Examples

[1662] "I want to relax today"

[1663] "I want to go somewhere that makes me feel happy."

[1664] According to the present invention, users can enjoy greater convenience and flexibility in planning and troubleshooting. In particular, by combining it with an emotion engine, it is possible to provide an optimal plan that takes into account the user's emotional state, thereby providing a more fulfilling experience.

[1665] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1666] Step 1:

[1667] The user inputs their wishes and conditions into the device. For example, they input text in natural language such as "I want to relax today" or "I want to go to a place that makes me feel happy." The device receives this input and temporarily stores the data.

[1668] Input: User's wishes and requirements (natural language text)

[1669] Output: Temporarily saved user preferences and conditions (natural language text)

[1670] Step 2:

[1671] The device sends the temporarily saved text data to the server, where it becomes the raw material for analysis.

[1672] Input: Temporarily saved user preferences and conditions (natural language text)

[1673] Output: The desired and required data sent to the server (natural language text)

[1674] Step 3:

[1675] The server receives the data and passes it to a natural language processing (NLP) engine, which analyzes the text and extracts key keywords and requirements, such as "relaxing" and "fun places."

[1676] Input: Desired and condition data sent to the server (natural language text)

[1677] Output: Extracted keywords and requirements (list format)

[1678] Step 4:

[1679] The server sends the user's input data and movement data (such as input speed and facial expression recognition data) to the emotion engine, which analyzes this data and recognizes the user's current emotional state (such as stress, relaxation, or excitement).

[1680] Input: User input and action data

[1681] Output: Recognized emotional state (list format)

[1682] Step 5:

[1683] The server searches for corresponding spots and activities from an internal database and external API based on the analyzed keywords and emotional state. For example, relaxation facilities and fun activity spots are suggested.

[1684] Input: extracted keywords and requirements, recognized emotional states

[1685] Output: Plan candidates (list format)

[1686] Step 6:

[1687] The server evaluates the generated plan candidates based on weather information, congestion levels, user reviews, etc., and selects the optimal plan. A scoring algorithm is used for the evaluation.

[1688] Input: Plan options, weather information, congestion level, user reviews

[1689] Output: Best plan (list format)

[1690] Step 7:

[1691] The server calculates an efficient route based on the selected optimal plan, taking into account the user's mode of transportation (walking, car, train, etc.), current traffic conditions, weather information, and emotional state.

[1692] Input: Best plan, transportation method, traffic conditions, weather information, emotional state

[1693] Output: Optimal travel route (route information)

[1694] Step 8:

[1695] The server sends the calculated route and plan information in JSON format to the device, which displays the received information on the user interface. The user confirms this and starts navigation.

[1696] Input: Optimal travel route, plan information

[1697] Output: Navigation information displayed on the user's device

[1698] Step 9:

[1699] The device periodically sends the user's current location, traffic information, and emotional state to the server, which then recalculates the route as needed and sends the new route information to the device.

[1700] Input: User's current location, traffic information, emotional state

[1701] Output: Recalculated route information

[1702] Step 10:

[1703] If a user encounters an unexpected situation, for example by inputting "I'm going to be late for my reservation" into their device, the device will send this information to the server. The server will then use this information to generate an optimal plan or adjust the schedule, and send the new plan information to the user's device.

[1704] Input: User input for unplanned situations

[1705] Output: New plan information

[1706] (Application example 2)

[1707] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1708] Conventional plan generation systems only considered the user's preferences and conditions, making it difficult to provide optimal plans that reflected the user's emotional state. Food delivery services also faced the problem of being unable to provide suggestions tailored to the user's emotions and mood. The present invention aims to solve this problem by providing optimal plans and food delivery plans that reflect the user's emotional state.

[1709] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting a user's preferences and conditions, means for analyzing the input preferences and conditions and extracting key information, means for generating multiple plan candidates based on the analyzed information, means for evaluating the generated plan candidates and selecting an optimal plan, means for calculating a travel route based on the optimal...

Claims

1. A means for inputting user preferences and requirements; A means of analyzing input requests and conditions and extracting key information; means for generating a plurality of candidate plans based on the analyzed information; A means for evaluating the generated plan candidates and selecting the optimal plan; a means for calculating a travel route based on an optimal plan; means for transmitting the calculated route information and plan information to a user's terminal; A means for optimizing a route based on a user's current location and traffic information; means for generating a new plan in response to an unexpected situation and transmitting the new plan to the user terminal; A system including a means for making a reservation with an external service when a reservation is required.

2. 2. The system according to claim 1, further comprising means for analyzing the user's wishes and conditions using natural language processing.

3. 2. The system according to claim 1, further comprising means for taking into account weather information and congestion conditions when calculating a user's travel route.

Images