System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving information on a travel destination, a period, an activity, and a meal desired by a user; means for generating an optimal visiting place and schedule based on the received information; means for cooperating with a map information providing service to automatically calculate a moving time; means for collectively reserving accommodation facilities, transportation facilities, and activities; means for acquiring weather information and a situation of a tourist spot in real time and proposing a change plan of a travel schedule; and means for coping with a plurality of languages.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern travelers face many challenges when planning their trips. The main challenge is the tedious process of selecting destinations, activities, and dining options, calculating travel times between each destination, and creating an optimal schedule. It also requires the time-consuming task of booking accommodations and transportation using multiple booking sites. Furthermore, it is difficult to flexibly respond to changes in weather or traffic conditions during a trip. These situations pose significant challenges, particularly for inbound tourists and the elderly, who require multilingual support. This invention aims to solve these challenges. [Means for solving the problem]

[0005] This invention provides a system that receives information on a user's desired travel destinations, duration, activities, and meals, and generates optimal places to visit and a schedule based on the received information. It also provides a system that includes a means for linking with a map information service to automatically calculate travel times, a means for making reservations for accommodations, transportation, and activities all at once, a means for acquiring real-time weather information and tourist spot conditions to suggest changes to the travel schedule, and a means for supporting multiple languages. This system also includes a function for analyzing images uploaded by the user and suggesting places to visit and eat based on the analysis results, a function for providing the user with an interface for confirming and confirming their travel plans, and a function for generating and sending a reservation confirmation based on the confirmed schedule. This allows for more efficient and flexible travel planning, and realizes a system that can deal with uncertainties during travel.

[0006] "Travel destination" refers to a place that is a destination that a traveler intends to visit.

[0007] "Duration" refers to the number of days and period over which the trip takes place.

[0008] "Activities" refers to specific actions or events that take place during a trip, such as sightseeing, sports, or cultural experiences.

[0009] "Meals" refers to the food and drinks consumed during travel and the places that serve them.

[0010] "Information" means data provided by a user, including details about travel destinations, duration, activities and meals.

[0011] "Means" refers to methods or functions for achieving a purpose, and includes specific technical techniques and devices for realizing each function.

[0012] A "schedule" refers to the arrangement of plans and activities to be carried out during a trip in a time and order.

[0013] "Map information service" refers to a type of online service that provides maps and geographic information, enabling route searches and travel time calculations.

[0014] "Reservation" refers to the act of a traveler securing accommodation, transportation, activities, etc. in advance.

[0015] "Real-time" refers to the current state of things, where information is acquired and updated instantly without delay.

[0016] "Weather Information" refers to data regarding weather conditions at your destination during your trip.

[0017] "Tourist destinations" refer to attractive places that tourists visit, such as famous historical sites, natural landscapes, and cultural facilities.

[0018] A "situation" refers to an event, circumstance, or state of affairs at a particular point in time.

[0019] "Changes" refers to new proposals or ideas that incorporate changes to the original plan or schedule.

[0020] A "system" refers to an integrated technological infrastructure consisting of multiple interrelated components and means to achieve a specific purpose. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[0043] Overall structure

[0044] This system consists of a user's device, a server, and many external API services. The user's device is the device that runs the application, and the server is responsible for generating and optimizing travel plans and obtaining external information. In addition, by utilizing map information services and reservation service APIs, each step of the travel plan can be carried out smoothly.

[0045] Travel planning

[0046] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[0047] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[0048] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[0049] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[0050] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[0051] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0052] Reservations and real-time information

[0053] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0054] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[0055] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0056] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[0057] Multilingual support and customer support

[0058] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[0059] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[0060] Specific examples

[0061] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[0062] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[0063] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it will suggest an alternative schedule. In this way, the user can enjoy their trip efficiently and hassle-free.

[0064] This system greatly improves convenience for travelers by automating and providing flexibility in travel planning.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The user launches the application and the login screen appears.

[0068] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[0069] Step 2:

[0070] The user enters their ID and password and clicks the "Login" button.

[0071] The device sends the entered ID and password to the server.

[0072] Step 3:

[0073] The server authenticates the user information.

[0074] The database is accessed to search for user information that matches the entered ID and password.

[0075] Step 4:

[0076] If the server is successful in authenticating, it generates an authentication token.

[0077] A session is established for the authenticated user, and a token is generated and sent to the terminal.

[0078] Step 5:

[0079] The device displays the main menu.

[0080] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[0081] Step 6:

[0082] The user selects "Plan a Trip" from the main menu.

[0083] Tap the "Travel Plans" option to proceed to the next form.

[0084] Step 7:

[0085] The device will display a screen for entering travel destination, duration, activities, and meals.

[0086] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[0087] Step 8:

[0088] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[0089] Enter the required information using the input form and image upload feature.

[0090] Step 9:

[0091] The terminal transmits the input information and image data to the server.

[0092] Send structured data and image data in JSON format to the server.

[0093] Step 10:

[0094] The server analyzes the input using generated AI.

[0095] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[0096] Step 11:

[0097] The server determines the strategy and calculates travel time using the Google Maps API.

[0098] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[0099] Step 12:

[0100] The server retrieves reservation options for accommodation, transportation, and activities.

[0101] A list of accommodation, transportation, and activity options is created through each booking site's API.

[0102] Step 13:

[0103] The server sends the optimized schedule to the terminal.

[0104] The final schedule data is sent to the terminal and prepared for display to the user.

[0105] Step 14:

[0106] The terminal displays the proposed schedule to the user.

[0107] The schedule contents are displayed on the screen so that the user can check them.

[0108] Step 15:

[0109] The user checks the schedule and presses the "Confirm" button.

[0110] Check the contents of the proposed schedule and tap the "Confirm" button.

[0111] Step 16:

[0112] The server batches reservations for accommodation, transportation, and activities.

[0113] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[0114] Step 17:

[0115] The server sends a reservation confirmation to the terminal.

[0116] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[0117] Step 18:

[0118] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[0119] Display the booking confirmation in the UI and provide an "Email" option.

[0120] Step 19:

[0121] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[0122] Real-time data is obtained through weather information services and tourist destination APIs.

[0123] Step 20:

[0124] The server detects the fluctuations and generates alternatives using generative AI.

[0125] Generate new schedules that adapt to changes in weather and traffic conditions.

[0126] Step 21:

[0127] The server sends alternatives to the terminal.

[0128] The new schedule proposal is sent to the device and a notification is sent.

[0129] Step 22:

[0130] The terminal notifies the user of the alternatives.

[0131] Use popups and notifications to let users know about alternatives.

[0132] Step 23:

[0133] The user selects the desired language from the application settings.

[0134] Use the language settings option to select your preferred language.

[0135] Step 24:

[0136] The device will display the UI in the selected language.

[0137] The text is displayed according to the language you set.

[0138] Step 25:

[0139] The user presses the "Customer Support" button.

[0140] Tap on the "Customer Support" option.

[0141] Step 26:

[0142] The server provides customer support based on the user's inquiry.

[0143] Resolve user issues through chatbots and human support.

[0144] Example 1

[0145] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Conventional travel planning support systems often require a lot of effort to input and analyze information about users' desired travel destinations, duration, activities, and meals. Furthermore, it is difficult to comprehensively provide multiple functions, such as optimizing travel schedules, calculating travel times, managing reservations, acquiring and responding to real-time information, and supporting multiple languages. As a result, users' convenience is reduced, and travel planning does not proceed smoothly.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0148] In this invention, the server includes means for receiving information about a user's desired destination, period, activities, and meals, means for using a generative AI model to generate optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring meteorological information and tourist spot conditions in real time and providing suggested changes to the travel schedule, means for supporting multiple languages, and means for analyzing multimedia data uploaded by the user and suggesting places to visit and eat based on the analysis results, thereby enabling users to efficiently plan their trips and flexibly respond to changes in the situation in real time.

[0149] "User" means an individual or organization that uses the system to make travel plans.

[0150] A "destination" is a place that a user wishes to visit during a trip.

[0151] "Duration" is the specific number of days or hours that the user plans to spend traveling.

[0152] An "activity" is an event or experience that a user wants to participate in during their trip.

[0153] "Meals" refers to the foods and places to eat that the user wants to eat during their trip.

[0154] A "generative AI model" is an artificial intelligence algorithm that automatically generates optimal places to visit and schedules based on user input.

[0155] "Map information service" refers to a geographic information system and its API used to calculate travel time and distance.

[0156] "Accommodation" refers to a hotel or lodging facility that a User reserves for stay during their trip.

[0157] "Transportation" refers to the vehicles such as airplanes, trains, buses, and taxis that users use to travel.

[0158] "Weather information" refers to information about the weather forecast and weather conditions during the travel period.

[0159] "Tourist destination situation" refers to the congestion situation at tourist destinations, business information, event information, etc.

[0160] "Multiple languages" refers to the ability to provide user interfaces and system functionality in different languages.

[0161] "Multimedia data" refers to information in digital form, such as images, audio, and video, uploaded by users.

[0162] "Analysis results" are information and recommendations obtained as a result of analyzing multimedia data.

[0163] A "reservation confirmation" is a document that is generated after an itinerary is confirmed and details the reservation details for accommodation, transportation, and activities.

[0164] This invention relates to an interactive travel concierge system that assists users in planning their trips. This system is composed of a user's device, a server, and multiple external API services. The user's device runs an application, while the server is responsible for generating and optimizing the trip plan and acquiring external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be smoothly carried out.

[0165] Hardware and software configuration

[0166] The user's terminal is a device capable of running applications, such as a smartphone, tablet, or personal computer. This device provides an interface for inputting travel plans, transmits the user's input information to the server, and receives and displays responses from the server to the user.

[0167] The server has high-performance computing resources, analyzes the information entered by the user, and generates optimal places to visit and a schedule using a generative AI model that incorporates natural language processing and image analysis technologies, for example.

[0168] The external API services used include map information services, such as a map information API and a reservation service API, which allow users to calculate travel times between locations and make reservations for accommodations and transportation.

[0169] Program operation explanation

[0170] 1. The user launches the travel concierge application from their device and logs in. On the login screen, the user enters their username and password, which are then sent to the server. The server then authenticates and verifies the user's identity.

[0171] 2. After authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images.

[0172] 3. The device sends this input information to the server, which analyzes it and uses a generative AI model to generate optimal places to visit and a schedule.

[0173] For example, the prompt text might look like this:

[0174] "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Sushi images"

[0175] 4. The server uses a map information API to calculate travel time between locations and optimize the overall schedule.

[0176] 5. The server sends the generated schedule to the terminal, which displays the proposed schedule to the user for confirmation.

[0177] 6. Once the user confirms and confirms the proposed schedule, the server will make all reservations for accommodation, transportation, and activities at once, and generate a reservation confirmation based on this reservation information and send it to the terminal.

[0178] 7. During the trip, the server periodically obtains real-time weather information and tourist destination information. It uses the weather information service and tourist destination APIs to obtain real-time information and generates alternatives for changing the schedule as needed. This information is sent to the terminal and notified to the user.

[0179] 8. The device provides a multilingual UI to display applications in the language the user needs. If a user needs assistance during their trip, they can contact the server by pressing the "Customer Support" button, and the server will respond to the inquiry with a chatbot or customer support representative.

[0180] This system allows users to plan trips efficiently and effortlessly, and responds flexibly to real-time changes in the situation. By linking with various APIs, it eliminates travel time between destinations and the hassle of making reservations, enriching the user's travel experience.

[0181] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0182] Step 1:

[0183] A user starts an application and logs in. The terminal prompts for a username and password, and the server receives this authentication information. The server accesses a database and collates the user information to perform authentication. The input data is the username and password, and the output data is the authentication result.

[0184] Step 2:

[0185] After successful login, the terminal displays the main menu. The user selects the "Travel Planning" option. The terminal displays menu selection information and accepts the user's selection as input. The displayed menu contains various items related to travel planning.

[0186] Step 3:

[0187] The user selects a travel planning option and inputs information about the travel destination, duration, desired activities, and dining locations. The terminal receives the input information and sends it to the server. The input data is information about the travel destination, duration, activities, and dining locations, and the output data is the data sent to the server.

[0188] Step 4:

[0189] The server analyzes the received information. The server analyzes the data using a generative AI model and generates the optimal places to visit and schedule based on the user's preferences. An example of a prompt sentence would be "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Images of sushi." The output data are the analysis results and the generated schedule.

[0190] Step 5:

[0191] The server uses a map information API to calculate travel times between visit locations, optimizing the overall schedule. The input data is the coordinate information of the visit locations and the start and destination points of the trip, and the output data is the travel time between each visit location.

[0192] Step 6:

[0193] The server sends the optimized schedule to the terminal, which receives it and displays it to the user. The input data is the optimized schedule received from the server, and the output data is the schedule to display to the user.

[0194] Step 7:

[0195] The user confirms the proposed schedule and presses the confirm button. This input operation causes the terminal to send the schedule confirmation information to the server. The input data is the schedule confirmation operation by the user, and the output data is the schedule confirmation information sent to the server.

[0196] Step 8:

[0197] The server makes reservations for accommodation, transportation, and activities in bulk based on the confirmed schedule. It generates a reservation confirmation and sends it to the terminal. The input data is the confirmed schedule information, and the output data is the reservation confirmation.

[0198] Step 9:

[0199] The terminal displays the reservation confirmation to the user. If desired, the user can receive the confirmation by email, etc. The input data is the reservation confirmation received from the server, and the output data is the information displayed and sent to the user.

[0200] Step 10:

[0201] During the trip, the server periodically obtains weather information and real-time information on tourist destinations. This is done using the weather information service and tourist destination APIs. The input data is data from the real-time information API, and the output data is analyzed real-time information.

[0202] Step 11:

[0203] When changes in weather or traffic conditions are detected, the server uses a generative AI model to generate alternative plans and sends them to the device. The device then notifies the user of the alternative plans. The input data is the change information, and the output data is the alternative plans.

[0204] Step 12:

[0205] The user selects the desired language from the application settings. The device displays the UI in the selected language. The input data is the user's language selection information, and the output data is the UI in the selected language.

[0206] Step 13:

[0207] If a user needs assistance during their trip, they press the "Customer Support" button. The server accepts the inquiry and responds via a chatbot or customer support representative. The input data is the user's inquiry information, and the output data is the response information from the server.

[0208] (Application example 1)

[0209] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] Conventional travel planning support systems lacked the means to perform detailed simulations of travel destinations, making it difficult for users to inspect or experience the places they would actually visit in advance. Furthermore, the generation of reservation confirmations and final confirmation of travel plans were performed only in real space, making flexible responses difficult. Furthermore, there was no interface available to accept complex user input and intuitively and efficiently plan travel in a virtual space. To solve these problems, there is a need for a system that can intuitively create travel plans in a virtual space and display 3D models of travel destinations.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0212] In this invention, the server includes: means for receiving information on a user's desired travel destination, duration, activities, and meals; means for generating optimal visit locations and schedules based on the received information; means for linking with a map information service to automatically calculate travel times; means for making reservations for accommodations, transportation, and activities all at once; means for acquiring real-time weather information and tourist spot conditions and proposing travel schedule changes; means for supporting multiple languages; means for providing an interface through which a user inputs travel destination information using a head-mounted display and makes a travel plan in a virtual space; and means for displaying a 3D model of the travel destination in the virtual space and allowing the user to simulate the places they will visit. This allows users to intuitively plan their trip in the virtual space and allows them to inspect and experience the destination in advance. Furthermore, flexible reservation confirmation can be ensured through confirmation in the virtual space.

[0213] "Means for receiving information on the user's desired travel destination, duration, activities, and meals" is a function that collects data regarding the destination, duration of stay, activities the user wishes to participate in, and meals the user wishes to eat, which are input to plan the trip.

[0214] "Means for generating optimal places to visit and schedules based on received information" refers to a function that uses algorithms and AI to create travel schedules including optimal tourist spots and activities based on information entered by the user.

[0215] The "means for linking with a map information service to automatically calculate travel time" is a function that links with a map service and automatically calculates travel time between each visited location.

[0216] "A means of booking accommodation, transportation, and activities all at once" is a function that allows users to book accommodation, transportation, and activities all at once based on their confirmed travel schedule.

[0217] "Means for obtaining weather information and tourist destination conditions in real time and proposing changes to the travel schedule" is a function that provides users with suggested schedule changes as needed based on weather information and tourist destination conditions obtained in real time during the trip.

[0218] "Means for supporting multiple languages" refers to a function that provides interfaces and guidance in multiple languages ​​to users who speak different languages.

[0219] "A means of displaying a 3D model of a travel destination in a virtual space, allowing users to simulate visiting places" is a function that uses virtual reality technology to present a 3D model of a travel destination to users, allowing users to simulate visiting places in advance.

[0220] "A means for providing an interface that allows a user to input travel destination information using a head-mounted display and make travel plans in a virtual space" is a function that provides an interface that allows a user to input travel destination and activity information in a virtual space while wearing a head-mounted display and make travel plans.

[0221] "Means for generating and sending a reservation confirmation based on a confirmed schedule and prompting the user to confirm it within the virtual space" is a function that creates a reservation confirmation based on the travel schedule confirmed by the user and prompts the user to confirm it through an interface within the virtual space.

[0222] "A means of analyzing images uploaded by users and suggesting places to visit and eat based on the analysis results" is a function that analyzes image data provided by users using AI, etc., and suggests the most suitable tourist spots and restaurants based on the analysis results.

[0223] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[0224] Overall structure

[0225] This system consists of a device used by the user, such as a head-mounted display or smart glasses, a server, and an external API service. The user's device provides an interface for planning the trip, while the server is responsible for generating and optimizing the trip plan, as well as obtaining external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be carried out smoothly.

[0226] Travel planning

[0227] 1. The user wears the head-mounted display and starts the application. The user performs the login operation, the device requests the ID and password, and the server performs authentication.

[0228] 2. Once authentication is complete, the device displays the virtual space's main menu. The user can select the "Plan Your Trip" option and enter information about their travel destination, duration, desired activities, and dining options.

[0229] 3. Users can select the places they want to visit while looking at a 3D model of the destination in the virtual space. This information is sent from the device to the server.

[0230] 4. The server analyzes the input information and generates optimal places to visit and a schedule using a generative AI model, such as GPT-3.5.

[0231] 5. Using the map information service API, the server calculates travel times between locations and optimizes the overall schedule. The server then sends the generated schedule to the device.

[0232] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0233] Reservations and real-time information

[0234] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0235] 2. A reservation confirmation is generated based on this reservation information and sent to the device. The device displays the confirmation to the user and prompts them to confirm it in the virtual space if necessary.

[0236] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0237] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[0238] Specific examples

[0239] For example, suppose a user plans a three-day trip to a certain location in Japan in a virtual space. The user puts on a head-mounted display, launches the application, logs in, and selects the travel planning options. The user uploads a "certain location in Japan" as the travel destination, "three days" as the duration, "historical buildings" and "famous places" as places to visit, and "images of traditional cuisine" as food they want to eat. Once this information is sent to the server, the server uses a generative AI model to generate an optimal travel itinerary. Using a map information service API, the server calculates the optimal route and travel time between each destination. The server obtains reservation information for appropriate accommodations, transportation, and activities, and sends the optimal schedule to the device. Once the user confirms this schedule and presses the confirm button, the server makes all reservations and generates a reservation confirmation. During the trip, the server periodically obtains real-time information on weather and tourist attractions, and suggests an alternative itinerary if there are any changes. In this way, users can enjoy their trip efficiently and hassle-free.

[0240] Prompt Sentence Examples

[0241] An example of a prompt that the user can enter into the AI ​​generator is:

[0242] "Information to help you plan your trip:

[0243] Travel destination: Somewhere in Japan

[0244] Duration: 3 days

[0245] Places I want to visit: Historical buildings, famous places

[0246] What to eat: Images of traditional dishes

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] The user puts on the head-mounted display and launches the travel planning application. The user enters their ID and password, and the server performs authentication. The input is the user's ID and password, and the output is the authentication result. The server checks the authentication information against the database.

[0250] Step 2:

[0251] After successful authentication, the device displays the main menu of the virtual space. The user selects the "Plan a Trip" option and enters information about the travel destination, duration, desired activities, and dining locations. The input is the user's travel destination information, and the output is the transmission of the input information. The device receives the user's input through the virtual interface and sends it to the server.

[0252] Step 3:

[0253] The server analyzes the information it receives and uses a generative AI model to generate optimal places to visit and a schedule. The input is the user's travel information, and the output is an optimal travel schedule. The server inputs a prompt into the generative AI model to generate a schedule. GPT-3.5 is used as the generative AI model.

[0254] Step 4:

[0255] The server uses the map information service API to calculate the travel time between visited locations. The input is a list of visited locations, and the output is the travel time between each visited location. The data is calculated using API calls.

[0256] Step 5:

[0257] The server generates an optimal travel schedule and sends it to the terminal. The input is the optimal schedule and the output is a transmission to the terminal. The server sends the optimal schedule to the terminal as a data packet.

[0258] Step 6:

[0259] The terminal displays the proposed schedule to the user, who then confirms it. The received schedule is the input, and the user's confirmation result is the output. The terminal visually displays the schedule in a virtual space, and the user presses the confirmation and agreement buttons.

[0260] Step 7:

[0261] Once the user confirms the schedule, the server makes reservations for accommodation, transportation, and activities in one go. The input is the confirmed schedule, and the output is a reservation confirmation. The server uses an external reservation service API to make the necessary reservations and generate a confirmation.

[0262] Step 8:

[0263] The server sends the reservation confirmation to the terminal, which then displays it to the user. The input is the reservation confirmation, and the output is a display to the user. The terminal displays the confirmation in the virtual space and asks the user for confirmation.

[0264] Step 9:

[0265] During the trip, the server periodically obtains real-time weather information and tourist destination information, and generates suggested changes to the travel schedule if any abnormalities are detected. The real-time information is the input, and the suggested changes are the output. The server uses weather information services and tourist destination APIs to generate suggested changes using a generative AI model.

[0266] Step 10:

[0267] The device notifies the user of the alternative plan, and if the user accepts it, the new schedule is applied. The alternative plan is the input, and the new schedule is presented as the output. The device displays the alternative plan in the virtual space, and applies the new schedule after confirmation.

[0268] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0269] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[0270] Overall structure

[0271] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes the user's input and reactions to recognize emotions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[0272] Travel planning

[0273] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[0274] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[0275] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[0276] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[0277] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[0278] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0279] Introducing and Adapting the Emotion Engine

[0280] 1. The emotion engine analyzes the reactions and comments entered by users when checking their schedules.

[0281] 2. The emotion engine recognizes the user's emotional state and identifies states such as joy, surprise, excitement, and anxiety.

[0282] 3. The server uses the analysis results of the emotion engine to readjust the suggested places to visit and schedule.

[0283] For example, if a user is feeling anxious, the app will suggest more relaxing tourist spots and activities.

[0284] If the user is excited, increase the adventurous activity.

[0285] Reservations and real-time information

[0286] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0287] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[0288] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0289] 4. If changes in weather or traffic conditions are detected, the server uses generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[0290] Multilingual support and customer support

[0291] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[0292] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[0293] Specific examples

[0294] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[0295] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[0296] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it uses the emotion engine to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[0297] This system greatly improves convenience for travelers by automating trip planning and adding flexibility through emotion recognition.

[0298] The processing flow will be explained below.

[0299] Step 1:

[0300] The user launches the application and the login screen appears.

[0301] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[0302] Step 2:

[0303] The user enters their ID and password and clicks the "Login" button.

[0304] The device sends the entered ID and password to the server.

[0305] Step 3:

[0306] The server authenticates the user information.

[0307] The database is accessed to search for user information that matches the entered ID and password.

[0308] Step 4:

[0309] If the server is successful in authenticating, it generates an authentication token.

[0310] A session is established for the authenticated user, and an authentication token is generated and sent to the terminal.

[0311] Step 5:

[0312] The device displays the main menu.

[0313] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[0314] Step 6:

[0315] The user selects "Plan a Trip" from the main menu.

[0316] Tap the "Travel Plans" option to proceed to the next form.

[0317] Step 7:

[0318] The device will display a screen for entering travel destination, duration, activities, and meals.

[0319] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[0320] Step 8:

[0321] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[0322] Enter the required information using the input form and image upload feature.

[0323] Step 9:

[0324] The terminal transmits the input information and image data to the server.

[0325] Send structured data and image data in JSON format to the server.

[0326] Step 10:

[0327] The server analyzes the input using generated AI.

[0328] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[0329] Step 11:

[0330] The server determines the strategy and calculates travel time using the Google Maps API.

[0331] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[0332] Step 12:

[0333] The server retrieves reservation options for accommodation, transportation, and activities.

[0334] A list of accommodation, transportation, and activity options is created through each booking site's API.

[0335] Step 13:

[0336] The server sends the optimized schedule to the terminal.

[0337] The final schedule data is sent to the terminal and prepared for display to the user.

[0338] Step 14:

[0339] The terminal displays the proposed schedule to the user.

[0340] The schedule contents are displayed on the screen so that the user can check them.

[0341] Step 15:

[0342] The user checks the schedule and presses the "Confirm" button.

[0343] Check the contents of the proposed schedule and tap the "Confirm" button.

[0344] Step 16:

[0345] The server uses an emotion engine to analyze users' reactions and comments.

[0346] The emotion engine analyzes the reactions and comments entered by the user to identify the user's emotional state.

[0347] Step 17:

[0348] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[0349] It generates a new schedule that takes into account the user's emotional state, and suggests a relaxing plan for a user who is feeling anxious, for example.

[0350] Step 18:

[0351] The server transmits the readjusted schedule to the terminal.

[0352] The adjusted schedule data is sent again to the terminal and the user is asked to confirm it.

[0353] Step 19:

[0354] The user reconfirms the schedule and presses the "Final Confirm" button.

[0355] Check the adjusted schedule and tap the "Final" button.

[0356] Step 20:

[0357] The server batches reservations for accommodation, transportation, and activities.

[0358] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[0359] Step 21:

[0360] The server sends a reservation confirmation to the terminal.

[0361] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[0362] Step 22:

[0363] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[0364] Display the booking confirmation in the UI and provide an "Email" option.

[0365] Step 23:

[0366] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[0367] Real-time data is obtained through weather information services and tourist destination APIs.

[0368] Step 24:

[0369] The server detects the fluctuations and generates alternatives using generative AI.

[0370] Generate new schedules based on changes in weather and traffic conditions, including considering the emotional state to incorporate using an emotion engine.

[0371] Step 25:

[0372] The server sends alternatives to the terminal.

[0373] The new schedule proposal is sent to the device and a notification is sent.

[0374] Step 26:

[0375] The terminal notifies the user of the alternatives.

[0376] Use popups and notifications to let users know about alternatives.

[0377] Step 27:

[0378] The user selects the desired language from the application settings.

[0379] Use the language settings option to select your preferred language.

[0380] Step 28:

[0381] The device will display the UI in the selected language.

[0382] The text is displayed according to the language you set.

[0383] Step 29:

[0384] The user presses the "Customer Support" button.

[0385] Tap on the "Customer Support" option.

[0386] Step 30:

[0387] The server provides customer support based on the user's inquiry.

[0388] Resolve user issues through chatbots and human support.

[0389] Example 2

[0390] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] Conventional travel planning systems generate schedules based on a user's desired travel destinations, duration, and activity information, but lack the flexibility to respond to user emotions or real-time information fluctuations. Furthermore, they do not support multiple languages, limiting their usability for global users. This makes it difficult for users to flexibly adjust plans to accommodate sudden changes during their trip or their individual emotional states.

[0392] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0393] In this invention, the server includes a means for analyzing the user's emotional state and readjusting the proposed content based on the emotion, a means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, and a means for supporting multiple languages, thereby enabling the provision of flexible travel plans that respond to the user's emotions and real-time changes in the situation.

[0394] A "user" is an end user who uses the travel planning system to select a travel destination and adjust a schedule.

[0395] "Destination" refers to a geographic location that a user wishes to visit.

[0396] "Duration" refers to the specific length of time that a user has set for planning a trip.

[0397] "Activities" refer to events or activities that a user wants to experience during their trip.

[0398] "Meal information" refers to information about the food and drink that the user desires to eat and drink during the trip.

[0399] "Map information service" refers to a third-party service that provides geographic information and calculates travel times between locations.

[0400] "Accommodation" refers to a facility where a user can stay while traveling.

[0401] "Transportation" refers to the vehicles and services used for transportation during a trip.

[0402] "Emotional state" refers to the psychological state a user feels about a travel plan or proposal.

[0403] "Real-time information" refers to the latest information at that time, such as weather information and the status of tourist attractions.

[0404] An "emotion engine" refers to software that analyzes user input and reactions to recognize emotions.

[0405] "Generative AI" refers to an artificial intelligence model that creates optimal travel schedules based on user input.

[0406] "Language support" refers to the ability to provide interfaces and notifications in the language selected by the user.

[0407] "Proposed Changes" means any proposed modifications or alternatives to your travel plans or schedule.

[0408] MODE FOR CARRYING OUT THE INVENTION

[0409] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[0410] Overall structure

[0411] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine recognizes emotions by analyzing the user's input and reactions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[0412] Travel planning

[0413] First, the user launches the application and logs in. The device prompts for an ID and password, and the server authenticates them. Once authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images. The device then sends this information to the server. The server analyzes the entered information and uses a generative AI (e.g., GPT-4) to generate an optimal itinerary and itinerary. An example prompt might be, "The user is planning a three-day trip to Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and eat sushi. Please suggest an optimal itinerary." The server uses the Google Maps API to calculate travel times between locations and optimize the overall itinerary. The server then generates the optimized itinerary and sends it to the device.

[0414] The device displays the proposed schedule to the user, who then confirms it. The emotion engine analyzes the user's reactions and comments when checking the schedule. The emotion engine recognizes the user's emotional state and sends the analysis results to the server. The server readjusts the proposed content based on the emotion engine's analysis results and sends the prompt text back to the generation AI to update the schedule.

[0415] Reservations and real-time information

[0416] Once the user confirms and confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities. The server generates a reservation confirmation based on the reservation information and sends it to the terminal, which displays it to the user. If the user wishes, the confirmation can be sent by email or other means.

[0417] Furthermore, during the trip, the server periodically obtains real-time weather information and tourist destination information. It obtains the latest information using weather information services and tourist destination APIs. If changes in weather or traffic conditions are detected, the server uses a generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[0418] Multilingual support and customer support

[0419] The user selects their preferred language in the application settings. The device displays the UI in the selected language. If the user needs assistance during their trip, they can press the "Customer Support" button, which the server will then handle via a chatbot or customer support representative.

[0420] Specific examples

[0421] For example, if a user plans a three-day trip to Tokyo, they launch the application, log in, and select the "Plan a Trip" option. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Sensoji Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. Using the Google Maps API, the server calculates the optimal route and travel time from Sensoji Temple to Tokyo Skytree. The server obtains reservation information for appropriate accommodation, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all the reservations in one go and generates a reservation confirmation.

[0422] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, the emotion engine is used to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[0423] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0424] Step 1:

[0425] The user launches the application and the login screen appears.

[0426] Specific operation: The user enters a user ID and password into the terminal, and the terminal sends the entered information to the server.

[0427] Input: User ID, Password

[0428] Output: Authentication request

[0429] Step 2:

[0430] The server compares the received ID and password with the database and performs authentication.

[0431] Specific operation: The server checks the ID and password against the database, and if they match, it sends an authentication success message to the terminal.

[0432] Input: User ID, Password

[0433] Output: Authentication success message or authentication failure message

[0434] Step 3:

[0435] The device receives a successful authentication message and displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options.

[0436] Specific operation: The user inputs information about the travel destination, duration, activities, and places to eat into the device, which then sends it to the server.

[0437] Input: Travel destination, duration, activities, and dining information

[0438] Output: Travel planning request

[0439] Step 4:

[0440] The server analyzes the user's input information and uses generative AI to generate the optimal places to visit and schedule.

[0441] Specific operation: The server analyzes the user's input information and sends prompt text to the generation AI to generate a schedule.

[0442] Example prompt: The user is planning a three-day trip in Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and he wants to eat sushi. Please suggest the best itinerary.

[0443] Input: Travel planning request (destination, duration, activities, dining information)

[0444] Output: Initial schedule

[0445] Step 5:

[0446] The server uses the Google Maps API to calculate travel times between locations and optimize the overall schedule.

[0447] Specific operation: The server calculates the travel time between each location using the Google Maps API, optimizes the overall schedule based on the calculation results, and sends the optimized schedule to the device.

[0448] Input: Location information of the place to be visited

[0449] Output: Optimized schedule

[0450] Step 6:

[0451] The terminal displays the proposed schedule to the user, who then checks the schedule and enters comments or requests for adjustments as necessary.

[0452] Specific operation: The terminal displays the schedule screen and the user confirms it. If the user inputs an adjustment request, the terminal sends it to the server.

[0453] Input: Initial Schedule

[0454] Output: User response and adjustment request

[0455] Step 7:

[0456] The server analyzes the user's reactions and comments using an emotion engine to identify their emotional state.

[0457] Specific operation: The server sends the user's comments and reactions to the emotion engine. The emotion engine analyzes the user's emotional state and sends the results to the server.

[0458] Input: User responses and comments

[0459] Output: User's emotional state

[0460] Step 8:

[0461] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[0462] Specific operation: The server uses the results of the emotion engine to send a prompt to the generation AI again and update the proposal content.

[0463] Input: User's emotional state, prompt sentence

[0464] Output: Re-adjusted schedule

[0465] Step 9:

[0466] Once the user confirms and confirms the rescheduled schedule, the server collectively makes reservations for accommodation, transportation, and activities.

[0467] Specific operation: The user checks the rescheduled schedule on the terminal and presses the confirm button. The server makes various reservations through the reservation system.

[0468] Input: Re-arranged schedule

[0469] Output: Reservation information and confirmation

[0470] Step 10:

[0471] The server periodically acquires real-time information on weather and tourist spots during the travel period.

[0472] Specific operation: The server periodically calls weather information services and tourist destination APIs to obtain the latest information.

[0473] Input: None (regular operation)

[0474] Output: Real-time information

[0475] Step 11:

[0476] The server generates alternatives by sending prompts to the generation AI based on the fluctuation information, and then uses an emotion engine to generate alternatives that are optimized to the user's emotions.

[0477] Specific operation: The server obtains the latest weather information and tourist destination conditions, and sends prompts to the generation AI as needed. The emotion engine optimizes alternatives and notifies the device.

[0478] Input: Real-time information, prompt text

[0479] Output: Optimized alternatives

[0480] Step 12:

[0481] The user selects the desired language from the application settings.

[0482] Specific behavior: The device displays the language settings screen, the user selects the desired language, and the device updates the UI in the selected language.

[0483] Input: Desired language

[0484] Output: UI in selected language

[0485] Step 13:

[0486] When a user requests customer support, the server initiates the response.

[0487] Specific operation: The user presses the support button, the server accepts the inquiry, and responds via a chatbot or customer support representative.

[0488] Input: Support Request

[0489] Output: Supported

[0490] (Application example 2)

[0491] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0492] Conventional travel planning support systems propose itineraries based on user preferences, but because they are unable to consider the user's emotional state, the proposed itinerary often does not match the user's expectations. Furthermore, there are only a limited number of systems that allow flexible schedule changes in real time during a trip, leaving room for improvement in the user experience. Another problem is that the proposed alternatives are uniform and not optimized for each individual user.

[0493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0494] In this invention, the server includes means for receiving information on the user's desired travel destinations, period, activities, and meals, means for generating optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, means for recognizing the user's emotional state using an emotion recognition engine and optimizing the travel plan and alternatives, and means for supporting multiple languages, thereby enabling the server to propose flexible and optimal travel plans that take the user's emotional state into consideration.

[0495] "User's desired travel destination, duration, activities, and meal information"

[0496] This refers to details of the places and duration a user wants to visit during their trip, the activities they want to participate in, and the meals they want to eat.

[0497] "Optimal visiting locations and schedules"

[0498] This refers to the allocation of destinations and time planned to make the trip efficient and enjoyable based on the user's desired information.

[0499] "Map information provision service"

[0500] This refers to external data services that provide location and route information, such as services that use map APIs.

[0501] "A one-stop solution for booking accommodation, transport and activities"

[0502] This refers to the ability for users to book accommodation, transportation, and local activities for their planned trip all at once.

[0503] "Weather information and tourist destination conditions"

[0504] This refers to real-time weather forecasts, congestion status at tourist spots, business status, etc.

[0505] Emotion recognition engine

[0506] This refers to a technology that analyzes and recognizes a user's emotional state (happiness, surprise, excitement, anxiety, etc.) from their facial expressions and behavior.

[0507] "A way to support multiple languages"

[0508] It refers to the ability of an application or system to provide a user interface in different languages.

[0509] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and is combined with an emotion engine that recognizes the user's emotions and optimizes the plan.

[0510] System configuration

[0511] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, and can be a smartphone, tablet, or PC. The server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes user input and reactions and recognizes emotions. The external API service includes a map information service and a reservation service API.

[0512] The process of emotion recognition

[0513] When a user plans a trip, facial expression recognition technology is used to analyze the user's emotional state in real time. This technology uses OpenCV to detect faces and TensorFlow / Keras to classify emotions. For example, emotional states such as joy, anxiety, and surprise can be detected in real time.

[0514] Travel plan example

[0515] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application on their smartphone, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. In this process, the Google Maps API is used to calculate travel times between each place to be visited and optimize the overall schedule.

[0516] Optimizing recommendations with an emotion engine

[0517] When the user reviews the proposed schedule, the emotion engine analyzes their reaction. If the emotion engine detects that the user is feeling anxious, it will suggest alternatives, including relaxing sightseeing spots and activities. Conversely, if the user is feeling excited, it will increase the suggestions of adventurous activities. For example, if the weather worsens, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed.

[0518] Examples of specific examples and prompts

[0519] In one embodiment of the invention, when a user is watching a horror movie and sentiment analysis is performed, a prompt is generated as follows:

[0520] Example prompt: If a user is feeling "scared," recommend another video with more action and thrills.

[0521] In this way, real-time emotion recognition can be performed to optimize the user experience and provide more personalized travel plans.

[0522] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0523] Step 1:

[0524] The server receives information about the user's desired travel destination, duration, activities, and meals. The user uses a smartphone or tablet device to input details such as the travel destination, duration, and desired activities. This input information is sent from the device to the server. As an output, the server stores the user's input information.

[0525] Step 2:

[0526] The server generates optimal places to visit and a schedule based on the information it receives. Specifically, it uses a generative AI model to formulate places to visit and a schedule that meets the user's preferences. During this process, it analyzes the user's desired information as data processing and calculates optimal places to visit and routes. As an output, the server generates an initial travel schedule.

[0527] Step 3:

[0528] The server works with a map information service to automatically calculate travel times. Specifically, it uses a map API to obtain travel times between each visit location. Based on this information, it optimizes the overall schedule and calculates efficient routes between visit locations. The output is an optimized schedule that reflects travel times.

[0529] Step 4:

[0530] The server sends the generated optimized schedule to the terminal. The proposed schedule is displayed on the user's terminal, and the user confirms its contents. The input is the schedule data from the server, and the output is a confirmation screen displayed on the user's terminal.

[0531] Step 5:

[0532] The device captures the user's facial expression and sends the video data to the server. Specifically, the device's camera function is used to capture the user's facial expression in real time and sends the data to the server. The output is the facial expression data sent to the server.

[0533] Step 6:

[0534] The server uses an emotion recognition engine to analyze the user's facial expression data and recognize emotions. Specifically, it uses a face detection algorithm (OpenCV) to identify the face area, and then uses a TensorFlow / Keras model to classify the emotion. The result of this operation is the recognized emotion data. The output is the recognized emotion (e.g., joy, anxiety, excitement, uneasiness, etc.).

[0535] Step 7:

[0536] The server optimizes travel plans and alternatives based on the recognized emotion data. For example, if the user is feeling "anxious," it will suggest more relaxing sightseeing spots and activities. It uses a generative AI model to generate these optimized plans and alternatives. The output is an optimized travel schedule tailored to the user's emotions.

[0537] Step 8:

[0538] The server continuously obtains weather information and real-time conditions of tourist destinations, and generates proposed schedule changes as needed. Specifically, it uses weather information services and tourist destination APIs to monitor local conditions. Based on this information, a generative AI model creates a new schedule and uses an emotion engine to suggest alternatives based on the user's emotions. The output is the latest alternative schedule.

[0539] Step 9:

[0540] The optimized schedule and alternative plans are notified to the terminal. The terminal displays the proposals to the user in real time and the user confirms the contents. The user confirms and decides on the new plan. The input is the optimized schedule from the server, and the output is a notification screen to the user.

[0541] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0542] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0543] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0544] [Second embodiment]

[0545] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0546] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0548] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0552] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0555] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0556] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0557] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[0558] Overall structure

[0559] This system consists of a user's device, a server, and many external API services. The user's device is the device that runs the application, and the server is responsible for generating and optimizing travel plans and obtaining external information. In addition, by utilizing map information services and reservation service APIs, each step of the travel plan can be carried out smoothly.

[0560] Travel planning

[0561] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[0562] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[0563] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[0564] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[0565] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[0566] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0567] Reservations and real-time information

[0568] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0569] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[0570] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0571] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[0572] Multilingual support and customer support

[0573] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[0574] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[0575] Specific examples

[0576] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[0577] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[0578] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it will suggest an alternative schedule. In this way, the user can enjoy their trip efficiently and hassle-free.

[0579] This system greatly improves convenience for travelers by automating and providing flexibility in travel planning.

[0580] The processing flow will be explained below.

[0581] Step 1:

[0582] The user launches the application and the login screen appears.

[0583] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[0584] Step 2:

[0585] The user enters their ID and password and clicks the "Login" button.

[0586] The device sends the entered ID and password to the server.

[0587] Step 3:

[0588] The server authenticates the user information.

[0589] The database is accessed to search for user information that matches the entered ID and password.

[0590] Step 4:

[0591] If the server is successful in authenticating, it generates an authentication token.

[0592] A session is established for the authenticated user, and a token is generated and sent to the terminal.

[0593] Step 5:

[0594] The device displays the main menu.

[0595] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[0596] Step 6:

[0597] The user selects "Plan a Trip" from the main menu.

[0598] Tap the "Travel Plans" option to proceed to the next form.

[0599] Step 7:

[0600] The device will display a screen for entering travel destination, duration, activities, and meals.

[0601] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[0602] Step 8:

[0603] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[0604] Enter the required information using the input form and image upload feature.

[0605] Step 9:

[0606] The terminal transmits the input information and image data to the server.

[0607] Send structured data and image data in JSON format to the server.

[0608] Step 10:

[0609] The server analyzes the input using generated AI.

[0610] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[0611] Step 11:

[0612] The server determines the strategy and calculates travel time using the Google Maps API.

[0613] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[0614] Step 12:

[0615] The server retrieves reservation options for accommodation, transportation, and activities.

[0616] A list of accommodation, transportation, and activity options is created through each booking site's API.

[0617] Step 13:

[0618] The server sends the optimized schedule to the terminal.

[0619] The final schedule data is sent to the terminal and prepared for display to the user.

[0620] Step 14:

[0621] The terminal displays the proposed schedule to the user.

[0622] The schedule contents are displayed on the screen so that the user can check them.

[0623] Step 15:

[0624] The user checks the schedule and presses the "Confirm" button.

[0625] Check the contents of the proposed schedule and tap the "Confirm" button.

[0626] Step 16:

[0627] The server batches reservations for accommodation, transportation, and activities.

[0628] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[0629] Step 17:

[0630] The server sends a reservation confirmation to the terminal.

[0631] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[0632] Step 18:

[0633] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[0634] Display the booking confirmation in the UI and provide an "Email" option.

[0635] Step 19:

[0636] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[0637] Real-time data is obtained through weather information services and tourist destination APIs.

[0638] Step 20:

[0639] The server detects the fluctuations and generates alternatives using generative AI.

[0640] Generate new schedules that adapt to changes in weather and traffic conditions.

[0641] Step 21:

[0642] The server sends alternatives to the terminal.

[0643] The new schedule proposal is sent to the device and a notification is sent.

[0644] Step 22:

[0645] The terminal notifies the user of the alternatives.

[0646] Use popups and notifications to let users know about alternatives.

[0647] Step 23:

[0648] The user selects the desired language from the application settings.

[0649] Use the language settings option to select your preferred language.

[0650] Step 24:

[0651] The device will display the UI in the selected language.

[0652] The text is displayed according to the language you set.

[0653] Step 25:

[0654] The user presses the "Customer Support" button.

[0655] Tap on the "Customer Support" option.

[0656] Step 26:

[0657] The server provides customer support based on the user's inquiry.

[0658] Resolve user issues through chatbots and human support.

[0659] Example 1

[0660] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0661] Conventional travel planning support systems often require a lot of effort to input and analyze information about users' desired travel destinations, duration, activities, and meals. Furthermore, it is difficult to comprehensively provide multiple functions, such as optimizing travel schedules, calculating travel times, managing reservations, acquiring and responding to real-time information, and supporting multiple languages. As a result, users' convenience is reduced, and travel planning does not proceed smoothly.

[0662] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0663] In this invention, the server includes means for receiving information about a user's desired destination, period, activities, and meals, means for using a generative AI model to generate optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring meteorological information and tourist spot conditions in real time and providing suggested changes to the travel schedule, means for supporting multiple languages, and means for analyzing multimedia data uploaded by the user and suggesting places to visit and eat based on the analysis results, thereby enabling users to efficiently plan their trips and flexibly respond to changes in the situation in real time.

[0664] "User" means an individual or organization that uses the system to make travel plans.

[0665] A "destination" is a place that a user wishes to visit during a trip.

[0666] "Duration" is the specific number of days or hours that the user plans to spend traveling.

[0667] An "activity" is an event or experience that a user wants to participate in during their trip.

[0668] "Meals" refers to the foods and places to eat that the user wants to eat during their trip.

[0669] A "generative AI model" is an artificial intelligence algorithm that automatically generates optimal places to visit and schedules based on user input.

[0670] "Map information service" refers to a geographic information system and its API used to calculate travel time and distance.

[0671] "Accommodation" refers to a hotel or lodging facility that a User reserves for stay during their trip.

[0672] "Transportation" refers to the vehicles such as airplanes, trains, buses, and taxis that users use to travel.

[0673] "Weather information" refers to information about the weather forecast and weather conditions during the travel period.

[0674] "Tourist destination situation" refers to the congestion situation at tourist destinations, business information, event information, etc.

[0675] "Multiple languages" refers to the ability to provide user interfaces and system functionality in different languages.

[0676] "Multimedia data" refers to information in digital form, such as images, audio, and video, uploaded by users.

[0677] "Analysis results" are information and recommendations obtained as a result of analyzing multimedia data.

[0678] A "reservation confirmation" is a document that is generated after an itinerary is confirmed and details the reservation details for accommodation, transportation, and activities.

[0679] This invention relates to an interactive travel concierge system that assists users in planning their trips. This system is composed of a user's device, a server, and multiple external API services. The user's device runs an application, while the server is responsible for generating and optimizing the trip plan and acquiring external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be smoothly carried out.

[0680] Hardware and software configuration

[0681] The user's terminal is a device capable of running applications, such as a smartphone, tablet, or personal computer. This device provides an interface for inputting travel plans, transmits the user's input information to the server, and receives and displays responses from the server to the user.

[0682] The server has high-performance computing resources, analyzes the information entered by the user, and generates optimal places to visit and a schedule using a generative AI model that incorporates natural language processing and image analysis technologies, for example.

[0683] The external API services used include map information services, such as a map information API and a reservation service API, which allow users to calculate travel times between locations and make reservations for accommodations and transportation.

[0684] Program operation explanation

[0685] 1. The user launches the travel concierge application from their device and logs in. On the login screen, the user enters their username and password, which are then sent to the server. The server then authenticates and verifies the user's identity.

[0686] 2. After authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images.

[0687] 3. The device sends this input information to the server, which analyzes it and uses a generative AI model to generate optimal places to visit and a schedule.

[0688] For example, the prompt text might look like this:

[0689] "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Sushi images"

[0690] 4. The server uses a map information API to calculate travel time between locations and optimize the overall schedule.

[0691] 5. The server sends the generated schedule to the terminal, which displays the proposed schedule to the user for confirmation.

[0692] 6. Once the user confirms and confirms the proposed schedule, the server will make all reservations for accommodation, transportation, and activities at once, and generate a reservation confirmation based on this reservation information and send it to the terminal.

[0693] 7. During the trip, the server periodically obtains real-time weather information and tourist destination information. It uses the weather information service and tourist destination APIs to obtain real-time information and generates alternatives for changing the schedule as needed. This information is sent to the terminal and notified to the user.

[0694] 8. The device provides a multilingual UI to display applications in the language the user needs. If a user needs assistance during their trip, they can contact the server by pressing the "Customer Support" button, and the server will respond to the inquiry with a chatbot or customer support representative.

[0695] This system allows users to plan trips efficiently and effortlessly, and responds flexibly to real-time changes in the situation. By linking with various APIs, it eliminates travel time between destinations and the hassle of making reservations, enriching the user's travel experience.

[0696] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0697] Step 1:

[0698] A user starts an application and logs in. The terminal prompts for a username and password, and the server receives this authentication information. The server accesses a database and collates the user information to perform authentication. The input data is the username and password, and the output data is the authentication result.

[0699] Step 2:

[0700] After successful login, the terminal displays the main menu. The user selects the "Travel Planning" option. The terminal displays menu selection information and accepts the user's selection as input. The displayed menu contains various items related to travel planning.

[0701] Step 3:

[0702] The user selects a travel planning option and inputs information about the travel destination, duration, desired activities, and dining locations. The terminal receives the input information and sends it to the server. The input data is information about the travel destination, duration, activities, and dining locations, and the output data is the data sent to the server.

[0703] Step 4:

[0704] The server analyzes the received information. The server analyzes the data using a generative AI model and generates the optimal places to visit and schedule based on the user's preferences. An example of a prompt sentence would be "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Images of sushi." The output data are the analysis results and the generated schedule.

[0705] Step 5:

[0706] The server uses a map information API to calculate travel times between visit locations, optimizing the overall schedule. The input data is the coordinate information of the visit locations and the start and destination points of the trip, and the output data is the travel time between each visit location.

[0707] Step 6:

[0708] The server sends the optimized schedule to the terminal, which receives it and displays it to the user. The input data is the optimized schedule received from the server, and the output data is the schedule to display to the user.

[0709] Step 7:

[0710] The user confirms the proposed schedule and presses the confirm button. This input operation causes the terminal to send the schedule confirmation information to the server. The input data is the schedule confirmation operation by the user, and the output data is the schedule confirmation information sent to the server.

[0711] Step 8:

[0712] The server makes reservations for accommodation, transportation, and activities in bulk based on the confirmed schedule. It generates a reservation confirmation and sends it to the terminal. The input data is the confirmed schedule information, and the output data is the reservation confirmation.

[0713] Step 9:

[0714] The terminal displays the reservation confirmation to the user. If desired, the user can receive the confirmation by email, etc. The input data is the reservation confirmation received from the server, and the output data is the information displayed and sent to the user.

[0715] Step 10:

[0716] During the trip, the server periodically obtains weather information and real-time information on tourist destinations. This is done using the weather information service and tourist destination APIs. The input data is data from the real-time information API, and the output data is analyzed real-time information.

[0717] Step 11:

[0718] When changes in weather or traffic conditions are detected, the server uses a generative AI model to generate alternative plans and sends them to the device. The device then notifies the user of the alternative plans. The input data is the change information, and the output data is the alternative plans.

[0719] Step 12:

[0720] The user selects the desired language from the application settings. The device displays the UI in the selected language. The input data is the user's language selection information, and the output data is the UI in the selected language.

[0721] Step 13:

[0722] If a user needs assistance during their trip, they press the "Customer Support" button. The server accepts the inquiry and responds via a chatbot or customer support representative. The input data is the user's inquiry information, and the output data is the response information from the server.

[0723] (Application example 1)

[0724] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0725] Conventional travel planning support systems lacked the means to perform detailed simulations of travel destinations, making it difficult for users to inspect or experience the places they would actually visit in advance. Furthermore, the generation of reservation confirmations and final confirmation of travel plans were performed only in real space, making flexible responses difficult. Furthermore, there was no interface available to accept complex user input and intuitively and efficiently plan travel in a virtual space. To solve these problems, there is a need for a system that can intuitively create travel plans in a virtual space and display 3D models of travel destinations.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0727] In this invention, the server includes: means for receiving information on a user's desired travel destination, duration, activities, and meals; means for generating optimal visit locations and schedules based on the received information; means for linking with a map information service to automatically calculate travel times; means for making reservations for accommodations, transportation, and activities all at once; means for acquiring real-time weather information and tourist spot conditions and proposing travel schedule changes; means for supporting multiple languages; means for providing an interface through which a user inputs travel destination information using a head-mounted display and makes a travel plan in a virtual space; and means for displaying a 3D model of the travel destination in the virtual space and allowing the user to simulate the places they will visit. This allows users to intuitively plan their trip in the virtual space and allows them to inspect and experience the destination in advance. Furthermore, flexible reservation confirmation can be ensured through confirmation in the virtual space.

[0728] "Means for receiving information on the user's desired travel destination, duration, activities, and meals" is a function that collects data regarding the destination, duration of stay, activities the user wishes to participate in, and meals the user wishes to eat, which are input to plan the trip.

[0729] "Means for generating optimal places to visit and schedules based on received information" refers to a function that uses algorithms and AI to create travel schedules including optimal tourist spots and activities based on information entered by the user.

[0730] The "means for linking with a map information service to automatically calculate travel time" is a function that links with a map service and automatically calculates travel time between each visited location.

[0731] "A means of booking accommodation, transportation, and activities all at once" is a function that allows users to book accommodation, transportation, and activities all at once based on their confirmed travel schedule.

[0732] "Means for obtaining weather information and tourist destination conditions in real time and proposing changes to the travel schedule" is a function that provides users with suggested schedule changes as needed based on weather information and tourist destination conditions obtained in real time during the trip.

[0733] "Means for supporting multiple languages" refers to a function that provides interfaces and guidance in multiple languages ​​to users who speak different languages.

[0734] "A means of displaying a 3D model of a travel destination in a virtual space, allowing users to simulate visiting places" is a function that uses virtual reality technology to present a 3D model of a travel destination to users, allowing users to simulate visiting places in advance.

[0735] "A means for providing an interface that allows a user to input travel destination information using a head-mounted display and make travel plans in a virtual space" is a function that provides an interface that allows a user to input travel destination and activity information in a virtual space while wearing a head-mounted display and make travel plans.

[0736] "Means for generating and sending a reservation confirmation based on a confirmed schedule and prompting the user to confirm it within the virtual space" is a function that creates a reservation confirmation based on the travel schedule confirmed by the user and prompts the user to confirm it through an interface within the virtual space.

[0737] "A means of analyzing images uploaded by users and suggesting places to visit and eat based on the analysis results" is a function that analyzes image data provided by users using AI, etc., and suggests the most suitable tourist spots and restaurants based on the analysis results.

[0738] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[0739] Overall structure

[0740] This system consists of a device used by the user, such as a head-mounted display or smart glasses, a server, and an external API service. The user's device provides an interface for planning the trip, while the server is responsible for generating and optimizing the trip plan, as well as obtaining external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be carried out smoothly.

[0741] Travel planning

[0742] 1. The user wears the head-mounted display and starts the application. The user performs the login operation, the device requests the ID and password, and the server performs authentication.

[0743] 2. Once authentication is complete, the device displays the virtual space's main menu. The user can select the "Plan Your Trip" option and enter information about their travel destination, duration, desired activities, and dining options.

[0744] 3. Users can select the places they want to visit while looking at a 3D model of the destination in the virtual space. This information is sent from the device to the server.

[0745] 4. The server analyzes the input information and generates optimal places to visit and a schedule using a generative AI model, such as GPT-3.5.

[0746] 5. Using the map information service API, the server calculates travel times between locations and optimizes the overall schedule. The server then sends the generated schedule to the device.

[0747] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0748] Reservations and real-time information

[0749] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0750] 2. A reservation confirmation is generated based on this reservation information and sent to the device. The device displays the confirmation to the user and prompts them to confirm it in the virtual space if necessary.

[0751] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0752] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[0753] Specific examples

[0754] For example, suppose a user plans a three-day trip to a certain location in Japan in a virtual space. The user puts on a head-mounted display, launches the application, logs in, and selects the travel planning options. The user uploads a "certain location in Japan" as the travel destination, "three days" as the duration, "historical buildings" and "famous places" as places to visit, and "images of traditional cuisine" as food they want to eat. Once this information is sent to the server, the server uses a generative AI model to generate an optimal travel itinerary. Using a map information service API, the server calculates the optimal route and travel time between each destination. The server obtains reservation information for appropriate accommodations, transportation, and activities, and sends the optimal schedule to the device. Once the user confirms this schedule and presses the confirm button, the server makes all reservations and generates a reservation confirmation. During the trip, the server periodically obtains real-time information on weather and tourist attractions, and suggests an alternative itinerary if there are any changes. In this way, users can enjoy their trip efficiently and hassle-free.

[0755] Prompt Sentence Examples

[0756] An example of a prompt that the user can enter into the AI ​​generator is:

[0757] "Information to help you plan your trip:

[0758] Travel destination: Somewhere in Japan

[0759] Duration: 3 days

[0760] Places I want to visit: Historical buildings, famous places

[0761] What to eat: Images of traditional dishes

[0762] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0763] Step 1:

[0764] The user puts on the head-mounted display and launches the travel planning application. The user enters their ID and password, and the server performs authentication. The input is the user's ID and password, and the output is the authentication result. The server checks the authentication information against the database.

[0765] Step 2:

[0766] After successful authentication, the device displays the main menu of the virtual space. The user selects the "Plan a Trip" option and enters information about the travel destination, duration, desired activities, and dining locations. The input is the user's travel destination information, and the output is the transmission of the input information. The device receives the user's input through the virtual interface and sends it to the server.

[0767] Step 3:

[0768] The server analyzes the information it receives and uses a generative AI model to generate optimal places to visit and a schedule. The input is the user's travel information, and the output is an optimal travel schedule. The server inputs a prompt into the generative AI model to generate a schedule. GPT-3.5 is used as the generative AI model.

[0769] Step 4:

[0770] The server uses the map information service API to calculate the travel time between visited locations. The input is a list of visited locations, and the output is the travel time between each visited location. The data is calculated using API calls.

[0771] Step 5:

[0772] The server generates an optimal travel schedule and sends it to the terminal. The input is the optimal schedule and the output is a transmission to the terminal. The server sends the optimal schedule to the terminal as a data packet.

[0773] Step 6:

[0774] The terminal displays the proposed schedule to the user, who then confirms it. The received schedule is the input, and the user's confirmation result is the output. The terminal visually displays the schedule in a virtual space, and the user presses the confirmation and agreement buttons.

[0775] Step 7:

[0776] Once the user confirms the schedule, the server makes reservations for accommodation, transportation, and activities in one go. The input is the confirmed schedule, and the output is a reservation confirmation. The server uses an external reservation service API to make the necessary reservations and generate a confirmation.

[0777] Step 8:

[0778] The server sends the reservation confirmation to the terminal, which then displays it to the user. The input is the reservation confirmation, and the output is a display to the user. The terminal displays the confirmation in the virtual space and asks the user for confirmation.

[0779] Step 9:

[0780] During the trip, the server periodically obtains real-time weather information and tourist destination information, and generates suggested changes to the travel schedule if any abnormalities are detected. The real-time information is the input, and the suggested changes are the output. The server uses weather information services and tourist destination APIs to generate suggested changes using a generative AI model.

[0781] Step 10:

[0782] The device notifies the user of the alternative plan, and if the user accepts it, the new schedule is applied. The alternative plan is the input, and the new schedule is presented as the output. The device displays the alternative plan in the virtual space, and applies the new schedule after confirmation.

[0783] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0784] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[0785] Overall structure

[0786] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes the user's input and reactions to recognize emotions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[0787] Travel planning

[0788] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[0789] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[0790] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[0791] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[0792] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[0793] 6. The terminal displays the proposed schedule to the user, who confirms it.

[0794] Introducing and Adapting the Emotion Engine

[0795] 1. The emotion engine analyzes the reactions and comments entered by users when checking their schedules.

[0796] 2. The emotion engine recognizes the user's emotional state and identifies states such as joy, surprise, excitement, and anxiety.

[0797] 3. The server uses the analysis results of the emotion engine to readjust the suggested places to visit and schedule.

[0798] For example, if a user is feeling anxious, the app will suggest more relaxing tourist spots and activities.

[0799] If the user is excited, increase the adventurous activity.

[0800] Reservations and real-time information

[0801] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[0802] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[0803] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[0804] 4. If changes in weather or traffic conditions are detected, the server uses generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[0805] Multilingual support and customer support

[0806] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[0807] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[0808] Specific examples

[0809] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[0810] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[0811] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it uses the emotion engine to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[0812] This system greatly improves convenience for travelers by automating trip planning and adding flexibility through emotion recognition.

[0813] The processing flow will be explained below.

[0814] Step 1:

[0815] The user launches the application and the login screen appears.

[0816] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[0817] Step 2:

[0818] The user enters their ID and password and clicks the "Login" button.

[0819] The device sends the entered ID and password to the server.

[0820] Step 3:

[0821] The server authenticates the user information.

[0822] The database is accessed to search for user information that matches the entered ID and password.

[0823] Step 4:

[0824] If the server is successful in authenticating, it generates an authentication token.

[0825] A session is established for the authenticated user, and an authentication token is generated and sent to the terminal.

[0826] Step 5:

[0827] The device displays the main menu.

[0828] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[0829] Step 6:

[0830] The user selects "Plan a Trip" from the main menu.

[0831] Tap the "Travel Plans" option to proceed to the next form.

[0832] Step 7:

[0833] The device will display a screen for entering travel destination, duration, activities, and meals.

[0834] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[0835] Step 8:

[0836] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[0837] Enter the required information using the input form and image upload feature.

[0838] Step 9:

[0839] The terminal transmits the input information and image data to the server.

[0840] Send structured data and image data in JSON format to the server.

[0841] Step 10:

[0842] The server analyzes the input using generated AI.

[0843] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[0844] Step 11:

[0845] The server determines the strategy and calculates travel time using the Google Maps API.

[0846] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[0847] Step 12:

[0848] The server retrieves reservation options for accommodation, transportation, and activities.

[0849] A list of accommodation, transportation, and activity options is created through each booking site's API.

[0850] Step 13:

[0851] The server sends the optimized schedule to the terminal.

[0852] The final schedule data is sent to the terminal and prepared for display to the user.

[0853] Step 14:

[0854] The terminal displays the proposed schedule to the user.

[0855] The schedule contents are displayed on the screen so that the user can check them.

[0856] Step 15:

[0857] The user checks the schedule and presses the "Confirm" button.

[0858] Check the contents of the proposed schedule and tap the "Confirm" button.

[0859] Step 16:

[0860] The server uses an emotion engine to analyze users' reactions and comments.

[0861] The emotion engine analyzes the reactions and comments entered by the user to identify the user's emotional state.

[0862] Step 17:

[0863] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[0864] It generates a new schedule that takes into account the user's emotional state, and suggests a relaxing plan for a user who is feeling anxious, for example.

[0865] Step 18:

[0866] The server transmits the readjusted schedule to the terminal.

[0867] The adjusted schedule data is sent again to the terminal and the user is asked to confirm it.

[0868] Step 19:

[0869] The user reconfirms the schedule and presses the "Final Confirm" button.

[0870] Check the adjusted schedule and tap the "Final" button.

[0871] Step 20:

[0872] The server batches reservations for accommodation, transportation, and activities.

[0873] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[0874] Step 21:

[0875] The server sends a reservation confirmation to the terminal.

[0876] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[0877] Step 22:

[0878] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[0879] Display the booking confirmation in the UI and provide an "Email" option.

[0880] Step 23:

[0881] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[0882] Real-time data is obtained through weather information services and tourist destination APIs.

[0883] Step 24:

[0884] The server detects the fluctuations and generates alternatives using generative AI.

[0885] Generate new schedules based on changes in weather and traffic conditions, including considering the emotional state to incorporate using an emotion engine.

[0886] Step 25:

[0887] The server sends alternatives to the terminal.

[0888] The new schedule proposal is sent to the device and a notification is sent.

[0889] Step 26:

[0890] The terminal notifies the user of the alternatives.

[0891] Use popups and notifications to let users know about alternatives.

[0892] Step 27:

[0893] The user selects the desired language from the application settings.

[0894] Use the language settings option to select your preferred language.

[0895] Step 28:

[0896] The device will display the UI in the selected language.

[0897] The text is displayed according to the language you set.

[0898] Step 29:

[0899] The user presses the "Customer Support" button.

[0900] Tap on the "Customer Support" option.

[0901] Step 30:

[0902] The server provides customer support based on the user's inquiry.

[0903] Resolve user issues through chatbots and human support.

[0904] Example 2

[0905] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0906] Conventional travel planning systems generate schedules based on a user's desired travel destinations, duration, and activity information, but lack the flexibility to respond to user emotions or real-time information fluctuations. Furthermore, they do not support multiple languages, limiting their usability for global users. This makes it difficult for users to flexibly adjust plans to accommodate sudden changes during their trip or their individual emotional states.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0908] In this invention, the server includes a means for analyzing the user's emotional state and readjusting the proposed content based on the emotion, a means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, and a means for supporting multiple languages, thereby enabling the provision of flexible travel plans that respond to the user's emotions and real-time changes in the situation.

[0909] A "user" is an end user who uses the travel planning system to select a travel destination and adjust a schedule.

[0910] "Destination" refers to a geographic location that a user wishes to visit.

[0911] "Duration" refers to the specific length of time that a user has set for planning a trip.

[0912] "Activities" refer to events or activities that a user wants to experience during their trip.

[0913] "Meal information" refers to information about the food and drink that the user desires to eat and drink during the trip.

[0914] "Map information service" refers to a third-party service that provides geographic information and calculates travel times between locations.

[0915] "Accommodation" refers to a facility where a user can stay while traveling.

[0916] "Transportation" refers to the vehicles and services used for transportation during a trip.

[0917] "Emotional state" refers to the psychological state a user feels about a travel plan or proposal.

[0918] "Real-time information" refers to the latest information at that time, such as weather information and the status of tourist attractions.

[0919] An "emotion engine" refers to software that analyzes user input and reactions to recognize emotions.

[0920] "Generative AI" refers to an artificial intelligence model that creates optimal travel schedules based on user input.

[0921] "Language support" refers to the ability to provide interfaces and notifications in the language selected by the user.

[0922] "Proposed Changes" means any proposed modifications or alternatives to your travel plans or schedule.

[0923] MODE FOR CARRYING OUT THE INVENTION

[0924] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[0925] Overall structure

[0926] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine recognizes emotions by analyzing the user's input and reactions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[0927] Travel planning

[0928] First, the user launches the application and logs in. The device prompts for an ID and password, and the server authenticates them. Once authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images. The device then sends this information to the server. The server analyzes the entered information and uses a generative AI (e.g., GPT-4) to generate an optimal itinerary and itinerary. An example prompt might be, "The user is planning a three-day trip to Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and eat sushi. Please suggest an optimal itinerary." The server uses the Google Maps API to calculate travel times between locations and optimize the overall itinerary. The server then generates the optimized itinerary and sends it to the device.

[0929] The device displays the proposed schedule to the user, who then confirms it. The emotion engine analyzes the user's reactions and comments when checking the schedule. The emotion engine recognizes the user's emotional state and sends the analysis results to the server. The server readjusts the proposed content based on the emotion engine's analysis results and sends the prompt text back to the generation AI to update the schedule.

[0930] Reservations and real-time information

[0931] Once the user confirms and confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities. The server generates a reservation confirmation based on the reservation information and sends it to the terminal, which displays it to the user. If the user wishes, the confirmation can be sent by email or other means.

[0932] Furthermore, during the trip, the server periodically obtains real-time weather information and tourist destination information. It obtains the latest information using weather information services and tourist destination APIs. If changes in weather or traffic conditions are detected, the server uses a generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[0933] Multilingual support and customer support

[0934] The user selects their preferred language in the application settings. The device displays the UI in the selected language. If the user needs assistance during their trip, they can press the "Customer Support" button, which the server will then handle via a chatbot or customer support representative.

[0935] Specific examples

[0936] For example, if a user plans a three-day trip to Tokyo, they launch the application, log in, and select the "Plan a Trip" option. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Sensoji Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. Using the Google Maps API, the server calculates the optimal route and travel time from Sensoji Temple to Tokyo Skytree. The server obtains reservation information for appropriate accommodation, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all the reservations in one go and generates a reservation confirmation.

[0937] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, the emotion engine is used to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[0938] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0939] Step 1:

[0940] The user launches the application and the login screen appears.

[0941] Specific operation: The user enters a user ID and password into the terminal, and the terminal sends the entered information to the server.

[0942] Input: User ID, Password

[0943] Output: Authentication request

[0944] Step 2:

[0945] The server compares the received ID and password with the database and performs authentication.

[0946] Specific operation: The server checks the ID and password against the database, and if they match, it sends an authentication success message to the terminal.

[0947] Input: User ID, Password

[0948] Output: Authentication success message or authentication failure message

[0949] Step 3:

[0950] The device receives a successful authentication message and displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options.

[0951] Specific operation: The user inputs information about the travel destination, duration, activities, and places to eat into the device, which then sends it to the server.

[0952] Input: Travel destination, duration, activities, and dining information

[0953] Output: Travel planning request

[0954] Step 4:

[0955] The server analyzes the user's input information and uses generative AI to generate the optimal places to visit and schedule.

[0956] Specific operation: The server analyzes the user's input information and sends prompt text to the generation AI to generate a schedule.

[0957] Example prompt: The user is planning a three-day trip in Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and he wants to eat sushi. Please suggest the best itinerary.

[0958] Input: Travel planning request (destination, duration, activities, dining information)

[0959] Output: Initial schedule

[0960] Step 5:

[0961] The server uses the Google Maps API to calculate travel times between locations and optimize the overall schedule.

[0962] Specific operation: The server calculates the travel time between each location using the Google Maps API, optimizes the overall schedule based on the calculation results, and sends the optimized schedule to the device.

[0963] Input: Location information of the place to be visited

[0964] Output: Optimized schedule

[0965] Step 6:

[0966] The terminal displays the proposed schedule to the user, who then checks the schedule and enters comments or requests for adjustments as necessary.

[0967] Specific operation: The terminal displays the schedule screen and the user confirms it. If the user inputs an adjustment request, the terminal sends it to the server.

[0968] Input: Initial Schedule

[0969] Output: User response and adjustment request

[0970] Step 7:

[0971] The server analyzes the user's reactions and comments using an emotion engine to identify their emotional state.

[0972] Specific operation: The server sends the user's comments and reactions to the emotion engine. The emotion engine analyzes the user's emotional state and sends the results to the server.

[0973] Input: User responses and comments

[0974] Output: User's emotional state

[0975] Step 8:

[0976] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[0977] Specific operation: The server uses the results of the emotion engine to send a prompt to the generation AI again and update the proposal content.

[0978] Input: User's emotional state, prompt sentence

[0979] Output: Re-adjusted schedule

[0980] Step 9:

[0981] Once the user confirms and confirms the rescheduled schedule, the server collectively makes reservations for accommodation, transportation, and activities.

[0982] Specific operation: The user checks the rescheduled schedule on the terminal and presses the confirm button. The server makes various reservations through the reservation system.

[0983] Input: Re-arranged schedule

[0984] Output: Reservation information and confirmation

[0985] Step 10:

[0986] The server periodically acquires real-time information on weather and tourist spots during the travel period.

[0987] Specific operation: The server periodically calls weather information services and tourist destination APIs to obtain the latest information.

[0988] Input: None (regular operation)

[0989] Output: Real-time information

[0990] Step 11:

[0991] The server generates alternatives by sending prompts to the generation AI based on the fluctuation information, and then uses an emotion engine to generate alternatives that are optimized to the user's emotions.

[0992] Specific operation: The server obtains the latest weather information and tourist destination conditions, and sends prompts to the generation AI as needed. The emotion engine optimizes alternatives and notifies the device.

[0993] Input: Real-time information, prompt text

[0994] Output: Optimized alternatives

[0995] Step 12:

[0996] The user selects the desired language from the application settings.

[0997] Specific behavior: The device displays the language settings screen, the user selects the desired language, and the device updates the UI in the selected language.

[0998] Input: Desired language

[0999] Output: UI in selected language

[1000] Step 13:

[1001] When a user requests customer support, the server initiates the response.

[1002] Specific operation: The user presses the support button, the server accepts the inquiry, and responds via a chatbot or customer support representative.

[1003] Input: Support Request

[1004] Output: Supported

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1007] Conventional travel planning support systems propose itineraries based on user preferences, but because they are unable to consider the user's emotional state, the proposed itinerary often does not match the user's expectations. Furthermore, there are only a limited number of systems that allow flexible schedule changes in real time during a trip, leaving room for improvement in the user experience. Another problem is that the proposed alternatives are uniform and not optimized for each individual user.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1009] In this invention, the server includes means for receiving information on the user's desired travel destinations, period, activities, and meals, means for generating optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, means for recognizing the user's emotional state using an emotion recognition engine and optimizing the travel plan and alternatives, and means for supporting multiple languages, thereby enabling the server to propose flexible and optimal travel plans that take the user's emotional state into consideration.

[1010] "User's desired travel destination, duration, activities, and meal information"

[1011] This refers to details of the places and duration a user wants to visit during their trip, the activities they want to participate in, and the meals they want to eat.

[1012] "Optimal visiting locations and schedules"

[1013] This refers to the allocation of destinations and time planned to make the trip efficient and enjoyable based on the user's desired information.

[1014] "Map information provision service"

[1015] This refers to external data services that provide location and route information, such as services that use map APIs.

[1016] "A one-stop solution for booking accommodation, transport and activities"

[1017] This refers to the ability for users to book accommodation, transportation, and local activities for their planned trip all at once.

[1018] "Weather information and tourist destination conditions"

[1019] This refers to real-time weather forecasts, congestion status at tourist spots, business status, etc.

[1020] Emotion recognition engine

[1021] This refers to a technology that analyzes and recognizes a user's emotional state (happiness, surprise, excitement, anxiety, etc.) from their facial expressions and behavior.

[1022] "A way to support multiple languages"

[1023] It refers to the ability of an application or system to provide a user interface in different languages.

[1024] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and is combined with an emotion engine that recognizes the user's emotions and optimizes the plan.

[1025] System configuration

[1026] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, and can be a smartphone, tablet, or PC. The server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes user input and reactions and recognizes emotions. The external API service includes a map information service and a reservation service API.

[1027] The process of emotion recognition

[1028] When a user plans a trip, facial expression recognition technology is used to analyze the user's emotional state in real time. This technology uses OpenCV to detect faces and TensorFlow / Keras to classify emotions. For example, emotional states such as joy, anxiety, and surprise can be detected in real time.

[1029] Travel plan example

[1030] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application on their smartphone, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. In this process, the Google Maps API is used to calculate travel times between each place to be visited and optimize the overall schedule.

[1031] Optimizing recommendations with an emotion engine

[1032] When the user reviews the proposed schedule, the emotion engine analyzes their reaction. If the emotion engine detects that the user is feeling anxious, it will suggest alternatives, including relaxing sightseeing spots and activities. Conversely, if the user is feeling excited, it will increase the suggestions of adventurous activities. For example, if the weather worsens, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed.

[1033] Examples of specific examples and prompts

[1034] In one embodiment of the invention, when a user is watching a horror movie and sentiment analysis is performed, a prompt is generated as follows:

[1035] Example prompt: If a user is feeling "scared," recommend another video with more action and thrills.

[1036] In this way, real-time emotion recognition can be performed to optimize the user experience and provide more personalized travel plans.

[1037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1038] Step 1:

[1039] The server receives information about the user's desired travel destination, duration, activities, and meals. The user uses a smartphone or tablet device to input details such as the travel destination, duration, and desired activities. This input information is sent from the device to the server. As an output, the server stores the user's input information.

[1040] Step 2:

[1041] The server generates optimal places to visit and a schedule based on the information it receives. Specifically, it uses a generative AI model to formulate places to visit and a schedule that meets the user's preferences. During this process, it analyzes the user's desired information as data processing and calculates optimal places to visit and routes. As an output, the server generates an initial travel schedule.

[1042] Step 3:

[1043] The server works with a map information service to automatically calculate travel times. Specifically, it uses a map API to obtain travel times between each visit location. Based on this information, it optimizes the overall schedule and calculates efficient routes between visit locations. The output is an optimized schedule that reflects travel times.

[1044] Step 4:

[1045] The server sends the generated optimized schedule to the terminal. The proposed schedule is displayed on the user's terminal, and the user confirms its contents. The input is the schedule data from the server, and the output is a confirmation screen displayed on the user's terminal.

[1046] Step 5:

[1047] The device captures the user's facial expression and sends the video data to the server. Specifically, the device's camera function is used to capture the user's facial expression in real time and sends the data to the server. The output is the facial expression data sent to the server.

[1048] Step 6:

[1049] The server uses an emotion recognition engine to analyze the user's facial expression data and recognize emotions. Specifically, it uses a face detection algorithm (OpenCV) to identify the face area, and then uses a TensorFlow / Keras model to classify the emotion. The result of this operation is the recognized emotion data. The output is the recognized emotion (e.g., joy, anxiety, excitement, uneasiness, etc.).

[1050] Step 7:

[1051] The server optimizes travel plans and alternatives based on the recognized emotion data. For example, if the user is feeling "anxious," it will suggest more relaxing sightseeing spots and activities. It uses a generative AI model to generate these optimized plans and alternatives. The output is an optimized travel schedule tailored to the user's emotions.

[1052] Step 8:

[1053] The server continuously obtains weather information and real-time conditions of tourist destinations, and generates proposed schedule changes as needed. Specifically, it uses weather information services and tourist destination APIs to monitor local conditions. Based on this information, a generative AI model creates a new schedule and uses an emotion engine to suggest alternatives based on the user's emotions. The output is the latest alternative schedule.

[1054] Step 9:

[1055] The optimized schedule and alternative plans are notified to the terminal. The terminal displays the proposals to the user in real time and the user confirms the contents. The user confirms and decides on the new plan. The input is the optimized schedule from the server, and the output is a notification screen to the user.

[1056] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1058] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1059] [Third embodiment]

[1060] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1061] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1062] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1063] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1064] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1065] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1066] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1067] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1070] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1071] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1072] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[1073] Overall structure

[1074] This system consists of a user's device, a server, and many external API services. The user's device is the device that runs the application, and the server is responsible for generating and optimizing travel plans and obtaining external information. In addition, by utilizing map information services and reservation service APIs, each step of the travel plan can be carried out smoothly.

[1075] Travel planning

[1076] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[1077] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[1078] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[1079] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[1080] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[1081] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1082] Reservations and real-time information

[1083] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1084] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[1085] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1086] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[1087] Multilingual support and customer support

[1088] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[1089] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[1090] Specific examples

[1091] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[1092] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[1093] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it will suggest an alternative schedule. In this way, the user can enjoy their trip efficiently and hassle-free.

[1094] This system greatly improves convenience for travelers by automating and providing flexibility in travel planning.

[1095] The processing flow will be explained below.

[1096] Step 1:

[1097] The user launches the application and the login screen appears.

[1098] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[1099] Step 2:

[1100] The user enters their ID and password and clicks the "Login" button.

[1101] The device sends the entered ID and password to the server.

[1102] Step 3:

[1103] The server authenticates the user information.

[1104] The database is accessed to search for user information that matches the entered ID and password.

[1105] Step 4:

[1106] If the server is successful in authenticating, it generates an authentication token.

[1107] A session is established for the authenticated user, and a token is generated and sent to the terminal.

[1108] Step 5:

[1109] The device displays the main menu.

[1110] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[1111] Step 6:

[1112] The user selects "Plan a Trip" from the main menu.

[1113] Tap the "Travel Plans" option to proceed to the next form.

[1114] Step 7:

[1115] The device will display a screen for entering travel destination, duration, activities, and meals.

[1116] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[1117] Step 8:

[1118] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[1119] Enter the required information using the input form and image upload feature.

[1120] Step 9:

[1121] The terminal transmits the input information and image data to the server.

[1122] Send structured data and image data in JSON format to the server.

[1123] Step 10:

[1124] The server analyzes the input using generated AI.

[1125] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[1126] Step 11:

[1127] The server determines the strategy and calculates travel time using the Google Maps API.

[1128] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[1129] Step 12:

[1130] The server retrieves reservation options for accommodation, transportation, and activities.

[1131] A list of accommodation, transportation, and activity options is created through each booking site's API.

[1132] Step 13:

[1133] The server sends the optimized schedule to the terminal.

[1134] The final schedule data is sent to the terminal and prepared for display to the user.

[1135] Step 14:

[1136] The terminal displays the proposed schedule to the user.

[1137] The schedule contents are displayed on the screen so that the user can check them.

[1138] Step 15:

[1139] The user checks the schedule and presses the "Confirm" button.

[1140] Check the contents of the proposed schedule and tap the "Confirm" button.

[1141] Step 16:

[1142] The server batches reservations for accommodation, transportation, and activities.

[1143] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[1144] Step 17:

[1145] The server sends a reservation confirmation to the terminal.

[1146] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[1147] Step 18:

[1148] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[1149] Display the booking confirmation in the UI and provide an "Email" option.

[1150] Step 19:

[1151] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[1152] Real-time data is obtained through weather information services and tourist destination APIs.

[1153] Step 20:

[1154] The server detects the fluctuations and generates alternatives using generative AI.

[1155] Generate new schedules that adapt to changes in weather and traffic conditions.

[1156] Step 21:

[1157] The server sends alternatives to the terminal.

[1158] The new schedule proposal is sent to the device and a notification is sent.

[1159] Step 22:

[1160] The terminal notifies the user of the alternatives.

[1161] Use popups and notifications to let users know about alternatives.

[1162] Step 23:

[1163] The user selects the desired language from the application settings.

[1164] Use the language settings option to select your preferred language.

[1165] Step 24:

[1166] The device will display the UI in the selected language.

[1167] The text is displayed according to the language you set.

[1168] Step 25:

[1169] The user presses the "Customer Support" button.

[1170] Tap on the "Customer Support" option.

[1171] Step 26:

[1172] The server provides customer support based on the user's inquiry.

[1173] Resolve user issues through chatbots and human support.

[1174] Example 1

[1175] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] Conventional travel planning support systems often require a lot of effort to input and analyze information about users' desired travel destinations, duration, activities, and meals. Furthermore, it is difficult to comprehensively provide multiple functions, such as optimizing travel schedules, calculating travel times, managing reservations, acquiring and responding to real-time information, and supporting multiple languages. As a result, users' convenience is reduced, and travel planning does not proceed smoothly.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1178] In this invention, the server includes means for receiving information about a user's desired destination, period, activities, and meals, means for using a generative AI model to generate optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring meteorological information and tourist spot conditions in real time and providing suggested changes to the travel schedule, means for supporting multiple languages, and means for analyzing multimedia data uploaded by the user and suggesting places to visit and eat based on the analysis results, thereby enabling users to efficiently plan their trips and flexibly respond to changes in the situation in real time.

[1179] "User" means an individual or organization that uses the system to make travel plans.

[1180] A "destination" is a place that a user wishes to visit during a trip.

[1181] "Duration" is the specific number of days or hours that the user plans to spend traveling.

[1182] An "activity" is an event or experience that a user wants to participate in during their trip.

[1183] "Meals" refers to the foods and places to eat that the user wants to eat during their trip.

[1184] A "generative AI model" is an artificial intelligence algorithm that automatically generates optimal places to visit and schedules based on user input.

[1185] "Map information service" refers to a geographic information system and its API used to calculate travel time and distance.

[1186] "Accommodation" refers to a hotel or lodging facility that a User reserves for stay during their trip.

[1187] "Transportation" refers to the vehicles such as airplanes, trains, buses, and taxis that users use to travel.

[1188] "Weather information" refers to information about the weather forecast and weather conditions during the travel period.

[1189] "Tourist destination situation" refers to the congestion situation at tourist destinations, business information, event information, etc.

[1190] "Multiple languages" refers to the ability to provide user interfaces and system functionality in different languages.

[1191] "Multimedia data" refers to information in digital form, such as images, audio, and video, uploaded by users.

[1192] "Analysis results" are information and recommendations obtained as a result of analyzing multimedia data.

[1193] A "reservation confirmation" is a document that is generated after an itinerary is confirmed and details the reservation details for accommodation, transportation, and activities.

[1194] This invention relates to an interactive travel concierge system that assists users in planning their trips. This system is composed of a user's device, a server, and multiple external API services. The user's device runs an application, while the server is responsible for generating and optimizing the trip plan and acquiring external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be smoothly carried out.

[1195] Hardware and software configuration

[1196] The user's terminal is a device capable of running applications, such as a smartphone, tablet, or personal computer. This device provides an interface for inputting travel plans, transmits the user's input information to the server, and receives and displays responses from the server to the user.

[1197] The server has high-performance computing resources, analyzes the information entered by the user, and generates optimal places to visit and a schedule using a generative AI model that incorporates natural language processing and image analysis technologies, for example.

[1198] The external API services used include map information services, such as a map information API and a reservation service API, which allow users to calculate travel times between locations and make reservations for accommodations and transportation.

[1199] Program operation explanation

[1200] 1. The user launches the travel concierge application from their device and logs in. On the login screen, the user enters their username and password, which are then sent to the server. The server then authenticates and verifies the user's identity.

[1201] 2. After authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images.

[1202] 3. The device sends this input information to the server, which analyzes it and uses a generative AI model to generate optimal places to visit and a schedule.

[1203] For example, the prompt text might look like this:

[1204] "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Sushi images"

[1205] 4. The server uses a map information API to calculate travel time between locations and optimize the overall schedule.

[1206] 5. The server sends the generated schedule to the terminal, which displays the proposed schedule to the user for confirmation.

[1207] 6. Once the user confirms and confirms the proposed schedule, the server will make all reservations for accommodation, transportation, and activities at once, and generate a reservation confirmation based on this reservation information and send it to the terminal.

[1208] 7. During the trip, the server periodically obtains real-time weather information and tourist destination information. It uses the weather information service and tourist destination APIs to obtain real-time information and generates alternatives for changing the schedule as needed. This information is sent to the terminal and notified to the user.

[1209] 8. The device provides a multilingual UI to display applications in the language the user needs. If a user needs assistance during their trip, they can contact the server by pressing the "Customer Support" button, and the server will respond to the inquiry with a chatbot or customer support representative.

[1210] This system allows users to plan trips efficiently and effortlessly, and responds flexibly to real-time changes in the situation. By linking with various APIs, it eliminates travel time between destinations and the hassle of making reservations, enriching the user's travel experience.

[1211] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1212] Step 1:

[1213] A user starts an application and logs in. The terminal prompts for a username and password, and the server receives this authentication information. The server accesses a database and collates the user information to perform authentication. The input data is the username and password, and the output data is the authentication result.

[1214] Step 2:

[1215] After successful login, the terminal displays the main menu. The user selects the "Travel Planning" option. The terminal displays menu selection information and accepts the user's selection as input. The displayed menu contains various items related to travel planning.

[1216] Step 3:

[1217] The user selects a travel planning option and inputs information about the travel destination, duration, desired activities, and dining locations. The terminal receives the input information and sends it to the server. The input data is information about the travel destination, duration, activities, and dining locations, and the output data is the data sent to the server.

[1218] Step 4:

[1219] The server analyzes the received information. The server analyzes the data using a generative AI model and generates the optimal places to visit and schedule based on the user's preferences. An example of a prompt sentence would be "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Images of sushi." The output data are the analysis results and the generated schedule.

[1220] Step 5:

[1221] The server uses a map information API to calculate travel times between visit locations, optimizing the overall schedule. The input data is the coordinate information of the visit locations and the start and destination points of the trip, and the output data is the travel time between each visit location.

[1222] Step 6:

[1223] The server sends the optimized schedule to the terminal, which receives it and displays it to the user. The input data is the optimized schedule received from the server, and the output data is the schedule to display to the user.

[1224] Step 7:

[1225] The user confirms the proposed schedule and presses the confirm button. This input operation causes the terminal to send the schedule confirmation information to the server. The input data is the schedule confirmation operation by the user, and the output data is the schedule confirmation information sent to the server.

[1226] Step 8:

[1227] The server makes reservations for accommodation, transportation, and activities in bulk based on the confirmed schedule. It generates a reservation confirmation and sends it to the terminal. The input data is the confirmed schedule information, and the output data is the reservation confirmation.

[1228] Step 9:

[1229] The terminal displays the reservation confirmation to the user. If desired, the user can receive the confirmation by email, etc. The input data is the reservation confirmation received from the server, and the output data is the information displayed and sent to the user.

[1230] Step 10:

[1231] During the trip, the server periodically obtains weather information and real-time information on tourist destinations. This is done using the weather information service and tourist destination APIs. The input data is data from the real-time information API, and the output data is analyzed real-time information.

[1232] Step 11:

[1233] When changes in weather or traffic conditions are detected, the server uses a generative AI model to generate alternative plans and sends them to the device. The device then notifies the user of the alternative plans. The input data is the change information, and the output data is the alternative plans.

[1234] Step 12:

[1235] The user selects the desired language from the application settings. The device displays the UI in the selected language. The input data is the user's language selection information, and the output data is the UI in the selected language.

[1236] Step 13:

[1237] If a user needs assistance during their trip, they press the "Customer Support" button. The server accepts the inquiry and responds via a chatbot or customer support representative. The input data is the user's inquiry information, and the output data is the response information from the server.

[1238] (Application example 1)

[1239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1240] Conventional travel planning support systems lacked the means to perform detailed simulations of travel destinations, making it difficult for users to inspect or experience the places they would actually visit in advance. Furthermore, the generation of reservation confirmations and final confirmation of travel plans were performed only in real space, making flexible responses difficult. Furthermore, there was no interface available to accept complex user input and intuitively and efficiently plan travel in a virtual space. To solve these problems, there is a need for a system that can intuitively create travel plans in a virtual space and display 3D models of travel destinations.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1242] In this invention, the server includes: means for receiving information on a user's desired travel destination, duration, activities, and meals; means for generating optimal visit locations and schedules based on the received information; means for linking with a map information service to automatically calculate travel times; means for making reservations for accommodations, transportation, and activities all at once; means for acquiring real-time weather information and tourist spot conditions and proposing travel schedule changes; means for supporting multiple languages; means for providing an interface through which a user inputs travel destination information using a head-mounted display and makes a travel plan in a virtual space; and means for displaying a 3D model of the travel destination in the virtual space and allowing the user to simulate the places they will visit. This allows users to intuitively plan their trip in the virtual space and allows them to inspect and experience the destination in advance. Furthermore, flexible reservation confirmation can be ensured through confirmation in the virtual space.

[1243] "Means for receiving information on the user's desired travel destination, duration, activities, and meals" is a function that collects data regarding the destination, duration of stay, activities the user wishes to participate in, and meals the user wishes to eat, which are input to plan the trip.

[1244] "Means for generating optimal places to visit and schedules based on received information" refers to a function that uses algorithms and AI to create travel schedules including optimal tourist spots and activities based on information entered by the user.

[1245] The "means for linking with a map information service to automatically calculate travel time" is a function that links with a map service and automatically calculates travel time between each visited location.

[1246] "A means of booking accommodation, transportation, and activities all at once" is a function that allows users to book accommodation, transportation, and activities all at once based on their confirmed travel schedule.

[1247] "Means for obtaining weather information and tourist destination conditions in real time and proposing changes to the travel schedule" is a function that provides users with suggested schedule changes as needed based on weather information and tourist destination conditions obtained in real time during the trip.

[1248] "Means for supporting multiple languages" refers to a function that provides interfaces and guidance in multiple languages ​​to users who speak different languages.

[1249] "A means of displaying a 3D model of a travel destination in a virtual space, allowing users to simulate visiting places" is a function that uses virtual reality technology to present a 3D model of a travel destination to users, allowing users to simulate visiting places in advance.

[1250] "A means for providing an interface that allows a user to input travel destination information using a head-mounted display and make travel plans in a virtual space" is a function that provides an interface that allows a user to input travel destination and activity information in a virtual space while wearing a head-mounted display and make travel plans.

[1251] "Means for generating and sending a reservation confirmation based on a confirmed schedule and prompting the user to confirm it within the virtual space" is a function that creates a reservation confirmation based on the travel schedule confirmed by the user and prompts the user to confirm it through an interface within the virtual space.

[1252] "A means of analyzing images uploaded by users and suggesting places to visit and eat based on the analysis results" is a function that analyzes image data provided by users using AI, etc., and suggests the most suitable tourist spots and restaurants based on the analysis results.

[1253] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[1254] Overall structure

[1255] This system consists of a device used by the user, such as a head-mounted display or smart glasses, a server, and an external API service. The user's device provides an interface for planning the trip, while the server is responsible for generating and optimizing the trip plan, as well as obtaining external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be carried out smoothly.

[1256] Travel planning

[1257] 1. The user wears the head-mounted display and starts the application. The user performs the login operation, the device requests the ID and password, and the server performs authentication.

[1258] 2. Once authentication is complete, the device displays the virtual space's main menu. The user can select the "Plan Your Trip" option and enter information about their travel destination, duration, desired activities, and dining options.

[1259] 3. Users can select the places they want to visit while looking at a 3D model of the destination in the virtual space. This information is sent from the device to the server.

[1260] 4. The server analyzes the input information and generates optimal places to visit and a schedule using a generative AI model, such as GPT-3.5.

[1261] 5. Using the map information service API, the server calculates travel times between locations and optimizes the overall schedule. The server then sends the generated schedule to the device.

[1262] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1263] Reservations and real-time information

[1264] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1265] 2. A reservation confirmation is generated based on this reservation information and sent to the device. The device displays the confirmation to the user and prompts them to confirm it in the virtual space if necessary.

[1266] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1267] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[1268] Specific examples

[1269] For example, suppose a user plans a three-day trip to a certain location in Japan in a virtual space. The user puts on a head-mounted display, launches the application, logs in, and selects the travel planning options. The user uploads a "certain location in Japan" as the travel destination, "three days" as the duration, "historical buildings" and "famous places" as places to visit, and "images of traditional cuisine" as food they want to eat. Once this information is sent to the server, the server uses a generative AI model to generate an optimal travel itinerary. Using a map information service API, the server calculates the optimal route and travel time between each destination. The server obtains reservation information for appropriate accommodations, transportation, and activities, and sends the optimal schedule to the device. Once the user confirms this schedule and presses the confirm button, the server makes all reservations and generates a reservation confirmation. During the trip, the server periodically obtains real-time information on weather and tourist attractions, and suggests an alternative itinerary if there are any changes. In this way, users can enjoy their trip efficiently and hassle-free.

[1270] Prompt Sentence Examples

[1271] An example of a prompt that the user can enter into the AI ​​generator is:

[1272] "Information to help you plan your trip:

[1273] Travel destination: Somewhere in Japan

[1274] Duration: 3 days

[1275] Places I want to visit: Historical buildings, famous places

[1276] What to eat: Images of traditional dishes

[1277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1278] Step 1:

[1279] The user puts on the head-mounted display and launches the travel planning application. The user enters their ID and password, and the server performs authentication. The input is the user's ID and password, and the output is the authentication result. The server checks the authentication information against the database.

[1280] Step 2:

[1281] After successful authentication, the device displays the main menu of the virtual space. The user selects the "Plan a Trip" option and enters information about the travel destination, duration, desired activities, and dining locations. The input is the user's travel destination information, and the output is the transmission of the input information. The device receives the user's input through the virtual interface and sends it to the server.

[1282] Step 3:

[1283] The server analyzes the information it receives and uses a generative AI model to generate optimal places to visit and a schedule. The input is the user's travel information, and the output is an optimal travel schedule. The server inputs a prompt into the generative AI model to generate a schedule. GPT-3.5 is used as the generative AI model.

[1284] Step 4:

[1285] The server uses the map information service API to calculate the travel time between visited locations. The input is a list of visited locations, and the output is the travel time between each visited location. The data is calculated using API calls.

[1286] Step 5:

[1287] The server generates an optimal travel schedule and sends it to the terminal. The input is the optimal schedule and the output is a transmission to the terminal. The server sends the optimal schedule to the terminal as a data packet.

[1288] Step 6:

[1289] The terminal displays the proposed schedule to the user, who then confirms it. The received schedule is the input, and the user's confirmation result is the output. The terminal visually displays the schedule in a virtual space, and the user presses the confirmation and agreement buttons.

[1290] Step 7:

[1291] Once the user confirms the schedule, the server makes reservations for accommodation, transportation, and activities in one go. The input is the confirmed schedule, and the output is a reservation confirmation. The server uses an external reservation service API to make the necessary reservations and generate a confirmation.

[1292] Step 8:

[1293] The server sends the reservation confirmation to the terminal, which then displays it to the user. The input is the reservation confirmation, and the output is a display to the user. The terminal displays the confirmation in the virtual space and asks the user for confirmation.

[1294] Step 9:

[1295] During the trip, the server periodically obtains real-time weather information and tourist destination information, and generates suggested changes to the travel schedule if any abnormalities are detected. The real-time information is the input, and the suggested changes are the output. The server uses weather information services and tourist destination APIs to generate suggested changes using a generative AI model.

[1296] Step 10:

[1297] The device notifies the user of the alternative plan, and if the user accepts it, the new schedule is applied. The alternative plan is the input, and the new schedule is presented as the output. The device displays the alternative plan in the virtual space, and applies the new schedule after confirmation.

[1298] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1299] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[1300] Overall structure

[1301] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes the user's input and reactions to recognize emotions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[1302] Travel planning

[1303] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[1304] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[1305] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[1306] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[1307] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[1308] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1309] Introducing and Adapting the Emotion Engine

[1310] 1. The emotion engine analyzes the reactions and comments entered by users when checking their schedules.

[1311] 2. The emotion engine recognizes the user's emotional state and identifies states such as joy, surprise, excitement, and anxiety.

[1312] 3. The server uses the analysis results of the emotion engine to readjust the suggested places to visit and schedule.

[1313] For example, if a user is feeling anxious, the app will suggest more relaxing tourist spots and activities.

[1314] If the user is excited, increase the adventurous activity.

[1315] Reservations and real-time information

[1316] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1317] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[1318] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1319] 4. If changes in weather or traffic conditions are detected, the server uses generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[1320] Multilingual support and customer support

[1321] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[1322] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[1323] Specific examples

[1324] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[1325] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[1326] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it uses the emotion engine to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[1327] This system greatly improves convenience for travelers by automating trip planning and adding flexibility through emotion recognition.

[1328] The processing flow will be explained below.

[1329] Step 1:

[1330] The user launches the application and the login screen appears.

[1331] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[1332] Step 2:

[1333] The user enters their ID and password and clicks the "Login" button.

[1334] The device sends the entered ID and password to the server.

[1335] Step 3:

[1336] The server authenticates the user information.

[1337] The database is accessed to search for user information that matches the entered ID and password.

[1338] Step 4:

[1339] If the server is successful in authenticating, it generates an authentication token.

[1340] A session is established for the authenticated user, and an authentication token is generated and sent to the terminal.

[1341] Step 5:

[1342] The device displays the main menu.

[1343] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[1344] Step 6:

[1345] The user selects "Plan a Trip" from the main menu.

[1346] Tap the "Travel Plans" option to proceed to the next form.

[1347] Step 7:

[1348] The device will display a screen for entering travel destination, duration, activities, and meals.

[1349] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[1350] Step 8:

[1351] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[1352] Enter the required information using the input form and image upload feature.

[1353] Step 9:

[1354] The terminal transmits the input information and image data to the server.

[1355] Send structured data and image data in JSON format to the server.

[1356] Step 10:

[1357] The server analyzes the input using generated AI.

[1358] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[1359] Step 11:

[1360] The server determines the strategy and calculates travel time using the Google Maps API.

[1361] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[1362] Step 12:

[1363] The server retrieves reservation options for accommodation, transportation, and activities.

[1364] A list of accommodation, transportation, and activity options is created through each booking site's API.

[1365] Step 13:

[1366] The server sends the optimized schedule to the terminal.

[1367] The final schedule data is sent to the terminal and prepared for display to the user.

[1368] Step 14:

[1369] The terminal displays the proposed schedule to the user.

[1370] The schedule contents are displayed on the screen so that the user can check them.

[1371] Step 15:

[1372] The user checks the schedule and presses the "Confirm" button.

[1373] Check the contents of the proposed schedule and tap the "Confirm" button.

[1374] Step 16:

[1375] The server uses an emotion engine to analyze users' reactions and comments.

[1376] The emotion engine analyzes the reactions and comments entered by the user to identify the user's emotional state.

[1377] Step 17:

[1378] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[1379] It generates a new schedule that takes into account the user's emotional state, and suggests a relaxing plan for a user who is feeling anxious, for example.

[1380] Step 18:

[1381] The server transmits the readjusted schedule to the terminal.

[1382] The adjusted schedule data is sent again to the terminal and the user is asked to confirm it.

[1383] Step 19:

[1384] The user reconfirms the schedule and presses the "Final Confirm" button.

[1385] Check the adjusted schedule and tap the "Final" button.

[1386] Step 20:

[1387] The server batches reservations for accommodation, transportation, and activities.

[1388] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[1389] Step 21:

[1390] The server sends a reservation confirmation to the terminal.

[1391] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[1392] Step 22:

[1393] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[1394] Display the booking confirmation in the UI and provide an "Email" option.

[1395] Step 23:

[1396] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[1397] Real-time data is obtained through weather information services and tourist destination APIs.

[1398] Step 24:

[1399] The server detects the fluctuations and generates alternatives using generative AI.

[1400] Generate new schedules based on changes in weather and traffic conditions, including considering the emotional state to incorporate using an emotion engine.

[1401] Step 25:

[1402] The server sends alternatives to the terminal.

[1403] The new schedule proposal is sent to the device and a notification is sent.

[1404] Step 26:

[1405] The terminal notifies the user of the alternatives.

[1406] Use popups and notifications to let users know about alternatives.

[1407] Step 27:

[1408] The user selects the desired language from the application settings.

[1409] Use the language settings option to select your preferred language.

[1410] Step 28:

[1411] The device will display the UI in the selected language.

[1412] The text is displayed according to the language you set.

[1413] Step 29:

[1414] The user presses the "Customer Support" button.

[1415] Tap on the "Customer Support" option.

[1416] Step 30:

[1417] The server provides customer support based on the user's inquiry.

[1418] Resolve user issues through chatbots and human support.

[1419] Example 2

[1420] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1421] Conventional travel planning systems generate schedules based on a user's desired travel destinations, duration, and activity information, but lack the flexibility to respond to user emotions or real-time information fluctuations. Furthermore, they do not support multiple languages, limiting their usability for global users. This makes it difficult for users to flexibly adjust plans to accommodate sudden changes during their trip or their individual emotional states.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1423] In this invention, the server includes a means for analyzing the user's emotional state and readjusting the proposed content based on the emotion, a means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, and a means for supporting multiple languages, thereby enabling the provision of flexible travel plans that respond to the user's emotions and real-time changes in the situation.

[1424] A "user" is an end user who uses the travel planning system to select a travel destination and adjust a schedule.

[1425] "Destination" refers to a geographic location that a user wishes to visit.

[1426] "Duration" refers to the specific length of time that a user has set for planning a trip.

[1427] "Activities" refer to events or activities that a user wants to experience during their trip.

[1428] "Meal information" refers to information about the food and drink that the user desires to eat and drink during the trip.

[1429] "Map information service" refers to a third-party service that provides geographic information and calculates travel times between locations.

[1430] "Accommodation" refers to a facility where a user can stay while traveling.

[1431] "Transportation" refers to the vehicles and services used for transportation during a trip.

[1432] "Emotional state" refers to the psychological state a user feels about a travel plan or proposal.

[1433] "Real-time information" refers to the latest information at that time, such as weather information and the status of tourist attractions.

[1434] An "emotion engine" refers to software that analyzes user input and reactions to recognize emotions.

[1435] "Generative AI" refers to an artificial intelligence model that creates optimal travel schedules based on user input.

[1436] "Language support" refers to the ability to provide interfaces and notifications in the language selected by the user.

[1437] "Proposed Changes" means any proposed modifications or alternatives to your travel plans or schedule.

[1438] MODE FOR CARRYING OUT THE INVENTION

[1439] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[1440] Overall structure

[1441] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine recognizes emotions by analyzing the user's input and reactions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[1442] Travel planning

[1443] First, the user launches the application and logs in. The device prompts for an ID and password, and the server authenticates them. Once authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images. The device then sends this information to the server. The server analyzes the entered information and uses a generative AI (e.g., GPT-4) to generate an optimal itinerary and itinerary. An example prompt might be, "The user is planning a three-day trip to Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and eat sushi. Please suggest an optimal itinerary." The server uses the Google Maps API to calculate travel times between locations and optimize the overall itinerary. The server then generates the optimized itinerary and sends it to the device.

[1444] The device displays the proposed schedule to the user, who then confirms it. The emotion engine analyzes the user's reactions and comments when checking the schedule. The emotion engine recognizes the user's emotional state and sends the analysis results to the server. The server readjusts the proposed content based on the emotion engine's analysis results and sends the prompt text back to the generation AI to update the schedule.

[1445] Reservations and real-time information

[1446] Once the user confirms and confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities. The server generates a reservation confirmation based on the reservation information and sends it to the terminal, which displays it to the user. If the user wishes, the confirmation can be sent by email or other means.

[1447] Furthermore, during the trip, the server periodically obtains real-time weather information and tourist destination information. It obtains the latest information using weather information services and tourist destination APIs. If changes in weather or traffic conditions are detected, the server uses a generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[1448] Multilingual support and customer support

[1449] The user selects their preferred language in the application settings. The device displays the UI in the selected language. If the user needs assistance during their trip, they can press the "Customer Support" button, which the server will then handle via a chatbot or customer support representative.

[1450] Specific examples

[1451] For example, if a user plans a three-day trip to Tokyo, they launch the application, log in, and select the "Plan a Trip" option. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Sensoji Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. Using the Google Maps API, the server calculates the optimal route and travel time from Sensoji Temple to Tokyo Skytree. The server obtains reservation information for appropriate accommodation, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all the reservations in one go and generates a reservation confirmation.

[1452] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, the emotion engine is used to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[1453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1454] Step 1:

[1455] The user launches the application and the login screen appears.

[1456] Specific operation: The user enters a user ID and password into the terminal, and the terminal sends the entered information to the server.

[1457] Input: User ID, Password

[1458] Output: Authentication request

[1459] Step 2:

[1460] The server compares the received ID and password with the database and performs authentication.

[1461] Specific operation: The server checks the ID and password against the database, and if they match, it sends an authentication success message to the terminal.

[1462] Input: User ID, Password

[1463] Output: Authentication success message or authentication failure message

[1464] Step 3:

[1465] The device receives a successful authentication message and displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options.

[1466] Specific operation: The user inputs information about the travel destination, duration, activities, and places to eat into the device, which then sends it to the server.

[1467] Input: Travel destination, duration, activities, and dining information

[1468] Output: Travel planning request

[1469] Step 4:

[1470] The server analyzes the user's input information and uses generative AI to generate the optimal places to visit and schedule.

[1471] Specific operation: The server analyzes the user's input information and sends prompt text to the generation AI to generate a schedule.

[1472] Example prompt: The user is planning a three-day trip in Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and he wants to eat sushi. Please suggest the best itinerary.

[1473] Input: Travel planning request (destination, duration, activities, dining information)

[1474] Output: Initial schedule

[1475] Step 5:

[1476] The server uses the Google Maps API to calculate travel times between locations and optimize the overall schedule.

[1477] Specific operation: The server calculates the travel time between each location using the Google Maps API, optimizes the overall schedule based on the calculation results, and sends the optimized schedule to the device.

[1478] Input: Location information of the place to be visited

[1479] Output: Optimized schedule

[1480] Step 6:

[1481] The terminal displays the proposed schedule to the user, who then checks the schedule and enters comments or requests for adjustments as necessary.

[1482] Specific operation: The terminal displays the schedule screen and the user confirms it. If the user inputs an adjustment request, the terminal sends it to the server.

[1483] Input: Initial Schedule

[1484] Output: User response and adjustment request

[1485] Step 7:

[1486] The server analyzes the user's reactions and comments using an emotion engine to identify their emotional state.

[1487] Specific operation: The server sends the user's comments and reactions to the emotion engine. The emotion engine analyzes the user's emotional state and sends the results to the server.

[1488] Input: User responses and comments

[1489] Output: User's emotional state

[1490] Step 8:

[1491] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[1492] Specific operation: The server uses the results of the emotion engine to send a prompt to the generation AI again and update the proposal content.

[1493] Input: User's emotional state, prompt sentence

[1494] Output: Re-adjusted schedule

[1495] Step 9:

[1496] Once the user confirms and confirms the rescheduled schedule, the server collectively makes reservations for accommodation, transportation, and activities.

[1497] Specific operation: The user checks the rescheduled schedule on the terminal and presses the confirm button. The server makes various reservations through the reservation system.

[1498] Input: Re-arranged schedule

[1499] Output: Reservation information and confirmation

[1500] Step 10:

[1501] The server periodically acquires real-time information on weather and tourist spots during the travel period.

[1502] Specific operation: The server periodically calls weather information services and tourist destination APIs to obtain the latest information.

[1503] Input: None (regular operation)

[1504] Output: Real-time information

[1505] Step 11:

[1506] The server generates alternatives by sending prompts to the generation AI based on the fluctuation information, and then uses an emotion engine to generate alternatives that are optimized to the user's emotions.

[1507] Specific operation: The server obtains the latest weather information and tourist destination conditions, and sends prompts to the generation AI as needed. The emotion engine optimizes alternatives and notifies the device.

[1508] Input: Real-time information, prompt text

[1509] Output: Optimized alternatives

[1510] Step 12:

[1511] The user selects the desired language from the application settings.

[1512] Specific behavior: The device displays the language settings screen, the user selects the desired language, and the device updates the UI in the selected language.

[1513] Input: Desired language

[1514] Output: UI in selected language

[1515] Step 13:

[1516] When a user requests customer support, the server initiates the response.

[1517] Specific operation: The user presses the support button, the server accepts the inquiry, and responds via a chatbot or customer support representative.

[1518] Input: Support Request

[1519] Output: Supported

[1520] (Application example 2)

[1521] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1522] Conventional travel planning support systems propose itineraries based on user preferences, but because they are unable to consider the user's emotional state, the proposed itinerary often does not match the user's expectations. Furthermore, there are only a limited number of systems that allow flexible schedule changes in real time during a trip, leaving room for improvement in the user experience. Another problem is that the proposed alternatives are uniform and not optimized for each individual user.

[1523] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1524] In this invention, the server includes means for receiving information on the user's desired travel destinations, period, activities, and meals, means for generating optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, means for recognizing the user's emotional state using an emotion recognition engine and optimizing the travel plan and alternatives, and means for supporting multiple languages, thereby enabling the server to propose flexible and optimal travel plans that take the user's emotional state into consideration.

[1525] "User's desired travel destination, duration, activities, and meal information"

[1526] This refers to details of the places and duration a user wants to visit during their trip, the activities they want to participate in, and the meals they want to eat.

[1527] "Optimal visiting locations and schedules"

[1528] This refers to the allocation of destinations and time planned to make the trip efficient and enjoyable based on the user's desired information.

[1529] "Map information provision service"

[1530] This refers to external data services that provide location and route information, such as services that use map APIs.

[1531] "A one-stop solution for booking accommodation, transport and activities"

[1532] This refers to the ability for users to book accommodation, transportation, and local activities for their planned trip all at once.

[1533] "Weather information and tourist destination conditions"

[1534] This refers to real-time weather forecasts, congestion status at tourist spots, business status, etc.

[1535] Emotion recognition engine

[1536] This refers to a technology that analyzes and recognizes a user's emotional state (happiness, surprise, excitement, anxiety, etc.) from their facial expressions and behavior.

[1537] "A way to support multiple languages"

[1538] It refers to the ability of an application or system to provide a user interface in different languages.

[1539] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and is combined with an emotion engine that recognizes the user's emotions and optimizes the plan.

[1540] System configuration

[1541] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, and can be a smartphone, tablet, or PC. The server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes user input and reactions and recognizes emotions. The external API service includes a map information service and a reservation service API.

[1542] The process of emotion recognition

[1543] When a user plans a trip, facial expression recognition technology is used to analyze the user's emotional state in real time. This technology uses OpenCV to detect faces and TensorFlow / Keras to classify emotions. For example, emotional states such as joy, anxiety, and surprise can be detected in real time.

[1544] Travel plan example

[1545] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application on their smartphone, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. In this process, the Google Maps API is used to calculate travel times between each place to be visited and optimize the overall schedule.

[1546] Optimizing recommendations with an emotion engine

[1547] When the user reviews the proposed schedule, the emotion engine analyzes their reaction. If the emotion engine detects that the user is feeling anxious, it will suggest alternatives, including relaxing sightseeing spots and activities. Conversely, if the user is feeling excited, it will increase the suggestions of adventurous activities. For example, if the weather worsens, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed.

[1548] Examples of specific examples and prompts

[1549] In one embodiment of the invention, when a user is watching a horror movie and sentiment analysis is performed, a prompt is generated as follows:

[1550] Example prompt: If a user is feeling "scared," recommend another video with more action and thrills.

[1551] In this way, real-time emotion recognition can be performed to optimize the user experience and provide more personalized travel plans.

[1552] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1553] Step 1:

[1554] The server receives information about the user's desired travel destination, duration, activities, and meals. The user uses a smartphone or tablet device to input details such as the travel destination, duration, and desired activities. This input information is sent from the device to the server. As an output, the server stores the user's input information.

[1555] Step 2:

[1556] The server generates optimal places to visit and a schedule based on the information it receives. Specifically, it uses a generative AI model to formulate places to visit and a schedule that meets the user's preferences. During this process, it analyzes the user's desired information as data processing and calculates optimal places to visit and routes. As an output, the server generates an initial travel schedule.

[1557] Step 3:

[1558] The server works with a map information service to automatically calculate travel times. Specifically, it uses a map API to obtain travel times between each visit location. Based on this information, it optimizes the overall schedule and calculates efficient routes between visit locations. The output is an optimized schedule that reflects travel times.

[1559] Step 4:

[1560] The server sends the generated optimized schedule to the terminal. The proposed schedule is displayed on the user's terminal, and the user confirms its contents. The input is the schedule data from the server, and the output is a confirmation screen displayed on the user's terminal.

[1561] Step 5:

[1562] The device captures the user's facial expression and sends the video data to the server. Specifically, the device's camera function is used to capture the user's facial expression in real time and sends the data to the server. The output is the facial expression data sent to the server.

[1563] Step 6:

[1564] The server uses an emotion recognition engine to analyze the user's facial expression data and recognize emotions. Specifically, it uses a face detection algorithm (OpenCV) to identify the face area, and then uses a TensorFlow / Keras model to classify the emotion. The result of this operation is the recognized emotion data. The output is the recognized emotion (e.g., joy, anxiety, excitement, uneasiness, etc.).

[1565] Step 7:

[1566] The server optimizes travel plans and alternatives based on the recognized emotion data. For example, if the user is feeling "anxious," it will suggest more relaxing sightseeing spots and activities. It uses a generative AI model to generate these optimized plans and alternatives. The output is an optimized travel schedule tailored to the user's emotions.

[1567] Step 8:

[1568] The server continuously obtains weather information and real-time conditions of tourist destinations, and generates proposed schedule changes as needed. Specifically, it uses weather information services and tourist destination APIs to monitor local conditions. Based on this information, a generative AI model creates a new schedule and uses an emotion engine to suggest alternatives based on the user's emotions. The output is the latest alternative schedule.

[1569] Step 9:

[1570] The optimized schedule and alternative plans are notified to the terminal. The terminal displays the proposals to the user in real time and the user confirms the contents. The user confirms and decides on the new plan. The input is the optimized schedule from the server, and the output is a notification screen to the user.

[1571] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1572] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1573] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1574] [Fourth embodiment]

[1575] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1576] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1577] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1578] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1579] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1580] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1581] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1582] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1583] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1584] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1585] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1586] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1587] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1588] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[1589] Overall structure

[1590] This system consists of a user's device, a server, and many external API services. The user's device is the device that runs the application, and the server is responsible for generating and optimizing travel plans and obtaining external information. In addition, by utilizing map information services and reservation service APIs, each step of the travel plan can be carried out smoothly.

[1591] Travel planning

[1592] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[1593] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[1594] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[1595] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[1596] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[1597] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1598] Reservations and real-time information

[1599] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1600] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[1601] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1602] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[1603] Multilingual support and customer support

[1604] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[1605] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[1606] Specific examples

[1607] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[1608] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[1609] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it will suggest an alternative schedule. In this way, the user can enjoy their trip efficiently and hassle-free.

[1610] This system greatly improves convenience for travelers by automating and providing flexibility in travel planning.

[1611] The processing flow will be explained below.

[1612] Step 1:

[1613] The user launches the application and the login screen appears.

[1614] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[1615] Step 2:

[1616] The user enters their ID and password and clicks the "Login" button.

[1617] The device sends the entered ID and password to the server.

[1618] Step 3:

[1619] The server authenticates the user information.

[1620] The database is accessed to search for user information that matches the entered ID and password.

[1621] Step 4:

[1622] If the server is successful in authenticating, it generates an authentication token.

[1623] A session is established for the authenticated user, and a token is generated and sent to the terminal.

[1624] Step 5:

[1625] The device displays the main menu.

[1626] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[1627] Step 6:

[1628] The user selects "Plan a Trip" from the main menu.

[1629] Tap the "Travel Plans" option to proceed to the next form.

[1630] Step 7:

[1631] The device will display a screen for entering travel destination, duration, activities, and meals.

[1632] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[1633] Step 8:

[1634] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[1635] Enter the required information using the input form and image upload feature.

[1636] Step 9:

[1637] The terminal transmits the input information and image data to the server.

[1638] Send structured data and image data in JSON format to the server.

[1639] Step 10:

[1640] The server analyzes the input using generated AI.

[1641] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[1642] Step 11:

[1643] The server determines the strategy and calculates travel time using the Google Maps API.

[1644] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[1645] Step 12:

[1646] The server retrieves reservation options for accommodation, transportation, and activities.

[1647] A list of accommodation, transportation, and activity options is created through each booking site's API.

[1648] Step 13:

[1649] The server sends the optimized schedule to the terminal.

[1650] The final schedule data is sent to the terminal and prepared for display to the user.

[1651] Step 14:

[1652] The terminal displays the proposed schedule to the user.

[1653] The schedule contents are displayed on the screen so that the user can check them.

[1654] Step 15:

[1655] The user checks the schedule and presses the "Confirm" button.

[1656] Check the contents of the proposed schedule and tap the "Confirm" button.

[1657] Step 16:

[1658] The server batches reservations for accommodation, transportation, and activities.

[1659] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[1660] Step 17:

[1661] The server sends a reservation confirmation to the terminal.

[1662] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[1663] Step 18:

[1664] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[1665] Display the booking confirmation in the UI and provide an "Email" option.

[1666] Step 19:

[1667] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[1668] Real-time data is obtained through weather information services and tourist destination APIs.

[1669] Step 20:

[1670] The server detects the fluctuations and generates alternatives using generative AI.

[1671] Generate new schedules that adapt to changes in weather and traffic conditions.

[1672] Step 21:

[1673] The server sends alternatives to the terminal.

[1674] The new schedule proposal is sent to the device and a notification is sent.

[1675] Step 22:

[1676] The terminal notifies the user of the alternatives.

[1677] Use popups and notifications to let users know about alternatives.

[1678] Step 23:

[1679] The user selects the desired language from the application settings.

[1680] Use the language settings option to select your preferred language.

[1681] Step 24:

[1682] The device will display the UI in the selected language.

[1683] The text is displayed according to the language you set.

[1684] Step 25:

[1685] The user presses the "Customer Support" button.

[1686] Tap on the "Customer Support" option.

[1687] Step 26:

[1688] The server provides customer support based on the user's inquiry.

[1689] Resolve user issues through chatbots and human support.

[1690] Example 1

[1691] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1692] Conventional travel planning support systems often require a lot of effort to input and analyze information about users' desired travel destinations, duration, activities, and meals. Furthermore, it is difficult to comprehensively provide multiple functions, such as optimizing travel schedules, calculating travel times, managing reservations, acquiring and responding to real-time information, and supporting multiple languages. As a result, users' convenience is reduced, and travel planning does not proceed smoothly.

[1693] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1694] In this invention, the server includes means for receiving information about a user's desired destination, period, activities, and meals, means for using a generative AI model to generate optimal places to visit and a schedule based on the received information, means for linking with a map information providing service to automatically calculate travel times, means for making reservations for accommodations, transportation, and activities all at once, means for acquiring meteorological information and tourist spot conditions in real time and providing suggested changes to the travel schedule, means for supporting multiple languages, and means for analyzing multimedia data uploaded by the user and suggesting places to visit and eat based on the analysis results, thereby enabling users to efficiently plan their trips and flexibly respond to changes in the situation in real time.

[1695] "User" means an individual or organization that uses the system to make travel plans.

[1696] A "destination" is a place that a user wishes to visit during a trip.

[1697] "Duration" is the specific number of days or hours that the user plans to spend traveling.

[1698] An "activity" is an event or experience that a user wants to participate in during their trip.

[1699] "Meals" refers to the foods and places to eat that the user wants to eat during their trip.

[1700] A "generative AI model" is an artificial intelligence algorithm that automatically generates optimal places to visit and schedules based on user input.

[1701] "Map information service" refers to a geographic information system and its API used to calculate travel time and distance.

[1702] "Accommodation" refers to a hotel or lodging facility that a User reserves for stay during their trip.

[1703] "Transportation" refers to the vehicles such as airplanes, trains, buses, and taxis that users use to travel.

[1704] "Weather information" refers to information about the weather forecast and weather conditions during the travel period.

[1705] "Tourist destination situation" refers to the congestion situation at tourist destinations, business information, event information, etc.

[1706] "Multiple languages" refers to the ability to provide user interfaces and system functionality in different languages.

[1707] "Multimedia data" refers to information in digital form, such as images, audio, and video, uploaded by users.

[1708] "Analysis results" are information and recommendations obtained as a result of analyzing multimedia data.

[1709] A "reservation confirmation" is a document that is generated after an itinerary is confirmed and details the reservation details for accommodation, transportation, and activities.

[1710] This invention relates to an interactive travel concierge system that assists users in planning their trips. This system is composed of a user's device, a server, and multiple external API services. The user's device runs an application, while the server is responsible for generating and optimizing the trip plan and acquiring external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be smoothly carried out.

[1711] Hardware and software configuration

[1712] The user's terminal is a device capable of running applications, such as a smartphone, tablet, or personal computer. This device provides an interface for inputting travel plans, transmits the user's input information to the server, and receives and displays responses from the server to the user.

[1713] The server has high-performance computing resources, analyzes the information entered by the user, and generates optimal places to visit and a schedule using a generative AI model that incorporates natural language processing and image analysis technologies, for example.

[1714] The external API services used include map information services, such as a map information API and a reservation service API, which allow users to calculate travel times between locations and make reservations for accommodations and transportation.

[1715] Program operation explanation

[1716] 1. The user launches the travel concierge application from their device and logs in. On the login screen, the user enters their username and password, which are then sent to the server. The server then authenticates and verifies the user's identity.

[1717] 2. After authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about the trip destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images.

[1718] 3. The device sends this input information to the server, which analyzes it and uses a generative AI model to generate optimal places to visit and a schedule.

[1719] For example, the prompt text might look like this:

[1720] "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Sushi images"

[1721] 4. The server uses a map information API to calculate travel time between locations and optimize the overall schedule.

[1722] 5. The server sends the generated schedule to the terminal, which displays the proposed schedule to the user for confirmation.

[1723] 6. Once the user confirms and confirms the proposed schedule, the server will make all reservations for accommodation, transportation, and activities at once, and generate a reservation confirmation based on this reservation information and send it to the terminal.

[1724] 7. During the trip, the server periodically obtains real-time weather information and tourist destination information. It uses the weather information service and tourist destination APIs to obtain real-time information and generates alternatives for changing the schedule as needed. This information is sent to the terminal and notified to the user.

[1725] 8. The device provides a multilingual UI to display applications in the language the user needs. If a user needs assistance during their trip, they can contact the server by pressing the "Customer Support" button, and the server will respond to the inquiry with a chatbot or customer support representative.

[1726] This system allows users to plan trips efficiently and effortlessly, and responds flexibly to real-time changes in the situation. By linking with various APIs, it eliminates travel time between destinations and the hassle of making reservations, enriching the user's travel experience.

[1727] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1728] Step 1:

[1729] A user starts an application and logs in. The terminal prompts for a username and password, and the server receives this authentication information. The server accesses a database and collates the user information to perform authentication. The input data is the username and password, and the output data is the authentication result.

[1730] Step 2:

[1731] After successful login, the terminal displays the main menu. The user selects the "Travel Planning" option. The terminal displays menu selection information and accepts the user's selection as input. The displayed menu contains various items related to travel planning.

[1732] Step 3:

[1733] The user selects a travel planning option and inputs information about the travel destination, duration, desired activities, and dining locations. The terminal receives the input information and sends it to the server. The input data is information about the travel destination, duration, activities, and dining locations, and the output data is the data sent to the server.

[1734] Step 4:

[1735] The server analyzes the received information. The server analyzes the data using a generative AI model and generates the optimal places to visit and schedule based on the user's preferences. An example of a prompt sentence would be "Travel destination: Tokyo, Duration: 3 days, Places to visit: Sensoji Temple, Tokyo Skytree, Food: Images of sushi." The output data are the analysis results and the generated schedule.

[1736] Step 5:

[1737] The server uses a map information API to calculate travel times between visit locations, optimizing the overall schedule. The input data is the coordinate information of the visit locations and the start and destination points of the trip, and the output data is the travel time between each visit location.

[1738] Step 6:

[1739] The server sends the optimized schedule to the terminal, which receives it and displays it to the user. The input data is the optimized schedule received from the server, and the output data is the schedule to display to the user.

[1740] Step 7:

[1741] The user confirms the proposed schedule and presses the confirm button. This input operation causes the terminal to send the schedule confirmation information to the server. The input data is the schedule confirmation operation by the user, and the output data is the schedule confirmation information sent to the server.

[1742] Step 8:

[1743] The server makes reservations for accommodation, transportation, and activities in bulk based on the confirmed schedule. It generates a reservation confirmation and sends it to the terminal. The input data is the confirmed schedule information, and the output data is the reservation confirmation.

[1744] Step 9:

[1745] The terminal displays the reservation confirmation to the user. If desired, the user can receive the confirmation by email, etc. The input data is the reservation confirmation received from the server, and the output data is the information displayed and sent to the user.

[1746] Step 10:

[1747] During the trip, the server periodically obtains weather information and real-time information on tourist destinations. This is done using the weather information service and tourist destination APIs. The input data is data from the real-time information API, and the output data is analyzed real-time information.

[1748] Step 11:

[1749] When changes in weather or traffic conditions are detected, the server uses a generative AI model to generate alternative plans and sends them to the device. The device then notifies the user of the alternative plans. The input data is the change information, and the output data is the alternative plans.

[1750] Step 12:

[1751] The user selects the desired language from the application settings. The device displays the UI in the selected language. The input data is the user's language selection information, and the output data is the UI in the selected language.

[1752] Step 13:

[1753] If a user needs assistance during their trip, they press the "Customer Support" button. The server accepts the inquiry and responds via a chatbot or customer support representative. The input data is the user's inquiry information, and the output data is the response information from the server.

[1754] (Application example 1)

[1755] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1756] Conventional travel planning support systems lacked the means to perform detailed simulations of travel destinations, making it difficult for users to inspect or experience the places they would actually visit in advance. Furthermore, the generation of reservation confirmations and final confirmation of travel plans were performed only in real space, making flexible responses difficult. Furthermore, there was no interface available to accept complex user input and intuitively and efficiently plan travel in a virtual space. To solve these problems, there is a need for a system that can intuitively create travel plans in a virtual space and display 3D models of travel destinations.

[1757] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1758] In this invention, the server includes: means for receiving information on a user's desired travel destination, duration, activities, and meals; means for generating optimal visit locations and schedules based on the received information; means for linking with a map information service to automatically calculate travel times; means for making reservations for accommodations, transportation, and activities all at once; means for acquiring real-time weather information and tourist spot conditions and proposing travel schedule changes; means for supporting multiple languages; means for providing an interface through which a user inputs travel destination information using a head-mounted display and makes a travel plan in a virtual space; and means for displaying a 3D model of the travel destination in the virtual space and allowing the user to simulate the places they will visit. This allows users to intuitively plan their trip in the virtual space and allows them to inspect and experience the destination in advance. Furthermore, flexible reservation confirmation can be ensured through confirmation in the virtual space.

[1759] "Means for receiving information on the user's desired travel destination, duration, activities, and meals" is a function that collects data regarding the destination, duration of stay, activities the user wishes to participate in, and meals the user wishes to eat, which are input to plan the trip.

[1760] "Means for generating optimal places to visit and schedules based on received information" refers to a function that uses algorithms and AI to create travel schedules including optimal tourist spots and activities based on information entered by the user.

[1761] The "means for linking with a map information service to automatically calculate travel time" is a function that links with a map service and automatically calculates travel time between each visited location.

[1762] "A means of booking accommodation, transportation, and activities all at once" is a function that allows users to book accommodation, transportation, and activities all at once based on their confirmed travel schedule.

[1763] "Means for obtaining weather information and tourist destination conditions in real time and proposing changes to the travel schedule" is a function that provides users with suggested schedule changes as needed based on weather information and tourist destination conditions obtained in real time during the trip.

[1764] "Means for supporting multiple languages" refers to a function that provides interfaces and guidance in multiple languages ​​to users who speak different languages.

[1765] "A means of displaying a 3D model of a travel destination in a virtual space, allowing users to simulate visiting places" is a function that uses virtual reality technology to present a 3D model of a travel destination to users, allowing users to simulate visiting places in advance.

[1766] "A means for providing an interface that allows a user to input travel destination information using a head-mounted display and make travel plans in a virtual space" is a function that provides an interface that allows a user to input travel destination and activity information in a virtual space while wearing a head-mounted display and make travel plans.

[1767] "Means for generating and sending a reservation confirmation based on a confirmed schedule and prompting the user to confirm it within the virtual space" is a function that creates a reservation confirmation based on the travel schedule confirmed by the user and prompts the user to confirm it through an interface within the virtual space.

[1768] "A means of analyzing images uploaded by users and suggesting places to visit and eat based on the analysis results" is a function that analyzes image data provided by users using AI, etc., and suggests the most suitable tourist spots and restaurants based on the analysis results.

[1769] The present invention relates to an interactive travel concierge system that assists users in planning their trips. Specific embodiments of this system will be described below.

[1770] Overall structure

[1771] This system consists of a device used by the user, such as a head-mounted display or smart glasses, a server, and an external API service. The user's device provides an interface for planning the trip, while the server is responsible for generating and optimizing the trip plan, as well as obtaining external information. Furthermore, by utilizing map information services and reservation service APIs, each step of the trip plan can be carried out smoothly.

[1772] Travel planning

[1773] 1. The user wears the head-mounted display and starts the application. The user performs the login operation, the device requests the ID and password, and the server performs authentication.

[1774] 2. Once authentication is complete, the device displays the virtual space's main menu. The user can select the "Plan Your Trip" option and enter information about their travel destination, duration, desired activities, and dining options.

[1775] 3. Users can select the places they want to visit while looking at a 3D model of the destination in the virtual space. This information is sent from the device to the server.

[1776] 4. The server analyzes the input information and generates optimal places to visit and a schedule using a generative AI model, such as GPT-3.5.

[1777] 5. Using the map information service API, the server calculates travel times between locations and optimizes the overall schedule. The server then sends the generated schedule to the device.

[1778] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1779] Reservations and real-time information

[1780] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1781] 2. A reservation confirmation is generated based on this reservation information and sent to the device. The device displays the confirmation to the user and prompts them to confirm it in the virtual space if necessary.

[1782] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1783] 4. If changes in weather or traffic conditions are detected, the server uses generation AI to generate alternative plans and sends them to the device, which then notifies the user of the alternative plans.

[1784] Specific examples

[1785] For example, suppose a user plans a three-day trip to a certain location in Japan in a virtual space. The user puts on a head-mounted display, launches the application, logs in, and selects the travel planning options. The user uploads a "certain location in Japan" as the travel destination, "three days" as the duration, "historical buildings" and "famous places" as places to visit, and "images of traditional cuisine" as food they want to eat. Once this information is sent to the server, the server uses a generative AI model to generate an optimal travel itinerary. Using a map information service API, the server calculates the optimal route and travel time between each destination. The server obtains reservation information for appropriate accommodations, transportation, and activities, and sends the optimal schedule to the device. Once the user confirms this schedule and presses the confirm button, the server makes all reservations and generates a reservation confirmation. During the trip, the server periodically obtains real-time information on weather and tourist attractions, and suggests an alternative itinerary if there are any changes. In this way, users can enjoy their trip efficiently and hassle-free.

[1786] Prompt Sentence Examples

[1787] An example of a prompt that the user can enter into the AI ​​generator is:

[1788] "Information to help you plan your trip:

[1789] Travel destination: Somewhere in Japan

[1790] Duration: 3 days

[1791] Places I want to visit: Historical buildings, famous places

[1792] What to eat: Images of traditional dishes

[1793] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1794] Step 1:

[1795] The user puts on the head-mounted display and launches the travel planning application. The user enters their ID and password, and the server performs authentication. The input is the user's ID and password, and the output is the authentication result. The server checks the authentication information against the database.

[1796] Step 2:

[1797] After successful authentication, the device displays the main menu of the virtual space. The user selects the "Plan a Trip" option and enters information about the travel destination, duration, desired activities, and dining locations. The input is the user's travel destination information, and the output is the transmission of the input information. The device receives the user's input through the virtual interface and sends it to the server.

[1798] Step 3:

[1799] The server analyzes the information it receives and uses a generative AI model to generate optimal places to visit and a schedule. The input is the user's travel information, and the output is an optimal travel schedule. The server inputs a prompt into the generative AI model to generate a schedule. GPT-3.5 is used as the generative AI model.

[1800] Step 4:

[1801] The server uses the map information service API to calculate the travel time between visited locations. The input is a list of visited locations, and the output is the travel time between each visited location. The data is calculated using API calls.

[1802] Step 5:

[1803] The server generates an optimal travel schedule and sends it to the terminal. The input is the optimal schedule and the output is a transmission to the terminal. The server sends the optimal schedule to the terminal as a data packet.

[1804] Step 6:

[1805] The terminal displays the proposed schedule to the user, who then confirms it. The received schedule is the input, and the user's confirmation result is the output. The terminal visually displays the schedule in a virtual space, and the user presses the confirmation and agreement buttons.

[1806] Step 7:

[1807] Once the user confirms the schedule, the server makes reservations for accommodation, transportation, and activities in one go. The input is the confirmed schedule, and the output is a reservation confirmation. The server uses an external reservation service API to make the necessary reservations and generate a confirmation.

[1808] Step 8:

[1809] The server sends the reservation confirmation to the terminal, which then displays it to the user. The input is the reservation confirmation, and the output is a display to the user. The terminal displays the confirmation in the virtual space and asks the user for confirmation.

[1810] Step 9:

[1811] During the trip, the server periodically obtains real-time weather information and tourist destination information, and generates suggested changes to the travel schedule if any abnormalities are detected. The real-time information is the input, and the suggested changes are the output. The server uses weather information services and tourist destination APIs to generate suggested changes using a generative AI model.

[1812] Step 10:

[1813] The device notifies the user of the alternative plan, and if the user accepts it, the new schedule is applied. The alternative plan is the input, and the new schedule is presented as the output. The device displays the alternative plan in the virtual space, and applies the new schedule after confirmation.

[1814] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1815] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[1816] Overall structure

[1817] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine analyzes the user's input and reactions to recognize emotions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[1818] Travel planning

[1819] 1. The user starts the application and logs in. At this time, the device requests an ID and password, and the server performs authentication.

[1820] 2. Once logged in, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options.

[1821] 3. This information includes not only text data but also multimedia data such as images. The device sends this information to the server.

[1822] 4. The server analyzes the input information and uses generative AI to generate the optimal places to visit and schedule.

[1823] 5. Using the Google Maps API, the travel time between locations is calculated and the overall schedule is optimized. The server generates a schedule and sends it to the device.

[1824] 6. The terminal displays the proposed schedule to the user, who confirms it.

[1825] Introducing and Adapting the Emotion Engine

[1826] 1. The emotion engine analyzes the reactions and comments entered by users when checking their schedules.

[1827] 2. The emotion engine recognizes the user's emotional state and identifies states such as joy, surprise, excitement, and anxiety.

[1828] 3. The server uses the analysis results of the emotion engine to readjust the suggested places to visit and schedule.

[1829] For example, if a user is feeling anxious, the app will suggest more relaxing tourist spots and activities.

[1830] If the user is excited, increase the adventurous activity.

[1831] Reservations and real-time information

[1832] 1. Once the user confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities.

[1833] 2. A reservation confirmation is generated based on this reservation information and sent to the terminal. The terminal displays the confirmation to the user and, if desired, sends it by email or other means.

[1834] 3. During the trip, the server periodically obtains real-time weather information and tourist destination information, using the APIs of weather information services and tourist destinations.

[1835] 4. If changes in weather or traffic conditions are detected, the server uses generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[1836] Multilingual support and customer support

[1837] 1. The user selects the desired language in the application settings, and the device displays the UI in the selected language.

[1838] 2. If the user needs assistance during their trip, they can press the "Customer Support" button. The server will then handle the inquiry and respond via a chatbot or customer support representative.

[1839] Specific examples

[1840] For example, let's say a user is planning a three-day trip to Tokyo. The user launches the application, logs in, and selects the travel planning options. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Asakusa Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate the optimal travel schedule.

[1841] Utilizing the Google Maps API, the optimal route and travel time from Sensoji Temple to Tokyo Skytree is calculated. The server obtains reservation information for suitable accommodations, transportation, and activities, and sends the optimal schedule to the device. When the user confirms this schedule and presses the confirm button, the server makes all reservations in one go and generates a reservation confirmation.

[1842] During the trip, the server periodically obtains real-time information on weather and tourist spots, and if there are any changes, it uses the emotion engine to suggest an alternative schedule. For example, if the weather worsens on the day of the trip, the emotion engine will detect the user's anxiety and suggest indoor activities that can be enjoyed. In this way, the user can enjoy their trip efficiently and hassle-free.

[1843] This system greatly improves convenience for travelers by automating trip planning and adding flexibility through emotion recognition.

[1844] The processing flow will be explained below.

[1845] Step 1:

[1846] The user launches the application and the login screen appears.

[1847] It connects to the server and displays a form on the terminal for entering an ID and password for login authentication.

[1848] Step 2:

[1849] The user enters their ID and password and clicks the "Login" button.

[1850] The device sends the entered ID and password to the server.

[1851] Step 3:

[1852] The server authenticates the user information.

[1853] The database is accessed to search for user information that matches the entered ID and password.

[1854] Step 4:

[1855] If the server is successful in authenticating, it generates an authentication token.

[1856] A session is established for the authenticated user, and an authentication token is generated and sent to the terminal.

[1857] Step 5:

[1858] The device displays the main menu.

[1859] If the authentication is successful, the main menu screen is displayed and the user is ready to begin work.

[1860] Step 6:

[1861] The user selects "Plan a Trip" from the main menu.

[1862] Tap the "Travel Plans" option to proceed to the next form.

[1863] Step 7:

[1864] The device will display a screen for entering travel destination, duration, activities, and meals.

[1865] Displays a form for entering details about the trip (e.g., destination, duration, places to visit, food to eat).

[1866] Step 8:

[1867] Users enter their travel destination and duration, and upload images of places they want to go and foods they want to eat.

[1868] Enter the required information using the input form and image upload feature.

[1869] Step 9:

[1870] The terminal transmits the input information and image data to the server.

[1871] Send structured data and image data in JSON format to the server.

[1872] Step 10:

[1873] The server analyzes the input using generated AI.

[1874] Generate optimal places to visit and itineraries based on destination, duration, activities, and meals.

[1875] Step 11:

[1876] The server determines the strategy and calculates travel time using the Google Maps API.

[1877] Obtain the optimal route and travel time between each visit location to optimize the schedule.

[1878] Step 12:

[1879] The server retrieves reservation options for accommodation, transportation, and activities.

[1880] A list of accommodation, transportation, and activity options is created through each booking site's API.

[1881] Step 13:

[1882] The server sends the optimized schedule to the terminal.

[1883] The final schedule data is sent to the terminal and prepared for display to the user.

[1884] Step 14:

[1885] The terminal displays the proposed schedule to the user.

[1886] The schedule contents are displayed on the screen so that the user can check them.

[1887] Step 15:

[1888] The user checks the schedule and presses the "Confirm" button.

[1889] Check the contents of the proposed schedule and tap the "Confirm" button.

[1890] Step 16:

[1891] The server uses an emotion engine to analyze users' reactions and comments.

[1892] The emotion engine analyzes the reactions and comments entered by the user to identify the user's emotional state.

[1893] Step 17:

[1894] The server readjusts visit locations and schedules based on the analysis results of the emotion engine.

[1895] It generates a new schedule that takes into account the user's emotional state, and suggests a relaxing plan for a user who is feeling anxious, for example.

[1896] Step 18:

[1897] The server transmits the readjusted schedule to the terminal.

[1898] The adjusted schedule data is sent again to the terminal and the user is asked to confirm it.

[1899] Step 19:

[1900] The user reconfirms the schedule and presses the "Final Confirm" button.

[1901] Check the adjusted schedule and tap the "Final" button.

[1902] Step 20:

[1903] The server batches reservations for accommodation, transportation, and activities.

[1904] The reservation is confirmed through the booking site API and a reservation confirmation is generated.

[1905] Step 21:

[1906] The server sends a reservation confirmation to the terminal.

[1907] The generated reservation confirmation (PDF or HTML format) is sent to the device.

[1908] Step 22:

[1909] The terminal displays the reservation confirmation to the user and, if necessary, emails it.

[1910] Display the booking confirmation in the UI and provide an "Email" option.

[1911] Step 23:

[1912] The server periodically obtains real-time weather information and the status of tourist spots during the travel period.

[1913] Real-time data is obtained through weather information services and tourist destination APIs.

[1914] Step 24:

[1915] The server detects the fluctuations and generates alternatives using generative AI.

[1916] Generate new schedules based on changes in weather and traffic conditions, including considering the emotional state to incorporate using an emotion engine.

[1917] Step 25:

[1918] The server sends alternatives to the terminal.

[1919] The new schedule proposal is sent to the device and a notification is sent.

[1920] Step 26:

[1921] The terminal notifies the user of the alternatives.

[1922] Use popups and notifications to let users know about alternatives.

[1923] Step 27:

[1924] The user selects the desired language from the application settings.

[1925] Use the language settings option to select your preferred language.

[1926] Step 28:

[1927] The device will display the UI in the selected language.

[1928] The text is displayed according to the language you set.

[1929] Step 29:

[1930] The user presses the "Customer Support" button.

[1931] Tap on the "Customer Support" option.

[1932] Step 30:

[1933] The server provides customer support based on the user's inquiry.

[1934] Resolve user issues through chatbots and human support.

[1935] Example 2

[1936] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1937] Conventional travel planning systems generate schedules based on a user's desired travel destinations, duration, and activity information, but lack the flexibility to respond to user emotions or real-time information fluctuations. Furthermore, they do not support multiple languages, limiting their usability for global users. This makes it difficult for users to flexibly adjust plans to accommodate sudden changes during their trip or their individual emotional states.

[1938] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1939] In this invention, the server includes a means for analyzing the user's emotional state and readjusting the proposed content based on the emotion, a means for acquiring weather information and tourist spot conditions in real time and proposing changes to the travel schedule, and a means for supporting multiple languages, thereby enabling the provision of flexible travel plans that respond to the user's emotions and real-time changes in the situation.

[1940] A "user" is an end user who uses the travel planning system to select a travel destination and adjust a schedule.

[1941] "Destination" refers to a geographic location that a user wishes to visit.

[1942] "Duration" refers to the specific length of time that a user has set for planning a trip.

[1943] "Activities" refer to events or activities that a user wants to experience during their trip.

[1944] "Meal information" refers to information about the food and drink that the user desires to eat and drink during the trip.

[1945] "Map information service" refers to a third-party service that provides geographic information and calculates travel times between locations.

[1946] "Accommodation" refers to a facility where a user can stay while traveling.

[1947] "Transportation" refers to the vehicles and services used for transportation during a trip.

[1948] "Emotional state" refers to the psychological state a user feels about a travel plan or proposal.

[1949] "Real-time information" refers to the latest information at that time, such as weather information and the status of tourist attractions.

[1950] An "emotion engine" refers to software that analyzes user input and reactions to recognize emotions.

[1951] "Generative AI" refers to an artificial intelligence model that creates optimal travel schedules based on user input.

[1952] "Language support" refers to the ability to provide interfaces and notifications in the language selected by the user.

[1953] "Proposed Changes" means any proposed modifications or alternatives to your travel plans or schedule.

[1954] MODE FOR CARRYING OUT THE INVENTION

[1955] The present invention relates to an interactive travel concierge system that assists users in planning their trips, and in particular, combines an emotion engine that recognizes the user's emotions and optimizes the plan. Specific embodiments of this system will be described below.

[1956] Overall structure

[1957] This system consists of a user's device, a server, an emotion engine, and an external API service. The user's device is the device that runs the application, while the server is responsible for generating and optimizing travel plans, recognizing emotions, and acquiring external information. The emotion engine recognizes emotions by analyzing the user's input and reactions. Furthermore, by utilizing map information services and reservation service APIs, it is possible to smoothly proceed with each step of the travel plan.

[1958] Travel planning

[1959] First, the user launches the application and logs in. The device prompts for an ID and password, and the server authenticates them. Once authentication is complete, the device displays the main menu. The user selects the "Plan a Trip" option and enters information about their travel destination, duration, desired activities, and dining options. This information includes not only text data but also multimedia data such as images. The device then sends this information to the server. The server analyzes the entered information and uses a generative AI (e.g., GPT-4) to generate an optimal itinerary and itinerary. An example prompt might be, "The user is planning a three-day trip to Tokyo. He wants to visit Sensoji Temple and Tokyo Skytree, and eat sushi. Please suggest an optimal itinerary." The server uses the Google Maps API to calculate travel times between locations and optimize the overall itinerary. The server then generates the optimized itinerary and sends it to the device.

[1960] The device displays the proposed schedule to the user, who then confirms it. The emotion engine analyzes the user's reactions and comments when checking the schedule. The emotion engine recognizes the user's emotional state and sends the analysis results to the server. The server readjusts the proposed content based on the emotion engine's analysis results and sends the prompt text back to the generation AI to update the schedule.

[1961] Reservations and real-time information

[1962] Once the user confirms and confirms the proposed schedule, the server makes all reservations for accommodation, transportation, and activities. The server generates a reservation confirmation based on the reservation information and sends it to the terminal, which displays it to the user. If the user wishes, the confirmation can be sent by email or other means.

[1963] Furthermore, during the trip, the server periodically obtains real-time weather information and tourist destination information. It obtains the latest information using weather information services and tourist destination APIs. If changes in weather or traffic conditions are detected, the server uses a generative AI to generate alternatives and an emotion engine to generate alternatives that best fit the user's emotional state. The device then notifies the user.

[1964] Multilingual support and customer support

[1965] The user selects their preferred language in the application settings. The device displays the UI in the selected language. If the user needs assistance during their trip, they can press the "Customer Support" button, which the server will then handle via a chatbot or customer support representative.

[1966] Specific examples

[1967] For example, if a user plans a three-day trip to Tokyo, they launch the application, log in, and select the "Plan a Trip" option. They upload "Tokyo" as the travel destination, "3 days" as the duration, "Sensoji Temple" and "Tokyo Skytree" as places they want to visit, and "images of sushi" as food they want to eat. This information is sent to the server, which uses generative AI to generate an optimal travel itinerary. Using the Google Maps API, the server calculates the optimal route and travel time from Sensoji Temple to Tokyo Skytree. The server obtains reservation information for appropriate accommodation, transportation, and activities, and sends the optimal schedule to the device. When the user con...

Claims

1. A means for receiving information on the user's desired travel destination, duration, activities, and meals; means for generating optimal visiting locations and schedules based on the received information; A means for linking with a map information service to automatically calculate travel time; A one-stop solution for booking accommodation, transport and activities A means for obtaining real-time weather information and tourist spot conditions and proposing changes to the travel schedule; A means of supporting multiple languages; A system including:

2. The system of claim 1 , further comprising means for analyzing images uploaded by a user and suggesting places to visit and eat based on the analysis results.

3. means for providing a user with an interface for reviewing and confirming travel plans; 10. The system of claim 1, further comprising means for generating and transmitting a booking confirmation based on the confirmed schedule.

Images