System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting a destination and a schedule by a user, a means for generating a travel plan based on the inputted information, a means for executing reservation in cooperation with an external reservation system based on the generated travel plan, a means for receiving a question from the user during travel and providing an appropriate answer to the question, and a means for receiving trouble information during travel and taking an appropriate response to the trouble.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's travelers are troubled by the wide variety of travel destination options and the complexity of travel arrangements, and require an integrated system that can efficiently manage multiple requirements, such as creating travel plans, arranging reservations, and responding to questions and problems during the trip. However, existing systems can only handle these requirements individually, which lacks convenience for travelers. Therefore, it is important to improve traveler convenience by providing a consistent service from creating travel plans to making reservations and responding during the trip. [Means for solving the problem]

[0005] The present invention provides a means for users to input travel destinations and itineraries, a means for generating a travel plan based on the input information, and a means for making reservations in cooperation with an external reservation system based on the generated travel plan. It also includes a means for accepting questions from users during the trip and providing appropriate answers to those questions, and a means for receiving information about problems that occur during the trip and taking appropriate measures to resolve those problems. In this way, the system enables users to consistently create travel plans, make reservations, and respond to questions and problems during the trip, greatly improving convenience for travelers.

[0006] A "travel destination" is a geographic location that a traveler intends to visit.

[0007] An "itinary" is the start and end dates of a trip or a schedule of activities planned for a particular date.

[0008] "Input" is the act of a user providing information to a system.

[0009] "Travel Plan" means the overall travel schedule and arrangements, including destinations, dates, accommodations, transportation, etc.

[0010] "Generation" is the process of creating new data or plans based on input information.

[0011] "External reservation system" refers to an external system for making reservations for accommodations, transportation, etc. based on travel plans.

[0012] "Functioning" refers to the interaction between systems to exchange information and perform specific functions or services.

[0013] "Reservation" means the act of a user reserving services such as accommodation, transportation, and restaurants in advance.

[0014] A "question" is a query that a user makes to the system seeking information or advice.

[0015] An "answer" is information or advice that the system provides in response to a user's question.

[0016] "Trouble" refers to unforeseen circumstances or problems that arise during travel.

[0017] "Accept" refers to the system's act of accepting information or requests from users.

[0018] "Response" refers to the system's actions of providing appropriate support or assistance in response to a user's problem or request. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, links with external reservation systems, and responds to questions and troubles during the trip. The system of the present invention integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[0041] Travel planning assistance

[0042] The user inputs the travel destination and the dates of the visit. The terminal receives the user's input and sends the information to the server. The server receives the information, generates a travel plan, and saves it in a database. For example, if the user plans to visit "Paris" and "London" on dates "2023-12-20" and "2023-12-25," the server will create a travel plan based on that and provide it to the user.

[0043] API integration with external reservation systems

[0044] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information, confirming the reservation. For example, if a user wants to reserve a hotel in Paris from 2023-12-20 to 2023-12-25, the server sends the reservation information to the external reservation system, retrieves the results, and completes the reservation.

[0045] QA support

[0046] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device sends the question to the server, which then provides an appropriate answer. For example, if a user asks, "What restaurant do you recommend?", the server will recommend a restaurant called "Gourmet Delight."

[0047] Troubleshooting

[0048] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. For example, if the user's flight is canceled, the server will initiate the process of arranging a new flight and notify them, "Your flight has been canceled. We are arranging a new flight..."

[0049] This invention enables users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[0050] The processing flow will be explained below.

[0051] Travel planning assistance

[0052] Step 1:

[0053] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[0054] Step 2:

[0055] The device sends the user's input in JSON format to the server.

[0056] Example of transmitted data:

[0057] json

[0058] {

[0059] "user_id": "12345",

[0060] "destinations": ["Paris", "London"],

[0061] "dates": ["2023-12-20", "2023-12-25"]

[0062] }

[0063] Step 3:

[0064] The server generates a travel plan based on the received data and stores it in a database. For example, it stores each destination and visit date as follows:

[0065] python

[0066] self.plan = [{"destination": "Paris", "date": "2023-12-20"}, {"destination": "London", "date": "2023-12-25"}]

[0067] API integration with external reservation systems

[0068] Step 1:

[0069] The user inputs reservation information. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[0070] Step 2:

[0071] The device sends the reservation information to the server in JSON format.

[0072] Example of transmitted data:

[0073] json

[0074] {

[0075] "user_id": "12345",

[0076] "destination": "Paris",

[0077] "check_in": "2023-12-20",

[0078] "check_out": "2023-12-25"

[0079] }

[0080] Step 3:

[0081] The server receives the reservation information and sends a POST request to the external reservation system API.

[0082] Example API request:

[0083] python

[0084] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[0085] Step 4:

[0086] The server receives the response from the API and notifies the user of the reservation result.

[0087] QA support

[0088] Step 1:

[0089] The user types a question into the terminal, for example, "must_visit" (what are the must-see spots?).

[0090] Step 2:

[0091] The terminal sends a question to the server.

[0092] Example of transmitted data:

[0093] json

[0094] {

[0095] "user_id": "12345",

[0096] "question": "must_visit"

[0097] }

[0098] Step 3:

[0099] The server receives the question and provides the appropriate answer.

[0100] Example answer:

[0101] python

[0102] answers = {

[0103] "best_restaurant": "The best restaurant in town is 'Gourmet Delight'.",

[0104] "must_visit": "You must visit the 'Historic Cathedral'."

[0105] }

[0106] answer = answers.get(question, "Sorry, I don't have an answer for that.")

[0107] Step 4:

[0108] The server sends the answer in JSON format to the device, which displays it to the user.

[0109] Troubleshooting

[0110] Step 1:

[0111] The user inputs a problem into the terminal, for example, "flight_cancelled" (the flight was canceled).

[0112] Step 2:

[0113] The device sends the trouble information to the server.

[0114] Example of transmitted data:

[0115] json

[0116] {

[0117] "user_id": "12345",

[0118] "issue": "flight_cancelled"

[0119] }

[0120] Step 3:

[0121] The server receives the trouble information and takes appropriate action.

[0122] Examples of support:

[0123] python

[0124] if issue == 'flight_cancelled':

[0125] response = "Your flight has been canceled. Booking a new flight for you..."

[0126] elif issue == 'need_hotel_extension':

[0127] response = "Booking an extension for your hotel stay..."

[0128] else:

[0129] response = "Sorry, I cannot handle that issue."

[0130] Step 4:

[0131] The server transmits the correspondence result to the terminal, which displays it to the user.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Modern travelers need multiple services, such as creating travel plans, booking accommodations and transportation, and answering questions and dealing with problems while traveling, but managing these tasks individually is extremely cumbersome. Conventional systems are unable to provide centralized support for these tasks, placing a heavy burden on travelers, which is a problem.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for receiving questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to deal with those problems, and means for using a generative AI model to generate the travel plan and handle the problems. This allows travelers to centrally manage their travel plans and receive the services they need smoothly and quickly.

[0137] "User" means any individual or group of people who use the Travel Planning System.

[0138] "Destination" means a geographic location that you intend to visit.

[0139] "Dates" refers to the period of time for which the user plans to travel.

[0140] "Travel Plan" refers to an itinerary or schedule created based on a User's travel destination and dates.

[0141] "External Reservation System" refers to an external online system used to make reservations such as hotel or transportation reservations.

[0142] An "API" is something that provides an interface for different software systems to communicate with each other.

[0143] "Generative AI models" refer to algorithms and techniques that use artificial intelligence to generate optimal responses and plans from data.

[0144] "Push transmission" refers to a technology in which a server sends data or notifications directly to a user's device.

[0145] "Trouble Information" refers to information about problems or obstacles that users encounter during their trip.

[0146] "Appropriate response" refers to the optimal measures or solutions that the system takes in response to trouble information.

[0147] This is an integrated system that starts when a user inputs their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. This system integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[0148] Specifically, the system is implemented as follows.

[0149] Travel planning assistance

[0150] The user uses the device to input the travel destination and visit dates. If the user plans to visit "Paris" and "London" and the dates are "2023-12-20" and "2023-12-25," the user enters this information into the input form on the device. The device receives this information and sends it to the server using the HTTPS protocol. The server uses the generative AI model to create an optimal travel plan and stores it in a database. The generated travel plan is provided to the user via the device. For example, the server may suggest, "In Paris, we recommend visiting the Louvre Museum and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[0151] API integration with external reservation systems

[0152] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information from their device. For example, if they want to reserve a hotel in "Paris" from "2023-12-20" to "2023-12-25," they enter this information into an input form. The device sends that information to the server, and the server sends an API request to an external reservation system based on the reservation information. For example, the reservation is made using the API of an external reservation system (e.g., Booking.com). At this time, the server receives a reservation confirmation response from the external system and notifies the user of the result via the device. As a specific example, information such as "The hotel reserved is Hôtel Le Meurice" is provided.

[0153] QA support

[0154] While traveling, the user inputs questions into the device about recommended facilities and restaurants at the destination. For example, "What restaurant do you recommend?" The device sends the question to the server, which uses a generative AI model to generate an appropriate answer. For example, a recommendation for a restaurant called "Gourmet Delight" is generated. The answer generated by the server is provided to the user via the device.

[0155] Troubleshooting

[0156] If a user encounters a problem during their trip, they report the problem from their device. For example, if their flight is canceled, they enter "Flight canceled." The device then sends this trouble information to the server. The server then takes appropriate action, such as arranging a new flight, based on the trouble information. The server then notifies the user of the progress and results of the action via the device. For example, the user may be notified that "Your flight has been canceled. Arranging a new flight..."

[0157] Examples of prompt statements

[0158] Travel planning assistance:

[0159] Input: "Destinations are Paris and London, visit dates are 2023-12-20 to 2023-12-25. Please make travel plans."

[0160] Output: "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[0161] API integration with external booking systems:

[0162] Input: "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25."

[0163] Output: "The hotel booked is Hôtel Le Meurice."

[0164] QA Support:

[0165] Input: "What are the best restaurants in Paris?"

[0166] Output: "The restaurant I recommend is Gourmet Delight."

[0167] Troubleshooting:

[0168] Enter: "Flight canceled. Please find a new flight."

[0169] Output: "Your flight has been cancelled. We are arranging a new flight..."

[0170] As described above, this invention allows users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1: User enters travel information

[0173] Description: The user enters a travel destination and visit dates into the terminal.

[0174] Input: Travel destination (e.g., "Paris" or "London"), visit date (e.g., "2023-12-20" to "2023-12-25").

[0175] Specific operation: The user enters the travel destination and itinerary into the input form on the device screen. The device accurately records the entered data.

[0176] Step 2: Submit your trip information

[0177] Description: The device sends travel information to the server.

[0178] Input: User-entered travel destination and visit date information.

[0179] Output: JSON formatted travel information data.

[0180] What happens: The device converts the user's input into JSON format and sends it to the server's API endpoint using the HTTPS protocol.

[0181] Step 3: Generate a travel plan

[0182] Description: Based on the travel information received by the server, a generative AI model is used to generate a travel plan.

[0183] Input: JSON-formatted trip information data sent from the device.

[0184] Output: The generated itinerary.

[0185] Specific operation: The server extracts the travel destinations and dates from the received data and sends them as prompts to the generative AI model (e.g., "The travel destinations are Paris and London, and the visit dates are from 2023-12-20 to 2023-12-25. Please create a travel plan."). The plan obtained from the model is saved in a database.

[0186] Step 4: Provide your travel plans

[0187] Description: The server provides the travel plan generated by the server to the user via the terminal.

[0188] Input: The generated itinerary.

[0189] Output: Display of travel plan.

[0190] Specific operation: The server retrieves the generated travel plan from the database and sends it to the device. The device displays the received plan on the screen (e.g., "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace.").

[0191] Step 5: Enter reservation information

[0192] Description: A user enters accommodation and transportation reservation information into a device.

[0193] Input: The facility and period you want to book (e.g. "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25").

[0194] Specific operation: The user accesses the reservation form on the device and enters the reservation information.

[0195] Step 6: Submit your reservation information

[0196] Description: The terminal sends reservation information to the server.

[0197] Input: Booking information entered by the user.

[0198] Output: Reservation information data formatted in JSON.

[0199] Specific operation: The terminal converts the reservation information into JSON format and sends it to the server via HTTPS protocol.

[0200] Step 7: API Request to External System

[0201] Description: The server sends an API request to an external reservation system based on the reservation information.

[0202] Input: Reservation information data in JSON format sent from the terminal.

[0203] Output: The API request sent to the external booking system.

[0204] Specific operation: The server analyzes the received data and sends an HTTP POST request to the external reservation system (e.g., the API of the reservation site).

[0205] Step 8: Confirmation and notification of booking

[0206] Description: The server receives the response from the external system and notifies the user of the reservation result.

[0207] Input: Booking confirmation response from the external system.

[0208] Output: Display of reservation confirmation information.

[0209] Specific behavior: The server receives the response from the external system and sends it in JSON format to the terminal. The terminal displays the received information on the user interface and notifies the user that the reservation has been confirmed (e.g., "The reserved hotel is Hôtel Le Meurice.").

[0210] Step 9: Enter your question

[0211] Description: User types questions into terminal while traveling.

[0212] Input: The question (e.g., "What restaurants do you recommend?").

[0213] Specific behavior: The user types a question into the chatbot interface on the device.

[0214] Step 10: Submit your question

[0215] Description: The device sends a question to the server.

[0216] Input: The question typed by the user.

[0217] Output: Question data formatted in JSON.

[0218] Specific operation: The device converts the question into JSON format and sends it to the server via HTTPS protocol.

[0219] Step 11: Generate an answer

[0220] Description: The server uses a generative AI model to generate an appropriate answer based on the question.

[0221] Input: Question data sent from the terminal.

[0222] Output: The generated answer.

[0223] How it works: The server sends the question as a prompt to the generative AI model and receives an answer from the model (e.g., "The restaurant I recommend is Gourmet Delight.").

[0224] Step 12: Provide your answers

[0225] Description: The server provides the generated answer to the user via the terminal.

[0226] Input: The generated answer.

[0227] Output: Display of the answer to the question.

[0228] Specific operation: The server sends the generated answer to the device, and the device displays the received information on the screen.

[0229] Step 13: Report a problem

[0230] Description: Allows users to report any trouble they encounter while traveling from their device.

[0231] Input: Problem description (e.g. "My flight has been cancelled").

[0232] Specific actions: The user accesses the device's reporting form and enters specific trouble information.

[0233] Step 14: Submit your trouble report

[0234] Description: The device sends trouble information to the server.

[0235] Input: Trouble information entered by the user.

[0236] Output: Trouble information data formatted in JSON format.

[0237] Specific operation: The device converts the trouble information into JSON format and sends it to the server via HTTPS protocol.

[0238] Step 15: Initiate response

[0239] Description: The server will take appropriate action based on the trouble information.

[0240] Input: Trouble information data sent from the device.

[0241] Output: A request to start the response process.

[0242] Specific behavior: The server analyzes the received trouble information and sends a request to an external system to initiate appropriate action, such as arranging a new flight.

[0243] Step 16: Notification of troubleshoot

[0244] Description: The server notifies the user of the progress of the response.

[0245] Input: Progress of the response.

[0246] Output: Displays troubleshooting progress information.

[0247] Specific behavior: The server sends progress information in JSON format to the device, and the device displays the received information in the user interface (e.g., "Your flight has been canceled. Arranging a new flight...").

[0248] (Application example 1)

[0249] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0250] Conventional travel support systems handle the creation of travel plans, reservations, answering questions, and troubleshooting individually, making it difficult for users to centrally manage everything, and there was a problem of a lack of convenience, especially when using meal services at destinations, as users had to go through a separate platform.In addition, there was also the problem that recommendations for meal services at travel destinations and reservation management were not integrated, which increased the effort required to make meal arrangements during travel.

[0251] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0252] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for accepting questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to address those problems, means for recommending meal services at locations of stay based on the travel plan, and means for managing reservations for the recommended meal services. This allows users to perform everything from creating a travel plan to answering questions and responding to them during the trip, and even arranging meals at locations of stay, all within the system.

[0253] "User" refers to an individual who creates or changes travel plans, asks questions, or reports problems.

[0254] The "means for inputting travel destination and itinerary" is an interface for inputting the travel destination, departure date, and return date.

[0255] "Means for generating travel plans" refers to a system that has the function of automatically creating a travel itinerary and detailed schedule based on inputted travel destinations and dates.

[0256] "Means for executing reservations in cooperation with external reservation systems" refers to a system that communicates with an external reservation platform to complete reservations for accommodation, transportation, etc. based on the contents of travel plans.

[0257] "Means of accepting questions and providing appropriate answers to those questions" refers to a system that responds to various questions from users while traveling and provides answers using pre-registered data or external databases.

[0258] "Means for receiving trouble information and taking appropriate measures to resolve the trouble" refers to a system that receives from users information about problems or troubles that occur during their trip and provides appropriate measures to resolve the trouble.

[0259] The "means for recommending local food services" is a system that has the function of suggesting to users food services available in places to be visited based on a travel plan.

[0260] "Means for managing reservations for meal services" refers to a system that records and manages reservations made by users for recommended meal services, and makes changes or cancellations as necessary.

[0261] An "API integration means" is a component that provides a common interface for different systems to communicate with each other, enabling the sending and receiving of information.

[0262] "Means for sending push notifications of service cancellations" refers to a system that has the function of immediately notifying users of sudden service cancellations during travel to their devices.

[0263] To implement this invention, the travel support system must be configured with a server and terminals. The server comprehensively manages various functions, such as generating travel plans, linking with external reservation systems, answering questions, dealing with problems, recommending meal services, and managing reservations. The terminal functions as an interface for users to input travel destinations and itineraries and to make inquiries and respond to requests during the trip.

[0264] Program Generation

[0265] The program for this system must have the following functions:

[0266] First, we provide an interface for users to input their travel destination and itinerary. This interface is implemented as a smartphone application. For example, it includes a function to input "Paris" as the travel destination and "2023-12-20 to 2023-12-25" as the itinerary.

[0267] The server then generates a travel plan based on the input information by analyzing the received data and extracting information such as optimal travel routes, tourist spots, and accommodations from a database to generate the travel plan.

[0268] In addition, by linking with an external reservation system, reservations are made based on the generated travel plans. Specifically, the server sends requests to the external reservation service API to complete reservations for accommodations and transportation.

[0269] During the trip, the server will provide a function to accept questions from users, such as "What are the best restaurants in Paris?", and will refer to the database to provide an appropriate answer.

[0270] Furthermore, the system receives information about troubles that occur during travel and takes appropriate measures to resolve them. For example, if a flight is canceled, the server will start arranging a new flight and notify the user's device, "Your flight has been canceled. Arranging a new flight..."

[0271] Newly added features include a way to recommend dining options based on travel plans. The server extracts popular restaurants and delivery services from a database and suggests them to users. Once a user selects a restaurant, the server manages the reservation.

[0272] For example, if a user wants to order "Pasta Carbonara" and "Tiramisu" in "Paris," the server connects to an external food delivery service API and confirms the order.

[0273] Example prompt sentence:

[0274] Enter your travel plan data.

[0275] Destination: Paris

[0276] Start date: 2023-12-20

[0277] End date: 2023-12-25

[0278] Place a food delivery order from a specific restaurant.

[0279] Restaurant ID: 456

[0280] Order Details:

[0281] Pasta Carbonara (2)

[0282] Tiramisu (1)

[0283] Delivery address: 123 Champs-Élysées, Paris

[0284] As described above, the system of the present invention links the server and terminals to provide consistent support from planning to execution of the trip, troubleshooting, answering questions, and arranging meal services. This allows travelers to use many functions in a unified manner, greatly improving the convenience of the entire trip.

[0285] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0286] Step 1:

[0287] The user inputs the travel destination and dates.

[0288] As input, the user enters the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20 to 2023-12-25").

[0289] The terminal receives this information and sends it to the server. As output, the travel destination and itinerary data is sent to the server.

[0290] Step 2:

[0291] The server generates a travel plan based on the received information.

[0292] As input, the server receives destination and itinerary data.

[0293] The server references the database and generates a travel plan by combining the best sightseeing routes, accommodations, and means of transportation.

[0294] As an output, the generated itinerary is stored in a database and provided to the user.

[0295] Step 3:

[0296] Execute reservations in conjunction with an external reservation system.

[0297] As input, the server uses the generated itinerary data.

[0298] The server sends requests to an external reservation system API to make reservations for accommodation and transportation.

[0299] As an output, the reservation completion information is returned to the server and stored in a database.

[0300] Step 4:

[0301] Accept questions from users during their trip and provide appropriate answers.

[0302] As input, the user types a question through the terminal (e.g., "What are some recommended restaurants in Paris?").

[0303] The terminal sends a query to the server.

[0304] The server consults the database and extracts the appropriate answer (e.g., "a restaurant called Gourmet Delight") from the data.

[0305] As an output, the answer is displayed on the user's terminal.

[0306] Step 5:

[0307] Receive information about troubles that may occur during travel and take appropriate action.

[0308] As input, the user enters trouble information (e.g., "flight canceled") into the terminal.

[0309] The terminal transmits the trouble information to the server.

[0310] The server analyzes the trouble information and initiates procedures to take appropriate action (e.g., arranging a new flight).

[0311] As an output, the progress of troubleshooting is notified to the user's terminal.

[0312] Step 6:

[0313] Recommend food services at your destination based on your travel plans.

[0314] As input, the server uses travel plan data to extract meal services from a database that are tailored to the user's location and time of stay.

[0315] The server sends recommended restaurants and delivery services to the user's device.

[0316] As an output, recommended meal services are displayed on the user's device.

[0317] Step 7:

[0318] Manage reservations for recommended meal services.

[0319] As input, the user selects which of the recommended meal services they wish to book (e.g., ordering "Pasta Carbonara" and "Tiramisu").

[0320] The terminal transmits the order information to the server.

[0321] The server connects to an external food delivery service API and sends a request to confirm the order.

[0322] As an output, information about the completion of the order is returned to the server and displayed on the user's terminal.

[0323] The above are the specific processing steps of this system.

[0324] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0325] This is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions and adjusts services based on those emotions, user satisfaction can be further improved.

[0326] Travel planning support and emotion recognition

[0327] The user inputs their travel destination and visit date. For example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively. The device sends the user's input in JSON format to the server. The server generates a travel plan based on the received data and saves it in a database. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice when they enter their information, and adjusts the travel plan based on those emotions. For example, if the user indicates that they want to "relax," it will suggest relaxing tourist spots and activities.

[0328] API integration with external reservation systems and emotion recognition

[0329] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, the emotion engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation result notification based on those emotions. For example, if the user is in an "anxious" state, the reservation confirmation email will include additional support information.

[0330] QA and Emotion Recognition

[0331] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device then sends the question to the server, which then provides an appropriate answer. The emotion engine recognizes the emotion the user expressed when entering the question and adjusts the answer accordingly. For example, if the user is "excited," it will suggest special events or activities.

[0332] Troubleshooting and Emotion Recognition

[0333] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. At this time, the emotion engine recognizes the user's emotion when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a quick and detailed response will reduce the user's dissatisfaction.

[0334] This invention makes it possible to recognize users' emotions in real time and adjust service content based on those emotions. This system provides consistent support from planning a trip to responding during the trip, and by combining it with an emotion engine, it can significantly improve traveler satisfaction and reliability.

[0335] The processing flow will be explained below.

[0336] Travel planning support and emotion recognition

[0337] Step 1:

[0338] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[0339] Step 2:

[0340] The device sends the user input information to the server in JSON format.

[0341] Example of transmitted data:

[0342] json

[0343] {

[0344] "user_id": "12345",

[0345] "destinations": ["Paris", "London"],

[0346] "dates": ["2023-12-20", "2023-12-25"]

[0347] }

[0348] Step 3:

[0349] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice when inputting. For example, it analyzes whether the user is expressing the emotion "I want to relax."

[0350] Step 4:

[0351] The server generates a travel plan based on the recognized emotion and the input information. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots.

[0352] Step 5:

[0353] The server stores the generated travel plan in a database and sends it to the terminal, which displays the plan to the user.

[0354] API integration with external reservation systems and emotion recognition

[0355] Step 1:

[0356] The user inputs reservation information for accommodation and transportation. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[0357] Step 2:

[0358] The device sends the reservation information in JSON format to the server.

[0359] Example of transmitted data:

[0360] json

[0361] {

[0362] "user_id": "12345",

[0363] "destination": "Paris",

[0364] "check_in": "2023-12-20",

[0365] "check_out": "2023-12-25"

[0366] }

[0367] Step 3:

[0368] The server receives the reservation information and sends a POST request to the external reservation system API.

[0369] Example API request:

[0370] python

[0371] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[0372] Step 4:

[0373] The server uses an emotion engine to recognize the user's emotions during the reservation process, for example, analyzing whether the user is in an "anxious" state during the reservation process.

[0374] Step 5:

[0375] The server tailors the booking outcome based on the perceived emotion, for example including additional support information in the booking confirmation email if the user indicates "anxiety."

[0376] Step 6:

[0377] The server sends the reservation results in JSON format to the terminal, and the terminal notifies the user.

[0378] QA and Emotion Recognition

[0379] Step 1:

[0380] The user inputs questions about recommended facilities and restaurants at the destination into the terminal. For example, the user inputs the question "must_visit" (what are the must-see spots?).

[0381] Step 2:

[0382] The device sends the question in JSON format to the server.

[0383] Example of transmitted data:

[0384] json

[0385] {

[0386] "user_id": "12345",

[0387] "question": "must_visit"

[0388] }

[0389] Step 3:

[0390] The server receives the question and uses an emotion engine to recognize the user's emotion when entering the question. For example, it analyzes whether the user is "excited."

[0391] Step 4:

[0392] The server generates appropriate responses and tailors them based on the emotion, for example, suggesting special events or activities if the user is "excited."

[0393] Step 5:

[0394] The server sends the answer in JSON format to the device, which displays it to the user.

[0395] Troubleshooting and Emotion Recognition

[0396] Step 1:

[0397] The user enters the problem into the terminal, for example, "flight_cancelled."

[0398] Step 2:

[0399] The device sends trouble information to the server in JSON format.

[0400] Example of transmitted data:

[0401] json

[0402] {

[0403] "user_id": "12345",

[0404] "issue": "flight_cancelled"

[0405] }

[0406] Step 3:

[0407] The server receives the trouble information and uses an emotion engine to recognize the user's emotion when reporting the trouble. For example, it analyzes whether the user is expressing anger.

[0408] Step 4:

[0409] The server responds appropriately based on the recognized emotion, for example, if the user expresses anger, it responds quickly and in detail.

[0410] Step 5:

[0411] The server sends the response results in JSON format to the device, which then displays them to the user.

[0412] Example 2

[0413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] Conventional travel support systems have difficulty recognizing users' emotions in real time and adjusting service content based on those emotions. Furthermore, there is a need for a method to sufficiently improve user satisfaction in travel planning, linking with external reservation systems, and responding to questions and problems during the trip.

[0415] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for transmitting the input information in structured data format; a means for analyzing emotions based on the structured data and generating a travel plan; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for analyzing the user's emotions during the reservation process and adjusting notification content based on the emotions; a means for receiving questions from the user during the trip and providing appropriate answers to the questions; and a means for receiving information about problems that occur during the trip, taking appropriate measures to deal with the problems, and adjusting the response method based on the user's emotions. This makes it possible to recognize the user's emotions in real time and adjust service content based on the emotions.

[0416] "User" refers to an individual who uses the System to make travel plans, make reservations, ask questions, or report problems.

[0417] "Destination" refers to a geographic location that a user wishes to visit.

[0418] "Itinerary" refers to time information including the start and end dates of a trip or the duration of stay at each location visited.

[0419] A "structured data format" refers to a data format that represents information in a regular and unified format, making it easier to process within databases and information systems.

[0420] "Emotion" refers to the psychological state or reaction that a user shows toward a system, such as feelings of relaxation, excitement, or anxiety.

[0421] "Travel Itinerary" means a series of suggestions based on your travel destination and dates, including attractions, activities, and accommodations.

[0422] "Third-party booking systems" refers to online services provided by other companies for booking accommodation and transportation.

[0423] "API integration" refers to the use of application program interfaces to share data and functions between different software systems.

[0424] "Question" refers to an inquiry made by a user to the system to obtain information or ask for advice.

[0425] "Trouble Information" refers to information that users use to report to the system any problems or difficulties they encounter during their trip.

[0426] "Notification" refers to information or messages sent by the system to the user.

[0427] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with external reservation systems, and responds to questions and troubles during the trip. It also has an emotion engine that recognizes the user's emotions and adjusts services accordingly.

[0428] In order to implement the present invention, the following hardware and software are used.

[0429] Hardware:

[0430] Terminal: The device on which the user enters information (e.g., smartphone, tablet, PC)

[0431] Server: A host computer used to process and store data and run the emotion engine.

[0432] Database: a storage system for storing travel plans, reservation information, etc.

[0433] software:

[0434] Emotion engine: Software for recognizing user emotions and adjusting services

[0435] API with external booking systems: Interface for booking accommodations, transportation, etc.

[0436] Question-answering system: An AI model for answering questions from users

[0437] Specific examples of the present invention are given below.

[0438] Example 1:

[0439] A user inputs into the device that they are traveling to "Paris" and departing on "2023-12-20." The device converts this information into JSON format and sends it to the server. The server receives this information and uses an emotion engine to analyze the emotion of the user at the time of input. For example, if the emotion "I want to relax" is detected, the server creates a travel plan that suggests relaxing tourist spots and saves it in the database.

[0440] Example prompt: "The user wants to visit Paris and London and is looking for a relaxing trip."

[0441] Example 2:

[0442] The user enters the accommodation "Hotel A" and "2023-12-20~2023-12-25" and the transportation method "Flight B" into the terminal. The terminal sends this information to the server. The server sends an API request to an external reservation system based on the information and confirms the reservation. Furthermore, if the user expresses "anxiety," the server includes additional support information in the reservation confirmation email and sends it.

[0443] Example 3:

[0444] While traveling, a user can type "What are some good restaurants in this area?" into their device, which then sends the question to the server. The server generates an answer to the question and, if it determines that the user is "excited," suggests special events or activities.

[0445] Example 4:

[0446] When a problem occurs during a trip and the user types "my flight is delayed" into the device, the device sends the problem information to the server. The server responds quickly and in detail based on this information, and if the user shows anger, it further adjusts the response method and notifies the user.

[0447] In this way, a specific embodiment of the present invention can recognize a user's emotions in real time and adjust travel plans and various responses based on those emotions, thereby significantly improving user satisfaction.

[0448] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0449] Step 1:

[0450] The user inputs the travel destination and dates

[0451] Specific operation: The user uses the terminal to input the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20").

[0452] Input: Travel destination and itinerary information

[0453] Output: Structured data of destinations and dates

[0454] Step 2:

[0455] The terminal converts the input content into JSON format and sends it.

[0456] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"destination": "Paris", "date": "2023-12-20"}).

[0457] Input: Structured data of destination and dates

[0458] Output: JSON format data

[0459] Step 3:

[0460] The server receives the data and analyzes it with the emotion engine.

[0461] Specific operation: The server receives the JSON data sent from the device and analyzes the user's emotions using the emotion engine. For example, it detects the emotion "I want to relax."

[0462] Input: JSON format data

[0463] Output: Sentiment analysis results

[0464] Step 4:

[0465] The server generates the itinerary and stores it in the database.

[0466] Specific operation: The server generates a travel plan based on the results of the sentiment analysis, for example, a plan including relaxing tourist spots and activities, and stores it in a database.

[0467] Input: Sentiment analysis results

[0468] Output: Travel plan data

[0469] Step 5:

[0470] The user enters accommodation and transportation reservations

[0471] Specific operation: The user uses the terminal to enter information about accommodation (e.g., "Hotel A") and transportation (e.g., "Flight B").

[0472] Input: Accommodation and transportation information

[0473] Output: Accommodation and transportation structured data

[0474] Step 6:

[0475] The device sends the reservation information to the server.

[0476] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"hotel": "Hotel A", "flight": "Flight B"}).

[0477] Input: Accommodation and transportation structured data

[0478] Output: JSON format data

[0479] Step 7:

[0480] The server sends an API request to the external booking system

[0481] Specific operation: The server sends an API request to the external reservation system based on the reservation information and confirms the reservation.

[0482] Input: Reservation information in JSON format

[0483] Output: Reservation confirmation result

[0484] Step 8:

[0485] The server adjusts and notifies the reservation result based on emotions.

[0486] Specific operation: The server receives the reservation confirmation result and uses the emotion engine to analyze the user's current emotion. For example, if the user is in an "anxious" state, the server sends the reservation confirmation email to the user with additional support information.

[0487] Input: Booking confirmation results, sentiment analysis

[0488] Output: Adjusted booking confirmation

[0489] Step 9:

[0490] The user enters a question

[0491] Specific operation: The user uses the device to input questions such as recommended facilities and restaurants at the destination.

[0492] Input: Question content

[0493] Output: Question structured data

[0494] Step 10:

[0495] The device sends a question to the server

[0496] Specific operation: The device sends the question content to the server in JSON format (e.g., "{"question": "What are some good restaurants in this area?"}).

[0497] Input: Question structured data

[0498] Output: Question data in JSON format

[0499] Step 11:

[0500] The server generates the answer and adjusts it based on the emotion.

[0501] What it does: The server generates answers to questions and uses an emotion engine to tailor the content based on the user's emotions. If the user is "excited," it suggests special events and activities.

[0502] Input: JSON format question data, sentiment analysis

[0503] Output: Adjusted answer notification

[0504] Step 12:

[0505] A user reports a problem

[0506] Specific operation: The user uses the terminal to input trouble information (e.g., "The flight is delayed").

[0507] Input: Trouble information

[0508] Output: Structured data of trouble information

[0509] Step 13:

[0510] The device sends trouble information to the server

[0511] Specific operation: The device sends trouble information to the server in JSON format (e.g., {"trouble": "The flight was delayed"}).

[0512] Input: Structured data of trouble information

[0513] Output: Trouble information in JSON format

[0514] Step 14:

[0515] The server adjusts and notifies the response method based on the emotion.

[0516] Specific operation: The server takes appropriate action based on the trouble information it receives. If the emotion engine recognizes that the user's emotion is "anger," it takes a swift and detailed action, adjusts the response method, and notifies the user.

[0517] Input: JSON format trouble information, sentiment analysis

[0518] Output: Coordinated response notification

[0519] (Application example 2)

[0520] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0521] Conventional travel plan generation systems do not take into account user emotions and lack full support during travel, especially in autonomous vehicles. Therefore, it is important to reduce the anxiety and frustration users feel during travel and improve overall satisfaction with the travel experience.

[0522] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for generating a travel plan based on the input information; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for accepting questions from the user during the travel and providing appropriate answers to the questions; a means for receiving information about troubles during the travel and taking appropriate measures to deal with the troubles; a means for recognizing the user's emotions using an emotion recognition engine and adjusting service content based on the emotions; and a means for coordinating with the navigation system of an autonomous vehicle to provide route guidance based on the travel plan. This makes it possible to provide a high level of satisfaction that takes the user's emotions into consideration, from travel planning to responses during the trip.

[0523] The "means for inputting travel destinations and itineraries" is an interface for the user to input the places he or she wishes to visit and the dates on which he or she will visit those places.

[0524] The "means for generating a travel plan" is a system for automatically creating a plan based on the travel destination and itinerary input by the user.

[0525] The "means of executing reservations in cooperation with external reservation systems" is a function of exchanging information with other external systems in order to reserve accommodations and transportation based on travel plans.

[0526] "Means of accepting questions and providing appropriate answers to those questions" refers to a function that quickly responds to questions and information requests that users have while traveling and provides appropriate information.

[0527] "A means of receiving trouble information and taking appropriate action against the trouble" is a system that proposes and responds to problems and difficulties that arise during travel.

[0528] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0529] "Means for recognizing the user's emotions and adjusting the service content based on those emotions" refers to a function that uses an emotion recognition engine to understand the user's emotions and optimize the service according to that state.

[0530] "Means for linking with the navigation system of an autonomous vehicle and providing route guidance based on a travel plan" refers to a system that communicates with the navigation system installed in an autonomous vehicle and provides guidance on the optimal route based on the generated travel plan.

[0531] Travel planning support and emotion recognition

[0532] The user first enters their travel destination and visit dates. For example, suppose the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The user's device sends this information in JSON format to the server. The server generates a travel plan based on the received data and stores it in a database. Furthermore, an emotion recognition engine recognizes emotions from the user's facial expressions and voice when entering information, and adjusts the travel plan based on those emotions. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots and activities.

[0533] API integration with external reservation systems and emotion recognition

[0534] When a user makes a reservation for accommodation or transportation, they input the reservation information on their device. The device then sends that information to a server, which then sends an API request to an external reservation system based on the reservation information to confirm the reservation. An emotion recognition engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation notification based on those emotions. For example, if the user is anxious, the reservation confirmation email can include additional support information.

[0535] QA and Emotion Recognition

[0536] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the content of the answer based on that emotion. For example, if the user is excited, it can suggest special events or activities.

[0537] Troubleshooting and Emotion Recognition

[0538] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. At this time, an emotion recognition engine recognizes the user's emotions when reporting the problem and adjusts the response method based on those emotions. For example, if the user expresses anger, a quick and detailed response can be provided to reduce the user's dissatisfaction.

[0539] Collaboration with autonomous vehicles

[0540] This system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary, and can adjust in real time based on the user's emotions by providing the navigation system with the best tourist spots and routes at each point during the trip.

[0541] Hardware and software used

[0542] 1. Hardware:

[0543] Camera: Recognizes the user's facial expressions.

[0544] Microphone: Recognizes the user's voice.

[0545] Display: Shows information.

[0546] Autonomous driving system: Provides route guidance based on trip plans.

[0547] 2. Software:

[0548] Emotion recognition engine: Analyzes emotions from facial expressions and voice.

[0549] Database: Stores travel planning and booking information.

[0550] API: Integrates with external reservation systems.

[0551] Examples of concrete examples and prompts

[0552] Travel planning prompt:

[0553] The user's travel destinations are Tokyo and Osaka, and the visit dates are December 20th to December 25th, 2023. This indicates a desire to relax. Please suggest a relaxing travel plan.

[0554] Booking anxiety detection prompt:

[0555] The user is booking a triple room at a hotel and is expressing anxiety. Please generate a booking confirmation email with additional support information.

[0556] This will make it possible to provide services that take into account the user's emotions at each stage of travel planning and troubleshooting.

[0557] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0558] Step 1:

[0559] The user enters the travel destination and itinerary. For example, the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The device sends the entered information to the server in JSON format. The data format of the entered information is JSON data as shown below.

[0560] json

[0561] {

[0562] "destinations": ["Tokyo", "Osaka"],

[0563] "dates": ["2023-12-20", "2023-12-25"]

[0564] }

[0565] Step 2:

[0566] The server analyzes the received JSON data and generates a travel plan. The generated travel plan is saved in a database. An emotion recognition engine analyzes the user's facial expressions and voice when inputting the plan and recognizes their emotion. For example, if the user indicates that they want to relax, relaxing tourist spots and activities are added to the travel plan.

[0567] Input: Travel destination and schedule, user facial expression and voice data

[0568] Output: Adjusted itinerary

[0569] Specific behavior:

[0570] The server uses an emotion recognition engine to extract emotional data from the user's facial expressions and voice, and then customizes the travel plan based on that emotional data.

[0571] Step 3:

[0572] Reservation information for accommodation and transportation is entered on the device. The device then sends the entered reservation information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, an emotion recognition engine recognizes the user's emotions during the reservation process, and if the user is anxious, a reservation confirmation email containing additional support information is sent.

[0573] Input: Reservation information, user facial and voice data

[0574] Output: Reservation confirmation email

[0575] Specific behavior:

[0576] The device inputs reservation information and sends it to the server. The server connects to the external reservation system via API to confirm the reservation, and an emotion recognition engine analyzes the user's emotions and sends a reservation confirmation email.

[0577] Step 4:

[0578] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the answer based on that emotion. For example, if the user is excited, it will suggest special events or activities.

[0579] Input: Question content, user facial expression and voice data

[0580] Output: Adjusted answer

[0581] Specific behavior:

[0582] The device inputs a question and sends it to the server, which then searches a database for the best answer and uses an emotion recognition engine to check the user's emotions and adjust the answer accordingly.

[0583] Step 5:

[0584] When a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. An emotion recognition engine recognizes the emotion the user expressed when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a prompt and detailed response is provided.

[0585] Input: Trouble information, user facial expression and voice data

[0586] Output: Coordinated response

[0587] Specific behavior:

[0588] The device inputs trouble information and sends it to the server, which uses an emotion recognition engine to analyze the user's emotions, adjusts countermeasures, and promptly communicates them.

[0589] Step 6:

[0590] The system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary. The navigation system displays route guidance based on the itinerary in real time, and an emotion recognition engine monitors the user's emotions and adjusts route guidance as needed.

[0591] Input: Travel plans, real-time user sentiment data

[0592] Output: Adjusted route guidance

[0593] Specific behavior:

[0594] The navigation system captures your trip plans and provides real-time directions, while an emotion recognition engine monitors your emotions and adjusts directions as needed.

[0595] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0597] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0598] [Second embodiment]

[0599] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0600] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0601] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0602] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0603] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0605] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0606] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0607] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0610] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0611] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, links with external reservation systems, and responds to questions and troubles during the trip. The system of the present invention integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[0612] Travel planning assistance

[0613] The user inputs the travel destination and the dates of the visit. The terminal receives the user's input and sends the information to the server. The server receives the information, generates a travel plan, and saves it in a database. For example, if the user plans to visit "Paris" and "London" on dates "2023-12-20" and "2023-12-25," the server will create a travel plan based on that and provide it to the user.

[0614] API integration with external reservation systems

[0615] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information, confirming the reservation. For example, if a user wants to reserve a hotel in Paris from 2023-12-20 to 2023-12-25, the server sends the reservation information to the external reservation system, retrieves the results, and completes the reservation.

[0616] QA support

[0617] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device sends the question to the server, which then provides an appropriate answer. For example, if a user asks, "What restaurant do you recommend?", the server will recommend a restaurant called "Gourmet Delight."

[0618] Troubleshooting

[0619] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. For example, if the user's flight is canceled, the server will initiate the process of arranging a new flight and notify them, "Your flight has been canceled. We are arranging a new flight..."

[0620] This invention enables users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[0621] The processing flow will be explained below.

[0622] Travel planning assistance

[0623] Step 1:

[0624] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[0625] Step 2:

[0626] The device sends the user's input in JSON format to the server.

[0627] Example of transmitted data:

[0628] json

[0629] {

[0630] "user_id": "12345",

[0631] "destinations": ["Paris", "London"],

[0632] "dates": ["2023-12-20", "2023-12-25"]

[0633] }

[0634] Step 3:

[0635] The server generates a travel plan based on the received data and stores it in a database. For example, it stores each destination and visit date as follows:

[0636] python

[0637] self.plan = [{"destination": "Paris", "date": "2023-12-20"}, {"destination": "London", "date": "2023-12-25"}]

[0638] API integration with external reservation systems

[0639] Step 1:

[0640] The user inputs reservation information. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[0641] Step 2:

[0642] The device sends the reservation information to the server in JSON format.

[0643] Example of transmitted data:

[0644] json

[0645] {

[0646] "user_id": "12345",

[0647] "destination": "Paris",

[0648] "check_in": "2023-12-20",

[0649] "check_out": "2023-12-25"

[0650] }

[0651] Step 3:

[0652] The server receives the reservation information and sends a POST request to the external reservation system API.

[0653] Example API request:

[0654] python

[0655] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[0656] Step 4:

[0657] The server receives the response from the API and notifies the user of the reservation result.

[0658] QA support

[0659] Step 1:

[0660] The user types a question into the terminal, for example, "must_visit" (what are the must-see spots?).

[0661] Step 2:

[0662] The terminal sends a question to the server.

[0663] Example of transmitted data:

[0664] json

[0665] {

[0666] "user_id": "12345",

[0667] "question": "must_visit"

[0668] }

[0669] Step 3:

[0670] The server receives the question and provides the appropriate answer.

[0671] Example answer:

[0672] python

[0673] answers = {

[0674] "best_restaurant": "The best restaurant in town is 'Gourmet Delight'.",

[0675] "must_visit": "You must visit the 'Historic Cathedral'."

[0676] }

[0677] answer = answers.get(question, "Sorry, I don't have an answer for that.")

[0678] Step 4:

[0679] The server sends the answer in JSON format to the device, which displays it to the user.

[0680] Troubleshooting

[0681] Step 1:

[0682] The user inputs a problem into the terminal, for example, "flight_cancelled" (the flight was canceled).

[0683] Step 2:

[0684] The device sends the trouble information to the server.

[0685] Example of transmitted data:

[0686] json

[0687] {

[0688] "user_id": "12345",

[0689] "issue": "flight_cancelled"

[0690] }

[0691] Step 3:

[0692] The server receives the trouble information and takes appropriate action.

[0693] Examples of support:

[0694] python

[0695] if issue == 'flight_cancelled':

[0696] response = "Your flight has been canceled. Booking a new flight for you..."

[0697] elif issue == 'need_hotel_extension':

[0698] response = "Booking an extension for your hotel stay..."

[0699] else:

[0700] response = "Sorry, I cannot handle that issue."

[0701] Step 4:

[0702] The server transmits the correspondence result to the terminal, which displays it to the user.

[0703] Example 1

[0704] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Modern travelers need multiple services, such as creating travel plans, booking accommodations and transportation, and answering questions and dealing with problems while traveling, but managing these tasks individually is extremely cumbersome. Conventional systems are unable to provide centralized support for these tasks, placing a heavy burden on travelers, which is a problem.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0707] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for receiving questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to deal with those problems, and means for using a generative AI model to generate the travel plan and handle the problems. This allows travelers to centrally manage their travel plans and receive the services they need smoothly and quickly.

[0708] "User" means any individual or group of people who use the Travel Planning System.

[0709] "Destination" means a geographic location that you intend to visit.

[0710] "Dates" refers to the period of time for which the user plans to travel.

[0711] "Travel Plan" refers to an itinerary or schedule created based on a User's travel destination and dates.

[0712] "External Reservation System" refers to an external online system used to make reservations such as hotel or transportation reservations.

[0713] An "API" is something that provides an interface for different software systems to communicate with each other.

[0714] "Generative AI models" refer to algorithms and techniques that use artificial intelligence to generate optimal responses and plans from data.

[0715] "Push transmission" refers to a technology in which a server sends data or notifications directly to a user's device.

[0716] "Trouble Information" refers to information about problems or obstacles that users encounter during their trip.

[0717] "Appropriate response" refers to the optimal measures or solutions that the system takes in response to trouble information.

[0718] This is an integrated system that starts when a user inputs their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. This system integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[0719] Specifically, the system is implemented as follows.

[0720] Travel planning assistance

[0721] The user uses the device to input the travel destination and visit dates. If the user plans to visit "Paris" and "London" and the dates are "2023-12-20" and "2023-12-25," the user enters this information into the input form on the device. The device receives this information and sends it to the server using the HTTPS protocol. The server uses the generative AI model to create an optimal travel plan and stores it in a database. The generated travel plan is provided to the user via the device. For example, the server may suggest, "In Paris, we recommend visiting the Louvre Museum and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[0722] API integration with external reservation systems

[0723] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information from their device. For example, if they want to reserve a hotel in "Paris" from "2023-12-20" to "2023-12-25," they enter this information into an input form. The device sends that information to the server, and the server sends an API request to an external reservation system based on the reservation information. For example, the reservation is made using the API of an external reservation system (e.g., Booking.com). At this time, the server receives a reservation confirmation response from the external system and notifies the user of the result via the device. As a specific example, information such as "The hotel reserved is Hôtel Le Meurice" is provided.

[0724] QA support

[0725] While traveling, the user inputs questions into the device about recommended facilities and restaurants at the destination. For example, "What restaurant do you recommend?" The device sends the question to the server, which uses a generative AI model to generate an appropriate answer. For example, a recommendation for a restaurant called "Gourmet Delight" is generated. The answer generated by the server is provided to the user via the device.

[0726] Troubleshooting

[0727] If a user encounters a problem during their trip, they report the problem from their device. For example, if their flight is canceled, they enter "Flight canceled." The device then sends this trouble information to the server. The server then takes appropriate action, such as arranging a new flight, based on the trouble information. The server then notifies the user of the progress and results of the action via the device. For example, the user may be notified that "Your flight has been canceled. Arranging a new flight..."

[0728] Examples of prompt statements

[0729] Travel planning assistance:

[0730] Input: "Destinations are Paris and London, visit dates are 2023-12-20 to 2023-12-25. Please make travel plans."

[0731] Output: "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[0732] API integration with external booking systems:

[0733] Input: "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25."

[0734] Output: "The hotel booked is Hôtel Le Meurice."

[0735] QA Support:

[0736] Input: "What are the best restaurants in Paris?"

[0737] Output: "The restaurant I recommend is Gourmet Delight."

[0738] Troubleshooting:

[0739] Enter: "Flight canceled. Please find a new flight."

[0740] Output: "Your flight has been cancelled. We are arranging a new flight..."

[0741] As described above, this invention allows users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[0742] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0743] Step 1: User enters travel information

[0744] Description: The user enters a travel destination and visit dates into the terminal.

[0745] Input: Travel destination (e.g., "Paris" or "London"), visit date (e.g., "2023-12-20" to "2023-12-25").

[0746] Specific operation: The user enters the travel destination and itinerary into the input form on the device screen. The device accurately records the entered data.

[0747] Step 2: Submit your trip information

[0748] Description: The device sends travel information to the server.

[0749] Input: User-entered travel destination and visit date information.

[0750] Output: JSON formatted travel information data.

[0751] What happens: The device converts the user's input into JSON format and sends it to the server's API endpoint using the HTTPS protocol.

[0752] Step 3: Generate a travel plan

[0753] Description: Based on the travel information received by the server, a generative AI model is used to generate a travel plan.

[0754] Input: JSON-formatted trip information data sent from the device.

[0755] Output: The generated itinerary.

[0756] Specific operation: The server extracts the travel destinations and dates from the received data and sends them as prompts to the generative AI model (e.g., "The travel destinations are Paris and London, and the visit dates are from 2023-12-20 to 2023-12-25. Please create a travel plan."). The plan obtained from the model is saved in a database.

[0757] Step 4: Provide your travel plans

[0758] Description: The server provides the travel plan generated by the server to the user via the terminal.

[0759] Input: The generated itinerary.

[0760] Output: Display of travel plan.

[0761] Specific operation: The server retrieves the generated travel plan from the database and sends it to the device. The device displays the received plan on the screen (e.g., "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace.").

[0762] Step 5: Enter reservation information

[0763] Description: A user enters accommodation and transportation reservation information into a device.

[0764] Input: The facility and period you want to book (e.g. "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25").

[0765] Specific operation: The user accesses the reservation form on the device and enters the reservation information.

[0766] Step 6: Submit your reservation information

[0767] Description: The terminal sends reservation information to the server.

[0768] Input: Booking information entered by the user.

[0769] Output: Reservation information data formatted in JSON.

[0770] Specific operation: The terminal converts the reservation information into JSON format and sends it to the server via HTTPS protocol.

[0771] Step 7: API Request to External System

[0772] Description: The server sends an API request to an external reservation system based on the reservation information.

[0773] Input: Reservation information data in JSON format sent from the terminal.

[0774] Output: The API request sent to the external booking system.

[0775] Specific operation: The server analyzes the received data and sends an HTTP POST request to the external reservation system (e.g., the API of the reservation site).

[0776] Step 8: Confirmation and notification of booking

[0777] Description: The server receives the response from the external system and notifies the user of the reservation result.

[0778] Input: Booking confirmation response from the external system.

[0779] Output: Display of reservation confirmation information.

[0780] Specific behavior: The server receives the response from the external system and sends it in JSON format to the terminal. The terminal displays the received information on the user interface and notifies the user that the reservation has been confirmed (e.g., "The reserved hotel is Hôtel Le Meurice.").

[0781] Step 9: Enter your question

[0782] Description: User types questions into terminal while traveling.

[0783] Input: The question (e.g., "What restaurants do you recommend?").

[0784] Specific behavior: The user types a question into the chatbot interface on the device.

[0785] Step 10: Submit your question

[0786] Description: The device sends a question to the server.

[0787] Input: The question typed by the user.

[0788] Output: Question data formatted in JSON.

[0789] Specific operation: The device converts the question into JSON format and sends it to the server via HTTPS protocol.

[0790] Step 11: Generate an answer

[0791] Description: The server uses a generative AI model to generate an appropriate answer based on the question.

[0792] Input: Question data sent from the terminal.

[0793] Output: The generated answer.

[0794] How it works: The server sends the question as a prompt to the generative AI model and receives an answer from the model (e.g., "The restaurant I recommend is Gourmet Delight.").

[0795] Step 12: Provide your answers

[0796] Description: The server provides the generated answer to the user via the terminal.

[0797] Input: The generated answer.

[0798] Output: Display of the answer to the question.

[0799] Specific operation: The server sends the generated answer to the device, and the device displays the received information on the screen.

[0800] Step 13: Report a problem

[0801] Description: Allows users to report any trouble they encounter while traveling from their device.

[0802] Input: Problem description (e.g. "My flight has been cancelled").

[0803] Specific actions: The user accesses the device's reporting form and enters specific trouble information.

[0804] Step 14: Submit your trouble report

[0805] Description: The device sends trouble information to the server.

[0806] Input: Trouble information entered by the user.

[0807] Output: Trouble information data formatted in JSON format.

[0808] Specific operation: The device converts the trouble information into JSON format and sends it to the server via HTTPS protocol.

[0809] Step 15: Initiate response

[0810] Description: The server will take appropriate action based on the trouble information.

[0811] Input: Trouble information data sent from the device.

[0812] Output: A request to start the response process.

[0813] Specific behavior: The server analyzes the received trouble information and sends a request to an external system to initiate appropriate action, such as arranging a new flight.

[0814] Step 16: Notification of troubleshoot

[0815] Description: The server notifies the user of the progress of the response.

[0816] Input: Progress of the response.

[0817] Output: Displays troubleshooting progress information.

[0818] Specific behavior: The server sends progress information in JSON format to the device, and the device displays the received information in the user interface (e.g., "Your flight has been canceled. Arranging a new flight...").

[0819] (Application example 1)

[0820] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0821] Conventional travel support systems handle the creation of travel plans, reservations, answering questions, and troubleshooting individually, making it difficult for users to centrally manage everything, and there was a problem of a lack of convenience, especially when using meal services at destinations, as users had to go through a separate platform.In addition, there was also the problem that recommendations for meal services at travel destinations and reservation management were not integrated, which increased the effort required to make meal arrangements during travel.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0823] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for accepting questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to address those problems, means for recommending meal services at locations of stay based on the travel plan, and means for managing reservations for the recommended meal services. This allows users to perform everything from creating a travel plan to answering questions and responding to them during the trip, and even arranging meals at locations of stay, all within the system.

[0824] "User" refers to an individual who creates or changes travel plans, asks questions, or reports problems.

[0825] The "means for inputting travel destination and itinerary" is an interface for inputting the travel destination, departure date, and return date.

[0826] "Means for generating travel plans" refers to a system that has the function of automatically creating a travel itinerary and detailed schedule based on inputted travel destinations and dates.

[0827] "Means for executing reservations in cooperation with external reservation systems" refers to a system that communicates with an external reservation platform to complete reservations for accommodation, transportation, etc. based on the contents of travel plans.

[0828] "Means of accepting questions and providing appropriate answers to those questions" refers to a system that responds to various questions from users while traveling and provides answers using pre-registered data or external databases.

[0829] "Means for receiving trouble information and taking appropriate measures to resolve the trouble" refers to a system that receives from users information about problems or troubles that occur during their trip and provides appropriate measures to resolve the trouble.

[0830] The "means for recommending local food services" is a system that has the function of suggesting to users food services available in places to be visited based on a travel plan.

[0831] "Means for managing reservations for meal services" refers to a system that records and manages reservations made by users for recommended meal services, and makes changes or cancellations as necessary.

[0832] An "API integration means" is a component that provides a common interface for different systems to communicate with each other, enabling the sending and receiving of information.

[0833] "Means for sending push notifications of service cancellations" refers to a system that has the function of immediately notifying users of sudden service cancellations during travel to their devices.

[0834] To implement this invention, the travel support system must be configured with a server and terminals. The server comprehensively manages various functions, such as generating travel plans, linking with external reservation systems, answering questions, dealing with problems, recommending meal services, and managing reservations. The terminal functions as an interface for users to input travel destinations and itineraries and to make inquiries and respond to requests during the trip.

[0835] Program Generation

[0836] The program for this system must have the following functions:

[0837] First, we provide an interface for users to input their travel destination and itinerary. This interface is implemented as a smartphone application. For example, it includes a function to input "Paris" as the travel destination and "2023-12-20 to 2023-12-25" as the itinerary.

[0838] The server then generates a travel plan based on the input information by analyzing the received data and extracting information such as optimal travel routes, tourist spots, and accommodations from a database to generate the travel plan.

[0839] In addition, by linking with an external reservation system, reservations are made based on the generated travel plans. Specifically, the server sends requests to the external reservation service API to complete reservations for accommodations and transportation.

[0840] During the trip, the server will provide a function to accept questions from users, such as "What are the best restaurants in Paris?", and will refer to the database to provide an appropriate answer.

[0841] Furthermore, the system receives information about troubles that occur during travel and takes appropriate measures to resolve them. For example, if a flight is canceled, the server will start arranging a new flight and notify the user's device, "Your flight has been canceled. Arranging a new flight..."

[0842] Newly added features include a way to recommend dining options based on travel plans. The server extracts popular restaurants and delivery services from a database and suggests them to users. Once a user selects a restaurant, the server manages the reservation.

[0843] For example, if a user wants to order "Pasta Carbonara" and "Tiramisu" in "Paris," the server connects to an external food delivery service API and confirms the order.

[0844] Example prompt sentence:

[0845] Enter your travel plan data.

[0846] Destination: Paris

[0847] Start date: 2023-12-20

[0848] End date: 2023-12-25

[0849] Place a food delivery order from a specific restaurant.

[0850] Restaurant ID: 456

[0851] Order Details:

[0852] Pasta Carbonara (2)

[0853] Tiramisu (1)

[0854] Delivery address: 123 Champs-Élysées, Paris

[0855] As described above, the system of the present invention links the server and terminals to provide consistent support from planning to execution of the trip, troubleshooting, answering questions, and arranging meal services. This allows travelers to use many functions in a unified manner, greatly improving the convenience of the entire trip.

[0856] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0857] Step 1:

[0858] The user inputs the travel destination and dates.

[0859] As input, the user enters the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20 to 2023-12-25").

[0860] The terminal receives this information and sends it to the server. As output, the travel destination and itinerary data is sent to the server.

[0861] Step 2:

[0862] The server generates a travel plan based on the received information.

[0863] As input, the server receives destination and itinerary data.

[0864] The server references the database and generates a travel plan by combining the best sightseeing routes, accommodations, and means of transportation.

[0865] As an output, the generated itinerary is stored in a database and provided to the user.

[0866] Step 3:

[0867] Execute reservations in conjunction with an external reservation system.

[0868] As input, the server uses the generated itinerary data.

[0869] The server sends requests to an external reservation system API to make reservations for accommodation and transportation.

[0870] As an output, the reservation completion information is returned to the server and stored in a database.

[0871] Step 4:

[0872] Accept questions from users during their trip and provide appropriate answers.

[0873] As input, the user types a question through the terminal (e.g., "What are some recommended restaurants in Paris?").

[0874] The terminal sends a query to the server.

[0875] The server consults the database and extracts the appropriate answer (e.g., "a restaurant called Gourmet Delight") from the data.

[0876] As an output, the answer is displayed on the user's terminal.

[0877] Step 5:

[0878] Receive information about troubles that may occur during travel and take appropriate action.

[0879] As input, the user enters trouble information (e.g., "flight canceled") into the terminal.

[0880] The terminal transmits the trouble information to the server.

[0881] The server analyzes the trouble information and initiates procedures to take appropriate action (e.g., arranging a new flight).

[0882] As an output, the progress of troubleshooting is notified to the user's terminal.

[0883] Step 6:

[0884] Recommend food services at your destination based on your travel plans.

[0885] As input, the server uses travel plan data to extract meal services from a database that are tailored to the user's location and time of stay.

[0886] The server sends recommended restaurants and delivery services to the user's device.

[0887] As an output, recommended meal services are displayed on the user's device.

[0888] Step 7:

[0889] Manage reservations for recommended meal services.

[0890] As input, the user selects which of the recommended meal services they wish to book (e.g., ordering "Pasta Carbonara" and "Tiramisu").

[0891] The terminal transmits the order information to the server.

[0892] The server connects to an external food delivery service API and sends a request to confirm the order.

[0893] As an output, information about the completion of the order is returned to the server and displayed on the user's terminal.

[0894] The above are the specific processing steps of this system.

[0895] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0896] This is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions and adjusts services based on those emotions, user satisfaction can be further improved.

[0897] Travel planning support and emotion recognition

[0898] The user inputs their travel destination and visit date. For example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively. The device sends the user's input in JSON format to the server. The server generates a travel plan based on the received data and saves it in a database. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice when they enter their information, and adjusts the travel plan based on those emotions. For example, if the user indicates that they want to "relax," it will suggest relaxing tourist spots and activities.

[0899] API integration with external reservation systems and emotion recognition

[0900] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, the emotion engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation result notification based on those emotions. For example, if the user is in an "anxious" state, the reservation confirmation email will include additional support information.

[0901] QA and Emotion Recognition

[0902] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device then sends the question to the server, which then provides an appropriate answer. The emotion engine recognizes the emotion the user expressed when entering the question and adjusts the answer accordingly. For example, if the user is "excited," it will suggest special events or activities.

[0903] Troubleshooting and Emotion Recognition

[0904] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. At this time, the emotion engine recognizes the user's emotion when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a quick and detailed response will reduce the user's dissatisfaction.

[0905] This invention makes it possible to recognize users' emotions in real time and adjust service content based on those emotions. This system provides consistent support from planning a trip to responding during the trip, and by combining it with an emotion engine, it can significantly improve traveler satisfaction and reliability.

[0906] The processing flow will be explained below.

[0907] Travel planning support and emotion recognition

[0908] Step 1:

[0909] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[0910] Step 2:

[0911] The device sends the user input information to the server in JSON format.

[0912] Example of transmitted data:

[0913] json

[0914] {

[0915] "user_id": "12345",

[0916] "destinations": ["Paris", "London"],

[0917] "dates": ["2023-12-20", "2023-12-25"]

[0918] }

[0919] Step 3:

[0920] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice when inputting. For example, it analyzes whether the user is expressing the emotion "I want to relax."

[0921] Step 4:

[0922] The server generates a travel plan based on the recognized emotion and the input information. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots.

[0923] Step 5:

[0924] The server stores the generated travel plan in a database and sends it to the terminal, which displays the plan to the user.

[0925] API integration with external reservation systems and emotion recognition

[0926] Step 1:

[0927] The user inputs reservation information for accommodation and transportation. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[0928] Step 2:

[0929] The device sends the reservation information in JSON format to the server.

[0930] Example of transmitted data:

[0931] json

[0932] {

[0933] "user_id": "12345",

[0934] "destination": "Paris",

[0935] "check_in": "2023-12-20",

[0936] "check_out": "2023-12-25"

[0937] }

[0938] Step 3:

[0939] The server receives the reservation information and sends a POST request to the external reservation system API.

[0940] Example API request:

[0941] python

[0942] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[0943] Step 4:

[0944] The server uses an emotion engine to recognize the user's emotions during the reservation process, for example, analyzing whether the user is in an "anxious" state during the reservation process.

[0945] Step 5:

[0946] The server tailors the booking outcome based on the perceived emotion, for example including additional support information in the booking confirmation email if the user indicates "anxiety."

[0947] Step 6:

[0948] The server sends the reservation results in JSON format to the terminal, and the terminal notifies the user.

[0949] QA and Emotion Recognition

[0950] Step 1:

[0951] The user inputs questions about recommended facilities and restaurants at the destination into the terminal. For example, the user inputs the question "must_visit" (what are the must-see spots?).

[0952] Step 2:

[0953] The device sends the question in JSON format to the server.

[0954] Example of transmitted data:

[0955] json

[0956] {

[0957] "user_id": "12345",

[0958] "question": "must_visit"

[0959] }

[0960] Step 3:

[0961] The server receives the question and uses an emotion engine to recognize the user's emotion when entering the question. For example, it analyzes whether the user is "excited."

[0962] Step 4:

[0963] The server generates appropriate responses and tailors them based on the emotion, for example, suggesting special events or activities if the user is "excited."

[0964] Step 5:

[0965] The server sends the answer in JSON format to the device, which displays it to the user.

[0966] Troubleshooting and Emotion Recognition

[0967] Step 1:

[0968] The user enters the problem into the terminal, for example, "flight_cancelled."

[0969] Step 2:

[0970] The device sends trouble information to the server in JSON format.

[0971] Example of transmitted data:

[0972] json

[0973] {

[0974] "user_id": "12345",

[0975] "issue": "flight_cancelled"

[0976] }

[0977] Step 3:

[0978] The server receives the trouble information and uses an emotion engine to recognize the user's emotion when reporting the trouble. For example, it analyzes whether the user is expressing anger.

[0979] Step 4:

[0980] The server responds appropriately based on the recognized emotion, for example, if the user expresses anger, it responds quickly and in detail.

[0981] Step 5:

[0982] The server sends the response results in JSON format to the device, which then displays them to the user.

[0983] Example 2

[0984] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0985] Conventional travel support systems have difficulty recognizing users' emotions in real time and adjusting service content based on those emotions. Furthermore, there is a need for a method to sufficiently improve user satisfaction in travel planning, linking with external reservation systems, and responding to questions and problems during the trip.

[0986] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for transmitting the input information in structured data format; a means for analyzing emotions based on the structured data and generating a travel plan; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for analyzing the user's emotions during the reservation process and adjusting notification content based on the emotions; a means for receiving questions from the user during the trip and providing appropriate answers to the questions; and a means for receiving information about problems that occur during the trip, taking appropriate measures to deal with the problems, and adjusting the response method based on the user's emotions. This makes it possible to recognize the user's emotions in real time and adjust service content based on the emotions.

[0987] "User" refers to an individual who uses the System to make travel plans, make reservations, ask questions, or report problems.

[0988] "Destination" refers to a geographic location that a user wishes to visit.

[0989] "Itinerary" refers to time information including the start and end dates of a trip or the duration of stay at each location visited.

[0990] A "structured data format" refers to a data format that represents information in a regular and unified format, making it easier to process within databases and information systems.

[0991] "Emotion" refers to the psychological state or reaction that a user shows toward a system, such as feelings of relaxation, excitement, or anxiety.

[0992] "Travel Itinerary" means a series of suggestions based on your travel destination and dates, including attractions, activities, and accommodations.

[0993] "Third-party booking systems" refers to online services provided by other companies for booking accommodation and transportation.

[0994] "API integration" refers to the use of application program interfaces to share data and functions between different software systems.

[0995] "Question" refers to an inquiry made by a user to the system to obtain information or ask for advice.

[0996] "Trouble Information" refers to information that users use to report to the system any problems or difficulties they encounter during their trip.

[0997] "Notification" refers to information or messages sent by the system to the user.

[0998] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with external reservation systems, and responds to questions and troubles during the trip. It also has an emotion engine that recognizes the user's emotions and adjusts services accordingly.

[0999] In order to implement the present invention, the following hardware and software are used.

[1000] Hardware:

[1001] Terminal: The device on which the user enters information (e.g., smartphone, tablet, PC)

[1002] Server: A host computer used to process and store data and run the emotion engine.

[1003] Database: a storage system for storing travel plans, reservation information, etc.

[1004] software:

[1005] Emotion engine: Software for recognizing user emotions and adjusting services

[1006] API with external booking systems: Interface for booking accommodations, transportation, etc.

[1007] Question-answering system: An AI model for answering questions from users

[1008] Specific examples of the present invention are given below.

[1009] Example 1:

[1010] A user inputs into the device that they are traveling to "Paris" and departing on "2023-12-20." The device converts this information into JSON format and sends it to the server. The server receives this information and uses an emotion engine to analyze the emotion of the user at the time of input. For example, if the emotion "I want to relax" is detected, the server creates a travel plan that suggests relaxing tourist spots and saves it in the database.

[1011] Example prompt: "The user wants to visit Paris and London and is looking for a relaxing trip."

[1012] Example 2:

[1013] The user enters the accommodation "Hotel A" and "2023-12-20~2023-12-25" and the transportation method "Flight B" into the terminal. The terminal sends this information to the server. The server sends an API request to an external reservation system based on the information and confirms the reservation. Furthermore, if the user expresses "anxiety," the server includes additional support information in the reservation confirmation email and sends it.

[1014] Example 3:

[1015] While traveling, a user can type "What are some good restaurants in this area?" into their device, which then sends the question to the server. The server generates an answer to the question and, if it determines that the user is "excited," suggests special events or activities.

[1016] Example 4:

[1017] When a problem occurs during a trip and the user types "my flight is delayed" into the device, the device sends the problem information to the server. The server responds quickly and in detail based on this information, and if the user shows anger, it further adjusts the response method and notifies the user.

[1018] In this way, a specific embodiment of the present invention can recognize a user's emotions in real time and adjust travel plans and various responses based on those emotions, thereby significantly improving user satisfaction.

[1019] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1020] Step 1:

[1021] The user inputs the travel destination and dates

[1022] Specific operation: The user uses the terminal to input the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20").

[1023] Input: Travel destination and itinerary information

[1024] Output: Structured data of destinations and dates

[1025] Step 2:

[1026] The terminal converts the input content into JSON format and sends it.

[1027] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"destination": "Paris", "date": "2023-12-20"}).

[1028] Input: Structured data of destination and dates

[1029] Output: JSON format data

[1030] Step 3:

[1031] The server receives the data and analyzes it with the emotion engine.

[1032] Specific operation: The server receives the JSON data sent from the device and analyzes the user's emotions using the emotion engine. For example, it detects the emotion "I want to relax."

[1033] Input: JSON format data

[1034] Output: Sentiment analysis results

[1035] Step 4:

[1036] The server generates the itinerary and stores it in the database.

[1037] Specific operation: The server generates a travel plan based on the results of the sentiment analysis, for example, a plan including relaxing tourist spots and activities, and stores it in a database.

[1038] Input: Sentiment analysis results

[1039] Output: Travel plan data

[1040] Step 5:

[1041] The user enters accommodation and transportation reservations

[1042] Specific operation: The user uses the terminal to enter information about accommodation (e.g., "Hotel A") and transportation (e.g., "Flight B").

[1043] Input: Accommodation and transportation information

[1044] Output: Accommodation and transportation structured data

[1045] Step 6:

[1046] The device sends the reservation information to the server.

[1047] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"hotel": "Hotel A", "flight": "Flight B"}).

[1048] Input: Accommodation and transportation structured data

[1049] Output: JSON format data

[1050] Step 7:

[1051] The server sends an API request to the external booking system

[1052] Specific operation: The server sends an API request to the external reservation system based on the reservation information and confirms the reservation.

[1053] Input: Reservation information in JSON format

[1054] Output: Reservation confirmation result

[1055] Step 8:

[1056] The server adjusts and notifies the reservation result based on emotions.

[1057] Specific operation: The server receives the reservation confirmation result and uses the emotion engine to analyze the user's current emotion. For example, if the user is in an "anxious" state, the server sends the reservation confirmation email to the user with additional support information.

[1058] Input: Booking confirmation results, sentiment analysis

[1059] Output: Adjusted booking confirmation

[1060] Step 9:

[1061] The user enters a question

[1062] Specific operation: The user uses the device to input questions such as recommended facilities and restaurants at the destination.

[1063] Input: Question content

[1064] Output: Question structured data

[1065] Step 10:

[1066] The device sends a question to the server

[1067] Specific operation: The device sends the question content to the server in JSON format (e.g., "{"question": "What are some good restaurants in this area?"}).

[1068] Input: Question structured data

[1069] Output: Question data in JSON format

[1070] Step 11:

[1071] The server generates the answer and adjusts it based on the emotion.

[1072] What it does: The server generates answers to questions and uses an emotion engine to tailor the content based on the user's emotions. If the user is "excited," it suggests special events and activities.

[1073] Input: JSON format question data, sentiment analysis

[1074] Output: Adjusted answer notification

[1075] Step 12:

[1076] A user reports a problem

[1077] Specific operation: The user uses the terminal to input trouble information (e.g., "The flight is delayed").

[1078] Input: Trouble information

[1079] Output: Structured data of trouble information

[1080] Step 13:

[1081] The device sends trouble information to the server

[1082] Specific operation: The device sends trouble information to the server in JSON format (e.g., {"trouble": "The flight was delayed"}).

[1083] Input: Structured data of trouble information

[1084] Output: Trouble information in JSON format

[1085] Step 14:

[1086] The server adjusts and notifies the response method based on the emotion.

[1087] Specific operation: The server takes appropriate action based on the trouble information it receives. If the emotion engine recognizes that the user's emotion is "anger," it takes a swift and detailed action, adjusts the response method, and notifies the user.

[1088] Input: JSON format trouble information, sentiment analysis

[1089] Output: Coordinated response notification

[1090] (Application example 2)

[1091] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1092] Conventional travel plan generation systems do not take into account user emotions and lack full support during travel, especially in autonomous vehicles. Therefore, it is important to reduce the anxiety and frustration users feel during travel and improve overall satisfaction with the travel experience.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for generating a travel plan based on the input information; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for accepting questions from the user during the travel and providing appropriate answers to the questions; a means for receiving information about troubles during the travel and taking appropriate measures to deal with the troubles; a means for recognizing the user's emotions using an emotion recognition engine and adjusting service content based on the emotions; and a means for coordinating with the navigation system of an autonomous vehicle to provide route guidance based on the travel plan. This makes it possible to provide a high level of satisfaction that takes the user's emotions into consideration, from travel planning to responses during the trip.

[1094] The "means for inputting travel destinations and itineraries" is an interface for the user to input the places he or she wishes to visit and the dates on which he or she will visit those places.

[1095] The "means for generating a travel plan" is a system for automatically creating a plan based on the travel destination and itinerary input by the user.

[1096] The "means of executing reservations in cooperation with external reservation systems" is a function of exchanging information with other external systems in order to reserve accommodations and transportation based on travel plans.

[1097] "Means of accepting questions and providing appropriate answers to those questions" refers to a function that quickly responds to questions and information requests that users have while traveling and provides appropriate information.

[1098] "A means of receiving trouble information and taking appropriate action against the trouble" is a system that proposes and responds to problems and difficulties that arise during travel.

[1099] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice and recognizes their emotional state.

[1100] "Means for recognizing the user's emotions and adjusting the service content based on those emotions" refers to a function that uses an emotion recognition engine to understand the user's emotions and optimize the service according to that state.

[1101] "Means for linking with the navigation system of an autonomous vehicle and providing route guidance based on a travel plan" refers to a system that communicates with the navigation system installed in an autonomous vehicle and provides guidance on the optimal route based on the generated travel plan.

[1102] Travel planning support and emotion recognition

[1103] The user first enters their travel destination and visit dates. For example, suppose the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The user's device sends this information in JSON format to the server. The server generates a travel plan based on the received data and stores it in a database. Furthermore, an emotion recognition engine recognizes emotions from the user's facial expressions and voice when entering information, and adjusts the travel plan based on those emotions. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots and activities.

[1104] API integration with external reservation systems and emotion recognition

[1105] When a user makes a reservation for accommodation or transportation, they input the reservation information on their device. The device then sends that information to a server, which then sends an API request to an external reservation system based on the reservation information to confirm the reservation. An emotion recognition engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation notification based on those emotions. For example, if the user is anxious, the reservation confirmation email can include additional support information.

[1106] QA and Emotion Recognition

[1107] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the content of the answer based on that emotion. For example, if the user is excited, it can suggest special events or activities.

[1108] Troubleshooting and Emotion Recognition

[1109] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. At this time, an emotion recognition engine recognizes the user's emotions when reporting the problem and adjusts the response method based on those emotions. For example, if the user expresses anger, a quick and detailed response can be provided to reduce the user's dissatisfaction.

[1110] Collaboration with autonomous vehicles

[1111] This system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary, and can adjust in real time based on the user's emotions by providing the navigation system with the best tourist spots and routes at each point during the trip.

[1112] Hardware and software used

[1113] 1. Hardware:

[1114] Camera: Recognizes the user's facial expressions.

[1115] Microphone: Recognizes the user's voice.

[1116] Display: Shows information.

[1117] Autonomous driving system: Provides route guidance based on trip plans.

[1118] 2. Software:

[1119] Emotion recognition engine: Analyzes emotions from facial expressions and voice.

[1120] Database: Stores travel planning and booking information.

[1121] API: Integrates with external reservation systems.

[1122] Examples of concrete examples and prompts

[1123] Travel planning prompt:

[1124] The user's travel destinations are Tokyo and Osaka, and the visit dates are December 20th to December 25th, 2023. This indicates a desire to relax. Please suggest a relaxing travel plan.

[1125] Booking anxiety detection prompt:

[1126] The user is booking a triple room at a hotel and is expressing anxiety. Please generate a booking confirmation email with additional support information.

[1127] This will make it possible to provide services that take into account the user's emotions at each stage of travel planning and troubleshooting.

[1128] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1129] Step 1:

[1130] The user enters the travel destination and itinerary. For example, the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The device sends the entered information to the server in JSON format. The data format of the entered information is JSON data as shown below.

[1131] json

[1132] {

[1133] "destinations": ["Tokyo", "Osaka"],

[1134] "dates": ["2023-12-20", "2023-12-25"]

[1135] }

[1136] Step 2:

[1137] The server analyzes the received JSON data and generates a travel plan. The generated travel plan is saved in a database. An emotion recognition engine analyzes the user's facial expressions and voice when inputting the plan and recognizes their emotion. For example, if the user indicates that they want to relax, relaxing tourist spots and activities are added to the travel plan.

[1138] Input: Travel destination and schedule, user facial expression and voice data

[1139] Output: Adjusted itinerary

[1140] Specific behavior:

[1141] The server uses an emotion recognition engine to extract emotional data from the user's facial expressions and voice, and then customizes the travel plan based on that emotional data.

[1142] Step 3:

[1143] Reservation information for accommodation and transportation is entered on the device. The device then sends the entered reservation information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, an emotion recognition engine recognizes the user's emotions during the reservation process, and if the user is anxious, a reservation confirmation email containing additional support information is sent.

[1144] Input: Reservation information, user facial and voice data

[1145] Output: Reservation confirmation email

[1146] Specific behavior:

[1147] The device inputs reservation information and sends it to the server. The server connects to the external reservation system via API to confirm the reservation, and an emotion recognition engine analyzes the user's emotions and sends a reservation confirmation email.

[1148] Step 4:

[1149] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the answer based on that emotion. For example, if the user is excited, it will suggest special events or activities.

[1150] Input: Question content, user facial expression and voice data

[1151] Output: Adjusted answer

[1152] Specific behavior:

[1153] The device inputs a question and sends it to the server, which then searches a database for the best answer and uses an emotion recognition engine to check the user's emotions and adjust the answer accordingly.

[1154] Step 5:

[1155] When a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. An emotion recognition engine recognizes the emotion the user expressed when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a prompt and detailed response is provided.

[1156] Input: Trouble information, user facial expression and voice data

[1157] Output: Coordinated response

[1158] Specific behavior:

[1159] The device inputs trouble information and sends it to the server, which uses an emotion recognition engine to analyze the user's emotions, adjusts countermeasures, and promptly communicates them.

[1160] Step 6:

[1161] The system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary. The navigation system displays route guidance based on the itinerary in real time, and an emotion recognition engine monitors the user's emotions and adjusts route guidance as needed.

[1162] Input: Travel plans, real-time user sentiment data

[1163] Output: Adjusted route guidance

[1164] Specific behavior:

[1165] The navigation system captures your trip plans and provides real-time directions, while an emotion recognition engine monitors your emotions and adjusts directions as needed.

[1166] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1167] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1168] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1169] [Third embodiment]

[1170] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1171] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1172] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1173] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1174] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1175] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1176] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1177] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1178] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1181] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1182] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, links with external reservation systems, and responds to questions and troubles during the trip. The system of the present invention integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[1183] Travel planning assistance

[1184] The user inputs the travel destination and the dates of the visit. The terminal receives the user's input and sends the information to the server. The server receives the information, generates a travel plan, and saves it in a database. For example, if the user plans to visit "Paris" and "London" on dates "2023-12-20" and "2023-12-25," the server will create a travel plan based on that and provide it to the user.

[1185] API integration with external reservation systems

[1186] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information, confirming the reservation. For example, if a user wants to reserve a hotel in Paris from 2023-12-20 to 2023-12-25, the server sends the reservation information to the external reservation system, retrieves the results, and completes the reservation.

[1187] QA support

[1188] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device sends the question to the server, which then provides an appropriate answer. For example, if a user asks, "What restaurant do you recommend?", the server will recommend a restaurant called "Gourmet Delight."

[1189] Troubleshooting

[1190] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. For example, if the user's flight is canceled, the server will initiate the process of arranging a new flight and notify them, "Your flight has been canceled. We are arranging a new flight..."

[1191] This invention enables users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[1192] The processing flow will be explained below.

[1193] Travel planning assistance

[1194] Step 1:

[1195] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[1196] Step 2:

[1197] The device sends the user's input in JSON format to the server.

[1198] Example of transmitted data:

[1199] json

[1200] {

[1201] "user_id": "12345",

[1202] "destinations": ["Paris", "London"],

[1203] "dates": ["2023-12-20", "2023-12-25"]

[1204] }

[1205] Step 3:

[1206] The server generates a travel plan based on the received data and stores it in a database. For example, it stores each destination and visit date as follows:

[1207] python

[1208] self.plan = [{"destination": "Paris", "date": "2023-12-20"}, {"destination": "London", "date": "2023-12-25"}]

[1209] API integration with external reservation systems

[1210] Step 1:

[1211] The user inputs reservation information. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[1212] Step 2:

[1213] The device sends the reservation information to the server in JSON format.

[1214] Example of transmitted data:

[1215] json

[1216] {

[1217] "user_id": "12345",

[1218] "destination": "Paris",

[1219] "check_in": "2023-12-20",

[1220] "check_out": "2023-12-25"

[1221] }

[1222] Step 3:

[1223] The server receives the reservation information and sends a POST request to the external reservation system API.

[1224] Example API request:

[1225] python

[1226] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[1227] Step 4:

[1228] The server receives the response from the API and notifies the user of the reservation result.

[1229] QA support

[1230] Step 1:

[1231] The user types a question into the terminal, for example, "must_visit" (what are the must-see spots?).

[1232] Step 2:

[1233] The terminal sends a question to the server.

[1234] Example of transmitted data:

[1235] json

[1236] {

[1237] "user_id": "12345",

[1238] "question": "must_visit"

[1239] }

[1240] Step 3:

[1241] The server receives the question and provides the appropriate answer.

[1242] Example answer:

[1243] python

[1244] answers = {

[1245] "best_restaurant": "The best restaurant in town is 'Gourmet Delight'.",

[1246] "must_visit": "You must visit the 'Historic Cathedral'."

[1247] }

[1248] answer = answers.get(question, "Sorry, I don't have an answer for that.")

[1249] Step 4:

[1250] The server sends the answer in JSON format to the device, which displays it to the user.

[1251] Troubleshooting

[1252] Step 1:

[1253] The user inputs a problem into the terminal, for example, "flight_cancelled" (the flight was canceled).

[1254] Step 2:

[1255] The device sends the trouble information to the server.

[1256] Example of transmitted data:

[1257] json

[1258] {

[1259] "user_id": "12345",

[1260] "issue": "flight_cancelled"

[1261] }

[1262] Step 3:

[1263] The server receives the trouble information and takes appropriate action.

[1264] Examples of support:

[1265] python

[1266] if issue == 'flight_cancelled':

[1267] response = "Your flight has been canceled. Booking a new flight for you..."

[1268] elif issue == 'need_hotel_extension':

[1269] response = "Booking an extension for your hotel stay..."

[1270] else:

[1271] response = "Sorry, I cannot handle that issue."

[1272] Step 4:

[1273] The server transmits the correspondence result to the terminal, which displays it to the user.

[1274] Example 1

[1275] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1276] Modern travelers need multiple services, such as creating travel plans, booking accommodations and transportation, and answering questions and dealing with problems while traveling, but managing these tasks individually is extremely cumbersome. Conventional systems are unable to provide centralized support for these tasks, placing a heavy burden on travelers, which is a problem.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1278] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for receiving questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to deal with those problems, and means for using a generative AI model to generate the travel plan and handle the problems. This allows travelers to centrally manage their travel plans and receive the services they need smoothly and quickly.

[1279] "User" means any individual or group of people who use the Travel Planning System.

[1280] "Destination" means a geographic location that you intend to visit.

[1281] "Dates" refers to the period of time for which the user plans to travel.

[1282] "Travel Plan" refers to an itinerary or schedule created based on a User's travel destination and dates.

[1283] "External Reservation System" refers to an external online system used to make reservations such as hotel or transportation reservations.

[1284] An "API" is something that provides an interface for different software systems to communicate with each other.

[1285] "Generative AI models" refer to algorithms and techniques that use artificial intelligence to generate optimal responses and plans from data.

[1286] "Push transmission" refers to a technology in which a server sends data or notifications directly to a user's device.

[1287] "Trouble Information" refers to information about problems or obstacles that users encounter during their trip.

[1288] "Appropriate response" refers to the optimal measures or solutions that the system takes in response to trouble information.

[1289] This is an integrated system that starts when a user inputs their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. This system integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[1290] Specifically, the system is implemented as follows.

[1291] Travel planning assistance

[1292] The user uses the device to input the travel destination and visit dates. If the user plans to visit "Paris" and "London" and the dates are "2023-12-20" and "2023-12-25," the user enters this information into the input form on the device. The device receives this information and sends it to the server using the HTTPS protocol. The server uses the generative AI model to create an optimal travel plan and stores it in a database. The generated travel plan is provided to the user via the device. For example, the server may suggest, "In Paris, we recommend visiting the Louvre Museum and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[1293] API integration with external reservation systems

[1294] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information from their device. For example, if they want to reserve a hotel in "Paris" from "2023-12-20" to "2023-12-25," they enter this information into an input form. The device sends that information to the server, and the server sends an API request to an external reservation system based on the reservation information. For example, the reservation is made using the API of an external reservation system (e.g., Booking.com). At this time, the server receives a reservation confirmation response from the external system and notifies the user of the result via the device. As a specific example, information such as "The hotel reserved is Hôtel Le Meurice" is provided.

[1295] QA support

[1296] While traveling, the user inputs questions into the device about recommended facilities and restaurants at the destination. For example, "What restaurant do you recommend?" The device sends the question to the server, which uses a generative AI model to generate an appropriate answer. For example, a recommendation for a restaurant called "Gourmet Delight" is generated. The answer generated by the server is provided to the user via the device.

[1297] Troubleshooting

[1298] If a user encounters a problem during their trip, they report the problem from their device. For example, if their flight is canceled, they enter "Flight canceled." The device then sends this trouble information to the server. The server then takes appropriate action, such as arranging a new flight, based on the trouble information. The server then notifies the user of the progress and results of the action via the device. For example, the user may be notified that "Your flight has been canceled. Arranging a new flight..."

[1299] Examples of prompt statements

[1300] Travel planning assistance:

[1301] Input: "Destinations are Paris and London, visit dates are 2023-12-20 to 2023-12-25. Please make travel plans."

[1302] Output: "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[1303] API integration with external booking systems:

[1304] Input: "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25."

[1305] Output: "The hotel booked is Hôtel Le Meurice."

[1306] QA Support:

[1307] Input: "What are the best restaurants in Paris?"

[1308] Output: "The restaurant I recommend is Gourmet Delight."

[1309] Troubleshooting:

[1310] Enter: "Flight canceled. Please find a new flight."

[1311] Output: "Your flight has been cancelled. We are arranging a new flight..."

[1312] As described above, this invention allows users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[1313] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1314] Step 1: User enters travel information

[1315] Description: The user enters a travel destination and visit dates into the terminal.

[1316] Input: Travel destination (e.g., "Paris" or "London"), visit date (e.g., "2023-12-20" to "2023-12-25").

[1317] Specific operation: The user enters the travel destination and itinerary into the input form on the device screen. The device accurately records the entered data.

[1318] Step 2: Submit your trip information

[1319] Description: The device sends travel information to the server.

[1320] Input: User-entered travel destination and visit date information.

[1321] Output: JSON formatted travel information data.

[1322] What happens: The device converts the user's input into JSON format and sends it to the server's API endpoint using the HTTPS protocol.

[1323] Step 3: Generate a travel plan

[1324] Description: Based on the travel information received by the server, a generative AI model is used to generate a travel plan.

[1325] Input: JSON-formatted trip information data sent from the device.

[1326] Output: The generated itinerary.

[1327] Specific operation: The server extracts the travel destinations and dates from the received data and sends them as prompts to the generative AI model (e.g., "The travel destinations are Paris and London, and the visit dates are from 2023-12-20 to 2023-12-25. Please create a travel plan."). The plan obtained from the model is saved in a database.

[1328] Step 4: Provide your travel plans

[1329] Description: The server provides the travel plan generated by the server to the user via the terminal.

[1330] Input: The generated itinerary.

[1331] Output: Display of travel plan.

[1332] Specific operation: The server retrieves the generated travel plan from the database and sends it to the device. The device displays the received plan on the screen (e.g., "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace.").

[1333] Step 5: Enter reservation information

[1334] Description: A user enters accommodation and transportation reservation information into a device.

[1335] Input: The facility and period you want to book (e.g. "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25").

[1336] Specific operation: The user accesses the reservation form on the device and enters the reservation information.

[1337] Step 6: Submit your reservation information

[1338] Description: The terminal sends reservation information to the server.

[1339] Input: Booking information entered by the user.

[1340] Output: Reservation information data formatted in JSON.

[1341] Specific operation: The terminal converts the reservation information into JSON format and sends it to the server via HTTPS protocol.

[1342] Step 7: API Request to External System

[1343] Description: The server sends an API request to an external reservation system based on the reservation information.

[1344] Input: Reservation information data in JSON format sent from the terminal.

[1345] Output: The API request sent to the external booking system.

[1346] Specific operation: The server analyzes the received data and sends an HTTP POST request to the external reservation system (e.g., the API of the reservation site).

[1347] Step 8: Confirmation and notification of booking

[1348] Description: The server receives the response from the external system and notifies the user of the reservation result.

[1349] Input: Booking confirmation response from the external system.

[1350] Output: Display of reservation confirmation information.

[1351] Specific behavior: The server receives the response from the external system and sends it in JSON format to the terminal. The terminal displays the received information on the user interface and notifies the user that the reservation has been confirmed (e.g., "The reserved hotel is Hôtel Le Meurice.").

[1352] Step 9: Enter your question

[1353] Description: User types questions into terminal while traveling.

[1354] Input: The question (e.g., "What restaurants do you recommend?").

[1355] Specific behavior: The user types a question into the chatbot interface on the device.

[1356] Step 10: Submit your question

[1357] Description: The device sends a question to the server.

[1358] Input: The question typed by the user.

[1359] Output: Question data formatted in JSON.

[1360] Specific operation: The device converts the question into JSON format and sends it to the server via HTTPS protocol.

[1361] Step 11: Generate an answer

[1362] Description: The server uses a generative AI model to generate an appropriate answer based on the question.

[1363] Input: Question data sent from the terminal.

[1364] Output: The generated answer.

[1365] How it works: The server sends the question as a prompt to the generative AI model and receives an answer from the model (e.g., "The restaurant I recommend is Gourmet Delight.").

[1366] Step 12: Provide your answers

[1367] Description: The server provides the generated answer to the user via the terminal.

[1368] Input: The generated answer.

[1369] Output: Display of the answer to the question.

[1370] Specific operation: The server sends the generated answer to the device, and the device displays the received information on the screen.

[1371] Step 13: Report a problem

[1372] Description: Allows users to report any trouble they encounter while traveling from their device.

[1373] Input: Problem description (e.g. "My flight has been cancelled").

[1374] Specific actions: The user accesses the device's reporting form and enters specific trouble information.

[1375] Step 14: Submit your trouble report

[1376] Description: The device sends trouble information to the server.

[1377] Input: Trouble information entered by the user.

[1378] Output: Trouble information data formatted in JSON format.

[1379] Specific operation: The device converts the trouble information into JSON format and sends it to the server via HTTPS protocol.

[1380] Step 15: Initiate response

[1381] Description: The server will take appropriate action based on the trouble information.

[1382] Input: Trouble information data sent from the device.

[1383] Output: A request to start the response process.

[1384] Specific behavior: The server analyzes the received trouble information and sends a request to an external system to initiate appropriate action, such as arranging a new flight.

[1385] Step 16: Notification of troubleshoot

[1386] Description: The server notifies the user of the progress of the response.

[1387] Input: Progress of the response.

[1388] Output: Displays troubleshooting progress information.

[1389] Specific behavior: The server sends progress information in JSON format to the device, and the device displays the received information in the user interface (e.g., "Your flight has been canceled. Arranging a new flight...").

[1390] (Application example 1)

[1391] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1392] Conventional travel support systems handle the creation of travel plans, reservations, answering questions, and troubleshooting individually, making it difficult for users to centrally manage everything, and there was a problem of a lack of convenience, especially when using meal services at destinations, as users had to go through a separate platform.In addition, there was also the problem that recommendations for meal services at travel destinations and reservation management were not integrated, which increased the effort required to make meal arrangements during travel.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1394] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for accepting questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to address those problems, means for recommending meal services at locations of stay based on the travel plan, and means for managing reservations for the recommended meal services. This allows users to perform everything from creating a travel plan to answering questions and responding to them during the trip, and even arranging meals at locations of stay, all within the system.

[1395] "User" refers to an individual who creates or changes travel plans, asks questions, or reports problems.

[1396] The "means for inputting travel destination and itinerary" is an interface for inputting the travel destination, departure date, and return date.

[1397] "Means for generating travel plans" refers to a system that has the function of automatically creating a travel itinerary and detailed schedule based on inputted travel destinations and dates.

[1398] "Means for executing reservations in cooperation with external reservation systems" refers to a system that communicates with an external reservation platform to complete reservations for accommodation, transportation, etc. based on the contents of travel plans.

[1399] "Means of accepting questions and providing appropriate answers to those questions" refers to a system that responds to various questions from users while traveling and provides answers using pre-registered data or external databases.

[1400] "Means for receiving trouble information and taking appropriate measures to resolve the trouble" refers to a system that receives from users information about problems or troubles that occur during their trip and provides appropriate measures to resolve the trouble.

[1401] The "means for recommending local food services" is a system that has the function of suggesting to users food services available in places to be visited based on a travel plan.

[1402] "Means for managing reservations for meal services" refers to a system that records and manages reservations made by users for recommended meal services, and makes changes or cancellations as necessary.

[1403] An "API integration means" is a component that provides a common interface for different systems to communicate with each other, enabling the sending and receiving of information.

[1404] "Means for sending push notifications of service cancellations" refers to a system that has the function of immediately notifying users of sudden service cancellations during travel to their devices.

[1405] To implement this invention, the travel support system must be configured with a server and terminals. The server comprehensively manages various functions, such as generating travel plans, linking with external reservation systems, answering questions, dealing with problems, recommending meal services, and managing reservations. The terminal functions as an interface for users to input travel destinations and itineraries and to make inquiries and respond to requests during the trip.

[1406] Program Generation

[1407] The program for this system must have the following functions:

[1408] First, we provide an interface for users to input their travel destination and itinerary. This interface is implemented as a smartphone application. For example, it includes a function to input "Paris" as the travel destination and "2023-12-20 to 2023-12-25" as the itinerary.

[1409] The server then generates a travel plan based on the input information by analyzing the received data and extracting information such as optimal travel routes, tourist spots, and accommodations from a database to generate the travel plan.

[1410] In addition, by linking with an external reservation system, reservations are made based on the generated travel plans. Specifically, the server sends requests to the external reservation service API to complete reservations for accommodations and transportation.

[1411] During the trip, the server will provide a function to accept questions from users, such as "What are the best restaurants in Paris?", and will refer to the database to provide an appropriate answer.

[1412] Furthermore, the system receives information about troubles that occur during travel and takes appropriate measures to resolve them. For example, if a flight is canceled, the server will start arranging a new flight and notify the user's device, "Your flight has been canceled. Arranging a new flight..."

[1413] Newly added features include a way to recommend dining options based on travel plans. The server extracts popular restaurants and delivery services from a database and suggests them to users. Once a user selects a restaurant, the server manages the reservation.

[1414] For example, if a user wants to order "Pasta Carbonara" and "Tiramisu" in "Paris," the server connects to an external food delivery service API and confirms the order.

[1415] Example prompt sentence:

[1416] Enter your travel plan data.

[1417] Destination: Paris

[1418] Start date: 2023-12-20

[1419] End date: 2023-12-25

[1420] Place a food delivery order from a specific restaurant.

[1421] Restaurant ID: 456

[1422] Order Details:

[1423] Pasta Carbonara (2)

[1424] Tiramisu (1)

[1425] Delivery address: 123 Champs-Élysées, Paris

[1426] As described above, the system of the present invention links the server and terminals to provide consistent support from planning to execution of the trip, troubleshooting, answering questions, and arranging meal services. This allows travelers to use many functions in a unified manner, greatly improving the convenience of the entire trip.

[1427] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1428] Step 1:

[1429] The user inputs the travel destination and dates.

[1430] As input, the user enters the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20 to 2023-12-25").

[1431] The terminal receives this information and sends it to the server. As output, the travel destination and itinerary data is sent to the server.

[1432] Step 2:

[1433] The server generates a travel plan based on the received information.

[1434] As input, the server receives destination and itinerary data.

[1435] The server references the database and generates a travel plan by combining the best sightseeing routes, accommodations, and means of transportation.

[1436] As an output, the generated itinerary is stored in a database and provided to the user.

[1437] Step 3:

[1438] Execute reservations in conjunction with an external reservation system.

[1439] As input, the server uses the generated itinerary data.

[1440] The server sends requests to an external reservation system API to make reservations for accommodation and transportation.

[1441] As an output, the reservation completion information is returned to the server and stored in a database.

[1442] Step 4:

[1443] Accept questions from users during their trip and provide appropriate answers.

[1444] As input, the user types a question through the terminal (e.g., "What are some recommended restaurants in Paris?").

[1445] The terminal sends a query to the server.

[1446] The server consults the database and extracts the appropriate answer (e.g., "a restaurant called Gourmet Delight") from the data.

[1447] As an output, the answer is displayed on the user's terminal.

[1448] Step 5:

[1449] Receive information about troubles that may occur during travel and take appropriate action.

[1450] As input, the user enters trouble information (e.g., "flight canceled") into the terminal.

[1451] The terminal transmits the trouble information to the server.

[1452] The server analyzes the trouble information and initiates procedures to take appropriate action (e.g., arranging a new flight).

[1453] As an output, the progress of troubleshooting is notified to the user's terminal.

[1454] Step 6:

[1455] Recommend food services at your destination based on your travel plans.

[1456] As input, the server uses travel plan data to extract meal services from a database that are tailored to the user's location and time of stay.

[1457] The server sends recommended restaurants and delivery services to the user's device.

[1458] As an output, recommended meal services are displayed on the user's device.

[1459] Step 7:

[1460] Manage reservations for recommended meal services.

[1461] As input, the user selects which of the recommended meal services they wish to book (e.g., ordering "Pasta Carbonara" and "Tiramisu").

[1462] The terminal transmits the order information to the server.

[1463] The server connects to an external food delivery service API and sends a request to confirm the order.

[1464] As an output, information about the completion of the order is returned to the server and displayed on the user's terminal.

[1465] The above are the specific processing steps of this system.

[1466] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1467] This is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions and adjusts services based on those emotions, user satisfaction can be further improved.

[1468] Travel planning support and emotion recognition

[1469] The user inputs their travel destination and visit date. For example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively. The device sends the user's input in JSON format to the server. The server generates a travel plan based on the received data and saves it in a database. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice when they enter their information, and adjusts the travel plan based on those emotions. For example, if the user indicates that they want to "relax," it will suggest relaxing tourist spots and activities.

[1470] API integration with external reservation systems and emotion recognition

[1471] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, the emotion engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation result notification based on those emotions. For example, if the user is in an "anxious" state, the reservation confirmation email will include additional support information.

[1472] QA and Emotion Recognition

[1473] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device then sends the question to the server, which then provides an appropriate answer. The emotion engine recognizes the emotion the user expressed when entering the question and adjusts the answer accordingly. For example, if the user is "excited," it will suggest special events or activities.

[1474] Troubleshooting and Emotion Recognition

[1475] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. At this time, the emotion engine recognizes the user's emotion when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a quick and detailed response will reduce the user's dissatisfaction.

[1476] This invention makes it possible to recognize users' emotions in real time and adjust service content based on those emotions. This system provides consistent support from planning a trip to responding during the trip, and by combining it with an emotion engine, it can significantly improve traveler satisfaction and reliability.

[1477] The processing flow will be explained below.

[1478] Travel planning support and emotion recognition

[1479] Step 1:

[1480] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[1481] Step 2:

[1482] The device sends the user input information to the server in JSON format.

[1483] Example of transmitted data:

[1484] json

[1485] {

[1486] "user_id": "12345",

[1487] "destinations": ["Paris", "London"],

[1488] "dates": ["2023-12-20", "2023-12-25"]

[1489] }

[1490] Step 3:

[1491] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice when inputting. For example, it analyzes whether the user is expressing the emotion "I want to relax."

[1492] Step 4:

[1493] The server generates a travel plan based on the recognized emotion and the input information. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots.

[1494] Step 5:

[1495] The server stores the generated travel plan in a database and sends it to the terminal, which displays the plan to the user.

[1496] API integration with external reservation systems and emotion recognition

[1497] Step 1:

[1498] The user inputs reservation information for accommodation and transportation. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[1499] Step 2:

[1500] The device sends the reservation information in JSON format to the server.

[1501] Example of transmitted data:

[1502] json

[1503] {

[1504] "user_id": "12345",

[1505] "destination": "Paris",

[1506] "check_in": "2023-12-20",

[1507] "check_out": "2023-12-25"

[1508] }

[1509] Step 3:

[1510] The server receives the reservation information and sends a POST request to the external reservation system API.

[1511] Example API request:

[1512] python

[1513] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[1514] Step 4:

[1515] The server uses an emotion engine to recognize the user's emotions during the reservation process, for example, analyzing whether the user is in an "anxious" state during the reservation process.

[1516] Step 5:

[1517] The server tailors the booking outcome based on the perceived emotion, for example including additional support information in the booking confirmation email if the user indicates "anxiety."

[1518] Step 6:

[1519] The server sends the reservation results in JSON format to the terminal, and the terminal notifies the user.

[1520] QA and Emotion Recognition

[1521] Step 1:

[1522] The user inputs questions about recommended facilities and restaurants at the destination into the terminal. For example, the user inputs the question "must_visit" (what are the must-see spots?).

[1523] Step 2:

[1524] The device sends the question in JSON format to the server.

[1525] Example of transmitted data:

[1526] json

[1527] {

[1528] "user_id": "12345",

[1529] "question": "must_visit"

[1530] }

[1531] Step 3:

[1532] The server receives the question and uses an emotion engine to recognize the user's emotion when entering the question. For example, it analyzes whether the user is "excited."

[1533] Step 4:

[1534] The server generates appropriate responses and tailors them based on the emotion, for example, suggesting special events or activities if the user is "excited."

[1535] Step 5:

[1536] The server sends the answer in JSON format to the device, which displays it to the user.

[1537] Troubleshooting and Emotion Recognition

[1538] Step 1:

[1539] The user enters the problem into the terminal, for example, "flight_cancelled."

[1540] Step 2:

[1541] The device sends trouble information to the server in JSON format.

[1542] Example of transmitted data:

[1543] json

[1544] {

[1545] "user_id": "12345",

[1546] "issue": "flight_cancelled"

[1547] }

[1548] Step 3:

[1549] The server receives the trouble information and uses an emotion engine to recognize the user's emotion when reporting the trouble. For example, it analyzes whether the user is expressing anger.

[1550] Step 4:

[1551] The server responds appropriately based on the recognized emotion, for example, if the user expresses anger, it responds quickly and in detail.

[1552] Step 5:

[1553] The server sends the response results in JSON format to the device, which then displays them to the user.

[1554] Example 2

[1555] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1556] Conventional travel support systems have difficulty recognizing users' emotions in real time and adjusting service content based on those emotions. Furthermore, there is a need for a method to sufficiently improve user satisfaction in travel planning, linking with external reservation systems, and responding to questions and problems during the trip.

[1557] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for transmitting the input information in structured data format; a means for analyzing emotions based on the structured data and generating a travel plan; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for analyzing the user's emotions during the reservation process and adjusting notification content based on the emotions; a means for receiving questions from the user during the trip and providing appropriate answers to the questions; and a means for receiving information about problems that occur during the trip, taking appropriate measures to deal with the problems, and adjusting the response method based on the user's emotions. This makes it possible to recognize the user's emotions in real time and adjust service content based on the emotions.

[1558] "User" refers to an individual who uses the System to make travel plans, make reservations, ask questions, or report problems.

[1559] "Destination" refers to a geographic location that a user wishes to visit.

[1560] "Itinerary" refers to time information including the start and end dates of a trip or the duration of stay at each location visited.

[1561] A "structured data format" refers to a data format that represents information in a regular and unified format, making it easier to process within databases and information systems.

[1562] "Emotion" refers to the psychological state or reaction that a user shows toward a system, such as feelings of relaxation, excitement, or anxiety.

[1563] "Travel Itinerary" means a series of suggestions based on your travel destination and dates, including attractions, activities, and accommodations.

[1564] "Third-party booking systems" refers to online services provided by other companies for booking accommodation and transportation.

[1565] "API integration" refers to the use of application program interfaces to share data and functions between different software systems.

[1566] "Question" refers to an inquiry made by a user to the system to obtain information or ask for advice.

[1567] "Trouble Information" refers to information that users use to report to the system any problems or difficulties they encounter during their trip.

[1568] "Notification" refers to information or messages sent by the system to the user.

[1569] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with external reservation systems, and responds to questions and troubles during the trip. It also has an emotion engine that recognizes the user's emotions and adjusts services accordingly.

[1570] In order to implement the present invention, the following hardware and software are used.

[1571] Hardware:

[1572] Terminal: The device on which the user enters information (e.g., smartphone, tablet, PC)

[1573] Server: A host computer used to process and store data and run the emotion engine.

[1574] Database: a storage system for storing travel plans, reservation information, etc.

[1575] software:

[1576] Emotion engine: Software for recognizing user emotions and adjusting services

[1577] API with external booking systems: Interface for booking accommodations, transportation, etc.

[1578] Question-answering system: An AI model for answering questions from users

[1579] Specific examples of the present invention are given below.

[1580] Example 1:

[1581] A user inputs into the device that they are traveling to "Paris" and departing on "2023-12-20." The device converts this information into JSON format and sends it to the server. The server receives this information and uses an emotion engine to analyze the emotion of the user at the time of input. For example, if the emotion "I want to relax" is detected, the server creates a travel plan that suggests relaxing tourist spots and saves it in the database.

[1582] Example prompt: "The user wants to visit Paris and London and is looking for a relaxing trip."

[1583] Example 2:

[1584] The user enters the accommodation "Hotel A" and "2023-12-20~2023-12-25" and the transportation method "Flight B" into the terminal. The terminal sends this information to the server. The server sends an API request to an external reservation system based on the information and confirms the reservation. Furthermore, if the user expresses "anxiety," the server includes additional support information in the reservation confirmation email and sends it.

[1585] Example 3:

[1586] While traveling, a user can type "What are some good restaurants in this area?" into their device, which then sends the question to the server. The server generates an answer to the question and, if it determines that the user is "excited," suggests special events or activities.

[1587] Example 4:

[1588] When a problem occurs during a trip and the user types "my flight is delayed" into the device, the device sends the problem information to the server. The server responds quickly and in detail based on this information, and if the user shows anger, it further adjusts the response method and notifies the user.

[1589] In this way, a specific embodiment of the present invention can recognize a user's emotions in real time and adjust travel plans and various responses based on those emotions, thereby significantly improving user satisfaction.

[1590] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1591] Step 1:

[1592] The user inputs the travel destination and dates

[1593] Specific operation: The user uses the terminal to input the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20").

[1594] Input: Travel destination and itinerary information

[1595] Output: Structured data of destinations and dates

[1596] Step 2:

[1597] The terminal converts the input content into JSON format and sends it.

[1598] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"destination": "Paris", "date": "2023-12-20"}).

[1599] Input: Structured data of destination and dates

[1600] Output: JSON format data

[1601] Step 3:

[1602] The server receives the data and analyzes it with the emotion engine.

[1603] Specific operation: The server receives the JSON data sent from the device and analyzes the user's emotions using the emotion engine. For example, it detects the emotion "I want to relax."

[1604] Input: JSON format data

[1605] Output: Sentiment analysis results

[1606] Step 4:

[1607] The server generates the itinerary and stores it in the database.

[1608] Specific operation: The server generates a travel plan based on the results of the sentiment analysis, for example, a plan including relaxing tourist spots and activities, and stores it in a database.

[1609] Input: Sentiment analysis results

[1610] Output: Travel plan data

[1611] Step 5:

[1612] The user enters accommodation and transportation reservations

[1613] Specific operation: The user uses the terminal to enter information about accommodation (e.g., "Hotel A") and transportation (e.g., "Flight B").

[1614] Input: Accommodation and transportation information

[1615] Output: Accommodation and transportation structured data

[1616] Step 6:

[1617] The device sends the reservation information to the server.

[1618] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"hotel": "Hotel A", "flight": "Flight B"}).

[1619] Input: Accommodation and transportation structured data

[1620] Output: JSON format data

[1621] Step 7:

[1622] The server sends an API request to the external booking system

[1623] Specific operation: The server sends an API request to the external reservation system based on the reservation information and confirms the reservation.

[1624] Input: Reservation information in JSON format

[1625] Output: Reservation confirmation result

[1626] Step 8:

[1627] The server adjusts and notifies the reservation result based on emotions.

[1628] Specific operation: The server receives the reservation confirmation result and uses the emotion engine to analyze the user's current emotion. For example, if the user is in an "anxious" state, the server sends the reservation confirmation email to the user with additional support information.

[1629] Input: Booking confirmation results, sentiment analysis

[1630] Output: Adjusted booking confirmation

[1631] Step 9:

[1632] The user enters a question

[1633] Specific operation: The user uses the device to input questions such as recommended facilities and restaurants at the destination.

[1634] Input: Question content

[1635] Output: Question structured data

[1636] Step 10:

[1637] The device sends a question to the server

[1638] Specific operation: The device sends the question content to the server in JSON format (e.g., "{"question": "What are some good restaurants in this area?"}).

[1639] Input: Question structured data

[1640] Output: Question data in JSON format

[1641] Step 11:

[1642] The server generates the answer and adjusts it based on the emotion.

[1643] What it does: The server generates answers to questions and uses an emotion engine to tailor the content based on the user's emotions. If the user is "excited," it suggests special events and activities.

[1644] Input: JSON format question data, sentiment analysis

[1645] Output: Adjusted answer notification

[1646] Step 12:

[1647] A user reports a problem

[1648] Specific operation: The user uses the terminal to input trouble information (e.g., "The flight is delayed").

[1649] Input: Trouble information

[1650] Output: Structured data of trouble information

[1651] Step 13:

[1652] The device sends trouble information to the server

[1653] Specific operation: The device sends trouble information to the server in JSON format (e.g., {"trouble": "The flight was delayed"}).

[1654] Input: Structured data of trouble information

[1655] Output: Trouble information in JSON format

[1656] Step 14:

[1657] The server adjusts and notifies the response method based on the emotion.

[1658] Specific operation: The server takes appropriate action based on the trouble information it receives. If the emotion engine recognizes that the user's emotion is "anger," it takes a swift and detailed action, adjusts the response method, and notifies the user.

[1659] Input: JSON format trouble information, sentiment analysis

[1660] Output: Coordinated response notification

[1661] (Application example 2)

[1662] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1663] Conventional travel plan generation systems do not take into account user emotions and lack full support during travel, especially in autonomous vehicles. Therefore, it is important to reduce the anxiety and frustration users feel during travel and improve overall satisfaction with the travel experience.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for generating a travel plan based on the input information; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for accepting questions from the user during the travel and providing appropriate answers to the questions; a means for receiving information about troubles during the travel and taking appropriate measures to deal with the troubles; a means for recognizing the user's emotions using an emotion recognition engine and adjusting service content based on the emotions; and a means for coordinating with the navigation system of an autonomous vehicle to provide route guidance based on the travel plan. This makes it possible to provide a high level of satisfaction that takes the user's emotions into consideration, from travel planning to responses during the trip.

[1665] The "means for inputting travel destinations and itineraries" is an interface for the user to input the places he or she wishes to visit and the dates on which he or she will visit those places.

[1666] The "means for generating a travel plan" is a system for automatically creating a plan based on the travel destination and itinerary input by the user.

[1667] The "means of executing reservations in cooperation with external reservation systems" is a function of exchanging information with other external systems in order to reserve accommodations and transportation based on travel plans.

[1668] "Means of accepting questions and providing appropriate answers to those questions" refers to a function that quickly responds to questions and information requests that users have while traveling and provides appropriate information.

[1669] "A means of receiving trouble information and taking appropriate action against the trouble" is a system that proposes and responds to problems and difficulties that arise during travel.

[1670] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice and recognizes their emotional state.

[1671] "Means for recognizing the user's emotions and adjusting the service content based on those emotions" refers to a function that uses an emotion recognition engine to understand the user's emotions and optimize the service according to that state.

[1672] "Means for linking with the navigation system of an autonomous vehicle and providing route guidance based on a travel plan" refers to a system that communicates with the navigation system installed in an autonomous vehicle and provides guidance on the optimal route based on the generated travel plan.

[1673] Travel planning support and emotion recognition

[1674] The user first enters their travel destination and visit dates. For example, suppose the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The user's device sends this information in JSON format to the server. The server generates a travel plan based on the received data and stores it in a database. Furthermore, an emotion recognition engine recognizes emotions from the user's facial expressions and voice when entering information, and adjusts the travel plan based on those emotions. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots and activities.

[1675] API integration with external reservation systems and emotion recognition

[1676] When a user makes a reservation for accommodation or transportation, they input the reservation information on their device. The device then sends that information to a server, which then sends an API request to an external reservation system based on the reservation information to confirm the reservation. An emotion recognition engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation notification based on those emotions. For example, if the user is anxious, the reservation confirmation email can include additional support information.

[1677] QA and Emotion Recognition

[1678] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the content of the answer based on that emotion. For example, if the user is excited, it can suggest special events or activities.

[1679] Troubleshooting and Emotion Recognition

[1680] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. At this time, an emotion recognition engine recognizes the user's emotions when reporting the problem and adjusts the response method based on those emotions. For example, if the user expresses anger, a quick and detailed response can be provided to reduce the user's dissatisfaction.

[1681] Collaboration with autonomous vehicles

[1682] This system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary, and can adjust in real time based on the user's emotions by providing the navigation system with the best tourist spots and routes at each point during the trip.

[1683] Hardware and software used

[1684] 1. Hardware:

[1685] Camera: Recognizes the user's facial expressions.

[1686] Microphone: Recognizes the user's voice.

[1687] Display: Shows information.

[1688] Autonomous driving system: Provides route guidance based on trip plans.

[1689] 2. Software:

[1690] Emotion recognition engine: Analyzes emotions from facial expressions and voice.

[1691] Database: Stores travel planning and booking information.

[1692] API: Integrates with external reservation systems.

[1693] Examples of concrete examples and prompts

[1694] Travel planning prompt:

[1695] The user's travel destinations are Tokyo and Osaka, and the visit dates are December 20th to December 25th, 2023. This indicates a desire to relax. Please suggest a relaxing travel plan.

[1696] Booking anxiety detection prompt:

[1697] The user is booking a triple room at a hotel and is expressing anxiety. Please generate a booking confirmation email with additional support information.

[1698] This will make it possible to provide services that take into account the user's emotions at each stage of travel planning and troubleshooting.

[1699] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1700] Step 1:

[1701] The user enters the travel destination and itinerary. For example, the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The device sends the entered information to the server in JSON format. The data format of the entered information is JSON data as shown below.

[1702] json

[1703] {

[1704] "destinations": ["Tokyo", "Osaka"],

[1705] "dates": ["2023-12-20", "2023-12-25"]

[1706] }

[1707] Step 2:

[1708] The server analyzes the received JSON data and generates a travel plan. The generated travel plan is saved in a database. An emotion recognition engine analyzes the user's facial expressions and voice when inputting the plan and recognizes their emotion. For example, if the user indicates that they want to relax, relaxing tourist spots and activities are added to the travel plan.

[1709] Input: Travel destination and schedule, user facial expression and voice data

[1710] Output: Adjusted itinerary

[1711] Specific behavior:

[1712] The server uses an emotion recognition engine to extract emotional data from the user's facial expressions and voice, and then customizes the travel plan based on that emotional data.

[1713] Step 3:

[1714] Reservation information for accommodation and transportation is entered on the device. The device then sends the entered reservation information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, an emotion recognition engine recognizes the user's emotions during the reservation process, and if the user is anxious, a reservation confirmation email containing additional support information is sent.

[1715] Input: Reservation information, user facial and voice data

[1716] Output: Reservation confirmation email

[1717] Specific behavior:

[1718] The device inputs reservation information and sends it to the server. The server connects to the external reservation system via API to confirm the reservation, and an emotion recognition engine analyzes the user's emotions and sends a reservation confirmation email.

[1719] Step 4:

[1720] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the answer based on that emotion. For example, if the user is excited, it will suggest special events or activities.

[1721] Input: Question content, user facial expression and voice data

[1722] Output: Adjusted answer

[1723] Specific behavior:

[1724] The device inputs a question and sends it to the server, which then searches a database for the best answer and uses an emotion recognition engine to check the user's emotions and adjust the answer accordingly.

[1725] Step 5:

[1726] When a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. An emotion recognition engine recognizes the emotion the user expressed when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a prompt and detailed response is provided.

[1727] Input: Trouble information, user facial expression and voice data

[1728] Output: Coordinated response

[1729] Specific behavior:

[1730] The device inputs trouble information and sends it to the server, which uses an emotion recognition engine to analyze the user's emotions, adjusts countermeasures, and promptly communicates them.

[1731] Step 6:

[1732] The system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary. The navigation system displays route guidance based on the itinerary in real time, and an emotion recognition engine monitors the user's emotions and adjusts route guidance as needed.

[1733] Input: Travel plans, real-time user sentiment data

[1734] Output: Adjusted route guidance

[1735] Specific behavior:

[1736] The navigation system captures your trip plans and provides real-time directions, while an emotion recognition engine monitors your emotions and adjusts directions as needed.

[1737] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1738] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1739] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1740] [Fourth embodiment]

[1741] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1742] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1743] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1744] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1745] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1746] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1747] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1748] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1749] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1750] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1751] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1752] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1753] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1754] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, links with external reservation systems, and responds to questions and troubles during the trip. The system of the present invention integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[1755] Travel planning assistance

[1756] The user inputs the travel destination and the dates of the visit. The terminal receives the user's input and sends the information to the server. The server receives the information, generates a travel plan, and saves it in a database. For example, if the user plans to visit "Paris" and "London" on dates "2023-12-20" and "2023-12-25," the server will create a travel plan based on that and provide it to the user.

[1757] API integration with external reservation systems

[1758] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information, confirming the reservation. For example, if a user wants to reserve a hotel in Paris from 2023-12-20 to 2023-12-25, the server sends the reservation information to the external reservation system, retrieves the results, and completes the reservation.

[1759] QA support

[1760] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device sends the question to the server, which then provides an appropriate answer. For example, if a user asks, "What restaurant do you recommend?", the server will recommend a restaurant called "Gourmet Delight."

[1761] Troubleshooting

[1762] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. For example, if the user's flight is canceled, the server will initiate the process of arranging a new flight and notify them, "Your flight has been canceled. We are arranging a new flight..."

[1763] This invention enables users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[1764] The processing flow will be explained below.

[1765] Travel planning assistance

[1766] Step 1:

[1767] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[1768] Step 2:

[1769] The device sends the user's input in JSON format to the server.

[1770] Example of transmitted data:

[1771] json

[1772] {

[1773] "user_id": "12345",

[1774] "destinations": ["Paris", "London"],

[1775] "dates": ["2023-12-20", "2023-12-25"]

[1776] }

[1777] Step 3:

[1778] The server generates a travel plan based on the received data and stores it in a database. For example, it stores each destination and visit date as follows:

[1779] python

[1780] self.plan = [{"destination": "Paris", "date": "2023-12-20"}, {"destination": "London", "date": "2023-12-25"}]

[1781] API integration with external reservation systems

[1782] Step 1:

[1783] The user inputs reservation information. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[1784] Step 2:

[1785] The device sends the reservation information to the server in JSON format.

[1786] Example of transmitted data:

[1787] json

[1788] {

[1789] "user_id": "12345",

[1790] "destination": "Paris",

[1791] "check_in": "2023-12-20",

[1792] "check_out": "2023-12-25"

[1793] }

[1794] Step 3:

[1795] The server receives the reservation information and sends a POST request to the external reservation system API.

[1796] Example API request:

[1797] python

[1798] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[1799] Step 4:

[1800] The server receives the response from the API and notifies the user of the reservation result.

[1801] QA support

[1802] Step 1:

[1803] The user types a question into the terminal, for example, "must_visit" (what are the must-see spots?).

[1804] Step 2:

[1805] The terminal sends a question to the server.

[1806] Example of transmitted data:

[1807] json

[1808] {

[1809] "user_id": "12345",

[1810] "question": "must_visit"

[1811] }

[1812] Step 3:

[1813] The server receives the question and provides the appropriate answer.

[1814] Example answer:

[1815] python

[1816] answers = {

[1817] "best_restaurant": "The best restaurant in town is 'Gourmet Delight'.",

[1818] "must_visit": "You must visit the 'Historic Cathedral'."

[1819] }

[1820] answer = answers.get(question, "Sorry, I don't have an answer for that.")

[1821] Step 4:

[1822] The server sends the answer in JSON format to the device, which displays it to the user.

[1823] Troubleshooting

[1824] Step 1:

[1825] The user inputs a problem into the terminal, for example, "flight_cancelled" (the flight was canceled).

[1826] Step 2:

[1827] The device sends the trouble information to the server.

[1828] Example of transmitted data:

[1829] json

[1830] {

[1831] "user_id": "12345",

[1832] "issue": "flight_cancelled"

[1833] }

[1834] Step 3:

[1835] The server receives the trouble information and takes appropriate action.

[1836] Examples of support:

[1837] python

[1838] if issue == 'flight_cancelled':

[1839] response = "Your flight has been canceled. Booking a new flight for you..."

[1840] elif issue == 'need_hotel_extension':

[1841] response = "Booking an extension for your hotel stay..."

[1842] else:

[1843] response = "Sorry, I cannot handle that issue."

[1844] Step 4:

[1845] The server transmits the correspondence result to the terminal, which displays it to the user.

[1846] Example 1

[1847] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1848] Modern travelers need multiple services, such as creating travel plans, booking accommodations and transportation, and answering questions and dealing with problems while traveling, but managing these tasks individually is extremely cumbersome. Conventional systems are unable to provide centralized support for these tasks, placing a heavy burden on travelers, which is a problem.

[1849] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1850] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for receiving questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to deal with those problems, and means for using a generative AI model to generate the travel plan and handle the problems. This allows travelers to centrally manage their travel plans and receive the services they need smoothly and quickly.

[1851] "User" means any individual or group of people who use the Travel Planning System.

[1852] "Destination" means a geographic location that you intend to visit.

[1853] "Dates" refers to the period of time for which the user plans to travel.

[1854] "Travel Plan" refers to an itinerary or schedule created based on a User's travel destination and dates.

[1855] "External Reservation System" refers to an external online system used to make reservations such as hotel or transportation reservations.

[1856] An "API" is something that provides an interface for different software systems to communicate with each other.

[1857] "Generative AI models" refer to algorithms and techniques that use artificial intelligence to generate optimal responses and plans from data.

[1858] "Push transmission" refers to a technology in which a server sends data or notifications directly to a user's device.

[1859] "Trouble Information" refers to information about problems or obstacles that users encounter during their trip.

[1860] "Appropriate response" refers to the optimal measures or solutions that the system takes in response to trouble information.

[1861] This is an integrated system that starts when a user inputs their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. This system integrates multiple functions that travelers need before and during their trip in one place, greatly improving convenience for travelers.

[1862] Specifically, the system is implemented as follows.

[1863] Travel planning assistance

[1864] The user uses the device to input the travel destination and visit dates. If the user plans to visit "Paris" and "London" and the dates are "2023-12-20" and "2023-12-25," the user enters this information into the input form on the device. The device receives this information and sends it to the server using the HTTPS protocol. The server uses the generative AI model to create an optimal travel plan and stores it in a database. The generated travel plan is provided to the user via the device. For example, the server may suggest, "In Paris, we recommend visiting the Louvre Museum and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[1865] API integration with external reservation systems

[1866] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information from their device. For example, if they want to reserve a hotel in "Paris" from "2023-12-20" to "2023-12-25," they enter this information into an input form. The device sends that information to the server, and the server sends an API request to an external reservation system based on the reservation information. For example, the reservation is made using the API of an external reservation system (e.g., Booking.com). At this time, the server receives a reservation confirmation response from the external system and notifies the user of the result via the device. As a specific example, information such as "The hotel reserved is Hôtel Le Meurice" is provided.

[1867] QA support

[1868] While traveling, the user inputs questions into the device about recommended facilities and restaurants at the destination. For example, "What restaurant do you recommend?" The device sends the question to the server, which uses a generative AI model to generate an appropriate answer. For example, a recommendation for a restaurant called "Gourmet Delight" is generated. The answer generated by the server is provided to the user via the device.

[1869] Troubleshooting

[1870] If a user encounters a problem during their trip, they report the problem from their device. For example, if their flight is canceled, they enter "Flight canceled." The device then sends this trouble information to the server. The server then takes appropriate action, such as arranging a new flight, based on the trouble information. The server then notifies the user of the progress and results of the action via the device. For example, the user may be notified that "Your flight has been canceled. Arranging a new flight..."

[1871] Examples of prompt statements

[1872] Travel planning assistance:

[1873] Input: "Destinations are Paris and London, visit dates are 2023-12-20 to 2023-12-25. Please make travel plans."

[1874] Output: "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace."

[1875] API integration with external booking systems:

[1876] Input: "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25."

[1877] Output: "The hotel booked is Hôtel Le Meurice."

[1878] QA Support:

[1879] Input: "What are the best restaurants in Paris?"

[1880] Output: "The restaurant I recommend is Gourmet Delight."

[1881] Troubleshooting:

[1882] Enter: "Flight canceled. Please find a new flight."

[1883] Output: "Your flight has been cancelled. We are arranging a new flight..."

[1884] As described above, this invention allows users to centrally manage various problems they may encounter during their trip, and respond smoothly and quickly. This system provides consistent support from creating travel plans to dealing with problems during the trip, greatly improving convenience for travelers.

[1885] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1886] Step 1: User enters travel information

[1887] Description: The user enters a travel destination and visit dates into the terminal.

[1888] Input: Travel destination (e.g., "Paris" or "London"), visit date (e.g., "2023-12-20" to "2023-12-25").

[1889] Specific operation: The user enters the travel destination and itinerary into the input form on the device screen. The device accurately records the entered data.

[1890] Step 2: Submit your trip information

[1891] Description: The device sends travel information to the server.

[1892] Input: User-entered travel destination and visit date information.

[1893] Output: JSON formatted travel information data.

[1894] What happens: The device converts the user's input into JSON format and sends it to the server's API endpoint using the HTTPS protocol.

[1895] Step 3: Generate a travel plan

[1896] Description: Based on the travel information received by the server, a generative AI model is used to generate a travel plan.

[1897] Input: JSON-formatted trip information data sent from the device.

[1898] Output: The generated itinerary.

[1899] Specific operation: The server extracts the travel destinations and dates from the received data and sends them as prompts to the generative AI model (e.g., "The travel destinations are Paris and London, and the visit dates are from 2023-12-20 to 2023-12-25. Please create a travel plan."). The plan obtained from the model is saved in a database.

[1900] Step 4: Provide your travel plans

[1901] Description: The server provides the travel plan generated by the server to the user via the terminal.

[1902] Input: The generated itinerary.

[1903] Output: Display of travel plan.

[1904] Specific operation: The server retrieves the generated travel plan from the database and sends it to the device. The device displays the received plan on the screen (e.g., "In Paris, we recommend visiting the Louvre and the Eiffel Tower. In London, you can visit Big Ben and Buckingham Palace.").

[1905] Step 5: Enter reservation information

[1906] Description: A user enters accommodation and transportation reservation information into a device.

[1907] Input: The facility and period you want to book (e.g. "I would like to book a hotel in Paris from 2023-12-20 to 2023-12-25").

[1908] Specific operation: The user accesses the reservation form on the device and enters the reservation information.

[1909] Step 6: Submit your reservation information

[1910] Description: The terminal sends reservation information to the server.

[1911] Input: Booking information entered by the user.

[1912] Output: Reservation information data formatted in JSON.

[1913] Specific operation: The terminal converts the reservation information into JSON format and sends it to the server via HTTPS protocol.

[1914] Step 7: API Request to External System

[1915] Description: The server sends an API request to an external reservation system based on the reservation information.

[1916] Input: Reservation information data in JSON format sent from the terminal.

[1917] Output: The API request sent to the external booking system.

[1918] Specific operation: The server analyzes the received data and sends an HTTP POST request to the external reservation system (e.g., the API of the reservation site).

[1919] Step 8: Confirmation and notification of booking

[1920] Description: The server receives the response from the external system and notifies the user of the reservation result.

[1921] Input: Booking confirmation response from the external system.

[1922] Output: Display of reservation confirmation information.

[1923] Specific behavior: The server receives the response from the external system and sends it in JSON format to the terminal. The terminal displays the received information on the user interface and notifies the user that the reservation has been confirmed (e.g., "The reserved hotel is Hôtel Le Meurice.").

[1924] Step 9: Enter your question

[1925] Description: User types questions into terminal while traveling.

[1926] Input: The question (e.g., "What restaurants do you recommend?").

[1927] Specific behavior: The user types a question into the chatbot interface on the device.

[1928] Step 10: Submit your question

[1929] Description: The device sends a question to the server.

[1930] Input: The question typed by the user.

[1931] Output: Question data formatted in JSON.

[1932] Specific operation: The device converts the question into JSON format and sends it to the server via HTTPS protocol.

[1933] Step 11: Generate an answer

[1934] Description: The server uses a generative AI model to generate an appropriate answer based on the question.

[1935] Input: Question data sent from the terminal.

[1936] Output: The generated answer.

[1937] How it works: The server sends the question as a prompt to the generative AI model and receives an answer from the model (e.g., "The restaurant I recommend is Gourmet Delight.").

[1938] Step 12: Provide your answers

[1939] Description: The server provides the generated answer to the user via the terminal.

[1940] Input: The generated answer.

[1941] Output: Display of the answer to the question.

[1942] Specific operation: The server sends the generated answer to the device, and the device displays the received information on the screen.

[1943] Step 13: Report a problem

[1944] Description: Allows users to report any trouble they encounter while traveling from their device.

[1945] Input: Problem description (e.g. "My flight has been cancelled").

[1946] Specific actions: The user accesses the device's reporting form and enters specific trouble information.

[1947] Step 14: Submit your trouble report

[1948] Description: The device sends trouble information to the server.

[1949] Input: Trouble information entered by the user.

[1950] Output: Trouble information data formatted in JSON format.

[1951] Specific operation: The device converts the trouble information into JSON format and sends it to the server via HTTPS protocol.

[1952] Step 15: Initiate response

[1953] Description: The server will take appropriate action based on the trouble information.

[1954] Input: Trouble information data sent from the device.

[1955] Output: A request to start the response process.

[1956] Specific behavior: The server analyzes the received trouble information and sends a request to an external system to initiate appropriate action, such as arranging a new flight.

[1957] Step 16: Notification of troubleshoot

[1958] Description: The server notifies the user of the progress of the response.

[1959] Input: Progress of the response.

[1960] Output: Displays troubleshooting progress information.

[1961] Specific behavior: The server sends progress information in JSON format to the device, and the device displays the received information in the user interface (e.g., "Your flight has been canceled. Arranging a new flight...").

[1962] (Application example 1)

[1963] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1964] Conventional travel support systems handle the creation of travel plans, reservations, answering questions, and troubleshooting individually, making it difficult for users to centrally manage everything, and there was a problem of a lack of convenience, especially when using meal services at destinations, as users had to go through a separate platform.In addition, there was also the problem that recommendations for meal services at travel destinations and reservation management were not integrated, which increased the effort required to make meal arrangements during travel.

[1965] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1966] In this invention, the server includes means for a user to input travel destinations and itineraries, means for generating a travel plan based on the input information, means for making reservations in cooperation with an external reservation system based on the generated travel plan, means for accepting questions from the user during the trip and providing appropriate answers to those questions, means for receiving information about problems during the trip and taking appropriate measures to address those problems, means for recommending meal services at locations of stay based on the travel plan, and means for managing reservations for the recommended meal services. This allows users to perform everything from creating a travel plan to answering questions and responding to them during the trip, and even arranging meals at locations of stay, all within the system.

[1967] "User" refers to an individual who creates or changes travel plans, asks questions, or reports problems.

[1968] The "means for inputting travel destination and itinerary" is an interface for inputting the travel destination, departure date, and return date.

[1969] "Means for generating travel plans" refers to a system that has the function of automatically creating a travel itinerary and detailed schedule based on inputted travel destinations and dates.

[1970] "Means for executing reservations in cooperation with external reservation systems" refers to a system that communicates with an external reservation platform to complete reservations for accommodation, transportation, etc. based on the contents of travel plans.

[1971] "Means of accepting questions and providing appropriate answers to those questions" refers to a system that responds to various questions from users while traveling and provides answers using pre-registered data or external databases.

[1972] "Means for receiving trouble information and taking appropriate measures to resolve the trouble" refers to a system that receives from users information about problems or troubles that occur during their trip and provides appropriate measures to resolve the trouble.

[1973] The "means for recommending local food services" is a system that has the function of suggesting to users food services available in places to be visited based on a travel plan.

[1974] "Means for managing reservations for meal services" refers to a system that records and manages reservations made by users for recommended meal services, and makes changes or cancellations as necessary.

[1975] An "API integration means" is a component that provides a common interface for different systems to communicate with each other, enabling the sending and receiving of information.

[1976] "Means for sending push notifications of service cancellations" refers to a system that has the function of immediately notifying users of sudden service cancellations during travel to their devices.

[1977] To implement this invention, the travel support system must be configured with a server and terminals. The server comprehensively manages various functions, such as generating travel plans, linking with external reservation systems, answering questions, dealing with problems, recommending meal services, and managing reservations. The terminal functions as an interface for users to input travel destinations and itineraries and to make inquiries and respond to requests during the trip.

[1978] Program Generation

[1979] The program for this system must have the following functions:

[1980] First, we provide an interface for users to input their travel destination and itinerary. This interface is implemented as a smartphone application. For example, it includes a function to input "Paris" as the travel destination and "2023-12-20 to 2023-12-25" as the itinerary.

[1981] The server then generates a travel plan based on the input information by analyzing the received data and extracting information such as optimal travel routes, tourist spots, and accommodations from a database to generate the travel plan.

[1982] In addition, by linking with an external reservation system, reservations are made based on the generated travel plans. Specifically, the server sends requests to the external reservation service API to complete reservations for accommodations and transportation.

[1983] During the trip, the server will provide a function to accept questions from users, such as "What are the best restaurants in Paris?", and will refer to the database to provide an appropriate answer.

[1984] Furthermore, the system receives information about troubles that occur during travel and takes appropriate measures to resolve them. For example, if a flight is canceled, the server will start arranging a new flight and notify the user's device, "Your flight has been canceled. Arranging a new flight..."

[1985] Newly added features include a way to recommend dining options based on travel plans. The server extracts popular restaurants and delivery services from a database and suggests them to users. Once a user selects a restaurant, the server manages the reservation.

[1986] For example, if a user wants to order "Pasta Carbonara" and "Tiramisu" in "Paris," the server connects to an external food delivery service API and confirms the order.

[1987] Example prompt sentence:

[1988] Enter your travel plan data.

[1989] Destination: Paris

[1990] Start date: 2023-12-20

[1991] End date: 2023-12-25

[1992] Place a food delivery order from a specific restaurant.

[1993] Restaurant ID: 456

[1994] Order Details:

[1995] Pasta Carbonara (2)

[1996] Tiramisu (1)

[1997] Delivery address: 123 Champs-Élysées, Paris

[1998] As described above, the system of the present invention links the server and terminals to provide consistent support from planning to execution of the trip, troubleshooting, answering questions, and arranging meal services. This allows travelers to use many functions in a unified manner, greatly improving the convenience of the entire trip.

[1999] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2000] Step 1:

[2001] The user inputs the travel destination and dates.

[2002] As input, the user enters the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20 to 2023-12-25").

[2003] The terminal receives this information and sends it to the server. As output, the travel destination and itinerary data is sent to the server.

[2004] Step 2:

[2005] The server generates a travel plan based on the received information.

[2006] As input, the server receives destination and itinerary data.

[2007] The server references the database and generates a travel plan by combining the best sightseeing routes, accommodations, and means of transportation.

[2008] As an output, the generated itinerary is stored in a database and provided to the user.

[2009] Step 3:

[2010] Execute reservations in conjunction with an external reservation system.

[2011] As input, the server uses the generated itinerary data.

[2012] The server sends requests to an external reservation system API to make reservations for accommodation and transportation.

[2013] As an output, the reservation completion information is returned to the server and stored in a database.

[2014] Step 4:

[2015] Accept questions from users during their trip and provide appropriate answers.

[2016] As input, the user types a question through the terminal (e.g., "What are some recommended restaurants in Paris?").

[2017] The terminal sends a query to the server.

[2018] The server consults the database and extracts the appropriate answer (e.g., "a restaurant called Gourmet Delight") from the data.

[2019] As an output, the answer is displayed on the user's terminal.

[2020] Step 5:

[2021] Receive information about troubles that may occur during travel and take appropriate action.

[2022] As input, the user enters trouble information (e.g., "flight canceled") into the terminal.

[2023] The terminal transmits the trouble information to the server.

[2024] The server analyzes the trouble information and initiates procedures to take appropriate action (e.g., arranging a new flight).

[2025] As an output, the progress of troubleshooting is notified to the user's terminal.

[2026] Step 6:

[2027] Recommend food services at your destination based on your travel plans.

[2028] As input, the server uses travel plan data to extract meal services from a database that are tailored to the user's location and time of stay.

[2029] The server sends recommended restaurants and delivery services to the user's device.

[2030] As an output, recommended meal services are displayed on the user's device.

[2031] Step 7:

[2032] Manage reservations for recommended meal services.

[2033] As input, the user selects which of the recommended meal services they wish to book (e.g., ordering "Pasta Carbonara" and "Tiramisu").

[2034] The terminal transmits the order information to the server.

[2035] The server connects to an external food delivery service API and sends a request to confirm the order.

[2036] As an output, information about the completion of the order is returned to the server and displayed on the user's terminal.

[2037] The above are the specific processing steps of this system.

[2038] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2039] This is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with an external reservation system, and responds to questions and troubles during the trip. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions and adjusts services based on those emotions, user satisfaction can be further improved.

[2040] Travel planning support and emotion recognition

[2041] The user inputs their travel destination and visit date. For example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively. The device sends the user's input in JSON format to the server. The server generates a travel plan based on the received data and saves it in a database. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice when they enter their information, and adjusts the travel plan based on those emotions. For example, if the user indicates that they want to "relax," it will suggest relaxing tourist spots and activities.

[2042] API integration with external reservation systems and emotion recognition

[2043] When a user wants to make a reservation for accommodation or transportation, they enter the reservation information on their device. The device then sends that information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, the emotion engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation result notification based on those emotions. For example, if the user is in an "anxious" state, the reservation confirmation email will include additional support information.

[2044] QA and Emotion Recognition

[2045] A traveling user inputs questions about recommended facilities and restaurants at their destination into their device. The device then sends the question to the server, which then provides an appropriate answer. The emotion engine recognizes the emotion the user expressed when entering the question and adjusts the answer accordingly. For example, if the user is "excited," it will suggest special events or activities.

[2046] Troubleshooting and Emotion Recognition

[2047] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to the server, which then takes appropriate action based on that information. At this time, the emotion engine recognizes the user's emotion when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a quick and detailed response will reduce the user's dissatisfaction.

[2048] This invention makes it possible to recognize users' emotions in real time and adjust service content based on those emotions. This system provides consistent support from planning a trip to responding during the trip, and by combining it with an emotion engine, it can significantly improve traveler satisfaction and reliability.

[2049] The processing flow will be explained below.

[2050] Travel planning support and emotion recognition

[2051] Step 1:

[2052] The user inputs the travel destination and visit date, for example, "Paris" and "London," with visit dates of "2023-12-20" and "2023-12-25," respectively.

[2053] Step 2:

[2054] The device sends the user input information to the server in JSON format.

[2055] Example of transmitted data:

[2056] json

[2057] {

[2058] "user_id": "12345",

[2059] "destinations": ["Paris", "London"],

[2060] "dates": ["2023-12-20", "2023-12-25"]

[2061] }

[2062] Step 3:

[2063] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice when inputting. For example, it analyzes whether the user is expressing the emotion "I want to relax."

[2064] Step 4:

[2065] The server generates a travel plan based on the recognized emotion and the input information. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots.

[2066] Step 5:

[2067] The server stores the generated travel plan in a database and sends it to the terminal, which displays the plan to the user.

[2068] API integration with external reservation systems and emotion recognition

[2069] Step 1:

[2070] The user inputs reservation information for accommodation and transportation. For example, a hotel reservation in "Paris" is made from "2023-12-20" to "2023-12-25."

[2071] Step 2:

[2072] The device sends the reservation information in JSON format to the server.

[2073] Example of transmitted data:

[2074] json

[2075] {

[2076] "user_id": "12345",

[2077] "destination": "Paris",

[2078] "check_in": "2023-12-20",

[2079] "check_out": "2023-12-25"

[2080] }

[2081] Step 3:

[2082] The server receives the reservation information and sends a POST request to the external reservation system API.

[2083] Example API request:

[2084] python

[2085] response = requests.post(api_url="http: / / booking-api.com / reserve", json=booking_info)

[2086] Step 4:

[2087] The server uses an emotion engine to recognize the user's emotions during the reservation process, for example, analyzing whether the user is in an "anxious" state during the reservation process.

[2088] Step 5:

[2089] The server tailors the booking outcome based on the perceived emotion, for example including additional support information in the booking confirmation email if the user indicates "anxiety."

[2090] Step 6:

[2091] The server sends the reservation results in JSON format to the terminal, and the terminal notifies the user.

[2092] QA and Emotion Recognition

[2093] Step 1:

[2094] The user inputs questions about recommended facilities and restaurants at the destination into the terminal. For example, the user inputs the question "must_visit" (what are the must-see spots?).

[2095] Step 2:

[2096] The device sends the question in JSON format to the server.

[2097] Example of transmitted data:

[2098] json

[2099] {

[2100] "user_id": "12345",

[2101] "question": "must_visit"

[2102] }

[2103] Step 3:

[2104] The server receives the question and uses an emotion engine to recognize the user's emotion when entering the question. For example, it analyzes whether the user is "excited."

[2105] Step 4:

[2106] The server generates appropriate responses and tailors them based on the emotion, for example, suggesting special events or activities if the user is "excited."

[2107] Step 5:

[2108] The server sends the answer in JSON format to the device, which displays it to the user.

[2109] Troubleshooting and Emotion Recognition

[2110] Step 1:

[2111] The user enters the problem into the terminal, for example, "flight_cancelled."

[2112] Step 2:

[2113] The device sends trouble information to the server in JSON format.

[2114] Example of transmitted data:

[2115] json

[2116] {

[2117] "user_id": "12345",

[2118] "issue": "flight_cancelled"

[2119] }

[2120] Step 3:

[2121] The server receives the trouble information and uses an emotion engine to recognize the user's emotion when reporting the trouble. For example, it analyzes whether the user is expressing anger.

[2122] Step 4:

[2123] The server responds appropriately based on the recognized emotion, for example, if the user expresses anger, it responds quickly and in detail.

[2124] Step 5:

[2125] The server sends the response results in JSON format to the device, which then displays them to the user.

[2126] Example 2

[2127] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2128] Conventional travel support systems have difficulty recognizing users' emotions in real time and adjusting service content based on those emotions. Furthermore, there is a need for a method to sufficiently improve user satisfaction in travel planning, linking with external reservation systems, and responding to questions and problems during the trip.

[2129] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for transmitting the input information in structured data format; a means for analyzing emotions based on the structured data and generating a travel plan; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for analyzing the user's emotions during the reservation process and adjusting notification content based on the emotions; a means for receiving questions from the user during the trip and providing appropriate answers to the questions; and a means for receiving information about problems that occur during the trip, taking appropriate measures to deal with the problems, and adjusting the response method based on the user's emotions. This makes it possible to recognize the user's emotions in real time and adjust service content based on the emotions.

[2130] "User" refers to an individual who uses the System to make travel plans, make reservations, ask questions, or report problems.

[2131] "Destination" refers to a geographic location that a user wishes to visit.

[2132] "Itinerary" refers to time information including the start and end dates of a trip or the duration of stay at each location visited.

[2133] A "structured data format" refers to a data format that represents information in a regular and unified format, making it easier to process within databases and information systems.

[2134] "Emotion" refers to the psychological state or reaction that a user shows toward a system, such as feelings of relaxation, excitement, or anxiety.

[2135] "Travel Itinerary" means a series of suggestions based on your travel destination and dates, including attractions, activities, and accommodations.

[2136] "Third-party booking systems" refers to online services provided by other companies for booking accommodation and transportation.

[2137] "API integration" refers to the use of application program interfaces to share data and functions between different software systems.

[2138] "Question" refers to an inquiry made by a user to the system to obtain information or ask for advice.

[2139] "Trouble Information" refers to information that users use to report to the system any problems or difficulties they encounter during their trip.

[2140] "Notification" refers to information or messages sent by the system to the user.

[2141] The present invention is an integrated system that starts with the user inputting their travel destination and itinerary, then generates a travel plan, connects with external reservation systems, and responds to questions and troubles during the trip. It also has an emotion engine that recognizes the user's emotions and adjusts services accordingly.

[2142] In order to implement the present invention, the following hardware and software are used.

[2143] Hardware:

[2144] Terminal: The device on which the user enters information (e.g., smartphone, tablet, PC)

[2145] Server: A host computer used to process and store data and run the emotion engine.

[2146] Database: a storage system for storing travel plans, reservation information, etc.

[2147] software:

[2148] Emotion engine: Software for recognizing user emotions and adjusting services

[2149] API with external booking systems: Interface for booking accommodations, transportation, etc.

[2150] Question-answering system: An AI model for answering questions from users

[2151] Specific examples of the present invention are given below.

[2152] Example 1:

[2153] A user inputs into the device that they are traveling to "Paris" and departing on "2023-12-20." The device converts this information into JSON format and sends it to the server. The server receives this information and uses an emotion engine to analyze the emotion of the user at the time of input. For example, if the emotion "I want to relax" is detected, the server creates a travel plan that suggests relaxing tourist spots and saves it in the database.

[2154] Example prompt: "The user wants to visit Paris and London and is looking for a relaxing trip."

[2155] Example 2:

[2156] The user enters the accommodation "Hotel A" and "2023-12-20~2023-12-25" and the transportation method "Flight B" into the terminal. The terminal sends this information to the server. The server sends an API request to an external reservation system based on the information and confirms the reservation. Furthermore, if the user expresses "anxiety," the server includes additional support information in the reservation confirmation email and sends it.

[2157] Example 3:

[2158] While traveling, a user can type "What are some good restaurants in this area?" into their device, which then sends the question to the server. The server generates an answer to the question and, if it determines that the user is "excited," suggests special events or activities.

[2159] Example 4:

[2160] When a problem occurs during a trip and the user types "my flight is delayed" into the device, the device sends the problem information to the server. The server responds quickly and in detail based on this information, and if the user shows anger, it further adjusts the response method and notifies the user.

[2161] In this way, a specific embodiment of the present invention can recognize a user's emotions in real time and adjust travel plans and various responses based on those emotions, thereby significantly improving user satisfaction.

[2162] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2163] Step 1:

[2164] The user inputs the travel destination and dates

[2165] Specific operation: The user uses the terminal to input the travel destination (e.g., "Paris") and dates (e.g., "2023-12-20").

[2166] Input: Travel destination and itinerary information

[2167] Output: Structured data of destinations and dates

[2168] Step 2:

[2169] The terminal converts the input content into JSON format and sends it.

[2170] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"destination": "Paris", "date": "2023-12-20"}).

[2171] Input: Structured data of destination and dates

[2172] Output: JSON format data

[2173] Step 3:

[2174] The server receives the data and analyzes it with the emotion engine.

[2175] Specific operation: The server receives the JSON data sent from the device and analyzes the user's emotions using the emotion engine. For example, it detects the emotion "I want to relax."

[2176] Input: JSON format data

[2177] Output: Sentiment analysis results

[2178] Step 4:

[2179] The server generates the itinerary and stores it in the database.

[2180] Specific operation: The server generates a travel plan based on the results of the sentiment analysis, for example, a plan including relaxing tourist spots and activities, and stores it in a database.

[2181] Input: Sentiment analysis results

[2182] Output: Travel plan data

[2183] Step 5:

[2184] The user enters accommodation and transportation reservations

[2185] Specific operation: The user uses the terminal to enter information about accommodation (e.g., "Hotel A") and transportation (e.g., "Flight B").

[2186] Input: Accommodation and transportation information

[2187] Output: Accommodation and transportation structured data

[2188] Step 6:

[2189] The device sends the reservation information to the server.

[2190] Specific operation: The device sends the user's input to the server in JSON format (e.g., {"hotel": "Hotel A", "flight": "Flight B"}).

[2191] Input: Accommodation and transportation structured data

[2192] Output: JSON format data

[2193] Step 7:

[2194] The server sends an API request to the external booking system

[2195] Specific operation: The server sends an API request to the external reservation system based on the reservation information and confirms the reservation.

[2196] Input: Reservation information in JSON format

[2197] Output: Reservation confirmation result

[2198] Step 8:

[2199] The server adjusts and notifies the reservation result based on emotions.

[2200] Specific operation: The server receives the reservation confirmation result and uses the emotion engine to analyze the user's current emotion. For example, if the user is in an "anxious" state, the server sends the reservation confirmation email to the user with additional support information.

[2201] Input: Booking confirmation results, sentiment analysis

[2202] Output: Adjusted booking confirmation

[2203] Step 9:

[2204] The user enters a question

[2205] Specific operation: The user uses the device to input questions such as recommended facilities and restaurants at the destination.

[2206] Input: Question content

[2207] Output: Question structured data

[2208] Step 10:

[2209] The device sends a question to the server

[2210] Specific operation: The device sends the question content to the server in JSON format (e.g., "{"question": "What are some good restaurants in this area?"}).

[2211] Input: Question structured data

[2212] Output: Question data in JSON format

[2213] Step 11:

[2214] The server generates the answer and adjusts it based on the emotion.

[2215] What it does: The server generates answers to questions and uses an emotion engine to tailor the content based on the user's emotions. If the user is "excited," it suggests special events and activities.

[2216] Input: JSON format question data, sentiment analysis

[2217] Output: Adjusted answer notification

[2218] Step 12:

[2219] A user reports a problem

[2220] Specific operation: The user uses the terminal to input trouble information (e.g., "The flight is delayed").

[2221] Input: Trouble information

[2222] Output: Structured data of trouble information

[2223] Step 13:

[2224] The device sends trouble information to the server

[2225] Specific operation: The device sends trouble information to the server in JSON format (e.g., {"trouble": "The flight was delayed"}).

[2226] Input: Structured data of trouble information

[2227] Output: Trouble information in JSON format

[2228] Step 14:

[2229] The server adjusts and notifies the response method based on the emotion.

[2230] Specific operation: The server takes appropriate action based on the trouble information it receives. If the emotion engine recognizes that the user's emotion is "anger," it takes a swift and detailed action, adjusts the response method, and notifies the user.

[2231] Input: JSON format trouble information, sentiment analysis

[2232] Output: Coordinated response notification

[2233] (Application example 2)

[2234] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2235] Conventional travel plan generation systems do not take into account user emotions and lack full support during travel, especially in autonomous vehicles. Therefore, it is important to reduce the anxiety and frustration users feel during travel and improve overall satisfaction with the travel experience.

[2236] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a travel destination and itinerary; a means for generating a travel plan based on the input information; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; a means for accepting questions from the user during the travel and providing appropriate answers to the questions; a means for receiving information about troubles during the travel and taking appropriate measures to deal with the troubles; a means for recognizing the user's emotions using an emotion recognition engine and adjusting service content based on the emotions; and a means for coordinating with the navigation system of an autonomous vehicle to provide route guidance based on the travel plan. This makes it possible to provide a high level of satisfaction that takes the user's emotions into consideration, from travel planning to responses during the trip.

[2237] The "means for inputting travel destinations and itineraries" is an interface for the user to input the places he or she wishes to visit and the dates on which he or she will visit those places.

[2238] The "means for generating a travel plan" is a system for automatically creating a plan based on the travel destination and itinerary input by the user.

[2239] The "means of executing reservations in cooperation with external reservation systems" is a function of exchanging information with other external systems in order to reserve accommodations and transportation based on travel plans.

[2240] "Means of accepting questions and providing appropriate answers to those questions" refers to a function that quickly responds to questions and information requests that users have while traveling and provides appropriate information.

[2241] "A means of receiving trouble information and taking appropriate action against the trouble" is a system that proposes and responds to problems and difficulties that arise during travel.

[2242] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice and recognizes their emotional state.

[2243] "Means for recognizing the user's emotions and adjusting the service content based on those emotions" refers to a function that uses an emotion recognition engine to understand the user's emotions and optimize the service according to that state.

[2244] "Means for linking with the navigation system of an autonomous vehicle and providing route guidance based on a travel plan" refers to a system that communicates with the navigation system installed in an autonomous vehicle and provides guidance on the optimal route based on the generated travel plan.

[2245] Travel planning support and emotion recognition

[2246] The user first enters their travel destination and visit dates. For example, suppose the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The user's device sends this information in JSON format to the server. The server generates a travel plan based on the received data and stores it in a database. Furthermore, an emotion recognition engine recognizes emotions from the user's facial expressions and voice when entering information, and adjusts the travel plan based on those emotions. For example, if the user indicates a desire to relax, it will suggest relaxing tourist spots and activities.

[2247] API integration with external reservation systems and emotion recognition

[2248] When a user makes a reservation for accommodation or transportation, they input the reservation information on their device. The device then sends that information to a server, which then sends an API request to an external reservation system based on the reservation information to confirm the reservation. An emotion recognition engine recognizes the user's emotions during the reservation process and adjusts the content of the reservation notification based on those emotions. For example, if the user is anxious, the reservation confirmation email can include additional support information.

[2249] QA and Emotion Recognition

[2250] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the content of the answer based on that emotion. For example, if the user is excited, it can suggest special events or activities.

[2251] Troubleshooting and Emotion Recognition

[2252] If a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. At this time, an emotion recognition engine recognizes the user's emotions when reporting the problem and adjusts the response method based on those emotions. For example, if the user expresses anger, a quick and detailed response can be provided to reduce the user's dissatisfaction.

[2253] Collaboration with autonomous vehicles

[2254] This system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary, and can adjust in real time based on the user's emotions by providing the navigation system with the best tourist spots and routes at each point during the trip.

[2255] Hardware and software used

[2256] 1. Hardware:

[2257] Camera: Recognizes the user's facial expressions.

[2258] Microphone: Recognizes the user's voice.

[2259] Display: Shows information.

[2260] Autonomous driving system: Provides route guidance based on trip plans.

[2261] 2. Software:

[2262] Emotion recognition engine: Analyzes emotions from facial expressions and voice.

[2263] Database: Stores travel planning and booking information.

[2264] API: Integrates with external reservation systems.

[2265] Examples of concrete examples and prompts

[2266] Travel planning prompt:

[2267] The user's travel destinations are Tokyo and Osaka, and the visit dates are December 20th to December 25th, 2023. This indicates a desire to relax. Please suggest a relaxing travel plan.

[2268] Booking anxiety detection prompt:

[2269] The user is booking a triple room at a hotel and is expressing anxiety. Please generate a booking confirmation email with additional support information.

[2270] This will make it possible to provide services that take into account the user's emotions at each stage of travel planning and troubleshooting.

[2271] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2272] Step 1:

[2273] The user enters the travel destination and itinerary. For example, the travel destinations are "Tokyo" and "Osaka," and the visit dates are "2023-12-20" and "2023-12-25," respectively. The device sends the entered information to the server in JSON format. The data format of the entered information is JSON data as shown below.

[2274] json

[2275] {

[2276] "destinations": ["Tokyo", "Osaka"],

[2277] "dates": ["2023-12-20", "2023-12-25"]

[2278] }

[2279] Step 2:

[2280] The server analyzes the received JSON data and generates a travel plan. The generated travel plan is saved in a database. An emotion recognition engine analyzes the user's facial expressions and voice when inputting the plan and recognizes their emotion. For example, if the user indicates that they want to relax, relaxing tourist spots and activities are added to the travel plan.

[2281] Input: Travel destination and schedule, user facial expression and voice data

[2282] Output: Adjusted itinerary

[2283] Specific behavior:

[2284] The server uses an emotion recognition engine to extract emotional data from the user's facial expressions and voice, and then customizes the travel plan based on that emotional data.

[2285] Step 3:

[2286] Reservation information for accommodation and transportation is entered on the device. The device then sends the entered reservation information to the server. The server then sends an API request to an external reservation system based on the reservation information to confirm the reservation. At this time, an emotion recognition engine recognizes the user's emotions during the reservation process, and if the user is anxious, a reservation confirmation email containing additional support information is sent.

[2287] Input: Reservation information, user facial and voice data

[2288] Output: Reservation confirmation email

[2289] Specific behavior:

[2290] The device inputs reservation information and sends it to the server. The server connects to the external reservation system via API to confirm the reservation, and an emotion recognition engine analyzes the user's emotions and sends a reservation confirmation email.

[2291] Step 4:

[2292] A traveling user inputs a question about recommended facilities and restaurants at their destination through their device. The device then sends the question to the server, which then provides an appropriate answer. At this time, an emotion recognition engine recognizes the emotion the user expressed when entering the question and adjusts the answer based on that emotion. For example, if the user is excited, it will suggest special events or activities.

[2293] Input: Question content, user facial expression and voice data

[2294] Output: Adjusted answer

[2295] Specific behavior:

[2296] The device inputs a question and sends it to the server, which then searches a database for the best answer and uses an emotion recognition engine to check the user's emotions and adjust the answer accordingly.

[2297] Step 5:

[2298] When a user encounters a problem during their trip, they report it from their device. The device then sends the problem information to a server, which then takes appropriate action based on that information. An emotion recognition engine recognizes the emotion the user expressed when reporting the problem and adjusts the response method based on that emotion. For example, if the user expresses anger, a prompt and detailed response is provided.

[2299] Input: Trouble information, user facial expression and voice data

[2300] Output: Coordinated response

[2301] Specific behavior:

[2302] The device inputs trouble information and sends it to the server, which uses an emotion recognition engine to analyze the user's emotions, adjusts countermeasures, and promptly communicates them.

[2303] Step 6:

[2304] The system works in conjunction with the autonomous vehicle's navigation system to provide route guidance based on the generated itinerary. The navigation system displays route guidance based on the itinerary in real time, and an emotion recognition engine monitors the user's emotions and adjusts route guidance as needed.

[2305] Input: Travel plans, real-time user sentiment data

[2306] Output: Adjusted route guidance

[2307] Specific behavior:

[2308] The navigation system captures your trip plans and provides real-time directions, while an emotion recognition engine monitors your emotions and adjusts directions as needed.

[2309] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2310] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2311] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2312] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2313] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2314] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2315] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2316] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2317] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2318] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2319] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2320] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2321] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2322] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processin...

Claims

1. A means for a user to input a travel destination and itinerary; means for generating a travel plan based on the input information; a means for executing a reservation in cooperation with an external reservation system based on the generated travel plan; A means for accepting questions from users during their travels and providing appropriate answers to those questions; A means of receiving information about troubles that occur during travel and taking appropriate measures to deal with the troubles; A system including:

2. and an API linking means for transmitting the travel plan to an external reservation system. The system of claim 1 .

3. means for sending a push notification of service suspension based on the trouble information; The system of claim 1 .

Images