system

Abstract

We provide the system. [Solution] A means of receiving travel-related information entered by a user and automatically generating a travel plan based on the user's interests, A means of translating user speech or text in real time and providing information that includes appropriate cultural context, A means of collecting safety information about travel destinations and notifying users of appropriate alerts, A means of monitoring the user's health status and providing information on the nearest medical facility in the event of an abnormality, A means of performing real-time currency conversion and securely supporting payments, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The main problems faced by travelers during overseas travel are travel planning, language barriers, misunderstandings based on cultural differences, confirmation of the safety situation at the travel destination, management of health conditions, and currency conversion and safe money transactions. These problems are factors that reduce the stress of travelers and the quality of travel. Therefore, in order for travelers to enjoy overseas travel with peace of mind, new means are needed to solve these diverse and complex problems in a unified manner.

Means for Solving the Problems

[0005] It should be noted that there is a misspelling in the original text where "

発明の概要

発明が解決しようとする課題

発明の概要

[0006] A "travel plan" is a systematically planned outline of the schedule, places to visit, and activities a traveler will engage in at their destination.

[0007] "Real-time translation" is a process that instantly converts between different languages, and is a technology that facilitates smooth communication among users.

[0008] "Cultural background" refers to social and cultural information such as history, customs, and values ​​that are unique to a particular country or region.

[0009] "Safety information" refers to data about dangers and risks in specific areas or situations, serving as a guide for travelers to act safely.

[0010] "Health monitoring" refers to the collection and analysis of biometric information that individuals use to understand their current physical condition.

[0011] "Currency conversion" refers to the calculation of converting the value of one currency to another, and it is an inconvenient operation to be without in international transactions and travel.

[0012] "Payment support" refers to assistance with payment methods and procedures to ensure that users can purchase goods and services with peace of mind. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention proposes an information processing system that centrally handles a variety of functions required when a user travels abroad. This system aims to provide consistent support from before the trip to during the trip and is implemented by a server that communicates with the user's portable device via the internet.

[0035] During the travel planning stage, users utilize the application to input information such as destination, dates, and interests. The device then sends this information to a server. Based on this information, the server searches a large database for relevant tourist attractions and activities and automatically generates a travel plan. This plan can be customized to the user's individual needs, and users can review and book their plan within the app.

[0036] While traveling, users can communicate smoothly in the local language using the real-time translation function. Voice and text input from the user is processed on the device, and the translation engine on the server instantly converts it. The translation results are provided along with cultural background information, helping users to gain a deeper understanding of the culture.

[0037] Furthermore, the safety alert function provides users with the latest safety information at their travel destination. The server collects and analyzes data in real time from external information sources to send necessary alerts to the device based on the traveler's current location. This allows users to enjoy their trip safely.

[0038] Health management during travel is also a crucial element of the system. The terminal receives data from the user's health monitoring device. If an abnormality is detected in heart rate or physical condition, the server immediately searches for the nearest medical facility and contacts the user. This allows users to manage their health with peace of mind.

[0039] Furthermore, the currency conversion function allows users to perform real-time conversions between the local currency and their own currency. The server obtains exchange rates from external financial data providers and provides the conversion results. In addition, the payment support function allows users to conduct local monetary transactions with peace of mind.

[0040] The various functions of this system work together to provide seamless and comprehensive support for users facing various issues while traveling abroad, thereby creating a more comfortable and safer travel experience.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user opens the application and enters information such as travel destination, dates, interests, and budget.

[0044] Step 2:

[0045] The terminal receives the entered information and sends it to the server.

[0046] Step 3:

[0047] Based on the received information, the server searches the database and automatically generates a travel plan.

[0048] Step 4:

[0049] The server sends the generated travel plan to the device.

[0050] Step 5:

[0051] Users can view travel plans on their devices and customize or book them as needed.

[0052] Step 6:

[0053] The user requests local communication and initiates translation via voice or text.

[0054] Step 7:

[0055] The terminal receives input data from the user and sends it to the server.

[0056] Step 8:

[0057] The server performs translation in real time, adds cultural context information, and returns the results to the terminal.

[0058] Step 9:

[0059] The user can view the translation results on their device and play them back as audio if necessary.

[0060] Step 10:

[0061] The server collects security information in real time from external sources.

[0062] Step 11:

[0063] The server analyzes the collected information and generates safety alerts based on the user's current location.

[0064] Step 12:

[0065] The server sends a security alert to the device, and the user checks the alert on the device.

[0066] Step 13:

[0067] The terminal periodically receives biometric data from the user's health monitoring device.

[0068] Step 14:

[0069] The server analyzes this data and, if it detects an abnormality in the health condition, sends information about the appropriate medical institution to the terminal.

[0070] Step 15:

[0071] Users can check medical institution information on their devices and take action as needed.

[0072] Step 16:

[0073] The server periodically retrieves the latest exchange rates from an external provider.

[0074] Step 17:

[0075] The user enters the amount they want to convert on their device and requests a currency conversion.

[0076] Step 18:

[0077] The server performs the conversion using the latest exchange rate and sends the result to the terminal.

[0078] Step 19:

[0079] The user checks the conversion result on their terminal and makes a secure payment.

[0080] (Example 1)

[0081] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0082] There is a need to provide comprehensive support for the challenges faced during overseas travel, such as multilingual communication, lack of safety information, health management, and the complexities of currency conversion. Furthermore, consistent support from the planning stage to during the trip requires the development of flexible travel plans tailored to individual user needs. Efficiently addressing these challenges and providing users with a comfortable and safe travel experience is crucial.

[0083] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0084] In this invention, the server includes means for receiving travel-related information entered by the user and generating related information from a database; means for translating the user's voice or text in real time and providing cultural context; and means for monitoring the user's current location, analyzing emergency information, and notifying the user. This enables the user to receive comprehensive support to enjoy their trip with peace of mind, from before departure to during the trip.

[0085] A "user" refers to an entity that uses the system to plan and carry out a trip, and also to a person who provides travel-related information.

[0086] A "travel plan" refers to an outline of the itinerary, including places to visit and activities, generated based on the user's interests, travel duration, and budget.

[0087] "Real-time translation" refers to technology that instantly converts audio or text in different languages ​​that users encounter while traveling, enabling them to understand them.

[0088] "Safety information" refers to data that includes the latest conditions and potential dangers at a travel destination, and is fundamental information for users to enjoy their trip safely.

[0089] "Health monitoring" refers to the process of monitoring the user's physical condition and responding quickly when an abnormality is detected.

[0090] "Currency conversion" refers to the process of converting amounts between the user's currency and the currency of their travel destination using the latest exchange rates.

[0091] "Transaction security support" refers to a service that helps users conduct financial transactions while traveling with reduced risk.

[0092] A "travel-related database" is a digital information source containing information on tourist attractions, activities, accommodations, etc., and is used to support users' travel planning.

[0093] "Emergency information analysis" refers to the process of evaluating data collected from external sources in order to provide necessary alerts and warnings based on the user's current location.

[0094] This invention is an information processing system that provides a centralized range of support needed by users when traveling abroad. It is primarily implemented using a server that communicates with the user's terminal via the internet.

[0095] Users download a dedicated application to their device to efficiently create travel plans, perform real-time translations, obtain safety information, manage their health, and convert currencies. The application processes the necessary information based on the user's input.

[0096] The device sends information entered by the user during the initial stages of travel, such as destination, itinerary, and activities of interest, to the server. The server receives this information, searches a large travel-related database in the cloud, and generates a travel plan tailored to the user's preferences. The generated plan can be reviewed and adjusted within the application.

[0097] During their travels, users receive assistance in communicating in the local language through a real-time translation function. Voice and text input is processed on the device and translated by advanced natural language processing software running on the server. The translation results are presented to the user along with relevant cultural background information.

[0098] Furthermore, to ensure safety during travel, the server constantly monitors the user's current location and gathers the latest safety information from external sources. This allows users to receive appropriate alerts as needed, ensuring a safe journey.

[0099] In terms of health management, the terminal acquires data from the user's health monitoring device and sends it to the server. The server monitors the user's health status in real time and, if an abnormality is detected, has the function to search for the nearest medical facility and notify the user.

[0100] For currency conversion and payment support, the terminal uses the latest exchange rates obtained from the server to provide real-time conversion results for all transactions the user makes while traveling. This allows users to conduct financial transactions locally with peace of mind.

[0101] For example, if a user enters information such as, "I want to visit Paris. I want to see the Louvre Museum and the Eiffel Tower," the server will automatically plan and provide related activities and visiting methods. Real-time translation can be used with prompts such as, "Please translate this conversation from French to Japanese."

[0102] This allows users to have a seamless experience, from planning their trip beforehand to receiving various support during their trip.

[0103] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0104] Step 1:

[0105] The user enters travel information. The user uses a dedicated application to input their travel destination, itinerary, and activities of interest into their mobile device. The device retrieves this information, converts it into a data format, and sends it to the server. Examples of information entered include the names of countries and cities to visit, specific tourist spots, and desired activities.

[0106] Step 2:

[0107] The server generates the travel plan. The server receives travel information sent from the terminal and searches for relevant tourist attractions and activities using a large database in the cloud. The server uses an optimization algorithm to automatically generate a travel plan based on the user's requests. This process also takes into account the user's travel duration and budget. The generated plan is customized for each user.

[0108] Step 3:

[0109] The device performs the translation function. When a user needs to use the local language while traveling, they input text or voice into the device. The device sends this input as text data to the server. The translation engine on the server performs the translation in real time and sends the result, converted to the user's language, back to the device. The translation result also includes relevant cultural information.

[0110] Step 4:

[0111] The server provides safety alerts. The server uses GPS to determine the user's current location and gathers up-to-date safety information from external sources. The server analyzes this information and generates alerts relevant to the user's location. These alerts are sent to the user's device, notifying them. This enhances the user's safety while traveling.

[0112] Step 5:

[0113] The device monitors the user's health status. The device periodically collects heart rate and other health data obtained from the user's health monitoring device. The collected data is sent from the device to a server, which analyzes the user's health status. If an abnormality is detected, the server searches for the nearest medical facility and notifies the user through the device.

[0114] Step 6:

[0115] The server performs currency conversion. When a user makes a purchase or payment while traveling, they enter the necessary currency information into their terminal. The server retrieves the latest exchange rates from an external financial data provider and converts the entered amount into the user's currency. The conversion result is displayed on the terminal, supporting the user in conducting financial transactions securely.

[0116] (Application Example 1)

[0117] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0118] For international travelers, it's crucial to have a unified solution for the various problems they face, from travel planning to during their trip. Current technology makes it difficult to integrate and utilize multiple challenges—such as travel planning, language barriers, safety information, health management, currency conversion, and payments—each requiring separate solutions. Furthermore, the lack of applications that allow for unified management of these aspects on smartphones makes it challenging for travelers to enjoy their trips efficiently and safely.

[0119] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0120] In this invention, the server includes means for automatically generating a travel plan based on travel-related information entered by the user, means for translating voice or text in real time and providing cultural information, means for collecting safety information about the travel destination and notifying warnings, and means for converting currency in real time and supporting payments. As a result, travelers can centrally manage various issues from travel preparation to during their trip on their smartphone, enabling safer and more comfortable travel.

[0121] "User-entered travel-related information" refers to information such as destination, itinerary, interests, and budget that travelers enter using their smartphones or other mobile devices when planning their trips.

[0122] A "means for automatically generating travel plans" is a system that uses entered travel-related information to search for tourist attractions and activities in a database and automatically creates a plan tailored to the traveler's preferences.

[0123] "A means of translating voice or text in real time and providing cultural information" refers to a function that translates what travelers input or speak on the spot, and not only translates it but also provides cultural background information of the relevant region, thereby supporting smooth communication.

[0124] A "means for collecting safety information and issuing warnings at travel destinations" is a system that collects the latest safety information from external sources based on the traveler's current location and immediately notifies them in case of an emergency or when a warning is needed.

[0125] "A means of converting currencies in real time and supporting payments" refers to a function that uses the latest exchange rates to accurately convert currencies between a traveler's home currency and the destination currency, enabling smooth monetary transactions in the destination country.

[0126] "Application software installed on a smartphone" refers to application software that runs on a smartphone, including the methods described above, and allows users to centrally manage and utilize a series of travel-related services.

[0127] This invention is a system that supports users' overseas travel, primarily through application software installed on a smartphone. The user uses their smartphone to input travel-related information into the app. This information includes destination, itinerary, interests, budget, etc. The smartphone then transmits the entered data to a server in the cloud.

[0128] The server runs on a cloud service provider (e.g., AWS®), uses DynamoDB to store travel-related data, and retrieves tourist destination information using the Google® Maps API. Based on this information, the server automatically generates travel plans tailored to the traveler's needs.

[0129] When users use their smartphones locally, real-time translation functionality is utilized. The Google Cloud Translate API operates within the smartphone, providing voice or text translation. Cultural information about the travel destination is also provided. Safety information is obtained from an external public API, analyzed by AWS Lambda, and notified to the smartphone.

[0130] Currency conversion and settlement are performed in real time based on exchange rate data from the Foreign Exchange Rates API. The system also includes mechanisms to support local currency settlements using payment APIs such as Stripe.

[0131] For example, if a user is visiting France, they can enter a prompt into their smartphone such as, "Please suggest a travel plan for my visit to Paris, France. Please include cultural recommendations." The system will then automatically generate a plan based on the input data. Similarly, when purchasing a menu at a local restaurant, entering a prompt like, "Please tell me the meaning of the French menu item for the dish I want to order," will provide a translation and background information.

[0132] This system allows users to enjoy overseas travel with peace of mind and comfort, from planning their trip to actually traveling.

[0133] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0134] Step 1:

[0135] The user enters travel-related information into their smartphone. This data includes destination, dates, interests, and budget. This information is then sent from the device to the server. The server stores the received data in a database. At this stage, a unique ID is generated to identify the user and stored in the database.

[0136] Step 2:

[0137] The server uses the Google Maps API to retrieve information on tourist attractions around the specified travel destination, based on the travel-related information it receives. It uses the user's interests and destinations as input data to select the most relevant tourist attractions. The retrieved information is filtered according to the user's preferences. A list of recommended tourist attractions is created as output and sent to the user's device.

[0138] Step 3:

[0139] Users request translations into the local language via voice or text through a smartphone application. The device calls the Google Cloud Translate API to translate the input in real time. The translated results are provided to the user along with cultural context information. This process enables users to communicate smoothly in the local area.

[0140] Step 4:

[0141] The server retrieves safety information for the travel destination from an external API and analyzes it using AWS Lambda. The server uses the user's current location information as input to collect safety information. Based on the analysis, it generates safety warnings and alerts as needed. The notifications are pushed to the device, and the user adjusts their actions based on that information.

[0142] Step 5:

[0143] When a user performs currency conversion or payment, the terminal calls an external exchange rate API to obtain current exchange rate data. Based on the obtained data, the terminal converts the specified amount into the user's home currency. The user can then confidently make the payment locally based on the calculation result. Payment processing utilizes payment APIs such as Stripe, ensuring secure payments.

[0144] Through these steps, users can obtain all the information and support they need for overseas travel from a single application, enabling a safer and more comfortable travel experience.

[0145] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0146] This invention provides a system that recognizes a user's emotions in real time while traveling and optimizes various services based on that information. By incorporating an emotion engine, the user experience becomes more personalized, supporting a safe and fulfilling travel experience.

[0147] This system consists of a mobile device and a server. Users input basic travel-related information through an application. The device collects this information, as well as voice and text data spoken by the user during their trip, and sends it to the emotion engine. The emotion engine analyzes this data to determine the user's emotional state in real time.

[0148] The server adjusts the pre-generated travel plan based on the identified emotional state. For example, if the user is excited, more active activities will be suggested. Conversely, if the user appears tired, relaxing places and activities will be recommended.

[0149] Furthermore, when users communicate, the translation results are provided in a tone that matches their emotional state. For example, if a user is feeling anxious, the translation tone can be adjusted to be calmer and more reassuring. The emotion engine can also provide advice that takes cultural background into account, thus reducing cross-cultural misunderstandings.

[0150] Furthermore, the user's emotional state is taken into consideration in safety alerts and health monitoring. For example, if a user is feeling stressed, the server will suggest a medical facility earlier or provide a gentler notification to reduce the user's burden.

[0151] Even in real-time currency conversion and payment, the emotional engine provides reassurance and emotionally tailored advice. Through this entire process, the user's travel experience is optimized to their individual emotional state, resulting in safer and more comfortable overseas travel.

[0152] The following describes the processing flow.

[0153] Step 1:

[0154] The user launches the application and enters travel-related information such as destination, duration, interests, and budget.

[0155] Step 2:

[0156] The terminal receives travel-related information entered by the user and sends it to the server.

[0157] Step 3:

[0158] The device acquires the user's voice and facial expressions in real time and converts them into data suitable for sentiment analysis.

[0159] Step 4:

[0160] The emotion engine analyzes voice and facial expression data transmitted from the device to determine the user's current emotional state.

[0161] Step 5:

[0162] Based on the emotional state information received from the emotion engine, the server reviews the existing travel plan and adjusts the activity content and suggestions as needed.

[0163] Step 6:

[0164] The server sends the adjusted travel plan to the device.

[0165] Step 7:

[0166] Users can view the customized travel plan on their device and select activities and places to visit based on the presented information.

[0167] Step 8:

[0168] When a user feels the need to communicate locally and requests voice or text translation, the device sends that input to the server.

[0169] Step 9:

[0170] The server performs translation in real time, creates a translation result in a tone appropriate to the user's emotional state, and sends it to the terminal.

[0171] Step 10:

[0172] Users can check the translation results on their devices and communicate with locals as needed.

[0173] Step 11:

[0174] Based on continuously collected safety information about the travel destination, the server generates alerts considering the user's emotional state and adjusts the timing and method of notification.

[0175] Step 12:

[0176] The device supports a safe travel experience by notifying users of safety alerts and prompting them to take action.

[0177] Step 13:

[0178] The device continues to monitor the user's health data, and if an anomaly is detected, it will support emergency response with the help of an emotion engine.

[0179] Step 14:

[0180] When a user's emotions indicate stress or anxiety, the server immediately searches for an appropriate medical institution and provides the terminal with information that enables a rapid response.

[0181] Step 15:

[0182] Users can view healthcare facility information on their devices and, if necessary, visit a facility or request additional support.

[0183] Step 16:

[0184] The server retrieves the latest exchange rates from external financial information providers and provides them to users along with information that allows them to trade with confidence, using a sentiment engine.

[0185] Step 17:

[0186] Users can check conversion results and payment information on their devices, make payments securely, and continue their trip seamlessly.

[0187] (Example 2)

[0188] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0189] Users traveling may experience emotional stress due to different cultural contexts and unexpected environmental changes. Furthermore, approaches to using local information and services that do not consider the user's current emotional state can reduce user satisfaction. Therefore, optimizing services based on the user's emotional state is essential to making the travel experience safer and more satisfying.

[0190] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0191] In this invention, the server includes means for analyzing the user's emotional state using an emotion inference engine based on travel-related information entered by the user and adjusting the travel plan in real time; means for dynamically adjusting the translation tone based on the inferred emotional state when translating the user's voice or text and providing information while considering the appropriate cultural background; and means for collecting safety information during travel based on the user's emotional state and notifying alerts at the optimal timing. This makes it possible to provide optimal services according to the user's emotional state.

[0192] "User-entered travel-related information" refers to basic travel information such as destination, duration, budget, and interests that travelers enter through their mobile devices or applications.

[0193] A "sentiment inference engine" refers to AI technology used to analyze voice and text data from users and determine the emotional state of travelers in real time.

[0194] "Translation tone" refers to the way the translation is expressed and the overall atmosphere of the translation. By adjusting this tone, especially according to the user's emotional state, appropriate communication for different emotions can be achieved.

[0195] A "travel plan" refers to an overall plan that includes places a traveler will visit, activities they will participate in, and their schedule during their stay.

[0196] "Safety information" refers to data related to the safety of travelers, such as the security situation at the travel destination, changes in weather, and information on medical facilities.

[0197] "Currency conversion" refers to the process of converting the currency of one country to the currency of another country, and includes means of providing real-time conversions at appropriate exchange rates.

[0198] This invention provides a system that combines a mobile device, a server, and a generative AI model to recognize the emotional state of a user during travel in real time and optimize the travel experience.

[0199] Users use mobile devices such as smartphones and tablets to input travel-related information, including destination, duration, budget, and interests, into the application. This information is collected on the device and sent to the server. The application includes voice and text input functions, and user speech and messages are transmitted to a sentiment inference engine using these functions.

[0200] The audio and text data collected by the device are transmitted to a server via the internet. The server runs a generative AI model as an emotion inference engine, analyzing this data to determine the user's emotional state in real time. This model incorporates natural language processing technology, enabling it to infer emotions from the input audio and text.

[0201] The server dynamically adjusts the travel plan based on the user's emotional state, which is determined in real time. For example, if the server determines that the user is "excited," it adds activities to the schedule and notifies the user's device with suggestions. When possible, it also considers local event information and tour availability when providing the plan.

[0202] Furthermore, the server adjusts the translation tone based on the user's emotional state. For example, if it determines that the user is feeling "anxious," it softens the translation tone to provide a sense of reassurance and ensures comfortable communication.

[0203] For example, if a user selects a city to visit and expresses concerns about language barriers, the prompt might read something like, "We'll support your communication at your travel destination. Please enter your specific inquiry." This helps alleviate their anxiety about communicating locally.

[0204] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0205] Step 1:

[0206] The user uses a travel application to input travel-related information such as destination, travel duration, budget, and interests. This becomes the input data. The terminal aggregates and formats this information. The output obtained here is travel plan data to be sent to the server.

[0207] Step 2:

[0208] The device collects user voice and text data. Specifically, it uses the device's microphone and text input field to record the user's speech and messages. This data is then sent to the server as input.

[0209] Step 3:

[0210] The server receives voice and text data sent from the terminal and inputs it into the sentiment inference engine. A generative AI model is used to analyze this data and estimate the user's emotional state. Data processing involves speech recognition and natural language processing. The output is data of the determined emotional state.

[0211] Step 4:

[0212] The server dynamically adjusts pre-generated travel plans based on the user's emotional state. Specifically, it suggests active activities for excited users and incorporates relaxing activities into the schedule for fatigued users. This adjusted travel plan becomes the new output and is sent to the device.

[0213] Step 5:

[0214] The server adjusts the translation output to match the user's emotional state. For example, if the user is identified as anxious, the server softens the tone of the translation. The output is the adjusted translation result.

[0215] Step 6:

[0216] The server provides safety and health information during travel, tailored to the user's emotional state. If the user is experiencing stress, it prioritizes providing information about medical facilities. The output consists of emotionally sensitive safety notifications and medical information.

[0217] Step 7:

[0218] The user performs payment and currency conversion based on instructions from the server. The server provides reassuring payment advice tailored to the user's emotional state. The output serves as a guideline to ensure the user can complete payments securely.

[0219] (Application Example 2)

[0220] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0221] Modern travelers seek diverse experiences while traveling, but they also need a sense of security and personalized support. However, traditional travel support systems have struggled to understand travelers' emotional states in real time and suggest the most appropriate services accordingly. Therefore, there is a need for information provision based on travelers' emotions and the provision of experiences that are suited to their emotional state.

[0222] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0223] In this invention, the server includes means for receiving information on areas of interest entered by the user and automatically generating an itinerary based on those preferences; means for analyzing the user's voice or text in real time and providing information including appropriate social context; and means for analyzing the user's emotional state and suggesting optimal products and services. This makes it possible to provide a personalized travel experience that responds to the user's emotional state and supports a safe and fulfilling trip.

[0224] "Information about user-entered areas of interest" refers to data that travelers specify reflecting their preferences and interests.

[0225] "Methods for automatically generating itineraries" refers to a function that automatically creates an appropriate travel plan based on the user's input information.

[0226] "Means for analyzing user voice or text in real time" refers to technologies that instantly process voice and text data emitted by users and analyze their content and emotions.

[0227] "Means of providing information that includes appropriate social context" refers to a function that presents users with relevant information that is appropriate to their cultural background and circumstances.

[0228] "Means of collecting safety information for destinations" refers to a system for gathering the latest safety information at travel destinations and informing users of it.

[0229] "Means of monitoring the user's health" refers to technology that continuously observes the health status of travelers and detects abnormalities.

[0230] "Means of providing information on the nearest medical facilities" refers to a function that displays information on the medical facility closest to the user's current location in an emergency.

[0231] "A means of performing real-time currency conversion" refers to a technology that instantly converts different currencies during travel and displays the accurate amount to the user.

[0232] "Means of securely supporting payments" refers to features that ensure user security in online and offline payments.

[0233] "Means for analyzing the emotional state of users" refers to technologies that determine and record a user's emotions from voice and text data.

[0234] "Means of presenting optimal products and services" refers to a function that recommends appropriate products and experiences based on the user's emotions and preferences.

[0235] The system that implements this application primarily consists of a mobile device and a server. The user uses the mobile device to input information about their interests and preferences. This information is sent to the server, and a travel itinerary is automatically generated based on the user's interests.

[0236] Mobile devices use microphones and cameras to acquire user voice and video data in real time. This data is sent to voice and image analysis services in the cloud to identify the user's emotional state. Specific examples of such services include Google Cloud's voice recognition API and image recognition API.

[0237] Based on the analysis results, the server provides users with information that takes social context into consideration. Furthermore, to ensure user safety, it collects the latest safety information for travel destinations and notifies users of warnings as needed. Health monitoring is conducted with the user's permission, and if an abnormality is detected, information on the nearest medical facility is provided promptly.

[0238] During payment, currency conversion is performed in real time, ensuring a safe and fast transaction. This process is supported by managing payment information through secure internet communication.

[0239] Furthermore, emotion analysis is used to suggest the most suitable products and services to the user. For example, if the user feels anxious, it will suggest products that provide reassurance or facilities that promote relaxation. As an example of a prompt, giving the AI ​​instructions such as, "The user may be feeling anxious while selecting a travel plan. Please suggest relaxing activities," will enable it to provide more appropriate suggestions.

[0240] As described above, by linking mobile devices and servers, it is possible to optimize the user's travel experience to their individual emotional state and provide a safe and comfortable experience.

[0241] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0242] Step 1:

[0243] Users use their mobile devices to input information about their interests and preferences. This information consists of data such as destinations, desired activities, and budget. The entered information is sent to a server and becomes the basis for the automated itinerary generation process.

[0244] Step 2:

[0245] The device acquires the user's voice and image data. It captures the user's facial expressions with a camera and records their voice with a microphone. This data is processed in real time using Google Cloud's speech recognition and image recognition APIs, serving as input for analyzing the user's emotional state. As a result, data on the user's current emotional state is generated.

[0246] Step 3:

[0247] The server uses an AI model to select and recommend the most suitable products and services to the user based on their analyzed emotional state. The input is emotional state data, and the output is a list of products and services corresponding to that emotion. In this process, prompt sentences are given to the AI ​​model, and suggestions that match the emotion are generated.

[0248] Step 4:

[0249] The server continuously collects safety information for the destination and assesses the safety level based on the user's current location. Inputs are destination information and the latest safety data, while output indicates the need for safety alerts for the current location. The user is notified of the assessment results to help maintain a safe journey.

[0250] Step 5:

[0251] The user's health status is monitored using data acquired from the device. If an abnormality is detected, the server provides information on the nearest medical facility based on the user's location. The input is the user's vital sign data, and the output is contact information for the nearest medical facility.

[0252] Step 6:

[0253] The server performs real-time currency conversion during payment, providing accurate exchange rates. Inputs are the payment amount and country-specific exchange rate data, while output is the converted amount in the local currency. This allows users to make payments with confidence.

[0254] Through the processing steps described above, the user's travel experience can be optimized based on emotions and safety, supporting a fulfilling trip.

[0255] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0256] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0257] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0258] [Second Embodiment]

[0259] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0260] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0261] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0262] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0263] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0264] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0265] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0266] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0267] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0268] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0269] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0270] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0271] This invention proposes an information processing system that centrally handles a variety of functions required when a user travels abroad. This system aims to provide consistent support from before the trip to during the trip and is implemented by a server that communicates with the user's portable device via the internet.

[0272] During the travel planning stage, users utilize the application to input information such as destination, dates, and interests. The device then sends this information to a server. Based on this information, the server searches a large database for relevant tourist attractions and activities and automatically generates a travel plan. This plan can be customized to the user's individual needs, and users can review and book their plan within the app.

[0273] While traveling, users can communicate smoothly in the local language using the real-time translation function. Voice and text input from the user is processed on the device, and the translation engine on the server instantly converts it. The translation results are provided along with cultural background information, helping users to gain a deeper understanding of the culture.

[0274] Furthermore, the safety alert function provides users with the latest safety information at their travel destination. The server collects and analyzes data in real time from external information sources to send necessary alerts to the device based on the traveler's current location. This allows users to enjoy their trip safely.

[0275] Health management during travel is also a crucial element of the system. The terminal receives data from the user's health monitoring device. If an abnormality is detected in heart rate or physical condition, the server immediately searches for the nearest medical facility and contacts the user. This allows users to manage their health with peace of mind.

[0276] Furthermore, the currency conversion function allows users to perform real-time conversions between the local currency and their own currency. The server obtains exchange rates from external financial data providers and provides the conversion results. In addition, the payment support function allows users to conduct local monetary transactions with peace of mind.

[0277] The various functions of this system work together to provide seamless and comprehensive support for users facing various issues while traveling abroad, thereby creating a more comfortable and safer travel experience.

[0278] The process flow will be described below.

[0279] Step 1:

[0280] The user opens the application and enters information such as the travel destination, schedule, interests, budget, etc.

[0281] Step 2:

[0282] The terminal receives the entered information and sends it to the server.

[0283] Step 3:

[0284] Based on the received information, the server searches the database and automatically generates a travel plan.

[0285] Step 4:

[0286] The server sends the generated travel plan to the terminal.

[0287] Step 5:

[0288] The user checks the travel plan on the terminal and makes customizations or reservations as needed.

[0289] Step 6:

[0290] The user requests local communication and starts translation via voice or text.

[0291] Step 7:

[0292] The terminal receives the input data from the user and sends it to the server.

[0293] Step 8:

[0294] The server performs real-time translation, adds cultural background information, and returns the result to the terminal.

[0295] Step 9:

[0296] The user checks the translation result on the terminal and plays it back in voice if necessary.

[0297] Step 10:

[0298] The server collects security information in real time from an external information source.

[0299] Step 11:

[0300] The server analyzes the collected information and generates a security alert based on the user's current location.

[0301] Step 12:

[0302] The server sends the security alert to the terminal, and the user checks the alert on the terminal.

[0303] Step 13:

[0304] The terminal periodically receives biometric data from the health monitoring device worn by the user.

[0305] Step 14:

[0306] If the server analyzes this data and detects an abnormality in the health condition, it sends appropriate medical institution information to the terminal.

[0307] Step 15:

[0308] The user checks the medical institution information on the terminal and takes action if necessary.

[0309] Step 16:

[0310] The server periodically obtains the latest exchange rates from an external provider.

[0311] Step 17:

[0312] The user enters the amount they want to convert on their device and requests a currency conversion.

[0313] Step 18:

[0314] The server performs the conversion using the latest exchange rate and sends the result to the terminal.

[0315] Step 19:

[0316] The user checks the conversion result on their terminal and makes a secure payment.

[0317] (Example 1)

[0318] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0319] There is a need to provide comprehensive support for the challenges faced during overseas travel, such as multilingual communication, lack of safety information, health management, and the complexities of currency conversion. Furthermore, consistent support from the planning stage to during the trip requires the development of flexible travel plans tailored to individual user needs. Efficiently addressing these challenges and providing users with a comfortable and safe travel experience is crucial.

[0320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0321] In this invention, the server includes means for receiving travel-related information entered by the user and generating related information from a database; means for translating the user's voice or text in real time and providing cultural context; and means for monitoring the user's current location, analyzing emergency information, and notifying the user. This enables the user to receive comprehensive support to enjoy their trip with peace of mind, from before departure to during the trip.

[0322] A "user" refers to an entity that uses the system to plan and carry out a trip, and also to a person who provides travel-related information.

[0323] A "travel plan" refers to an outline of the itinerary, including places to visit and activities, generated based on the user's interests, travel duration, and budget.

[0324] "Real-time translation" refers to technology that instantly converts audio or text in different languages ​​that users encounter while traveling, enabling them to understand them.

[0325] "Safety information" refers to data that includes the latest conditions and potential dangers at a travel destination, and is fundamental information for users to enjoy their trip safely.

[0326] "Health monitoring" refers to the process of monitoring the user's physical condition and responding quickly when an abnormality is detected.

[0327] "Currency conversion" refers to the process of converting amounts between the user's currency and the currency of their travel destination using the latest exchange rates.

[0328] "Transaction security support" refers to a service that helps users conduct financial transactions while traveling with reduced risk.

[0329] A "travel-related database" is a digital information source containing information on tourist attractions, activities, accommodations, etc., and is used to support users' travel planning.

[0330] "Emergency information analysis" refers to the process of evaluating data collected from external sources in order to provide necessary alerts and warnings based on the user's current location.

[0331] This invention is an information processing system that provides a centralized range of support needed by users when traveling abroad. It is primarily implemented using a server that communicates with the user's terminal via the internet.

[0332] Users download a dedicated application to their device to efficiently create travel plans, perform real-time translations, obtain safety information, manage their health, and convert currencies. The application processes the necessary information based on the user's input.

[0333] The device sends information entered by the user during the initial stages of travel, such as destination, itinerary, and activities of interest, to the server. The server receives this information, searches a large travel-related database in the cloud, and generates a travel plan tailored to the user's preferences. The generated plan can be reviewed and adjusted within the application.

[0334] During their travels, users receive assistance in communicating in the local language through a real-time translation function. Voice and text input is processed on the device and translated by advanced natural language processing software running on the server. The translation results are presented to the user along with relevant cultural background information.

[0335] Furthermore, to ensure safety during travel, the server constantly monitors the user's current location and gathers the latest safety information from external sources. This allows users to receive appropriate alerts as needed, ensuring a safe journey.

[0336] In terms of health management, the terminal acquires data from the user's health monitoring device and sends it to the server. The server monitors the user's health status in real time and, if an abnormality is detected, has the function to search for the nearest medical facility and notify the user.

[0337] For currency conversion and payment support, the terminal uses the latest exchange rates obtained from the server to provide real-time conversion results for all transactions the user makes while traveling. This allows users to conduct financial transactions locally with peace of mind.

[0338] For example, if a user enters information such as, "I want to visit Paris. I want to see the Louvre Museum and the Eiffel Tower," the server will automatically plan and provide related activities and visiting methods. Real-time translation can be used with prompts such as, "Please translate this conversation from French to Japanese."

[0339] This allows users to have a seamless experience, from planning their trip beforehand to receiving various support during their trip.

[0340] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0341] Step 1:

[0342] The user enters travel information. The user uses a dedicated application to input their travel destination, itinerary, and activities of interest into their mobile device. The device retrieves this information, converts it into a data format, and sends it to the server. Examples of information entered include the names of countries and cities to visit, specific tourist spots, and desired activities.

[0343] Step 2:

[0344] The server generates the travel plan. The server receives travel information sent from the terminal and searches for relevant tourist attractions and activities using a large database in the cloud. The server uses an optimization algorithm to automatically generate a travel plan based on the user's requests. This process also takes into account the user's travel duration and budget. The generated plan is customized for each user.

[0345] Step 3:

[0346] The device performs the translation function. When a user needs to use the local language while traveling, they input text or voice into the device. The device sends this input as text data to the server. The translation engine on the server performs the translation in real time and sends the result, converted to the user's language, back to the device. The translation result also includes relevant cultural information.

[0347] Step 4:

[0348] The server provides safety alerts. The server uses GPS to determine the user's current location and gathers up-to-date safety information from external sources. The server analyzes this information and generates alerts relevant to the user's location. These alerts are sent to the user's device, notifying them. This enhances the user's safety while traveling.

[0349] Step 5:

[0350] The device monitors the user's health status. The device periodically collects heart rate and other health data obtained from the user's health monitoring device. The collected data is sent from the device to a server, which analyzes the user's health status. If an abnormality is detected, the server searches for the nearest medical facility and notifies the user through the device.

[0351] Step 6:

[0352] The server performs currency conversion. When a user makes a purchase or payment while traveling, they enter the necessary currency information into their terminal. The server retrieves the latest exchange rates from an external financial data provider and converts the entered amount into the user's currency. The conversion result is displayed on the terminal, supporting the user in conducting financial transactions securely.

[0353] (Application Example 1)

[0354] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0355] For international travelers, it's crucial to have a unified solution for the various problems they face, from travel planning to during their trip. Current technology makes it difficult to integrate and utilize multiple challenges—such as travel planning, language barriers, safety information, health management, currency conversion, and payments—each requiring separate solutions. Furthermore, the lack of applications that allow for unified management of these aspects on smartphones makes it challenging for travelers to enjoy their trips efficiently and safely.

[0356] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0357] In this invention, the server includes means for automatically generating a travel plan based on travel-related information entered by the user, means for translating voice or text in real time and providing cultural information, means for collecting safety information about the travel destination and notifying warnings, and means for converting currency in real time and supporting payments. As a result, travelers can centrally manage various issues from travel preparation to during their trip on their smartphone, enabling safer and more comfortable travel.

[0358] "User-entered travel-related information" refers to information such as destination, itinerary, interests, and budget that travelers enter using their smartphones or other mobile devices when planning their trips.

[0359] A "means for automatically generating travel plans" is a system that uses entered travel-related information to search for tourist attractions and activities in a database and automatically creates a plan tailored to the traveler's preferences.

[0360] "A means of translating voice or text in real time and providing cultural information" refers to a function that translates what travelers input or speak on the spot, and not only translates it but also provides cultural background information of the relevant region, thereby supporting smooth communication.

[0361] A "means for collecting safety information and issuing warnings at travel destinations" is a system that collects the latest safety information from external sources based on the traveler's current location and immediately notifies them in case of an emergency or when a warning is needed.

[0362] "A means of converting currencies in real time and supporting payments" refers to a function that uses the latest exchange rates to accurately convert currencies between a traveler's home currency and the destination currency, enabling smooth monetary transactions in the destination country.

[0363] "Application software installed on a smartphone" refers to application software that runs on a smartphone, including the methods described above, and allows users to centrally manage and utilize a series of travel-related services.

[0364] This invention is a system that supports users' overseas travel, primarily through application software installed on a smartphone. The user uses their smartphone to input travel-related information into the app. This information includes destination, itinerary, interests, budget, etc. The smartphone then transmits the entered data to a server in the cloud.

[0365] The server runs on a cloud service provider (e.g., AWS), uses DynamoDB to store travel-related data, and retrieves tourist destination information using the Google Maps API. Based on this information, the server automatically generates travel plans tailored to the traveler's needs.

[0366] When users use their smartphones locally, real-time translation functionality is utilized. The Google Cloud Translate API operates within the smartphone, providing voice or text translation. Cultural information about the travel destination is also provided. Safety information is obtained from an external public API, analyzed by AWS Lambda, and notified to the smartphone.

[0367] Currency conversion and settlement are performed in real time based on exchange rate data from the Foreign Exchange Rates API. The system also includes mechanisms to support local currency settlements using payment APIs such as Stripe.

[0368] For example, if a user is visiting France, they can enter a prompt into their smartphone such as, "Please suggest a travel plan for my visit to Paris, France. Please include cultural recommendations." The system will then automatically generate a plan based on the input data. Similarly, when purchasing a menu at a local restaurant, entering a prompt like, "Please tell me the meaning of the French menu item for the dish I want to order," will provide a translation and background information.

[0369] This system allows users to enjoy overseas travel with peace of mind and comfort, from planning their trip to actually traveling.

[0370] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0371] Step 1:

[0372] The user enters travel-related information into their smartphone. This data includes destination, dates, interests, and budget. This information is then sent from the device to the server. The server stores the received data in a database. At this stage, a unique ID is generated to identify the user and stored in the database.

[0373] Step 2:

[0374] The server uses the Google Maps API to retrieve information on tourist attractions around the specified travel destination, based on the travel-related information it receives. It uses the user's interests and destinations as input data to select the most relevant tourist attractions. The retrieved information is filtered according to the user's preferences. A list of recommended tourist attractions is created as output and sent to the user's device.

[0375] Step 3:

[0376] Users request translations into the local language via voice or text through a smartphone application. The device calls the Google Cloud Translate API to translate the input in real time. The translated results are provided to the user along with cultural context information. This process enables users to communicate smoothly in the local area.

[0377] Step 4:

[0378] The server retrieves safety information for the travel destination from an external API and analyzes it using AWS Lambda. The server uses the user's current location information as input to collect safety information. Based on the analysis, it generates safety warnings and alerts as needed. The notifications are pushed to the device, and the user adjusts their actions based on that information.

[0379] Step 5:

[0380] When a user performs currency conversion or payment, the terminal calls an external exchange rate API to obtain current exchange rate data. Based on the obtained data, the terminal converts the specified amount into the user's home currency. The user can then confidently make the payment locally based on the calculation result. Payment processing utilizes payment APIs such as Stripe, ensuring secure payments.

[0381] Through these steps, users can obtain all the information and support they need for overseas travel from a single application, enabling a safer and more comfortable travel experience.

[0382] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0383] This invention provides a system that recognizes a user's emotions in real time while traveling and optimizes various services based on that information. By incorporating an emotion engine, the user experience becomes more personalized, supporting a safe and fulfilling travel experience.

[0384] This system consists of a mobile device and a server. Users input basic travel-related information through an application. The device collects this information, as well as voice and text data spoken by the user during their trip, and sends it to the emotion engine. The emotion engine analyzes this data to determine the user's emotional state in real time.

[0385] The server adjusts the pre-generated travel plan based on the identified emotional state. For example, if the user is excited, more active activities will be suggested. Conversely, if the user appears tired, relaxing places and activities will be recommended.

[0386] Furthermore, when users communicate, the translation results are provided in a tone that matches their emotional state. For example, if a user is feeling anxious, the translation tone can be adjusted to be calmer and more reassuring. The emotion engine can also provide advice that takes cultural background into account, thus reducing cross-cultural misunderstandings.

[0387] Furthermore, the user's emotional state is taken into consideration in safety alerts and health monitoring. For example, if a user is feeling stressed, the server will suggest a medical facility earlier or provide a gentler notification to reduce the user's burden.

[0388] Even in real-time currency conversion and payment, the emotional engine provides reassurance and emotionally tailored advice. Through this entire process, the user's travel experience is optimized to their individual emotional state, resulting in safer and more comfortable overseas travel.

[0389] The following describes the processing flow.

[0390] Step 1:

[0391] The user launches the application and enters travel-related information such as destination, duration, interests, and budget.

[0392] Step 2:

[0393] The terminal receives travel-related information entered by the user and sends it to the server.

[0394] Step 3:

[0395] The device acquires the user's voice and facial expressions in real time and converts them into data suitable for sentiment analysis.

[0396] Step 4:

[0397] The emotion engine analyzes voice and facial expression data transmitted from the device to determine the user's current emotional state.

[0398] Step 5:

[0399] Based on the emotional state information received from the emotion engine, the server reviews the existing travel plan and adjusts the activity content and suggestions as needed.

[0400] Step 6:

[0401] The server sends the adjusted travel plan to the device.

[0402] Step 7:

[0403] Users can view the customized travel plan on their device and select activities and places to visit based on the presented information.

[0404] Step 8:

[0405] When a user feels the need to communicate locally and requests voice or text translation, the device sends that input to the server.

[0406] Step 9:

[0407] The server performs translation in real time, creates a translation result in a tone appropriate to the user's emotional state, and sends it to the terminal.

[0408] Step 10:

[0409] Users can check the translation results on their devices and communicate with locals as needed.

[0410] Step 11:

[0411] Based on continuously collected safety information about the travel destination, the server generates alerts considering the user's emotional state and adjusts the timing and method of notification.

[0412] Step 12:

[0413] The device supports a safe travel experience by notifying users of safety alerts and prompting them to take action.

[0414] Step 13:

[0415] The device continues to monitor the user's health data, and if an anomaly is detected, it will support emergency response with the help of an emotion engine.

[0416] Step 14:

[0417] When a user's emotions indicate stress or anxiety, the server immediately searches for an appropriate medical institution and provides the terminal with information that enables a rapid response.

[0418] Step 15:

[0419] Users can view healthcare facility information on their devices and, if necessary, visit a facility or request additional support.

[0420] Step 16:

[0421] The server retrieves the latest exchange rates from external financial information providers and provides them to users along with information that allows them to trade with confidence, using a sentiment engine.

[0422] Step 17:

[0423] Users can check conversion results and payment information on their devices, make payments securely, and continue their trip seamlessly.

[0424] (Example 2)

[0425] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0426] Users traveling may experience emotional stress due to different cultural contexts and unexpected environmental changes. Furthermore, approaches to using local information and services that do not consider the user's current emotional state can reduce user satisfaction. Therefore, optimizing services based on the user's emotional state is essential to making the travel experience safer and more satisfying.

[0427] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0428] In this invention, the server includes means for analyzing the user's emotional state using an emotion inference engine based on travel-related information entered by the user and adjusting the travel plan in real time; means for dynamically adjusting the translation tone based on the inferred emotional state when translating the user's voice or text and providing information while considering the appropriate cultural background; and means for collecting safety information during travel based on the user's emotional state and notifying alerts at the optimal timing. This makes it possible to provide optimal services according to the user's emotional state.

[0429] "User-entered travel-related information" refers to basic travel information such as destination, duration, budget, and interests that travelers enter through their mobile devices or applications.

[0430] A "sentiment inference engine" refers to AI technology used to analyze voice and text data from users and determine the emotional state of travelers in real time.

[0431] "Translation tone" refers to the way the translation is expressed and the overall atmosphere of the translation. By adjusting this tone, especially according to the user's emotional state, appropriate communication for different emotions can be achieved.

[0432] A "travel plan" refers to an overall plan that includes places a traveler will visit, activities they will participate in, and their schedule during their stay.

[0433] "Safety information" refers to data related to the safety of travelers, such as the security situation at the travel destination, changes in weather, and information on medical facilities.

[0434] "Currency conversion" refers to the process of converting the currency of one country to the currency of another country, and includes means of providing real-time conversions at appropriate exchange rates.

[0435] This invention provides a system that combines a mobile device, a server, and a generative AI model to recognize the emotional state of a user during travel in real time and optimize the travel experience.

[0436] Users use mobile devices such as smartphones and tablets to input travel-related information, including destination, duration, budget, and interests, into the application. This information is collected on the device and sent to the server. The application includes voice and text input functions, and user speech and messages are transmitted to a sentiment inference engine using these functions.

[0437] The audio and text data collected by the device are transmitted to a server via the internet. The server runs a generative AI model as an emotion inference engine, analyzing this data to determine the user's emotional state in real time. This model incorporates natural language processing technology, enabling it to infer emotions from the input audio and text.

[0438] The server dynamically adjusts the travel plan based on the user's emotional state, which is determined in real time. For example, if the server determines that the user is "excited," it adds activities to the schedule and notifies the user's device with suggestions. When possible, it also considers local event information and tour availability when providing the plan.

[0439] Furthermore, the server adjusts the translation tone based on the user's emotional state. For example, if it determines that the user is feeling "anxious," it softens the translation tone to provide a sense of reassurance and ensures comfortable communication.

[0440] For example, if a user selects a city to visit and expresses concerns about language barriers, the prompt might read something like, "We'll support your communication at your travel destination. Please enter your specific inquiry." This helps alleviate their anxiety about communicating locally.

[0441] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0442] Step 1:

[0443] The user uses a travel application to input travel-related information such as destination, travel duration, budget, and interests. This becomes the input data. The terminal aggregates and formats this information. The output obtained here is travel plan data to be sent to the server.

[0444] Step 2:

[0445] The device collects user voice and text data. Specifically, it uses the device's microphone and text input field to record the user's speech and messages. This data is then sent to the server as input.

[0446] Step 3:

[0447] The server receives voice and text data sent from the terminal and inputs it into the sentiment inference engine. A generative AI model is used to analyze this data and estimate the user's emotional state. Data processing involves speech recognition and natural language processing. The output is data of the determined emotional state.

[0448] Step 4:

[0449] The server dynamically adjusts pre-generated travel plans based on the user's emotional state. Specifically, it suggests active activities for excited users and incorporates relaxing activities into the schedule for fatigued users. This adjusted travel plan becomes the new output and is sent to the device.

[0450] Step 5:

[0451] The server adjusts the translation output to match the user's emotional state. For example, if the user is identified as anxious, the server softens the tone of the translation. The output is the adjusted translation result.

[0452] Step 6:

[0453] The server provides safety and health information during travel, tailored to the user's emotional state. If the user is experiencing stress, it prioritizes providing information about medical facilities. The output consists of emotionally sensitive safety notifications and medical information.

[0454] Step 7:

[0455] The user performs payment and currency conversion based on instructions from the server. The server provides reassuring payment advice tailored to the user's emotional state. The output serves as a guideline to ensure the user can complete payments securely.

[0456] (Application Example 2)

[0457] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0458] Modern travelers seek diverse experiences while traveling, but they also need a sense of security and personalized support. However, traditional travel support systems have struggled to understand travelers' emotional states in real time and suggest the most appropriate services accordingly. Therefore, there is a need for information provision based on travelers' emotions and the provision of experiences that are suited to their emotional state.

[0459] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0460] In this invention, the server includes means for receiving information on areas of interest entered by the user and automatically generating an itinerary based on those preferences; means for analyzing the user's voice or text in real time and providing information including appropriate social context; and means for analyzing the user's emotional state and suggesting optimal products and services. This makes it possible to provide a personalized travel experience that responds to the user's emotional state and supports a safe and fulfilling trip.

[0461] "Information about user-entered areas of interest" refers to data that travelers specify reflecting their preferences and interests.

[0462] "Methods for automatically generating itineraries" refers to a function that automatically creates an appropriate travel plan based on the user's input information.

[0463] "Means for analyzing user voice or text in real time" refers to technologies that instantly process voice and text data emitted by users and analyze their content and emotions.

[0464] "Means of providing information that includes appropriate social context" refers to a function that presents users with relevant information that is appropriate to their cultural background and circumstances.

[0465] "Means of collecting safety information for destinations" refers to a system for gathering the latest safety information at travel destinations and informing users of it.

[0466] "Means of monitoring the user's health" refers to technology that continuously observes the health status of travelers and detects abnormalities.

[0467] "Means of providing information on the nearest medical facilities" refers to a function that displays information on the medical facility closest to the user's current location in an emergency.

[0468] "A means of performing real-time currency conversion" refers to a technology that instantly converts different currencies during travel and displays the accurate amount to the user.

[0469] "Means of securely supporting payments" refers to features that ensure user security in online and offline payments.

[0470] "Means for analyzing the emotional state of users" refers to technologies that determine and record a user's emotions from voice and text data.

[0471] "Means of presenting optimal products and services" refers to a function that recommends appropriate products and experiences based on the user's emotions and preferences.

[0472] The system that implements this application primarily consists of a mobile device and a server. The user uses the mobile device to input information about their interests and preferences. This information is sent to the server, and a travel itinerary is automatically generated based on the user's interests.

[0473] Mobile devices use microphones and cameras to acquire user voice and video data in real time. This data is sent to voice and image analysis services in the cloud to identify the user's emotional state. Specific examples of such services include Google Cloud's voice recognition API and image recognition API.

[0474] Based on the analysis results, the server provides users with information that takes social context into consideration. Furthermore, to ensure user safety, it collects the latest safety information for travel destinations and notifies users of warnings as needed. Health monitoring is conducted with the user's permission, and if an abnormality is detected, information on the nearest medical facility is provided promptly.

[0475] During payment, currency conversion is performed in real time, ensuring a safe and fast transaction. This process is supported by managing payment information through secure internet communication.

[0476] Furthermore, emotion analysis is used to suggest the most suitable products and services to the user. For example, if the user feels anxious, it will suggest products that provide reassurance or facilities that promote relaxation. As an example of a prompt, giving the AI ​​instructions such as, "The user may be feeling anxious while selecting a travel plan. Please suggest relaxing activities," will enable it to provide more appropriate suggestions.

[0477] As described above, by linking mobile devices and servers, it is possible to optimize the user's travel experience to their individual emotional state and provide a safe and comfortable experience.

[0478] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0479] Step 1:

[0480] Users use their mobile devices to input information about their interests and preferences. This information consists of data such as destinations, desired activities, and budget. The entered information is sent to a server and becomes the basis for the automated itinerary generation process.

[0481] Step 2:

[0482] The device acquires the user's voice and image data. It captures the user's facial expressions with a camera and records their voice with a microphone. This data is processed in real time using Google Cloud's speech recognition and image recognition APIs, serving as input for analyzing the user's emotional state. As a result, data on the user's current emotional state is generated.

[0483] Step 3:

[0484] The server uses an AI model to select and recommend the most suitable products and services to the user based on their analyzed emotional state. The input is emotional state data, and the output is a list of products and services corresponding to that emotion. In this process, prompt sentences are given to the AI ​​model, and suggestions that match the emotion are generated.

[0485] Step 4:

[0486] The server continuously collects safety information for the destination and assesses the safety level based on the user's current location. Inputs are destination information and the latest safety data, while output indicates the need for safety alerts for the current location. The user is notified of the assessment results to help maintain a safe journey.

[0487] Step 5:

[0488] The user's health status is monitored using data acquired from the device. If an abnormality is detected, the server provides information on the nearest medical facility based on the user's location. The input is the user's vital sign data, and the output is contact information for the nearest medical facility.

[0489] Step 6:

[0490] The server performs real-time currency conversion during payment, providing accurate exchange rates. Inputs are the payment amount and country-specific exchange rate data, while output is the converted amount in the local currency. This allows users to make payments with confidence.

[0491] Through the processing steps described above, the user's travel experience can be optimized based on emotions and safety, supporting a fulfilling trip.

[0492] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0493] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0494] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0495] [Third Embodiment]

[0496] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0497] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0498] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0499] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0500] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0501] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0502] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0503] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0504] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0505] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0506] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0507] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0508] This invention proposes an information processing system that centrally handles a variety of functions required when a user travels abroad. This system aims to provide consistent support from before the trip to during the trip and is implemented by a server that communicates with the user's portable device via the internet.

[0509] During the travel planning stage, users utilize the application to input information such as destination, dates, and interests. The device then sends this information to a server. Based on this information, the server searches a large database for relevant tourist attractions and activities and automatically generates a travel plan. This plan can be customized to the user's individual needs, and users can review and book their plan within the app.

[0510] While traveling, users can communicate smoothly in the local language using the real-time translation function. Voice and text input from the user is processed on the device, and the translation engine on the server instantly converts it. The translation results are provided along with cultural background information, helping users to gain a deeper understanding of the culture.

[0511] Furthermore, the safety alert function provides users with the latest safety information at their travel destination. The server collects and analyzes data in real time from external information sources to send necessary alerts to the device based on the traveler's current location. This allows users to enjoy their trip safely.

[0512] Health management during travel is also a crucial element of the system. The terminal receives data from the user's health monitoring device. If an abnormality is detected in heart rate or physical condition, the server immediately searches for the nearest medical facility and contacts the user. This allows users to manage their health with peace of mind.

[0513] Furthermore, the currency conversion function allows users to perform real-time conversions between the local currency and their own currency. The server obtains exchange rates from external financial data providers and provides the conversion results. In addition, the payment support function allows users to conduct local monetary transactions with peace of mind.

[0514] The various functions of this system work together to provide seamless and comprehensive support for users facing various issues while traveling abroad, thereby creating a more comfortable and safer travel experience.

[0515] The following describes the processing flow.

[0516] Step 1:

[0517] The user opens the application and enters information such as travel destination, dates, interests, and budget.

[0518] Step 2:

[0519] The terminal receives the entered information and sends it to the server.

[0520] Step 3:

[0521] Based on the received information, the server searches the database and automatically generates a travel plan.

[0522] Step 4:

[0523] The server sends the generated travel plan to the device.

[0524] Step 5:

[0525] Users can view travel plans on their devices and customize or book them as needed.

[0526] Step 6:

[0527] The user requests local communication and initiates translation via voice or text.

[0528] Step 7:

[0529] The terminal receives input data from the user and sends it to the server.

[0530] Step 8:

[0531] The server performs translation in real time, adds cultural context information, and returns the results to the terminal.

[0532] Step 9:

[0533] The user can view the translation results on their device and play them back as audio if necessary.

[0534] Step 10:

[0535] The server collects security information in real time from external sources.

[0536] Step 11:

[0537] The server analyzes the collected information and generates safety alerts based on the user's current location.

[0538] Step 12:

[0539] The server sends a security alert to the device, and the user checks the alert on the device.

[0540] Step 13:

[0541] The terminal periodically receives biometric data from the user's health monitoring device.

[0542] Step 14:

[0543] The server analyzes this data and, if it detects an abnormality in the health condition, sends information about the appropriate medical institution to the terminal.

[0544] Step 15:

[0545] Users can check medical institution information on their devices and take action as needed.

[0546] Step 16:

[0547] The server periodically retrieves the latest exchange rates from an external provider.

[0548] Step 17:

[0549] The user enters the amount they want to convert on their device and requests a currency conversion.

[0550] Step 18:

[0551] The server performs the conversion using the latest exchange rate and sends the result to the terminal.

[0552] Step 19:

[0553] The user checks the conversion result on their terminal and makes a secure payment.

[0554] (Example 1)

[0555] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0556] There is a need to provide comprehensive support for the challenges faced during overseas travel, such as multilingual communication, lack of safety information, health management, and the complexities of currency conversion. Furthermore, consistent support from the planning stage to during the trip requires the development of flexible travel plans tailored to individual user needs. Efficiently addressing these challenges and providing users with a comfortable and safe travel experience is crucial.

[0557] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0558] In this invention, the server includes means for receiving travel-related information entered by the user and generating related information from a database; means for translating the user's voice or text in real time and providing cultural context; and means for monitoring the user's current location, analyzing emergency information, and notifying the user. This enables the user to receive comprehensive support to enjoy their trip with peace of mind, from before departure to during the trip.

[0559] A "user" refers to an entity that uses the system to plan and carry out a trip, and also to a person who provides travel-related information.

[0560] A "travel plan" refers to an outline of the itinerary, including places to visit and activities, generated based on the user's interests, travel duration, and budget.

[0561] "Real-time translation" refers to technology that instantly converts audio or text in different languages ​​that users encounter while traveling, enabling them to understand them.

[0562] "Safety information" refers to data that includes the latest conditions and potential dangers at a travel destination, and is fundamental information for users to enjoy their trip safely.

[0563] "Health monitoring" refers to the process of monitoring the user's physical condition and responding quickly when an abnormality is detected.

[0564] "Currency conversion" refers to the process of converting amounts between the user's currency and the currency of their travel destination using the latest exchange rates.

[0565] "Transaction security support" refers to a service that helps users conduct financial transactions while traveling with reduced risk.

[0566] A "travel-related database" is a digital information source containing information on tourist attractions, activities, accommodations, etc., and is used to support users' travel planning.

[0567] "Emergency information analysis" refers to the process of evaluating data collected from external sources in order to provide necessary alerts and warnings based on the user's current location.

[0568] This invention is an information processing system that provides a centralized range of support needed by users when traveling abroad. It is primarily implemented using a server that communicates with the user's terminal via the internet.

[0569] Users download a dedicated application to their device to efficiently create travel plans, perform real-time translations, obtain safety information, manage their health, and convert currencies. The application processes the necessary information based on the user's input.

[0570] The device sends information entered by the user during the initial stages of travel, such as destination, itinerary, and activities of interest, to the server. The server receives this information, searches a large travel-related database in the cloud, and generates a travel plan tailored to the user's preferences. The generated plan can be reviewed and adjusted within the application.

[0571] During their travels, users receive assistance in communicating in the local language through a real-time translation function. Voice and text input is processed on the device and translated by advanced natural language processing software running on the server. The translation results are presented to the user along with relevant cultural background information.

[0572] Furthermore, to ensure safety during travel, the server constantly monitors the user's current location and gathers the latest safety information from external sources. This allows users to receive appropriate alerts as needed, ensuring a safe journey.

[0573] In terms of health management, the terminal acquires data from the user's health monitoring device and sends it to the server. The server monitors the user's health status in real time and, if an abnormality is detected, has the function to search for the nearest medical facility and notify the user.

[0574] For currency conversion and payment support, the terminal uses the latest exchange rates obtained from the server to provide real-time conversion results for all transactions the user makes while traveling. This allows users to conduct financial transactions locally with peace of mind.

[0575] For example, if a user enters information such as, "I want to visit Paris. I want to see the Louvre Museum and the Eiffel Tower," the server will automatically plan and provide related activities and visiting methods. Real-time translation can be used with prompts such as, "Please translate this conversation from French to Japanese."

[0576] This allows users to have a seamless experience, from planning their trip beforehand to receiving various support during their trip.

[0577] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0578] Step 1:

[0579] The user enters travel information. The user uses a dedicated application to input their travel destination, itinerary, and activities of interest into their mobile device. The device retrieves this information, converts it into a data format, and sends it to the server. Examples of information entered include the names of countries and cities to visit, specific tourist spots, and desired activities.

[0580] Step 2:

[0581] The server generates the travel plan. The server receives travel information sent from the terminal and searches for relevant tourist attractions and activities using a large database in the cloud. The server uses an optimization algorithm to automatically generate a travel plan based on the user's requests. This process also takes into account the user's travel duration and budget. The generated plan is customized for each user.

[0582] Step 3:

[0583] The device performs the translation function. When a user needs to use the local language while traveling, they input text or voice into the device. The device sends this input as text data to the server. The translation engine on the server performs the translation in real time and sends the result, converted to the user's language, back to the device. The translation result also includes relevant cultural information.

[0584] Step 4:

[0585] The server provides safety alerts. The server uses GPS to determine the user's current location and gathers up-to-date safety information from external sources. The server analyzes this information and generates alerts relevant to the user's location. These alerts are sent to the user's device, notifying them. This enhances the user's safety while traveling.

[0586] Step 5:

[0587] The device monitors the user's health status. The device periodically collects heart rate and other health data obtained from the user's health monitoring device. The collected data is sent from the device to a server, which analyzes the user's health status. If an abnormality is detected, the server searches for the nearest medical facility and notifies the user through the device.

[0588] Step 6:

[0589] The server performs currency conversion. When a user makes a purchase or payment while traveling, they enter the necessary currency information into their terminal. The server retrieves the latest exchange rates from an external financial data provider and converts the entered amount into the user's currency. The conversion result is displayed on the terminal, supporting the user in conducting financial transactions securely.

[0590] (Application Example 1)

[0591] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0592] For international travelers, it's crucial to have a unified solution for the various problems they face, from travel planning to during their trip. Current technology makes it difficult to integrate and utilize multiple challenges—such as travel planning, language barriers, safety information, health management, currency conversion, and payments—each requiring separate solutions. Furthermore, the lack of applications that allow for unified management of these aspects on smartphones makes it challenging for travelers to enjoy their trips efficiently and safely.

[0593] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0594] In this invention, the server includes means for automatically generating a travel plan based on travel-related information entered by the user, means for translating voice or text in real time and providing cultural information, means for collecting safety information about the travel destination and notifying warnings, and means for converting currency in real time and supporting payments. As a result, travelers can centrally manage various issues from travel preparation to during their trip on their smartphone, enabling safer and more comfortable travel.

[0595] "User-entered travel-related information" refers to information such as destination, itinerary, interests, and budget that travelers enter using their smartphones or other mobile devices when planning their trips.

[0596] A "means for automatically generating travel plans" is a system that uses entered travel-related information to search for tourist attractions and activities in a database and automatically creates a plan tailored to the traveler's preferences.

[0597] "A means of translating voice or text in real time and providing cultural information" refers to a function that translates what travelers input or speak on the spot, and not only translates it but also provides cultural background information of the relevant region, thereby supporting smooth communication.

[0598] A "means for collecting safety information and issuing warnings at travel destinations" is a system that collects the latest safety information from external sources based on the traveler's current location and immediately notifies them in case of an emergency or when a warning is needed.

[0599] "A means of converting currencies in real time and supporting payments" refers to a function that uses the latest exchange rates to accurately convert currencies between a traveler's home currency and the destination currency, enabling smooth monetary transactions in the destination country.

[0600] "Application software installed on a smartphone" refers to application software that runs on a smartphone, including the methods described above, and allows users to centrally manage and utilize a series of travel-related services.

[0601] This invention is a system that supports users' overseas travel, primarily through application software installed on a smartphone. The user uses their smartphone to input travel-related information into the app. This information includes destination, itinerary, interests, budget, etc. The smartphone then transmits the entered data to a server in the cloud.

[0602] The server runs on a cloud service provider (e.g., AWS), uses DynamoDB to store travel-related data, and retrieves tourist destination information using the Google Maps API. Based on this information, the server automatically generates travel plans tailored to the traveler's needs.

[0603] When users use their smartphones locally, real-time translation functionality is utilized. The Google Cloud Translate API operates within the smartphone, providing voice or text translation. Cultural information about the travel destination is also provided. Safety information is obtained from an external public API, analyzed by AWS Lambda, and notified to the smartphone.

[0604] Currency conversion and settlement are performed in real time based on exchange rate data from the Foreign Exchange Rates API. The system also includes mechanisms to support local currency settlements using payment APIs such as Stripe.

[0605] For example, if a user is visiting France, they can enter a prompt into their smartphone such as, "Please suggest a travel plan for my visit to Paris, France. Please include cultural recommendations." The system will then automatically generate a plan based on the input data. Similarly, when purchasing a menu at a local restaurant, entering a prompt like, "Please tell me the meaning of the French menu item for the dish I want to order," will provide a translation and background information.

[0606] This system allows users to enjoy overseas travel with peace of mind and comfort, from planning their trip to actually traveling.

[0607] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0608] Step 1:

[0609] The user enters travel-related information into their smartphone. This data includes destination, dates, interests, and budget. This information is then sent from the device to the server. The server stores the received data in a database. At this stage, a unique ID is generated to identify the user and stored in the database.

[0610] Step 2:

[0611] The server uses the Google Maps API to retrieve information on tourist attractions around the specified travel destination, based on the travel-related information it receives. It uses the user's interests and destinations as input data to select the most relevant tourist attractions. The retrieved information is filtered according to the user's preferences. A list of recommended tourist attractions is created as output and sent to the user's device.

[0612] Step 3:

[0613] Users request translations into the local language via voice or text through a smartphone application. The device calls the Google Cloud Translate API to translate the input in real time. The translated results are provided to the user along with cultural context information. This process enables users to communicate smoothly in the local area.

[0614] Step 4:

[0615] The server retrieves safety information for the travel destination from an external API and analyzes it using AWS Lambda. The server uses the user's current location information as input to collect safety information. Based on the analysis, it generates safety warnings and alerts as needed. The notifications are pushed to the device, and the user adjusts their actions based on that information.

[0616] Step 5:

[0617] When a user performs currency conversion or payment, the terminal calls an external exchange rate API to obtain current exchange rate data. Based on the obtained data, the terminal converts the specified amount into the user's home currency. The user can then confidently make the payment locally based on the calculation result. Payment processing utilizes payment APIs such as Stripe, ensuring secure payments.

[0618] Through these steps, users can obtain all the information and support they need for overseas travel from a single application, enabling a safer and more comfortable travel experience.

[0619] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0620] This invention provides a system that recognizes a user's emotions in real time while traveling and optimizes various services based on that information. By incorporating an emotion engine, the user experience becomes more personalized, supporting a safe and fulfilling travel experience.

[0621] This system consists of a mobile device and a server. Users input basic travel-related information through an application. The device collects this information, as well as voice and text data spoken by the user during their trip, and sends it to the emotion engine. The emotion engine analyzes this data to determine the user's emotional state in real time.

[0622] The server adjusts the pre-generated travel plan based on the identified emotional state. For example, if the user is excited, more active activities will be suggested. Conversely, if the user appears tired, relaxing places and activities will be recommended.

[0623] Furthermore, when users communicate, the translation results are provided in a tone that matches their emotional state. For example, if a user is feeling anxious, the translation tone can be adjusted to be calmer and more reassuring. The emotion engine can also provide advice that takes cultural background into account, thus reducing cross-cultural misunderstandings.

[0624] Furthermore, the user's emotional state is taken into consideration in safety alerts and health monitoring. For example, if a user is feeling stressed, the server will suggest a medical facility earlier or provide a gentler notification to reduce the user's burden.

[0625] Even in real-time currency conversion and payment, the emotional engine provides reassurance and emotionally tailored advice. Through this entire process, the user's travel experience is optimized to their individual emotional state, resulting in safer and more comfortable overseas travel.

[0626] The following describes the processing flow.

[0627] Step 1:

[0628] The user launches the application and enters travel-related information such as destination, duration, interests, and budget.

[0629] Step 2:

[0630] The terminal receives travel-related information entered by the user and sends it to the server.

[0631] Step 3:

[0632] The device acquires the user's voice and facial expressions in real time and converts them into data suitable for sentiment analysis.

[0633] Step 4:

[0634] The emotion engine analyzes voice and facial expression data transmitted from the device to determine the user's current emotional state.

[0635] Step 5:

[0636] Based on the emotional state information received from the emotion engine, the server reviews the existing travel plan and adjusts the activity content and suggestions as needed.

[0637] Step 6:

[0638] The server sends the adjusted travel plan to the device.

[0639] Step 7:

[0640] Users can view the customized travel plan on their device and select activities and places to visit based on the presented information.

[0641] Step 8:

[0642] When a user feels the need to communicate locally and requests voice or text translation, the device sends that input to the server.

[0643] Step 9:

[0644] The server performs translation in real time, creates a translation result in a tone appropriate to the user's emotional state, and sends it to the terminal.

[0645] Step 10:

[0646] Users can check the translation results on their devices and communicate with locals as needed.

[0647] Step 11:

[0648] Based on continuously collected safety information about the travel destination, the server generates alerts considering the user's emotional state and adjusts the timing and method of notification.

[0649] Step 12:

[0650] The device supports a safe travel experience by notifying users of safety alerts and prompting them to take action.

[0651] Step 13:

[0652] The device continues to monitor the user's health data, and if an anomaly is detected, it will support emergency response with the help of an emotion engine.

[0653] Step 14:

[0654] When a user's emotions indicate stress or anxiety, the server immediately searches for an appropriate medical institution and provides the terminal with information that enables a rapid response.

[0655] Step 15:

[0656] Users can view healthcare facility information on their devices and, if necessary, visit a facility or request additional support.

[0657] Step 16:

[0658] The server retrieves the latest exchange rates from external financial information providers and provides them to users along with information that allows them to trade with confidence, using a sentiment engine.

[0659] Step 17:

[0660] Users can check conversion results and payment information on their devices, make payments securely, and continue their trip seamlessly.

[0661] (Example 2)

[0662] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0663] Users traveling may experience emotional stress due to different cultural contexts and unexpected environmental changes. Furthermore, approaches to using local information and services that do not consider the user's current emotional state can reduce user satisfaction. Therefore, optimizing services based on the user's emotional state is essential to making the travel experience safer and more satisfying.

[0664] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0665] In this invention, the server includes means for analyzing the user's emotional state using an emotion inference engine based on travel-related information entered by the user and adjusting the travel plan in real time; means for dynamically adjusting the translation tone based on the inferred emotional state when translating the user's voice or text and providing information while considering the appropriate cultural background; and means for collecting safety information during travel based on the user's emotional state and notifying alerts at the optimal timing. This makes it possible to provide optimal services according to the user's emotional state.

[0666] "User-entered travel-related information" refers to basic travel information such as destination, duration, budget, and interests that travelers enter through their mobile devices or applications.

[0667] A "sentiment inference engine" refers to AI technology used to analyze voice and text data from users and determine the emotional state of travelers in real time.

[0668] "Translation tone" refers to the way the translation is expressed and the overall atmosphere of the translation. By adjusting this tone, especially according to the user's emotional state, appropriate communication for different emotions can be achieved.

[0669] A "travel plan" refers to an overall plan that includes places a traveler will visit, activities they will participate in, and their schedule during their stay.

[0670] "Safety information" refers to data related to the safety of travelers, such as the security situation at the travel destination, changes in weather, and information on medical facilities.

[0671] "Currency conversion" refers to the process of converting the currency of one country to the currency of another country, and includes means of providing real-time conversions at appropriate exchange rates.

[0672] This invention provides a system that combines a mobile device, a server, and a generative AI model to recognize the emotional state of a user during travel in real time and optimize the travel experience.

[0673] Users use mobile devices such as smartphones and tablets to input travel-related information, including destination, duration, budget, and interests, into the application. This information is collected on the device and sent to the server. The application includes voice and text input functions, and user speech and messages are transmitted to a sentiment inference engine using these functions.

[0674] The audio and text data collected by the device are transmitted to a server via the internet. The server runs a generative AI model as an emotion inference engine, analyzing this data to determine the user's emotional state in real time. This model incorporates natural language processing technology, enabling it to infer emotions from the input audio and text.

[0675] The server dynamically adjusts the travel plan based on the user's emotional state, which is determined in real time. For example, if the server determines that the user is "excited," it adds activities to the schedule and notifies the user's device with suggestions. When possible, it also considers local event information and tour availability when providing the plan.

[0676] Furthermore, the server adjusts the translation tone based on the user's emotional state. For example, if it determines that the user is feeling "anxious," it softens the translation tone to provide a sense of reassurance and ensures comfortable communication.

[0677] For example, if a user selects a city to visit and expresses concerns about language barriers, the prompt might read something like, "We'll support your communication at your travel destination. Please enter your specific inquiry." This helps alleviate their anxiety about communicating locally.

[0678] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0679] Step 1:

[0680] The user uses a travel application to input travel-related information such as destination, travel duration, budget, and interests. This becomes the input data. The terminal aggregates and formats this information. The output obtained here is travel plan data to be sent to the server.

[0681] Step 2:

[0682] The device collects user voice and text data. Specifically, it uses the device's microphone and text input field to record the user's speech and messages. This data is then sent to the server as input.

[0683] Step 3:

[0684] The server receives voice and text data sent from the terminal and inputs it into the sentiment inference engine. A generative AI model is used to analyze this data and estimate the user's emotional state. Data processing involves speech recognition and natural language processing. The output is data of the determined emotional state.

[0685] Step 4:

[0686] The server dynamically adjusts pre-generated travel plans based on the user's emotional state. Specifically, it suggests active activities for excited users and incorporates relaxing activities into the schedule for fatigued users. This adjusted travel plan becomes the new output and is sent to the device.

[0687] Step 5:

[0688] The server adjusts the translation output to match the user's emotional state. For example, if the user is identified as anxious, the server softens the tone of the translation. The output is the adjusted translation result.

[0689] Step 6:

[0690] The server provides safety and health information during travel, tailored to the user's emotional state. If the user is experiencing stress, it prioritizes providing information about medical facilities. The output consists of emotionally sensitive safety notifications and medical information.

[0691] Step 7:

[0692] The user performs payment and currency conversion based on instructions from the server. The server provides reassuring payment advice tailored to the user's emotional state. The output serves as a guideline to ensure the user can complete payments securely.

[0693] (Application Example 2)

[0694] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0695] Modern travelers seek diverse experiences while traveling, but they also need a sense of security and personalized support. However, traditional travel support systems have struggled to understand travelers' emotional states in real time and suggest the most appropriate services accordingly. Therefore, there is a need for information provision based on travelers' emotions and the provision of experiences that are suited to their emotional state.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0697] In this invention, the server includes means for receiving information on areas of interest entered by the user and automatically generating an itinerary based on those preferences; means for analyzing the user's voice or text in real time and providing information including appropriate social context; and means for analyzing the user's emotional state and suggesting optimal products and services. This makes it possible to provide a personalized travel experience that responds to the user's emotional state and supports a safe and fulfilling trip.

[0698] "Information about user-entered areas of interest" refers to data that travelers specify reflecting their preferences and interests.

[0699] "Methods for automatically generating itineraries" refers to a function that automatically creates an appropriate travel plan based on the user's input information.

[0700] "Means for analyzing user voice or text in real time" refers to technologies that instantly process voice and text data emitted by users and analyze their content and emotions.

[0701] "Means of providing information that includes appropriate social context" refers to a function that presents users with relevant information that is appropriate to their cultural background and circumstances.

[0702] "Means of collecting safety information for destinations" refers to a system for gathering the latest safety information at travel destinations and informing users of it.

[0703] "Means of monitoring the user's health" refers to technology that continuously observes the health status of travelers and detects abnormalities.

[0704] "Means of providing information on the nearest medical facilities" refers to a function that displays information on the medical facility closest to the user's current location in an emergency.

[0705] "A means of performing real-time currency conversion" refers to a technology that instantly converts different currencies during travel and displays the accurate amount to the user.

[0706] "Means of securely supporting payments" refers to features that ensure user security in online and offline payments.

[0707] "Means for analyzing the emotional state of users" refers to technologies that determine and record a user's emotions from voice and text data.

[0708] "Means of presenting optimal products and services" refers to a function that recommends appropriate products and experiences based on the user's emotions and preferences.

[0709] The system that implements this application primarily consists of a mobile device and a server. The user uses the mobile device to input information about their interests and preferences. This information is sent to the server, and a travel itinerary is automatically generated based on the user's interests.

[0710] Mobile devices use microphones and cameras to acquire user voice and video data in real time. This data is sent to voice and image analysis services in the cloud to identify the user's emotional state. Specific examples of such services include Google Cloud's voice recognition API and image recognition API.

[0711] Based on the analysis results, the server provides users with information that takes social context into consideration. Furthermore, to ensure user safety, it collects the latest safety information for travel destinations and notifies users of warnings as needed. Health monitoring is conducted with the user's permission, and if an abnormality is detected, information on the nearest medical facility is provided promptly.

[0712] During payment, currency conversion is performed in real time, ensuring a safe and fast transaction. This process is supported by managing payment information through secure internet communication.

[0713] Furthermore, emotion analysis is used to suggest the most suitable products and services to the user. For example, if the user feels anxious, it will suggest products that provide reassurance or facilities that promote relaxation. As an example of a prompt, giving the AI ​​instructions such as, "The user may be feeling anxious while selecting a travel plan. Please suggest relaxing activities," will enable it to provide more appropriate suggestions.

[0714] As described above, by linking mobile devices and servers, it is possible to optimize the user's travel experience to their individual emotional state and provide a safe and comfortable experience.

[0715] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0716] Step 1:

[0717] Users use their mobile devices to input information about their interests and preferences. This information consists of data such as destinations, desired activities, and budget. The entered information is sent to a server and becomes the basis for the automated itinerary generation process.

[0718] Step 2:

[0719] The device acquires the user's voice and image data. It captures the user's facial expressions with a camera and records their voice with a microphone. This data is processed in real time using Google Cloud's speech recognition and image recognition APIs, serving as input for analyzing the user's emotional state. As a result, data on the user's current emotional state is generated.

[0720] Step 3:

[0721] The server uses an AI model to select and recommend the most suitable products and services to the user based on their analyzed emotional state. The input is emotional state data, and the output is a list of products and services corresponding to that emotion. In this process, prompt sentences are given to the AI ​​model, and suggestions that match the emotion are generated.

[0722] Step 4:

[0723] The server continuously collects safety information for the destination and assesses the safety level based on the user's current location. Inputs are destination information and the latest safety data, while output indicates the need for safety alerts for the current location. The user is notified of the assessment results to help maintain a safe journey.

[0724] Step 5:

[0725] The user's health status is monitored using data acquired from the device. If an abnormality is detected, the server provides information on the nearest medical facility based on the user's location. The input is the user's vital sign data, and the output is contact information for the nearest medical facility.

[0726] Step 6:

[0727] The server performs real-time currency conversion during payment, providing accurate exchange rates. Inputs are the payment amount and country-specific exchange rate data, while output is the converted amount in the local currency. This allows users to make payments with confidence.

[0728] Through the processing steps described above, the user's travel experience can be optimized based on emotions and safety, supporting a fulfilling trip.

[0729] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0730] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0731] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0732] [Fourth Embodiment]

[0733] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0734] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0735] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0736] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0737] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0738] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0739] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0740] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0741] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0742] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0743] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0744] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0745] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0746] This invention proposes an information processing system that centrally handles a variety of functions required when a user travels abroad. This system aims to provide consistent support from before the trip to during the trip and is implemented by a server that communicates with the user's portable device via the internet.

[0747] During the travel planning stage, users utilize the application to input information such as destination, dates, and interests. The device then sends this information to a server. Based on this information, the server searches a large database for relevant tourist attractions and activities and automatically generates a travel plan. This plan can be customized to the user's individual needs, and users can review and book their plan within the app.

[0748] While traveling, users can communicate smoothly in the local language using the real-time translation function. Voice and text input from the user is processed on the device, and the translation engine on the server instantly converts it. The translation results are provided along with cultural background information, helping users to gain a deeper understanding of the culture.

[0749] Furthermore, the safety alert function provides users with the latest safety information at their travel destination. The server collects and analyzes data in real time from external information sources to send necessary alerts to the device based on the traveler's current location. This allows users to enjoy their trip safely.

[0750] Health management during travel is also a crucial element of the system. The terminal receives data from the user's health monitoring device. If an abnormality is detected in heart rate or physical condition, the server immediately searches for the nearest medical facility and contacts the user. This allows users to manage their health with peace of mind.

[0751] Furthermore, the currency conversion function allows users to perform real-time conversions between the local currency and their own currency. The server obtains exchange rates from external financial data providers and provides the conversion results. In addition, the payment support function allows users to conduct local monetary transactions with peace of mind.

[0752] The various functions of this system work together to provide seamless and comprehensive support for users facing various issues while traveling abroad, thereby creating a more comfortable and safer travel experience.

[0753] The following describes the processing flow.

[0754] Step 1:

[0755] The user opens the application and enters information such as travel destination, dates, interests, and budget.

[0756] Step 2:

[0757] The terminal receives the entered information and sends it to the server.

[0758] Step 3:

[0759] Based on the received information, the server searches the database and automatically generates a travel plan.

[0760] Step 4:

[0761] The server sends the generated travel plan to the device.

[0762] Step 5:

[0763] Users can view travel plans on their devices and customize or book them as needed.

[0764] Step 6:

[0765] The user requests local communication and initiates translation via voice or text.

[0766] Step 7:

[0767] The terminal receives input data from the user and sends it to the server.

[0768] Step 8:

[0769] The server performs translation in real time, adds cultural context information, and returns the results to the terminal.

[0770] Step 9:

[0771] The user can view the translation results on their device and play them back as audio if necessary.

[0772] Step 10:

[0773] The server collects security information in real time from external sources.

[0774] Step 11:

[0775] The server analyzes the collected information and generates safety alerts based on the user's current location.

[0776] Step 12:

[0777] The server sends a security alert to the device, and the user checks the alert on the device.

[0778] Step 13:

[0779] The terminal periodically receives biometric data from the user's health monitoring device.

[0780] Step 14:

[0781] The server analyzes this data and, if it detects an abnormality in the health condition, sends information about the appropriate medical institution to the terminal.

[0782] Step 15:

[0783] Users can check medical institution information on their devices and take action as needed.

[0784] Step 16:

[0785] The server periodically retrieves the latest exchange rates from an external provider.

[0786] Step 17:

[0787] The user enters the amount they want to convert on their device and requests a currency conversion.

[0788] Step 18:

[0789] The server performs the conversion using the latest exchange rate and sends the result to the terminal.

[0790] Step 19:

[0791] The user checks the conversion result on their terminal and makes a secure payment.

[0792] (Example 1)

[0793] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0794] There is a need to provide comprehensive support for the challenges faced during overseas travel, such as multilingual communication, lack of safety information, health management, and the complexities of currency conversion. Furthermore, consistent support from the planning stage to during the trip requires the development of flexible travel plans tailored to individual user needs. Efficiently addressing these challenges and providing users with a comfortable and safe travel experience is crucial.

[0795] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0796] In this invention, the server includes means for receiving travel-related information entered by the user and generating related information from a database; means for translating the user's voice or text in real time and providing cultural context; and means for monitoring the user's current location, analyzing emergency information, and notifying the user. This enables the user to receive comprehensive support to enjoy their trip with peace of mind, from before departure to during the trip.

[0797] A "user" refers to an entity that uses the system to plan and carry out a trip, and also to a person who provides travel-related information.

[0798] A "travel plan" refers to an outline of the itinerary, including places to visit and activities, generated based on the user's interests, travel duration, and budget.

[0799] "Real-time translation" refers to technology that instantly converts audio or text in different languages ​​that users encounter while traveling, enabling them to understand them.

[0800] "Safety information" refers to data that includes the latest conditions and potential dangers at a travel destination, and is fundamental information for users to enjoy their trip safely.

[0801] "Health monitoring" refers to the process of monitoring the user's physical condition and responding quickly when an abnormality is detected.

[0802] "Currency conversion" refers to the process of converting amounts between the user's currency and the currency of their travel destination using the latest exchange rates.

[0803] "Transaction security support" refers to a service that helps users conduct financial transactions while traveling with reduced risk.

[0804] A "travel-related database" is a digital information source containing information on tourist attractions, activities, accommodations, etc., and is used to support users' travel planning.

[0805] "Emergency information analysis" refers to the process of evaluating data collected from external sources in order to provide necessary alerts and warnings based on the user's current location.

[0806] This invention is an information processing system that provides a centralized range of support needed by users when traveling abroad. It is primarily implemented using a server that communicates with the user's terminal via the internet.

[0807] Users download a dedicated application to their device to efficiently create travel plans, perform real-time translations, obtain safety information, manage their health, and convert currencies. The application processes the necessary information based on the user's input.

[0808] The device sends information entered by the user during the initial stages of travel, such as destination, itinerary, and activities of interest, to the server. The server receives this information, searches a large travel-related database in the cloud, and generates a travel plan tailored to the user's preferences. The generated plan can be reviewed and adjusted within the application.

[0809] During their travels, users receive assistance in communicating in the local language through a real-time translation function. Voice and text input is processed on the device and translated by advanced natural language processing software running on the server. The translation results are presented to the user along with relevant cultural background information.

[0810] Furthermore, to ensure safety during travel, the server constantly monitors the user's current location and gathers the latest safety information from external sources. This allows users to receive appropriate alerts as needed, ensuring a safe journey.

[0811] In terms of health management, the terminal acquires data from the user's health monitoring device and sends it to the server. The server monitors the user's health status in real time and, if an abnormality is detected, has the function to search for the nearest medical facility and notify the user.

[0812] For currency conversion and payment support, the terminal uses the latest exchange rates obtained from the server to provide real-time conversion results for all transactions the user makes while traveling. This allows users to conduct financial transactions locally with peace of mind.

[0813] For example, if a user enters information such as, "I want to visit Paris. I want to see the Louvre Museum and the Eiffel Tower," the server will automatically plan and provide related activities and visiting methods. Real-time translation can be used with prompts such as, "Please translate this conversation from French to Japanese."

[0814] This allows users to have a seamless experience, from planning their trip beforehand to receiving various support during their trip.

[0815] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0816] Step 1:

[0817] The user enters travel information. The user uses a dedicated application to input their travel destination, itinerary, and activities of interest into their mobile device. The device retrieves this information, converts it into a data format, and sends it to the server. Examples of information entered include the names of countries and cities to visit, specific tourist spots, and desired activities.

[0818] Step 2:

[0819] The server generates the travel plan. The server receives travel information sent from the terminal and searches for relevant tourist attractions and activities using a large database in the cloud. The server uses an optimization algorithm to automatically generate a travel plan based on the user's requests. This process also takes into account the user's travel duration and budget. The generated plan is customized for each user.

[0820] Step 3:

[0821] The device performs the translation function. When a user needs to use the local language while traveling, they input text or voice into the device. The device sends this input as text data to the server. The translation engine on the server performs the translation in real time and sends the result, converted to the user's language, back to the device. The translation result also includes relevant cultural information.

[0822] Step 4:

[0823] The server provides safety alerts. The server uses GPS to determine the user's current location and gathers up-to-date safety information from external sources. The server analyzes this information and generates alerts relevant to the user's location. These alerts are sent to the user's device, notifying them. This enhances the user's safety while traveling.

[0824] Step 5:

[0825] The device monitors the user's health status. The device periodically collects heart rate and other health data obtained from the user's health monitoring device. The collected data is sent from the device to a server, which analyzes the user's health status. If an abnormality is detected, the server searches for the nearest medical facility and notifies the user through the device.

[0826] Step 6:

[0827] The server performs currency conversion. When a user makes a purchase or payment while traveling, they enter the necessary currency information into their terminal. The server retrieves the latest exchange rates from an external financial data provider and converts the entered amount into the user's currency. The conversion result is displayed on the terminal, supporting the user in conducting financial transactions securely.

[0828] (Application Example 1)

[0829] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0830] For international travelers, it's crucial to have a unified solution for the various problems they face, from travel planning to during their trip. Current technology makes it difficult to integrate and utilize multiple challenges—such as travel planning, language barriers, safety information, health management, currency conversion, and payments—each requiring separate solutions. Furthermore, the lack of applications that allow for unified management of these aspects on smartphones makes it challenging for travelers to enjoy their trips efficiently and safely.

[0831] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0832] In this invention, the server includes means for automatically generating a travel plan based on travel-related information entered by the user, means for translating voice or text in real time and providing cultural information, means for collecting safety information about the travel destination and notifying warnings, and means for converting currency in real time and supporting payments. As a result, travelers can centrally manage various issues from travel preparation to during their trip on their smartphone, enabling safer and more comfortable travel.

[0833] "User-entered travel-related information" refers to information such as destination, itinerary, interests, and budget that travelers enter using their smartphones or other mobile devices when planning their trips.

[0834] A "means for automatically generating travel plans" is a system that uses entered travel-related information to search for tourist attractions and activities in a database and automatically creates a plan tailored to the traveler's preferences.

[0835] "A means of translating voice or text in real time and providing cultural information" refers to a function that translates what travelers input or speak on the spot, and not only translates it but also provides cultural background information of the relevant region, thereby supporting smooth communication.

[0836] A "means for collecting safety information and issuing warnings at travel destinations" is a system that collects the latest safety information from external sources based on the traveler's current location and immediately notifies them in case of an emergency or when a warning is needed.

[0837] "A means of converting currencies in real time and supporting payments" refers to a function that uses the latest exchange rates to accurately convert currencies between a traveler's home currency and the destination currency, enabling smooth monetary transactions in the destination country.

[0838] "Application software installed on a smartphone" refers to application software that runs on a smartphone, including the methods described above, and allows users to centrally manage and utilize a series of travel-related services.

[0839] This invention is a system that supports users' overseas travel, primarily through application software installed on a smartphone. The user uses their smartphone to input travel-related information into the app. This information includes destination, itinerary, interests, budget, etc. The smartphone then transmits the entered data to a server in the cloud.

[0840] The server runs on a cloud service provider (e.g., AWS), uses DynamoDB to store travel-related data, and retrieves tourist destination information using the Google Maps API. Based on this information, the server automatically generates travel plans tailored to the traveler's needs.

[0841] When users use their smartphones locally, real-time translation functionality is utilized. The Google Cloud Translate API operates within the smartphone, providing voice or text translation. Cultural information about the travel destination is also provided. Safety information is obtained from an external public API, analyzed by AWS Lambda, and notified to the smartphone.

[0842] Currency conversion and settlement are performed in real time based on exchange rate data from the Foreign Exchange Rates API. The system also includes mechanisms to support local currency settlements using payment APIs such as Stripe.

[0843] For example, if a user is visiting France, they can enter a prompt into their smartphone such as, "Please suggest a travel plan for my visit to Paris, France. Please include cultural recommendations." The system will then automatically generate a plan based on the input data. Similarly, when purchasing a menu at a local restaurant, entering a prompt like, "Please tell me the meaning of the French menu item for the dish I want to order," will provide a translation and background information.

[0844] This system allows users to enjoy overseas travel with peace of mind and comfort, from planning their trip to actually traveling.

[0845] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0846] Step 1:

[0847] The user enters travel-related information into their smartphone. This data includes destination, dates, interests, and budget. This information is then sent from the device to the server. The server stores the received data in a database. At this stage, a unique ID is generated to identify the user and stored in the database.

[0848] Step 2:

[0849] The server uses the Google Maps API to retrieve information on tourist attractions around the specified travel destination, based on the travel-related information it receives. It uses the user's interests and destinations as input data to select the most relevant tourist attractions. The retrieved information is filtered according to the user's preferences. A list of recommended tourist attractions is created as output and sent to the user's device.

[0850] Step 3:

[0851] Users request translations into the local language via voice or text through a smartphone application. The device calls the Google Cloud Translate API to translate the input in real time. The translated results are provided to the user along with cultural context information. This process enables users to communicate smoothly in the local area.

[0852] Step 4:

[0853] The server retrieves safety information for the travel destination from an external API and analyzes it using AWS Lambda. The server uses the user's current location information as input to collect safety information. Based on the analysis, it generates safety warnings and alerts as needed. The notifications are pushed to the device, and the user adjusts their actions based on that information.

[0854] Step 5:

[0855] When a user performs currency conversion or payment, the terminal calls an external exchange rate API to obtain current exchange rate data. Based on the obtained data, the terminal converts the specified amount into the user's home currency. The user can then confidently make the payment locally based on the calculation result. Payment processing utilizes payment APIs such as Stripe, ensuring secure payments.

[0856] Through these steps, users can obtain all the information and support they need for overseas travel from a single application, enabling a safer and more comfortable travel experience.

[0857] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0858] This invention provides a system that recognizes a user's emotions in real time while traveling and optimizes various services based on that information. By incorporating an emotion engine, the user experience becomes more personalized, supporting a safe and fulfilling travel experience.

[0859] This system consists of a mobile device and a server. Users input basic travel-related information through an application. The device collects this information, as well as voice and text data spoken by the user during their trip, and sends it to the emotion engine. The emotion engine analyzes this data to determine the user's emotional state in real time.

[0860] The server adjusts the pre-generated travel plan based on the identified emotional state. For example, if the user is excited, more active activities will be suggested. Conversely, if the user appears tired, relaxing places and activities will be recommended.

[0861] Furthermore, when users communicate, the translation results are provided in a tone that matches their emotional state. For example, if a user is feeling anxious, the translation tone can be adjusted to be calmer and more reassuring. The emotion engine can also provide advice that takes cultural background into account, thus reducing cross-cultural misunderstandings.

[0862] Furthermore, the user's emotional state is taken into consideration in safety alerts and health monitoring. For example, if a user is feeling stressed, the server will suggest a medical facility earlier or provide a gentler notification to reduce the user's burden.

[0863] Even in real-time currency conversion and payment, the emotional engine provides reassurance and emotionally tailored advice. Through this entire process, the user's travel experience is optimized to their individual emotional state, resulting in safer and more comfortable overseas travel.

[0864] The following describes the processing flow.

[0865] Step 1:

[0866] The user launches the application and enters travel-related information such as destination, duration, interests, and budget.

[0867] Step 2:

[0868] The terminal receives travel-related information entered by the user and sends it to the server.

[0869] Step 3:

[0870] The device acquires the user's voice and facial expressions in real time and converts them into data suitable for sentiment analysis.

[0871] Step 4:

[0872] The emotion engine analyzes voice and facial expression data transmitted from the device to determine the user's current emotional state.

[0873] Step 5:

[0874] Based on the emotional state information received from the emotion engine, the server reviews the existing travel plan and adjusts the activity content and suggestions as needed.

[0875] Step 6:

[0876] The server sends the adjusted travel plan to the device.

[0877] Step 7:

[0878] Users can view the customized travel plan on their device and select activities and places to visit based on the presented information.

[0879] Step 8:

[0880] When a user feels the need to communicate locally and requests voice or text translation, the device sends that input to the server.

[0881] Step 9:

[0882] The server performs translation in real time, creates a translation result in a tone appropriate to the user's emotional state, and sends it to the terminal.

[0883] Step 10:

[0884] Users can check the translation results on their devices and communicate with locals as needed.

[0885] Step 11:

[0886] Based on continuously collected safety information about the travel destination, the server generates alerts considering the user's emotional state and adjusts the timing and method of notification.

[0887] Step 12:

[0888] The device supports a safe travel experience by notifying users of safety alerts and prompting them to take action.

[0889] Step 13:

[0890] The device continues to monitor the user's health data, and if an anomaly is detected, it will support emergency response with the help of an emotion engine.

[0891] Step 14:

[0892] When a user's emotions indicate stress or anxiety, the server immediately searches for an appropriate medical institution and provides the terminal with information that enables a rapid response.

[0893] Step 15:

[0894] Users can view healthcare facility information on their devices and, if necessary, visit a facility or request additional support.

[0895] Step 16:

[0896] The server retrieves the latest exchange rates from external financial information providers and provides them to users along with information that allows them to trade with confidence, using a sentiment engine.

[0897] Step 17:

[0898] Users can check conversion results and payment information on their devices, make payments securely, and continue their trip seamlessly.

[0899] (Example 2)

[0900] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0901] Users traveling may experience emotional stress due to different cultural contexts and unexpected environmental changes. Furthermore, approaches to using local information and services that do not consider the user's current emotional state can reduce user satisfaction. Therefore, optimizing services based on the user's emotional state is essential to making the travel experience safer and more satisfying.

[0902] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0903] In this invention, the server includes means for analyzing the user's emotional state using an emotion inference engine based on travel-related information entered by the user and adjusting the travel plan in real time; means for dynamically adjusting the translation tone based on the inferred emotional state when translating the user's voice or text and providing information while considering the appropriate cultural background; and means for collecting safety information during travel based on the user's emotional state and notifying alerts at the optimal timing. This makes it possible to provide optimal services according to the user's emotional state.

[0904] "User-entered travel-related information" refers to basic travel information such as destination, duration, budget, and interests that travelers enter through their mobile devices or applications.

[0905] A "sentiment inference engine" refers to AI technology used to analyze voice and text data from users and determine the emotional state of travelers in real time.

[0906] "Translation tone" refers to the way the translation is expressed and the overall atmosphere of the translation. By adjusting this tone, especially according to the user's emotional state, appropriate communication for different emotions can be achieved.

[0907] A "travel plan" refers to an overall plan that includes places a traveler will visit, activities they will participate in, and their schedule during their stay.

[0908] "Safety information" refers to data related to the safety of travelers, such as the security situation at the travel destination, changes in weather, and information on medical facilities.

[0909] "Currency conversion" refers to the process of converting the currency of one country to the currency of another country, and includes means of providing real-time conversions at appropriate exchange rates.

[0910] This invention provides a system that combines a mobile device, a server, and a generative AI model to recognize the emotional state of a user during travel in real time and optimize the travel experience.

[0911] Users use mobile devices such as smartphones and tablets to input travel-related information, including destination, duration, budget, and interests, into the application. This information is collected on the device and sent to the server. The application includes voice and text input functions, and user speech and messages are transmitted to a sentiment inference engine using these functions.

[0912] The audio and text data collected by the device are transmitted to a server via the internet. The server runs a generative AI model as an emotion inference engine, analyzing this data to determine the user's emotional state in real time. This model incorporates natural language processing technology, enabling it to infer emotions from the input audio and text.

[0913] The server dynamically adjusts the travel plan based on the user's emotional state, which is determined in real time. For example, if the server determines that the user is "excited," it adds activities to the schedule and notifies the user's device with suggestions. When possible, it also considers local event information and tour availability when providing the plan.

[0914] Furthermore, the server adjusts the translation tone based on the user's emotional state. For example, if it determines that the user is feeling "anxious," it softens the translation tone to provide a sense of reassurance and ensures comfortable communication.

[0915] For example, if a user selects a city to visit and expresses concerns about language barriers, the prompt might read something like, "We'll support your communication at your travel destination. Please enter your specific inquiry." This helps alleviate their anxiety about communicating locally.

[0916] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0917] Step 1:

[0918] The user uses a travel application to input travel-related information such as destination, travel duration, budget, and interests. This becomes the input data. The terminal aggregates and formats this information. The output obtained here is travel plan data to be sent to the server.

[0919] Step 2:

[0920] The device collects user voice and text data. Specifically, it uses the device's microphone and text input field to record the user's speech and messages. This data is then sent to the server as input.

[0921] Step 3:

[0922] The server receives voice and text data sent from the terminal and inputs it into the sentiment inference engine. A generative AI model is used to analyze this data and estimate the user's emotional state. Data processing involves speech recognition and natural language processing. The output is data of the determined emotional state.

[0923] Step 4:

[0924] The server dynamically adjusts pre-generated travel plans based on the user's emotional state. Specifically, it suggests active activities for excited users and incorporates relaxing activities into the schedule for fatigued users. This adjusted travel plan becomes the new output and is sent to the device.

[0925] Step 5:

[0926] The server adjusts the translation output to match the user's emotional state. For example, if the user is identified as anxious, the server softens the tone of the translation. The output is the adjusted translation result.

[0927] Step 6:

[0928] The server provides safety and health information during travel, tailored to the user's emotional state. If the user is experiencing stress, it prioritizes providing information about medical facilities. The output consists of emotionally sensitive safety notifications and medical information.

[0929] Step 7:

[0930] The user performs payment and currency conversion based on instructions from the server. The server provides reassuring payment advice tailored to the user's emotional state. The output serves as a guideline to ensure the user can complete payments securely.

[0931] (Application Example 2)

[0932] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0933] Modern travelers seek diverse experiences while traveling, but they also need a sense of security and personalized support. However, traditional travel support systems have struggled to understand travelers' emotional states in real time and suggest the most appropriate services accordingly. Therefore, there is a need for information provision based on travelers' emotions and the provision of experiences that are suited to their emotional state.

[0934] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0935] In this invention, the server includes means for receiving information on areas of interest entered by the user and automatically generating an itinerary based on those preferences; means for analyzing the user's voice or text in real time and providing information including appropriate social context; and means for analyzing the user's emotional state and suggesting optimal products and services. This makes it possible to provide a personalized travel experience that responds to the user's emotional state and supports a safe and fulfilling trip.

[0936] "Information about user-entered areas of interest" refers to data that travelers specify reflecting their preferences and interests.

[0937] "Methods for automatically generating itineraries" refers to a function that automatically creates an appropriate travel plan based on the user's input information.

[0938] "Means for analyzing user voice or text in real time" refers to technologies that instantly process voice and text data emitted by users and analyze their content and emotions.

[0939] "Means of providing information that includes appropriate social context" refers to a function that presents users with relevant information that is appropriate to their cultural background and circumstances.

[0940] "Means of collecting safety information for destinations" refers to a system for gathering the latest safety information at travel destinations and informing users of it.

[0941] "Means of monitoring the user's health" refers to technology that continuously observes the health status of travelers and detects abnormalities.

[0942] "Means of providing information on the nearest medical facilities" refers to a function that displays information on the medical facility closest to the user's current location in an emergency.

[0943] "A means of performing real-time currency conversion" refers to a technology that instantly converts different currencies during travel and displays the accurate amount to the user.

[0944] "Means of securely supporting payments" refers to features that ensure user security in online and offline payments.

[0945] "Means for analyzing the emotional state of users" refers to technologies that determine and record a user's emotions from voice and text data.

[0946] "Means of presenting optimal products and services" refers to a function that recommends appropriate products and experiences based on the user's emotions and preferences.

[0947] The system that implements this application primarily consists of a mobile device and a server. The user uses the mobile device to input information about their interests and preferences. This information is sent to the server, and a travel itinerary is automatically generated based on the user's interests.

[0948] Mobile devices use microphones and cameras to acquire user voice and video data in real time. This data is sent to voice and image analysis services in the cloud to identify the user's emotional state. Specific examples of such services include Google Cloud's voice recognition API and image recognition API.

[0949] Based on the analysis results, the server provides users with information that takes social context into consideration. Furthermore, to ensure user safety, it collects the latest safety information for travel destinations and notifies users of warnings as needed. Health monitoring is conducted with the user's permission, and if an abnormality is detected, information on the nearest medical facility is provided promptly.

[0950] During payment, currency conversion is performed in real time, ensuring a safe and fast transaction. This process is supported by managing payment information through secure internet communication.

[0951] Furthermore, emotion analysis is used to suggest the most suitable products and services to the user. For example, if the user feels anxious, it will suggest products that provide reassurance or facilities that promote relaxation. As an example of a prompt, giving the AI ​​instructions such as, "The user may be feeling anxious while selecting a travel plan. Please suggest relaxing activities," will enable it to provide more appropriate suggestions.

[0952] As described above, by linking mobile devices and servers, it is possible to optimize the user's travel experience to their individual emotional state and provide a safe and comfortable experience.

[0953] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0954] Step 1:

[0955] Users use their mobile devices to input information about their interests and preferences. This information consists of data such as destinations, desired activities, and budget. The entered information is sent to a server and becomes the basis for the automated itinerary generation process.

[0956] Step 2:

[0957] The device acquires the user's voice and image data. It captures the user's facial expressions with a camera and records their voice with a microphone. This data is processed in real time using Google Cloud's speech recognition and image recognition APIs, serving as input for analyzing the user's emotional state. As a result, data on the user's current emotional state is generated.

[0958] Step 3:

[0959] The server uses an AI model to select and recommend the most suitable products and services to the user based on their analyzed emotional state. The input is emotional state data, and the output is a list of products and services corresponding to that emotion. In this process, prompt sentences are given to the AI ​​model, and suggestions that match the emotion are generated.

[0960] Step 4:

[0961] The server continuously collects safety information for the destination and assesses the safety level based on the user's current location. Inputs are destination information and the latest safety data, while output indicates the need for safety alerts for the current location. The user is notified of the assessment results to help maintain a safe journey.

[0962] Step 5:

[0963] The user's health status is monitored using data acquired from the device. If an abnormality is detected, the server provides information on the nearest medical facility based on the user's location. The input is the user's vital sign data, and the output is contact information for the nearest medical facility.

[0964] Step 6:

[0965] The server performs real-time currency conversion during payment, providing accurate exchange rates. Inputs are the payment amount and country-specific exchange rate data, while output is the converted amount in the local currency. This allows users to make payments with confidence.

[0966] Through the processing steps described above, the user's travel experience can be optimized based on emotions and safety, supporting a fulfilling trip.

[0967] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0968] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0969] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0970] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0971] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0972] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0973] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0974] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0975] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0976] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0977] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0978] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0979] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0980] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0981] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0982] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0983] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0984] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0985] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0986] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0987] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0988] The following is further disclosed regarding the embodiments described above.

[0989] (Claim 1)

[0990] A means of receiving travel-related information entered by a user and automatically generating a travel plan based on the user's interests,

[0991] A means of translating user speech or text in real time and providing information that includes appropriate cultural context,

[0992] A means of collecting safety information about travel destinations and notifying users of appropriate alerts,

[0993] A means of monitoring the user's health status and providing information on the nearest medical facility in the event of an abnormality,

[0994] A means of performing real-time currency conversion and securely supporting payments,

[0995] A system that includes this.

[0996] (Claim 2)

[0997] The system according to claim 1, which allows the aforementioned travel plan to be customized based on the user's interests, travel duration, and budget.

[0998] (Claim 3)

[0999] The system according to claim 1, wherein the translation and provision of cultural information are dynamically changed in accordance with the user's input.

[1000] "Example 1"

[1001] (Claim 1)

[1002] A means for receiving travel-related information entered by a user and automatically generating a travel plan based on the user's interests,

[1003] A means of translating user speech or text in real time and providing information that includes appropriate cultural context,

[1004] A means of collecting safety information about travel destinations and notifying users of appropriate warnings,

[1005] A means of monitoring the user's health status and providing information on the nearest medical facility in the event of an abnormality,

[1006] A means to perform real-time currency conversion and securely support transactions,

[1007] A means of suggesting tourist attractions and activities based on interests identified by searching travel-related databases,

[1008] A means of monitoring the user's current location, analyzing related emergency information, and notifying the user,

[1009] A system that includes this.

[1010] (Claim 2)

[1011] The system according to claim 1, wherein the travel plan can be adjusted based on the user's interests, travel duration, and budget.

[1012] (Claim 3)

[1013] The system according to claim 1, wherein the translation and provision of cultural information change dynamically according to the user's input.

[1014] "Application Example 1"

[1015] (Claim 1)

[1016] A means for receiving travel-related information entered by a user and automatically generating a travel plan based on the user's interests,

[1017] A means of translating user speech or text in real time and providing information that includes appropriate cultural context,

[1018] A means of collecting safety information about travel destinations and notifying users of appropriate warnings,

[1019] A means of monitoring the user's health status and providing information on the nearest medical facility in the event of an abnormality,

[1020] A means of converting currencies in real time and supporting secure payments,

[1021] A means of providing the aforementioned means as application software installed on a smartphone,

[1022] A system that includes this.

[1023] (Claim 2)

[1024] The system according to claim 1, wherein the travel plan can be adjusted based on the user's interests, the duration of the trip, and their financial plan.

[1025] (Claim 3)

[1026] The system according to claim 1, wherein the translation and provision of cultural information are dynamically changed in accordance with the user's input.

[1027] "Example 2 of combining an emotion engine"

[1028] (Claim 1)

[1029] A means of analyzing the user's emotional state using an emotion inference engine based on travel-related information entered by the user, and adjusting the travel plan in real time,

[1030] A means of dynamically adjusting the translation tone based on the inferred emotional state when translating a user's voice or text, and providing information while considering the appropriate cultural background,

[1031] A method for collecting safety information during travel based on the user's emotional state and notifying alerts at the optimal time,

[1032] A means of simultaneously monitoring the user's health and emotional state, and providing information on the nearest medical facility in an emotionally sensitive manner when abnormalities occur,

[1033] A means of securely supporting payments by performing real-time currency conversion based on emotional state,

[1034] A system that includes this.

[1035] (Claim 2)

[1036] The system according to claim 1, which allows the travel plan to be customized based on the user's emotional state, interests, travel duration, and budget.

[1037] (Claim 3)

[1038] The system according to claim 1, wherein the translation and provision of cultural information are dynamically changed according to the user's emotional state and input.

[1039] "Application example 2 when combining with an emotional engine"

[1040] (Claim 1)

[1041] A means for receiving information about areas of interest entered by the user and automatically generating an itinerary based on those preferences,

[1042] A means of analyzing the user's voice or text in real time and providing information that includes appropriate social context,

[1043] A means of collecting safety information about the destination and notifying users of appropriate warnings,

[1044] A means of monitoring the user's health condition and providing information on the nearest medical facility in case of abnormalities,

[1045] A means of securely supporting payments by performing currency conversion in real time,

[1046] A means of analyzing the emotional state of users and suggesting the most suitable products and services,

[1047] A system that includes this.

[1048] (Claim 2)

[1049] The system according to claim 1, which can adapt the aforementioned itinerary based on the user's preferences, travel duration, and budget.

[1050] (Claim 3)

[1051] The system according to claim 1, wherein the translation and provision of social information are dynamically adjusted according to the user's input. [Explanation of symbols]

[1052] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving travel-related information entered by a user and automatically generating a travel plan based on the user's interests, A means of translating user speech or text in real time and providing information that includes appropriate cultural context, A means of collecting safety information about travel destinations and notifying users of appropriate alerts, A means of monitoring the user's health status and providing information on the nearest medical facility in the event of an abnormality, A means of performing real-time currency conversion and securely supporting payments, A system that includes this.

2. The system according to claim 1, which allows the aforementioned travel plan to be customized based on the user's interests, travel duration, and budget.

3. The system according to claim 1, wherein the translation and provision of cultural information are dynamically changed according to the user's input.

Images