system

Abstract

We provide the system. [Solution] A means of executing a travel support program that operates on a mobile communication terminal, A means for processing travel requests from users and retrieving information based on those requests from an external database, A means of suggesting a travel plan suitable for the user based on the acquired information, A means of translating input audio using a multilingual translation function to support communication between the user and third parties, A system that integrates with other service platforms and includes means for providing local information and payment information.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Means are required to eliminate the language and cultural barriers faced by foreign travelers across different cultures, the difficulty of accessing local information, and the anxiety regarding emergency response, so that they can enjoy their trips more smoothly and safely. Also, it is important to improve the convenience for effectively utilizing local services and information during the trip.

Means for Solving the Problems

[0005] This service provides a travel support program that operates on mobile communication terminals. This program processes travel requests from users, acquires information, and proposes appropriate travel plans. It also features multilingual translation capabilities to facilitate communication between users and local residents. Furthermore, it collaborates with other service platforms to provide local and payment information, and in emergencies, it offers users a means to quickly access local emergency services.

[0006] A "mobile communication terminal" refers to a portable electronic device capable of sending and receiving information via a communication network.

[0007] A "travel support program" refers to software designed to provide travel-related information and support access to various services.

[0008] A "user" refers to an individual who utilizes a travel support program and receives travel-related services by operating a mobile communication terminal.

[0009] An "external database" refers to a collection of information accessible via the internet or other means, and includes external information sources such as travel-related data.

[0010] A "travel plan" refers to a plan that includes the itinerary and activities of a trip, tailored to the user's preferences and needs.

[0011] "Multilingual translation function" refers to a software feature that enables automatic translation between different languages.

[0012] A "service platform" refers to the underlying software or system designed to provide specific services or functions.

[0013] "Emergency services" refer to services that require a rapid response in the event of an accident or disaster, and generally include police and emergency medical services. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention utilizes a travel support program that operates on a mobile communication terminal to enable foreign travelers to overcome language and cultural barriers in different cultural spheres and to quickly and easily obtain local information.

[0036] The user first launches a travel support program pre-installed on their mobile communication terminal. The terminal accepts the user's login and sends authentication information to the server, initiating the management of travel data associated with the individual. The server collects data based on the user's current location, past usage history, and interests, and suggests recommended travel plans to the user.

[0037] When a user selects a plan, the device requests related details from the server, displaying information such as tourist attractions, restaurants, and transportation. The server provides this information to the device in an up-to-date state, helping the user navigate smoothly.

[0038] In real-time communication support, the terminal receives voice input through its built-in microphone and sends it to the server. The server translates the voice data using its multilingual translation function and sends it back to the terminal. The terminal then provides the user with this translation result in voice or text, facilitating smooth communication with local residents.

[0039] Furthermore, through this program, users can utilize the linked service platform to handle local reservations and payments. This allows users to enjoy their stay in Japan comfortably without experiencing language or cultural barriers.

[0040] In emergencies, the device uses GPS to pinpoint the user's current location and provides information on the nearest emergency services based on that location. Users can quickly access these services via the app and receive appropriate assistance.

[0041] By utilizing the above functions, travelers can enjoy a safe and comfortable trip to Japan using mobile communication terminals and travel support programs. This invention significantly enhances the individual travel experience and will be a particularly beneficial infrastructure for foreign tourists visiting Japan.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The terminal launches the travel support program and displays an authentication screen to first-time users. The user enters their authentication information and sends it from the terminal to the server.

[0045] Step 2:

[0046] The server receives authentication information and verifies the user's account. If authentication is successful, it retrieves a personalized travel plan from the database based on the user's past history and current location, and sends it to the device.

[0047] Step 3:

[0048] The device displays travel plans received from the server on its home screen. The user selects a plan of interest and sends a request for more details from the device to the server.

[0049] Step 4:

[0050] The server receives the request and extracts detailed information about the selected plan from the database. This information covers a wide range of topics, including tourist attractions, restaurants, and activities, and is then sent to the terminal.

[0051] Step 5:

[0052] The device visually presents the received details to the user, allowing the user to further review and select desired activities.

[0053] Step 6:

[0054] When a user needs to communicate locally, they can use the device's voice input to send their speech to the server.

[0055] Step 7:

[0056] The server analyzes the received audio data and translates it into the selected language using its multilingual translation function. The translated information is then sent back to the terminal.

[0057] Step 8:

[0058] The device outputs the translation results to the user in either voice or text, facilitating smooth communication with local residents.

[0059] Step 9:

[0060] Through the app, users can utilize integrated service platforms to complete transportation reservations and payments for local services.

[0061] Step 10:

[0062] In the event of an emergency, the device uses GPS to pinpoint the user's current location and displays information about the nearest emergency services. This allows the user to respond quickly.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] While many travel support systems using mobile communication devices exist, they are insufficient to overcome language and cultural barriers in different cultural spheres and to enable foreign travelers to smoothly carry out their travel plans. Furthermore, there is a lack of means to respond comfortably and quickly in emergencies, so improvements in safety are needed.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes means for executing travel support software that operates on a mobile communication device, means for processing travel-related requests from users and obtaining information based on those requests from external sources, and means for translating input voice into different languages ​​in real time to assist communication between the user and others. This enables travelers to overcome language and cultural barriers in different cultural spheres and to have a travel experience that allows for quick and safe responses even in emergencies.

[0068] A "mobile communication device" is a portable electronic device that can send and receive information through a communication network.

[0069] "Travel support software" is a program that has functions to help travelers smoothly execute their travel plans.

[0070] "External information sources" refer to external databases and information services that provide data for travel support.

[0071] A "travel plan" is a set of information and schedules for travel, accommodation, and sightseeing activities that are optimized according to the traveler's purpose.

[0072] "Translating into different languages" is the process of converting content entered in one language into another language.

[0073] A "user" is an individual who uses this system and receives travel assistance.

[0074] "Other service platforms" refer to other online services and systems that the travel support software integrates with.

[0075] "Positioning technology" refers to technologies that can determine precise location information using technologies such as GPS.

[0076] A "different cultural sphere" refers to a region with a cultural background and language different from one's own country.

[0077] This invention is designed to enable users to travel smoothly in different cultural regions by using travel support software that operates on a mobile communication device. The following describes a specific embodiment of the system.

[0078] The user first launches the travel assistance software pre-installed on their mobile communication device. The device receives the user's login information and sends it to the server for authentication. The server then verifies the user's authentication information and grants access to their individual travel data. This process ensures data security.

[0079] The server continuously retrieves the latest travel-related information from external sources, generates an optimal travel plan based on the user's current location, past travel history, and interests, and provides it to the user's device. This allows the user to obtain information on tourist attractions, restaurants, and other destinations based on the suggested travel plan.

[0080] The device also uses its built-in microphone to collect voice information and sends it to the server. The server translates the input voice into different languages ​​through its multilingual translation function and sends the result back to the device. Through this function, users can communicate smoothly with people from different cultural backgrounds.

[0081] Furthermore, users can use their devices to connect with multiple service platforms and make reservations and payments locally. This connectivity feature allows users to arrange services in advance and enjoy their trip without worry.

[0082] For example, if a user visits Kyoto, a popular tourist destination in Japan, and books a tea ceremony experience, they can use their device to send their current location and past interest history to a server. The server can then suggest recommended tea ceremony facilities. Based on this information, the user can complete the booking and payment through the service platform.

[0083] As an example of a prompt, entering "Please suggest a travel plan that supports sightseeing and a tea ceremony experience in Kyoto" will cause the AI ​​model to provide the optimal plan. This system allows users to enjoy travel experiences in different cultural spheres more safely and comfortably.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] The user launches travel support software on their mobile communication terminal. The terminal displays a login screen and receives authentication information (e.g., user ID, password) from the user. This input is sent to the server as authentication data. The server verifies the authentication data and outputs whether authentication was successful or not. If authentication is successful, the user gains access to their unique travel data. The specific actions include pressing the login button and the subsequent display of an authentication success message.

[0087] Step 2:

[0088] The server receives the user's current location data and data indicating past usage history and interests as input. Based on this data, it collects relevant travel information from the database and generates an optimal travel plan. As a result, the generated travel plan is output and sent to the terminal. Specifically, it displays a list of travel plans on the screen and provides an interface for the user to select one.

[0089] Step 3:

[0090] The user selects a desired travel plan from those displayed on their device. This selection is sent from the device to the server, which retrieves detailed information related to that plan from external sources. As output, detailed information on tourist attractions, restaurants, and transportation is displayed on the user's device. Specifically, the system displays detailed information along with maps and photos and provides navigation functionality.

[0091] Step 4:

[0092] To facilitate communication with locals, users input voice using the device's built-in microphone. The device sends this voice data to a server. The server uses a multilingual translation function to translate the voice data into the specified language and send it back to the device. The output is provided to the user visually or audibly as translated audio or text. Specific actions include the translation results being scrolled as text or played back as audio.

[0093] Step 5:

[0094] Users access the service platform within the travel support software via their device to make reservations and payments at their destination. The server receives the relevant reservation data and payment information as input and processes it. The output is a reservation confirmation and payment completion notification, displayed on the device. Specific actions include receiving a reservation confirmation email and completing payment using a QR code (registered trademark).

[0095] Step 6:

[0096] In an emergency, the device's GPS function automatically determines the user's current location. This location information is sent to the server as input, and the server searches for the nearest emergency service information based on the location. The output displays the nearest emergency contact information and access methods on the device. Specifically, pressing the emergency contact button sends a phone call or message.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] When travelers are in a different cultural environment, they need to overcome challenges such as language barriers, communication difficulties due to cultural differences, difficulty accessing local food culture, and the need for quick responses in emergencies. In particular, ordering food locally can be confusing due to the language of the menu or differences in currency when paying. It is essential to overcome these problems and provide an environment where travelers can smoothly enjoy their local experiences.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for executing a travel support application that runs on a computer terminal, means for processing travel-related requests from users and obtaining information based on those requests from an external database, and means for taking pictures of menus and translating them into the user's native language through a translation function. This makes it possible for travelers to easily access local information and food culture, overcoming language barriers in different cultural spheres, and to enjoy a comfortable travel experience.

[0102] A "computer terminal" is an electronic device that performs information processing; it is a hardware device that can install software and run applications.

[0103] A "travel support application" is software designed to help travelers obtain local information and facilitate smooth travel and communication.

[0104] A "user" is someone who operates the travel support application and utilizes the service.

[0105] "Processing a request" is the process of receiving input from a user, analyzing its content, and providing the necessary information.

[0106] An "external database" is a collection of data that cannot be accessed directly but whose information can be obtained via a network.

[0107] "Taking a picture of a menu" refers to the action of using a camera to capture an image of a menu from a restaurant or similar establishment.

[0108] A "translation function" is a technology that converts input text or audio into another language.

[0109] "Cultural information" refers to data about traditions, customs, and social background in a particular region or country.

[0110] "Emergency services" refer to organizations and systems that provide the rapid support needed when users face a critical situation.

[0111] "Converting prices" refers to the process of converting the price of a product into a different currency unit and displaying it.

[0112] The system for implementing this invention consists of a computer terminal equipped with a travel support application and a server that works in the backend. The terminal provides various information according to the user's requests, and the server is responsible for data processing and communication with external databases.

[0113] Computer terminals include hardware such as cameras, microphones, and GPS, which are used to collect and process information. Software includes image recognition APIs, translation APIs, and speech recognition engines. For example, menus are scanned using the Google® Vision API and translated using the Google Translate API. User voice input is also converted to text using the Google Speech-to-Text API and translated into multiple languages.

[0114] The server uses the user's past activity history and current location information to suggest appropriate travel plans and local cuisine. It also uses a fixed exchange rate to convert prices in different currencies in real time.

[0115] As a concrete example, there is a function that automatically translates the image and displays the price in USD when a user scans a menu using their smartphone camera at a restaurant overseas. In this process, users can enjoy their meal more efficiently by using prompts such as "Take a picture of the menu and translate it" and "Convert the price to dollars and display it."

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] When the device boots up, the user launches the travel assistance application. The application authenticates the user, connects to the server to retrieve the current location and past history. Based on this data, the available functions are displayed on the screen.

[0119] Step 2:

[0120] The user takes a picture of a menu at a local restaurant using their camera. The device then uses an image recognition API to convert the menu into text data. The input is a camera image, and the output is text information.

[0121] Step 3:

[0122] After the text information is generated, the device calls a translation API to translate the text data into the user's native language. In this process, the input is text information, and the output is the translated text.

[0123] Step 4:

[0124] Translated menu information is displayed on the screen, and the user makes a meal selection. Based on the user's selection, the terminal retrieves local dish recommendations from the server, taking into account past history and allergy information. The input is the selected menu item, and the output is the recommendation information.

[0125] Step 5:

[0126] The terminal checks the price of the dish selected by the user and uses a currency conversion function to convert the local currency to the user's home currency. The terminal retrieves exchange rate information from the server, with the input being the local price and the output being the converted amount.

[0127] Step 6:

[0128] The user places an order using voice commands. The terminal uses a speech recognition engine to convert the speech into text and a translation API to translate it into the target language. The input is voice data, and the output is the translated order text.

[0129] Step 7:

[0130] Once the user confirms their order, the terminal sends the order information to the restaurant via the server. The server then verifies the order and sends a confirmation message to the user. The input is the order text, and the output is the order confirmation.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention aims to make the travel experience more personalized and comfortable for users by using a travel support program and emotion engine that operate on a mobile communication terminal. The emotion engine aims to analyze the user's emotional state and provide appropriate information and services based on the results.

[0133] The user first launches a travel support program installed on their mobile communication terminal. The terminal completes the authentication process and connects to the server to receive individual travel data. When suggesting travel plans, the server considers the user's emotional state as a parameter. An emotion engine analyzes the user's emotions through their input (voice and text data) and sends that information to the server.

[0134] The server generates travel plans and activity suggestions that reflect emotional data and sends them back to the device. For example, if the analysis indicates that the user is feeling stressed, it will suggest relaxing places and activities. Conversely, if feelings of excitement or joy are recognized, it will recommend more active activities and tourist destinations.

[0135] Real-time communication features can also utilize emotion recognition. For example, if a user is feeling anxious, the emotion engine prioritizes providing reassuring translations and local information. Furthermore, explanations of cultural backgrounds based on emotion data are enhanced to help users avoid misunderstandings in interpersonal relationships.

[0136] This system enriches the travel experience by understanding the user's emotions at any given moment through interaction with the device and supporting their actions and choices. Furthermore, users can connect with other service platforms through this program and access essential services in various situations while abroad.

[0137] Thus, the present invention provides more comfortable and personalized travel support through a travel assistance program equipped with an emotion engine, realizing services that meet the user's needs and emotions.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The user launches the travel support program on their mobile communication terminal. The terminal displays the user's login information on an input screen, and the user enters the necessary information.

[0141] Step 2:

[0142] The terminal sends the user's login information to the server for user authentication. The server verifies the authentication information and retrieves the user's personal profile from the database.

[0143] Step 3:

[0144] The server generates the optimal travel plan for the user based on their past travel history and current location. In addition, it utilizes the results of an emotion engine analysis to take into account the user's emotional information.

[0145] Step 4:

[0146] The device uses a microphone and camera to collect user voice and text input, and sends that data to a server and an emotion engine.

[0147] Step 5:

[0148] The emotion engine analyzes the received user input data and determines the user's current emotional state. This information is sent to the server as emotion parameters.

[0149] Step 6:

[0150] The server adjusts travel plans and suggestions based on emotional data received from the emotion engine. For example, if a user is feeling the need to relax, it will prioritize suggesting calm tourist destinations and relaxation activities.

[0151] Step 7:

[0152] The device visually presents users with customized travel plans and activity suggestions. Users can refer to these and select places they want to go and activities they want to participate in.

[0153] Step 8:

[0154] When users need to communicate locally, they can use the device's translation function to translate voice input into the local language. Furthermore, emotion-based translations and advice are provided.

[0155] Step 9:

[0156] Depending on the user's emotions and choices, the device will also interact with other service platforms to assist with processing related information, ticket reservations, payments, and other tasks.

[0157] Step 10:

[0158] In emergencies, the device will immediately provide necessary emergency service information based on the analysis results of the emotion engine and the user's current location, helping the user take swift action.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Providing travelers with a travel experience tailored to their individual emotional states presents challenges, including the difficulty of facilitating smooth multilingual communication and providing insufficient information based on the cultural background of their destination. Furthermore, it is necessary to ensure that there are means to respond quickly and appropriately to emergencies during travel.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for executing a travel support application that operates on a mobile communication device, means for analyzing the user's emotional data using an emotion analysis engine and acquiring information based on the analysis results from an external data storage device, and means for proposing a travel plan tailored to the user based on the acquired information and the emotion analysis results. As a result, the user can enjoy a travel experience that matches their emotions, communicate smoothly in multiple languages, and respond appropriately to emergencies during travel.

[0164] A "mobile communication device" is an electronic device that a user can carry and that has communication capabilities.

[0165] A "travel support application" is software designed to support a user's travel activities.

[0166] An "emotion analysis engine" is a technology that analyzes user input data to identify their emotional state.

[0167] An "external data storage device" is a database that stores information such as the results of user sentiment analysis and allows it to be retrieved as needed.

[0168] A "travel plan" is a proposed plan that includes schedules and activities related to a trip.

[0169] A "multilingual translation function" is a technology that translates between different languages, enabling users to communicate with speakers of other languages.

[0170] A "service platform" is a foundational technology that provides various services related to travel and can be integrated with other systems.

[0171] "Cultural description information" refers to information that provides users with an overview and details about the culture and customs of their destination.

[0172] "Local emergency services" refer to locally provided support organizations and facilities that can be used in the event of an emergency while traveling.

[0173] This invention is a system for providing travelers with personalized travel experiences. Users run a travel support application using a mobile communication terminal. This application is software developed to support the user's travels.

[0174] The device is equipped with an emotion analysis engine that can analyze voice and text input from the user in real time to identify their emotional state. The emotion analysis engine utilizes natural language processing technology to analyze the user's emotional data in detail and transmit the results to an external data storage device.

[0175] The server generates an optimal travel plan for the user based on the received emotional data. The generated travel plan takes into account the user's emotions and preferences, and the suggested activities are tailored to the user's state of mind. For example, if the user wants to relax, it can suggest quiet tourist destinations and relaxation spots.

[0176] Furthermore, the device features multilingual translation capabilities, facilitating smooth communication when users speak foreign languages. This translation function not only ensures that cultural information and interactions with locals at travel destinations are understood, but also enhances cultural descriptions based on sentiment data. As a result, users can minimize cultural misunderstandings during their travels.

[0177] This system also integrates with other service platforms, allowing users to easily access various local and payment information. This enables a quick response to emergencies during travel.

[0178] For example, if a user enters "I want to relax, tell me a quiet place" into the device, the emotion analysis engine analyzes the user's desire to relax, and the server generates suggestions accordingly. As a result, the user can visit a suitable tourist destination.

[0179] An example of a prompt to a generative AI model might be, "Based on the user's sentiment data, suggest activities for this weekend's trip." This prompt allows the system to provide information optimized for the user.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The user launches a travel assistance application on their mobile communication terminal. The terminal first obtains authentication information from the user. Specifically, it receives a user ID and password, or biometric authentication data such as fingerprint or facial recognition, as input. The terminal uses this data to perform the authentication process and, if successful, establishes a connection with the server. The output is the result of whether the authentication was successful or unsuccessful.

[0183] Step 2:

[0184] After the terminal connects to the server, it requests individual travel data. The server uses the user's profile data and past travel history as input to generate travel information tailored to the user. This data processing based on the profile yields individual travel information as output, which is then sent to the terminal. The user can then view this information on the terminal's display.

[0185] Step 3:

[0186] The user inputs emotional information into the terminal in voice or text format. The terminal uses an emotion analysis engine to analyze this input data and identify the user's current emotional state. Specifically, natural language processing techniques are applied to analyze the meaning of the text and the tone of the voice. As a result, an output representing the user's emotional state is obtained and sent to the server.

[0187] Step 4:

[0188] The server generates a travel plan optimized for the user based on the emotion analysis results. The server receives emotion data and user profile data as input and processes the data according to the emotional state. As output, for example, suggestions for relaxing tourist spots and activities are generated and sent to the terminal.

[0189] Step 5:

[0190] The device displays travel plans and activity suggestions received from the server to the user. The user can review them and choose whether to include the suggested activities in their schedule. The user's selection is sent back to the server as feedback, and the plan is adjusted as needed.

[0191] Step 6:

[0192] Because a user's emotional state may change during their trip, the device continuously performs emotion analysis. It analyzes any new emotional input from the user and sends the results to the server. Based on this new input, the server updates information and services in real time and provides them to the device. This continuously optimizes the user's travel experience.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] In today's restaurant and food delivery industries, it is difficult to suggest optimal food choices that are tailored to the individual emotional state and mood of each customer. Furthermore, there is a lack of means to provide appropriate information based on the customer's emotional state and to create sophisticated experiences through collaboration with other services.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes means for executing a support program that operates on a mobile communication device, means for analyzing the user's emotional state using an emotion analysis engine and suggesting food based on the analysis results, and means for coordinating with multiple service platforms to provide local information and payment information. This makes it possible to provide appropriate food and services according to the user's emotions.

[0198] A "mobile communication device" refers to a terminal that is portable by the user and has communication capabilities. Typical examples include smartphones and tablet devices.

[0199] A "support program" refers to software that provides information and service suggestions based on user requests. This program runs on the device and supports the diverse needs of users.

[0200] An "emotion analysis engine" refers to a technology that analyzes a user's emotional state based on their input data. This engine accurately determines the user's emotions by utilizing voice, text, and other data.

[0201] "Suggesting food" refers to the process of selecting and providing the most suitable dishes and dining locations to the user based on their emotional state and other parameters.

[0202] A "service platform" refers to a foundation for integrating multiple different services and providing information and functionality to users. This includes payment services and information provision services.

[0203] "Local information" refers to data related to a specific geographical area. This includes information on tourist attractions, shops, and transportation.

[0204] "Payment information" refers to payment-related data necessary for conducting transactions. This includes the user's payment method and transaction amount.

[0205] The system that implements this application consists of a server program that works in conjunction with a mobile communication device. The mobile communication device has a support program installed and incorporates an emotion analysis engine that analyzes the user's emotional state. Specifically, it can use emotion analysis technologies such as the Google Cloud Natural Language API.

[0206] When a user enters information, the device's emotion analysis engine processes the data and analyzes their emotional state. The analysis results are sent to the server. The server receives the analyzed emotion data and begins integrating with multiple service platforms. For example, it uses APIs to retrieve necessary information from external databases such as local information and payment information. In this case, to suggest food products, it might be possible to integrate with data service APIs for the food service industry (e.g., the Uber Eats API).

[0207] The server suggests services and food items suitable for the user based on the acquired information and the results of sentiment analysis. The suggested information and options are sent back to the terminal and displayed to the user. As a result, the user can obtain an optimal experience that matches their emotional state.

[0208] For example, if a user enters "I'm tired today," the emotion analysis engine recognizes their stress level and suggests a relaxing meal. The suggested dishes might be something like "warm soup" or "Japanese food," and these are selected based on the user's emotional state.

[0209] An example of a prompt message would be: "User's emotion: stressed, type of food suggested: warm and relaxing, e.g., ramen or hot pot."

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The terminal receives user input data. The user provides emotional information to the terminal via voice or text. This input consists of the user's voice messages or text messages. The input information is processed as text data by an emotion analysis engine.

[0213] Step 2:

[0214] The device uses an emotion analysis engine to analyze the user's emotional state from input data. A generative AI model is used for the analysis, specifically the Google Cloud Natural Language API. Text data is the target of the analysis, and an indicator of the emotional state is obtained as output. The analysis results are represented as integer values ​​or category labels, clearly indicating the user's current emotions.

[0215] Step 3:

[0216] The device sends the analysis results to the server. Based on the received emotional data, the server creates prompt messages to collect information relevant to the user. These prompt messages include recommendations based on the emotional state. For example, "User's emotion: stressed, desired type of food: warm and relaxing, e.g., ramen or hot pot."

[0217] Step 4:

[0218] The server uses prompts to call external APIs and retrieve the information needed for food recommendations. These APIs include a food delivery service API and a local information service API. The input is a request based on prompts, and the output is a list of foods corresponding to the user's emotional state. The server analyzes the responses from the APIs and extracts the appropriate data.

[0219] Step 5:

[0220] The server organizes suggestion information based on the data it has acquired and sends it to the terminal. After receiving the suggestion information, the terminal displays food and service options to the user. The user can choose what they want from the presented options, thereby providing an optimal experience tailored to their emotional state. The output is suggestions presented to the user visually or audibly.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention utilizes a travel support program that operates on a mobile communication terminal to enable foreign travelers to overcome language and cultural barriers in different cultural spheres and to quickly and easily obtain local information.

[0238] The user first launches a travel support program pre-installed on their mobile communication terminal. The terminal accepts the user's login and sends authentication information to the server, initiating the management of travel data associated with the individual. The server collects data based on the user's current location, past usage history, and interests, and suggests recommended travel plans to the user.

[0239] When a user selects a plan, the device requests related details from the server, displaying information such as tourist attractions, restaurants, and transportation. The server provides this information to the device in an up-to-date state, helping the user navigate smoothly.

[0240] In real-time communication support, the terminal receives voice input through its built-in microphone and sends it to the server. The server translates the voice data using its multilingual translation function and sends it back to the terminal. The terminal then provides the user with this translation result in voice or text, facilitating smooth communication with local residents.

[0241] Furthermore, through this program, users can utilize the linked service platform to handle local reservations and payments. This allows users to enjoy their stay in Japan comfortably without experiencing language or cultural barriers.

[0242] In emergencies, the device uses GPS to pinpoint the user's current location and provides information on the nearest emergency services based on that location. Users can quickly access these services via the app and receive appropriate assistance.

[0243] By utilizing the above functions, travelers can enjoy a safe and comfortable trip to Japan using mobile communication terminals and travel support programs. This invention significantly enhances the individual travel experience and will be a particularly beneficial infrastructure for foreign tourists visiting Japan.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The terminal launches the travel support program and displays an authentication screen to first-time users. The user enters their authentication information and sends it from the terminal to the server.

[0247] Step 2:

[0248] The server receives authentication information and verifies the user's account. If authentication is successful, it retrieves a personalized travel plan from the database based on the user's past history and current location, and sends it to the device.

[0249] Step 3:

[0250] The device displays travel plans received from the server on its home screen. The user selects a plan of interest and sends a request for more details from the device to the server.

[0251] Step 4:

[0252] The server receives the request and extracts detailed information about the selected plan from the database. This information covers a wide range of topics, including tourist attractions, restaurants, and activities, and is then sent to the terminal.

[0253] Step 5:

[0254] The device visually presents the received details to the user, allowing the user to further review and select desired activities.

[0255] Step 6:

[0256] When a user needs to communicate locally, they can use the device's voice input to send their speech to the server.

[0257] Step 7:

[0258] The server analyzes the received audio data and translates it into the selected language using its multilingual translation function. The translated information is then sent back to the terminal.

[0259] Step 8:

[0260] The device outputs the translation results to the user in either voice or text, facilitating smooth communication with local residents.

[0261] Step 9:

[0262] Through the app, users can utilize integrated service platforms to complete transportation reservations and payments for local services.

[0263] Step 10:

[0264] In the event of an emergency, the device uses GPS to pinpoint the user's current location and displays information about the nearest emergency services. This allows the user to respond quickly.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] While many travel support systems using mobile communication devices exist, they are insufficient to overcome language and cultural barriers in different cultural spheres and to enable foreign travelers to smoothly carry out their travel plans. Furthermore, there is a lack of means to respond comfortably and quickly in emergencies, so improvements in safety are needed.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for executing travel support software that operates on a mobile communication device, means for processing travel-related requests from users and obtaining information based on those requests from external sources, and means for translating input voice into different languages ​​in real time to assist communication between the user and others. This enables travelers to overcome language and cultural barriers in different cultural spheres and to have a travel experience that allows for quick and safe responses even in emergencies.

[0270] A "mobile communication device" is a portable electronic device that can send and receive information through a communication network.

[0271] "Travel support software" is a program that has functions to help travelers smoothly execute their travel plans.

[0272] "External information sources" refer to external databases and information services that provide data for travel support.

[0273] A "travel plan" is a set of information and schedules for travel, accommodation, and sightseeing activities that are optimized according to the traveler's purpose.

[0274] "Translating into different languages" is the process of converting content entered in one language into another language.

[0275] A "user" is an individual who uses this system and receives travel assistance.

[0276] "Other service platforms" refer to other online services and systems that the travel support software integrates with.

[0277] "Positioning technology" refers to technologies that can determine precise location information using technologies such as GPS.

[0278] A "different cultural sphere" refers to a region with a cultural background and language different from one's own country.

[0279] This invention is designed to enable users to travel smoothly in different cultural regions by using travel support software that operates on a mobile communication device. The following describes a specific embodiment of the system.

[0280] The user first launches the travel assistance software pre-installed on the mobile communication device. The terminal receives the user's login information and conducts authentication by sending it to the server. After that, the server verifies the user's authentication information and permits access to individual travel data. Through this process, the security of the data is maintained.

[0281] The server sequentially obtains the latest travel-related information from external information sources, generates an optimal travel plan based on the user's current location, past usage history, and interests, and provides it to the terminal. As a result, the user can obtain information on tourist attractions, restaurants, etc. based on the proposed travel plan.

[0282] Also, the terminal uses the built-in microphone to collect voice information and sends it to the server. The server converts the input voice into different languages through a multilingual translation function and returns the result to the terminal. Through this function, the user can communicate smoothly with the residents of different cultural regions.

[0283] Furthermore, the user can use the terminal to cooperate with multiple service platforms to make reservations and payments locally. With this cooperation function, the user can arrange services in advance and enjoy the trip without worry.

[0284] As a specific example, when the user visits Kyoto, a tourist destination in Japan, and reserves a tea ceremony experience, the server can propose recommended tea ceremony experience facilities by sending the current location information and past interest history to the server using the terminal. Based on this information, the user can complete the reservation and payment through the service platform.

[0285] As an example of the prompt text, by inputting "Please propose a travel plan to support sightseeing and tea ceremony experience in Kyoto.", the generative AI model provides an optimal plan. With this system, the user can enjoy the travel experience in different cultural regions more safely and comfortably.

[0286] The flow of the specific process in Example 1 will be described using FIG. 11.

[0287] Step 1:

[0288] The user starts the travel support software on the mobile communication terminal. The terminal displays a login screen and receives authentication information (e.g., user ID, password) from the user as input. This input is sent to the server as authentication data. The server verifies the authentication data and outputs whether the authentication is successful or not. When the authentication is successful, access to the user-specific travel data becomes possible. Specific operations include pressing the login button and subsequently displaying a success message for authentication.

[0289] Step 2:

[0290] The server receives the user's current location data and data indicating past usage history and interests as input. Based on this data, relevant travel information is collected from the database, and an optimal travel plan is generated. As a result, the generated travel plan is output and sent to the terminal. Specific operations include displaying a list of travel plans on the screen and providing an interface for the user to select.

[0291] Step 3:

[0292] The user selects the desired plan from the travel plans displayed on the terminal. This selection is sent from the terminal to the server, and the server obtains detailed information related to that plan from an external information source. As output, detailed information on tourist attractions, restaurants, and transportation means is displayed on the user's terminal. Specific operations include displaying the detailed information together with a map and photos and providing a navigation function.

[0293] Step 4:

[0294] To facilitate communication with locals, users input voice using the device's built-in microphone. The device sends this voice data to a server. The server uses a multilingual translation function to translate the voice data into the specified language and send it back to the device. The output is provided to the user visually or audibly as translated audio or text. Specific actions include the translation results being scrolled as text or played back as audio.

[0295] Step 5:

[0296] Users access the service platform within the travel support software via their device to make reservations and payments at their destination. The server receives the relevant reservation data and payment information as input and processes it. The output is a reservation confirmation and payment completion notification, displayed on the device. Specific actions include receiving a reservation confirmation email and completing payment using a QR code.

[0297] Step 6:

[0298] In an emergency, the device's GPS function automatically determines the user's current location. This location information is sent to the server as input, and the server searches for the nearest emergency service information based on the location. The output displays the nearest emergency contact information and access methods on the device. Specifically, pressing the emergency contact button sends a phone call or message.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] Travelers need to solve problems such as difficulties in communication due to language barriers and cultural differences in foreign cultural areas, the difficulty of accessing local food cultures, and the need for rapid response in emergencies. In particular, when ordering food locally, there may be confusion due to inability to understand the menu language or differences in currency at the time of payment. There is a need to overcome these problems and provide an environment in which travelers can smoothly enjoy the local experience.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes means for executing a travel support application operating on a computer terminal, means for processing requests related to travel from a user and obtaining information based on the requests from an external database, and means for photographing a menu and translating it into the user's native language through a translation function. As a result, travelers can easily access local information and food cultures across language barriers in foreign cultural areas and enjoy a comfortable travel experience.

[0304] A "computer terminal" is an electronic device that performs information processing and is a hardware device on which software can be installed to execute an application.

[0305] A "travel support application" is software designed for travelers to obtain local information and smoothly move and communicate.

[0306] A "user" is a person who operates a travel support application and uses the service.

[0307] "Processing a request" is a process of receiving an input from a user, analyzing its content, and providing necessary information.

[0308] An "external database" is a collection of data that cannot be accessed directly but whose information can be obtained via a network.

[0309] "Taking a picture of a menu" refers to the action of using a camera to capture an image of a menu from a restaurant or similar establishment.

[0310] A "translation function" is a technology that converts input text or audio into another language.

[0311] "Cultural information" refers to data about traditions, customs, and social background in a particular region or country.

[0312] "Emergency services" refer to organizations and systems that provide the rapid support needed when users face a critical situation.

[0313] "Converting prices" refers to the process of converting the price of a product into a different currency unit and displaying it.

[0314] The system for implementing this invention consists of a computer terminal equipped with a travel support application and a server that works in the backend. The terminal provides various information according to the user's requests, and the server is responsible for data processing and communication with external databases.

[0315] The computer terminal includes hardware such as a camera, microphone, and GPS, which are used to collect and process information. Software includes image recognition APIs, translation APIs, and speech recognition engines. For example, a menu can be scanned using the Google Vision API and translated using the Google Translate API. Additionally, user voice input is converted to text using the Google Speech-to-Text API and then translated into multiple languages.

[0316] The server uses the user's past activity history and current location information to suggest appropriate travel plans and local cuisine. It also uses a fixed exchange rate to convert prices in different currencies in real time.

[0317] As a concrete example, there is a function that automatically translates the image and displays the price in USD when a user scans a menu using their smartphone camera at a restaurant overseas. In this process, users can enjoy their meal more efficiently by using prompts such as "Take a picture of the menu and translate it" and "Convert the price to dollars and display it."

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] When the device boots up, the user launches the travel assistance application. The application authenticates the user, connects to the server to retrieve the current location and past history. Based on this data, the available functions are displayed on the screen.

[0321] Step 2:

[0322] The user takes a picture of a menu at a local restaurant using their camera. The device then uses an image recognition API to convert the menu into text data. The input is a camera image, and the output is text information.

[0323] Step 3:

[0324] After the text information is generated, the device calls a translation API to translate the text data into the user's native language. In this process, the input is text information, and the output is the translated text.

[0325] Step 4:

[0326] Translated menu information is displayed on the screen, and the user makes a meal selection. Based on the user's selection, the terminal retrieves local dish recommendations from the server, taking into account past history and allergy information. The input is the selected menu item, and the output is the recommendation information.

[0327] Step 5:

[0328] The terminal checks the price of the dish selected by the user and uses a currency conversion function to convert the local currency to the user's home currency. The terminal retrieves exchange rate information from the server, with the input being the local price and the output being the converted amount.

[0329] Step 6:

[0330] The user places an order using voice commands. The terminal uses a speech recognition engine to convert the speech into text and a translation API to translate it into the target language. The input is voice data, and the output is the translated order text.

[0331] Step 7:

[0332] Once the user confirms their order, the terminal sends the order information to the restaurant via the server. The server then verifies the order and sends a confirmation message to the user. The input is the order text, and the output is the order confirmation.

[0333] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0334] This invention aims to make the travel experience more personalized and comfortable for users by using a travel support program and emotion engine that operate on a mobile communication terminal. The emotion engine aims to analyze the user's emotional state and provide appropriate information and services based on the results.

[0335] The user first launches a travel support program installed on their mobile communication terminal. The terminal completes the authentication process and connects to the server to receive individual travel data. When suggesting travel plans, the server considers the user's emotional state as a parameter. An emotion engine analyzes the user's emotions through their input (voice and text data) and sends that information to the server.

[0336] The server generates travel plans and activity suggestions that reflect emotional data and sends them back to the device. For example, if the analysis indicates that the user is feeling stressed, it will suggest relaxing places and activities. Conversely, if feelings of excitement or joy are recognized, it will recommend more active activities and tourist destinations.

[0337] Real-time communication features can also utilize emotion recognition. For example, if a user is feeling anxious, the emotion engine prioritizes providing reassuring translations and local information. Furthermore, explanations of cultural backgrounds based on emotion data are enhanced to help users avoid misunderstandings in interpersonal relationships.

[0338] This system enriches the travel experience by understanding the user's emotions at any given moment through interaction with the device and supporting their actions and choices. Furthermore, users can connect with other service platforms through this program and access essential services in various situations while abroad.

[0339] Thus, the present invention provides more comfortable and personalized travel support through a travel assistance program equipped with an emotion engine, realizing services that meet the user's needs and emotions.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The user launches the travel support program on their mobile communication terminal. The terminal displays the user's login information on an input screen, and the user enters the necessary information.

[0343] Step 2:

[0344] The terminal sends the user's login information to the server for user authentication. The server verifies the authentication information and retrieves the user's personal profile from the database.

[0345] Step 3:

[0346] The server generates the optimal travel plan for the user based on their past travel history and current location. In addition, it utilizes the results of an emotion engine analysis to take into account the user's emotional information.

[0347] Step 4:

[0348] The device uses a microphone and camera to collect user voice and text input, and sends that data to a server and an emotion engine.

[0349] Step 5:

[0350] The emotion engine analyzes the received user input data and determines the user's current emotional state. This information is sent to the server as emotion parameters.

[0351] Step 6:

[0352] The server adjusts travel plans and suggestions based on emotional data received from the emotion engine. For example, if a user is feeling the need to relax, it will prioritize suggesting calm tourist destinations and relaxation activities.

[0353] Step 7:

[0354] The device visually presents users with customized travel plans and activity suggestions. Users can refer to these and select places they want to go and activities they want to participate in.

[0355] Step 8:

[0356] When users need to communicate locally, they can use the device's translation function to translate voice input into the local language. Furthermore, emotion-based translations and advice are provided.

[0357] Step 9:

[0358] Depending on the user's emotions and choices, the device will also interact with other service platforms to assist with processing related information, ticket reservations, payments, and other tasks.

[0359] Step 10:

[0360] In emergencies, the device will immediately provide necessary emergency service information based on the analysis results of the emotion engine and the user's current location, helping the user take swift action.

[0361] (Example 2)

[0362] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0363] Providing travelers with a travel experience tailored to their individual emotional states presents challenges, including the difficulty of facilitating smooth multilingual communication and providing insufficient information based on the cultural background of their destination. Furthermore, it is necessary to ensure that there are means to respond quickly and appropriately to emergencies during travel.

[0364] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0365] In this invention, the server includes means for executing a travel support application that operates on a mobile communication device, means for analyzing the user's emotional data using an emotion analysis engine and acquiring information based on the analysis results from an external data storage device, and means for proposing a travel plan tailored to the user based on the acquired information and the emotion analysis results. As a result, the user can enjoy a travel experience that matches their emotions, communicate smoothly in multiple languages, and respond appropriately to emergencies during travel.

[0366] A "mobile communication device" is an electronic device that a user can carry and that has communication capabilities.

[0367] A "travel support application" is software designed to support a user's travel activities.

[0368] An "emotion analysis engine" is a technology that analyzes user input data to identify their emotional state.

[0369] An "external data storage device" is a database that stores information such as the results of user sentiment analysis and allows it to be retrieved as needed.

[0370] A "travel plan" is a proposed plan that includes schedules and activities related to a trip.

[0371] A "multilingual translation function" is a technology that translates between different languages, enabling users to communicate with speakers of other languages.

[0372] A "service platform" is a foundational technology that provides various services related to travel and can be integrated with other systems.

[0373] "Cultural description information" refers to information that provides users with an overview and details about the culture and customs of their destination.

[0374] "Local emergency services" refer to locally provided support organizations and facilities that can be used in the event of an emergency while traveling.

[0375] This invention is a system for providing travelers with personalized travel experiences. Users run a travel support application using a mobile communication terminal. This application is software developed to support the user's travels.

[0376] The device is equipped with an emotion analysis engine that can analyze voice and text input from the user in real time to identify their emotional state. The emotion analysis engine utilizes natural language processing technology to analyze the user's emotional data in detail and transmit the results to an external data storage device.

[0377] The server generates an optimal travel plan for the user based on the received emotional data. The generated travel plan takes into account the user's emotions and preferences, and the suggested activities are tailored to the user's state of mind. For example, if the user wants to relax, it can suggest quiet tourist destinations and relaxation spots.

[0378] Furthermore, the device features multilingual translation capabilities, facilitating smooth communication when users speak foreign languages. This translation function not only ensures that cultural information and interactions with locals at travel destinations are understood, but also enhances cultural descriptions based on sentiment data. As a result, users can minimize cultural misunderstandings during their travels.

[0379] This system also integrates with other service platforms, allowing users to easily access various local and payment information. This enables a quick response to emergencies during travel.

[0380] For example, if a user enters "I want to relax, tell me a quiet place" into the device, the emotion analysis engine analyzes the user's desire to relax, and the server generates suggestions accordingly. As a result, the user can visit a suitable tourist destination.

[0381] An example of a prompt to a generative AI model might be, "Based on the user's sentiment data, suggest activities for this weekend's trip." This prompt allows the system to provide information optimized for the user.

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] The user launches a travel assistance application on their mobile communication terminal. The terminal first obtains authentication information from the user. Specifically, it receives a user ID and password, or biometric authentication data such as fingerprint or facial recognition, as input. The terminal uses this data to perform the authentication process and, if successful, establishes a connection with the server. The output is the result of whether the authentication was successful or unsuccessful.

[0385] Step 2:

[0386] After the terminal connects to the server, it requests individual travel data. The server uses the user's profile data and past travel history as input to generate travel information tailored to the user. This data processing based on the profile yields individual travel information as output, which is then sent to the terminal. The user can then view this information on the terminal's display.

[0387] Step 3:

[0388] The user inputs emotional information into the terminal in voice or text format. The terminal uses an emotion analysis engine to analyze this input data and identify the user's current emotional state. Specifically, natural language processing techniques are applied to analyze the meaning of the text and the tone of the voice. As a result, an output representing the user's emotional state is obtained and sent to the server.

[0389] Step 4:

[0390] The server generates a travel plan optimized for the user based on the emotion analysis results. The server receives emotion data and user profile data as input and processes the data according to the emotional state. As output, for example, suggestions for relaxing tourist spots and activities are generated and sent to the terminal.

[0391] Step 5:

[0392] The device displays travel plans and activity suggestions received from the server to the user. The user can review them and choose whether to include the suggested activities in their schedule. The user's selection is sent back to the server as feedback, and the plan is adjusted as needed.

[0393] Step 6:

[0394] Because a user's emotional state may change during their trip, the device continuously performs emotion analysis. It analyzes any new emotional input from the user and sends the results to the server. Based on this new input, the server updates information and services in real time and provides them to the device. This continuously optimizes the user's travel experience.

[0395] (Application Example 2)

[0396] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0397] In today's restaurant and food delivery industries, it is difficult to suggest optimal food choices that are tailored to the individual emotional state and mood of each customer. Furthermore, there is a lack of means to provide appropriate information based on the customer's emotional state and to create sophisticated experiences through collaboration with other services.

[0398] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0399] In this invention, the server includes means for executing a support program that operates on a mobile communication device, means for analyzing the user's emotional state using an emotion analysis engine and suggesting food based on the analysis results, and means for coordinating with multiple service platforms to provide local information and payment information. This makes it possible to provide appropriate food and services according to the user's emotions.

[0400] A "mobile communication device" refers to a terminal that is portable by the user and has communication capabilities. Typical examples include smartphones and tablet devices.

[0401] A "support program" refers to software that provides information and service suggestions based on user requests. This program runs on the device and supports the diverse needs of users.

[0402] An "emotion analysis engine" refers to a technology that analyzes a user's emotional state based on their input data. This engine accurately determines the user's emotions by utilizing voice, text, and other data.

[0403] "Suggesting food" refers to the process of selecting and providing the most suitable dishes and dining locations to the user based on their emotional state and other parameters.

[0404] A "service platform" refers to a foundation for integrating multiple different services and providing information and functionality to users. This includes payment services and information provision services.

[0405] "Local information" refers to data related to a specific geographical area. This includes information on tourist attractions, shops, and transportation.

[0406] "Payment information" refers to payment-related data necessary for conducting transactions. This includes the user's payment method and transaction amount.

[0407] The system that implements this application consists of a server program that works in conjunction with a mobile communication device. The mobile communication device has a support program installed and incorporates an emotion analysis engine that analyzes the user's emotional state. Specifically, it can use emotion analysis technologies such as the Google Cloud Natural Language API.

[0408] When a user enters information, the device's emotion analysis engine processes the data and analyzes their emotional state. The analysis results are sent to the server. The server receives the analyzed emotion data and begins integrating with multiple service platforms. For example, it uses APIs to retrieve necessary information from external databases such as local information and payment information. In this case, to suggest food products, it might be possible to integrate with data service APIs for the food service industry (e.g., the Uber Eats API).

[0409] The server suggests services and food items suitable for the user based on the acquired information and the results of sentiment analysis. The suggested information and options are sent back to the terminal and displayed to the user. As a result, the user can obtain an optimal experience that matches their emotional state.

[0410] For example, if a user enters "I'm tired today," the emotion analysis engine recognizes their stress level and suggests a relaxing meal. The suggested dishes might be something like "warm soup" or "Japanese food," and these are selected based on the user's emotional state.

[0411] An example of a prompt message would be: "User's emotion: stressed, type of food suggested: warm and relaxing, e.g., ramen or hot pot."

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The terminal receives user input data. The user provides emotional information to the terminal via voice or text. This input consists of the user's voice messages or text messages. The input information is processed as text data by an emotion analysis engine.

[0415] Step 2:

[0416] The device uses an emotion analysis engine to analyze the user's emotional state from input data. A generative AI model is used for the analysis, specifically the Google Cloud Natural Language API. Text data is the target of the analysis, and an indicator of the emotional state is obtained as output. The analysis results are represented as integer values ​​or category labels, clearly indicating the user's current emotions.

[0417] Step 3:

[0418] The device sends the analysis results to the server. Based on the received emotional data, the server creates prompt messages to collect information relevant to the user. These prompt messages include recommendations based on the emotional state. For example, "User's emotion: stressed, desired type of food: warm and relaxing, e.g., ramen or hot pot."

[0419] Step 4:

[0420] The server uses prompts to call external APIs and retrieve the information needed for food recommendations. These APIs include a food delivery service API and a local information service API. The input is a request based on prompts, and the output is a list of foods corresponding to the user's emotional state. The server analyzes the responses from the APIs and extracts the appropriate data.

[0421] Step 5:

[0422] The server organizes suggestion information based on the data it has acquired and sends it to the terminal. After receiving the suggestion information, the terminal displays food and service options to the user. The user can choose what they want from the presented options, thereby providing an optimal experience tailored to their emotional state. The output is suggestions presented to the user visually or audibly.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] This invention utilizes a travel support program that operates on a mobile communication terminal to enable foreign travelers to overcome language and cultural barriers in different cultural spheres and to quickly and easily obtain local information.

[0440] The user first launches a travel support program pre-installed on their mobile communication terminal. The terminal accepts the user's login and sends authentication information to the server, initiating the management of travel data associated with the individual. The server collects data based on the user's current location, past usage history, and interests, and suggests recommended travel plans to the user.

[0441] When a user selects a plan, the device requests related details from the server, displaying information such as tourist attractions, restaurants, and transportation. The server provides this information to the device in an up-to-date state, helping the user navigate smoothly.

[0442] In real-time communication support, the terminal receives voice input through its built-in microphone and sends it to the server. The server translates the voice data using its multilingual translation function and sends it back to the terminal. The terminal then provides the user with this translation result in voice or text, facilitating smooth communication with local residents.

[0443] Furthermore, through this program, users can utilize the linked service platform to handle local reservations and payments. This allows users to enjoy their stay in Japan comfortably without experiencing language or cultural barriers.

[0444] In emergencies, the device uses GPS to pinpoint the user's current location and provides information on the nearest emergency services based on that location. Users can quickly access these services via the app and receive appropriate assistance.

[0445] By utilizing the above functions, travelers can enjoy a safe and comfortable trip to Japan using mobile communication terminals and travel support programs. This invention significantly enhances the individual travel experience and will be a particularly beneficial infrastructure for foreign tourists visiting Japan.

[0446] The following describes the processing flow.

[0447] Step 1:

[0448] The terminal launches the travel support program and displays an authentication screen to first-time users. The user enters their authentication information and sends it from the terminal to the server.

[0449] Step 2:

[0450] The server receives authentication information and verifies the user's account. If authentication is successful, it retrieves a personalized travel plan from the database based on the user's past history and current location, and sends it to the device.

[0451] Step 3:

[0452] The device displays travel plans received from the server on its home screen. The user selects a plan of interest and sends a request for more details from the device to the server.

[0453] Step 4:

[0454] The server receives the request and extracts detailed information about the selected plan from the database. This information covers a wide range of topics, including tourist attractions, restaurants, and activities, and is then sent to the terminal.

[0455] Step 5:

[0456] The device visually presents the received details to the user, allowing the user to further review and select desired activities.

[0457] Step 6:

[0458] When a user needs to communicate locally, they can use the device's voice input to send their speech to the server.

[0459] Step 7:

[0460] The server analyzes the received audio data and translates it into the selected language using its multilingual translation function. The translated information is then sent back to the terminal.

[0461] Step 8:

[0462] The device outputs the translation results to the user in either voice or text, facilitating smooth communication with local residents.

[0463] Step 9:

[0464] Through the app, users can utilize integrated service platforms to complete transportation reservations and payments for local services.

[0465] Step 10:

[0466] In the event of an emergency, the device uses GPS to pinpoint the user's current location and displays information about the nearest emergency services. This allows the user to respond quickly.

[0467] (Example 1)

[0468] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0469] While many travel support systems using mobile communication devices exist, they are insufficient to overcome language and cultural barriers in different cultural spheres and to enable foreign travelers to smoothly carry out their travel plans. Furthermore, there is a lack of means to respond comfortably and quickly in emergencies, so improvements in safety are needed.

[0470] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0471] In this invention, the server includes means for executing travel support software that operates on a mobile communication device, means for processing travel-related requests from users and obtaining information based on those requests from external sources, and means for translating input voice into different languages ​​in real time to assist communication between the user and others. This enables travelers to overcome language and cultural barriers in different cultural spheres and to have a travel experience that allows for quick and safe responses even in emergencies.

[0472] A "mobile communication device" is a portable electronic device that can send and receive information through a communication network.

[0473] "Travel support software" is a program that has functions to help travelers smoothly execute their travel plans.

[0474] "External information sources" refer to external databases and information services that provide data for travel support.

[0475] A "travel plan" is a set of information and schedules for travel, accommodation, and sightseeing activities that are optimized according to the traveler's purpose.

[0476] "Translating into different languages" is the process of converting content entered in one language into another language.

[0477] A "user" is an individual who uses this system and receives travel assistance.

[0478] "Other service platforms" refer to other online services and systems that the travel support software integrates with.

[0479] "Positioning technology" refers to technologies that can determine precise location information using technologies such as GPS.

[0480] A "different cultural sphere" refers to a region with a cultural background and language different from one's own country.

[0481] This invention is designed to enable users to travel smoothly in different cultural regions by using travel support software that operates on a mobile communication device. The following describes a specific embodiment of the system.

[0482] The user first launches the travel assistance software pre-installed on their mobile communication device. The device receives the user's login information and sends it to the server for authentication. The server then verifies the user's authentication information and grants access to their individual travel data. This process ensures data security.

[0483] The server continuously retrieves the latest travel-related information from external sources, generates an optimal travel plan based on the user's current location, past travel history, and interests, and provides it to the user's device. This allows the user to obtain information on tourist attractions, restaurants, and other destinations based on the suggested travel plan.

[0484] The device also uses its built-in microphone to collect voice information and sends it to the server. The server translates the input voice into different languages ​​through its multilingual translation function and sends the result back to the device. Through this function, users can communicate smoothly with people from different cultural backgrounds.

[0485] Furthermore, users can use their devices to connect with multiple service platforms and make reservations and payments locally. This connectivity feature allows users to arrange services in advance and enjoy their trip without worry.

[0486] For example, if a user visits Kyoto, a popular tourist destination in Japan, and books a tea ceremony experience, they can use their device to send their current location and past interest history to a server. The server can then suggest recommended tea ceremony facilities. Based on this information, the user can complete the booking and payment through the service platform.

[0487] As an example of a prompt, entering "Please suggest a travel plan that supports sightseeing and a tea ceremony experience in Kyoto" will cause the AI ​​model to provide the optimal plan. This system allows users to enjoy travel experiences in different cultural spheres more safely and comfortably.

[0488] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0489] Step 1:

[0490] The user launches travel support software on their mobile communication terminal. The terminal displays a login screen and receives authentication information (e.g., user ID, password) from the user. This input is sent to the server as authentication data. The server verifies the authentication data and outputs whether authentication was successful or not. If authentication is successful, the user gains access to their unique travel data. The specific actions include pressing the login button and the subsequent display of an authentication success message.

[0491] Step 2:

[0492] The server receives the user's current location data and data indicating past usage history and interests as input. Based on this data, it collects relevant travel information from the database and generates an optimal travel plan. As a result, the generated travel plan is output and sent to the terminal. Specifically, it displays a list of travel plans on the screen and provides an interface for the user to select one.

[0493] Step 3:

[0494] The user selects a desired travel plan from those displayed on their device. This selection is sent from the device to the server, which retrieves detailed information related to that plan from external sources. As output, detailed information on tourist attractions, restaurants, and transportation is displayed on the user's device. Specifically, the system displays detailed information along with maps and photos and provides navigation functionality.

[0495] Step 4:

[0496] To facilitate communication with locals, users input voice using the device's built-in microphone. The device sends this voice data to a server. The server uses a multilingual translation function to translate the voice data into the specified language and send it back to the device. The output is provided to the user visually or audibly as translated audio or text. Specific actions include the translation results being scrolled as text or played back as audio.

[0497] Step 5:

[0498] Users access the service platform within the travel support software via their device to make reservations and payments at their destination. The server receives the relevant reservation data and payment information as input and processes it. The output is a reservation confirmation and payment completion notification, displayed on the device. Specific actions include receiving a reservation confirmation email and completing payment using a QR code.

[0499] Step 6:

[0500] In an emergency, the device's GPS function automatically determines the user's current location. This location information is sent to the server as input, and the server searches for the nearest emergency service information based on the location. The output displays the nearest emergency contact information and access methods on the device. Specifically, pressing the emergency contact button sends a phone call or message.

[0501] (Application Example 1)

[0502] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0503] When travelers are in a different cultural environment, they need to overcome challenges such as language barriers, communication difficulties due to cultural differences, difficulty accessing local food culture, and the need for quick responses in emergencies. In particular, ordering food locally can be confusing due to the language of the menu or differences in currency when paying. It is essential to overcome these problems and provide an environment where travelers can smoothly enjoy their local experiences.

[0504] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0505] In this invention, the server includes means for executing a travel support application that runs on a computer terminal, means for processing travel-related requests from users and obtaining information based on those requests from an external database, and means for taking pictures of menus and translating them into the user's native language through a translation function. This makes it possible for travelers to easily access local information and food culture, overcoming language barriers in different cultural spheres, and to enjoy a comfortable travel experience.

[0506] A "computer terminal" is an electronic device that performs information processing; it is a hardware device that can install software and run applications.

[0507] A "travel support application" is software designed to help travelers obtain local information and facilitate smooth travel and communication.

[0508] A "user" is someone who operates the travel support application and utilizes the service.

[0509] "Processing a request" is the process of receiving input from a user, analyzing its content, and providing the necessary information.

[0510] An "external database" is a collection of data that cannot be accessed directly but whose information can be obtained via a network.

[0511] "Taking a picture of a menu" refers to the action of using a camera to capture an image of a menu from a restaurant or similar establishment.

[0512] A "translation function" is a technology that converts input text or audio into another language.

[0513] "Cultural information" refers to data about traditions, customs, and social background in a particular region or country.

[0514] "Emergency services" refer to organizations and systems that provide the rapid support needed when users face a critical situation.

[0515] "Converting prices" refers to the process of converting the price of a product into a different currency unit and displaying it.

[0516] The system for implementing this invention consists of a computer terminal equipped with a travel support application and a server that works in the backend. The terminal provides various information according to the user's requests, and the server is responsible for data processing and communication with external databases.

[0517] The computer terminal includes hardware such as a camera, microphone, and GPS, which are used to collect and process information. Software includes image recognition APIs, translation APIs, and speech recognition engines. For example, a menu can be scanned using the Google Vision API and translated using the Google Translate API. Additionally, user voice input is converted to text using the Google Speech-to-Text API and then translated into multiple languages.

[0518] The server uses the user's past activity history and current location information to suggest appropriate travel plans and local cuisine. It also uses a fixed exchange rate to convert prices in different currencies in real time.

[0519] As a concrete example, there is a function that automatically translates the image and displays the price in USD when a user scans a menu using their smartphone camera at a restaurant overseas. In this process, users can enjoy their meal more efficiently by using prompts such as "Take a picture of the menu and translate it" and "Convert the price to dollars and display it."

[0520] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0521] Step 1:

[0522] When the device boots up, the user launches the travel assistance application. The application authenticates the user, connects to the server to retrieve the current location and past history. Based on this data, the available functions are displayed on the screen.

[0523] Step 2:

[0524] The user takes a picture of a menu at a local restaurant using their camera. The device then uses an image recognition API to convert the menu into text data. The input is a camera image, and the output is text information.

[0525] Step 3:

[0526] After the text information is generated, the device calls a translation API to translate the text data into the user's native language. In this process, the input is text information, and the output is the translated text.

[0527] Step 4:

[0528] Translated menu information is displayed on the screen, and the user makes a meal selection. Based on the user's selection, the terminal retrieves local dish recommendations from the server, taking into account past history and allergy information. The input is the selected menu item, and the output is the recommendation information.

[0529] Step 5:

[0530] The terminal checks the price of the dish selected by the user and uses a currency conversion function to convert the local currency to the user's home currency. The terminal retrieves exchange rate information from the server, with the input being the local price and the output being the converted amount.

[0531] Step 6:

[0532] The user places an order using voice commands. The terminal uses a speech recognition engine to convert the speech into text and a translation API to translate it into the target language. The input is voice data, and the output is the translated order text.

[0533] Step 7:

[0534] Once the user confirms their order, the terminal sends the order information to the restaurant via the server. The server then verifies the order and sends a confirmation message to the user. The input is the order text, and the output is the order confirmation.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] This invention aims to make the travel experience more personalized and comfortable for users by using a travel support program and emotion engine that operate on a mobile communication terminal. The emotion engine aims to analyze the user's emotional state and provide appropriate information and services based on the results.

[0537] The user first launches a travel support program installed on their mobile communication terminal. The terminal completes the authentication process and connects to the server to receive individual travel data. When suggesting travel plans, the server considers the user's emotional state as a parameter. An emotion engine analyzes the user's emotions through their input (voice and text data) and sends that information to the server.

[0538] The server generates travel plans and activity suggestions that reflect emotional data and sends them back to the device. For example, if the analysis indicates that the user is feeling stressed, it will suggest relaxing places and activities. Conversely, if feelings of excitement or joy are recognized, it will recommend more active activities and tourist destinations.

[0539] Real-time communication features can also utilize emotion recognition. For example, if a user is feeling anxious, the emotion engine prioritizes providing reassuring translations and local information. Furthermore, explanations of cultural backgrounds based on emotion data are enhanced to help users avoid misunderstandings in interpersonal relationships.

[0540] This system enriches the travel experience by understanding the user's emotions at any given moment through interaction with the device and supporting their actions and choices. Furthermore, users can connect with other service platforms through this program and access essential services in various situations while abroad.

[0541] Thus, the present invention provides more comfortable and personalized travel support through a travel assistance program equipped with an emotion engine, realizing services that meet the user's needs and emotions.

[0542] The following describes the processing flow.

[0543] Step 1:

[0544] The user launches the travel support program on their mobile communication terminal. The terminal displays the user's login information on an input screen, and the user enters the necessary information.

[0545] Step 2:

[0546] The terminal sends the user's login information to the server for user authentication. The server verifies the authentication information and retrieves the user's personal profile from the database.

[0547] Step 3:

[0548] The server generates the optimal travel plan for the user based on their past travel history and current location. In addition, it utilizes the results of an emotion engine analysis to take into account the user's emotional information.

[0549] Step 4:

[0550] The device uses a microphone and camera to collect user voice and text input, and sends that data to a server and an emotion engine.

[0551] Step 5:

[0552] The emotion engine analyzes the received user input data and determines the user's current emotional state. This information is sent to the server as emotion parameters.

[0553] Step 6:

[0554] The server adjusts travel plans and suggestions based on emotional data received from the emotion engine. For example, if a user is feeling the need to relax, it will prioritize suggesting calm tourist destinations and relaxation activities.

[0555] Step 7:

[0556] The device visually presents users with customized travel plans and activity suggestions. Users can refer to these and select places they want to go and activities they want to participate in.

[0557] Step 8:

[0558] When users need to communicate locally, they can use the device's translation function to translate voice input into the local language. Furthermore, emotion-based translations and advice are provided.

[0559] Step 9:

[0560] Depending on the user's emotions and choices, the device will also interact with other service platforms to assist with processing related information, ticket reservations, payments, and other tasks.

[0561] Step 10:

[0562] In emergencies, the device will immediately provide necessary emergency service information based on the analysis results of the emotion engine and the user's current location, helping the user take swift action.

[0563] (Example 2)

[0564] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0565] Providing travelers with a travel experience tailored to their individual emotional states presents challenges, including the difficulty of facilitating smooth multilingual communication and providing insufficient information based on the cultural background of their destination. Furthermore, it is necessary to ensure that there are means to respond quickly and appropriately to emergencies during travel.

[0566] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0567] In this invention, the server includes means for executing a travel support application that operates on a mobile communication device, means for analyzing the user's emotional data using an emotion analysis engine and acquiring information based on the analysis results from an external data storage device, and means for proposing a travel plan tailored to the user based on the acquired information and the emotion analysis results. As a result, the user can enjoy a travel experience that matches their emotions, communicate smoothly in multiple languages, and respond appropriately to emergencies during travel.

[0568] A "mobile communication device" is an electronic device that a user can carry and that has communication capabilities.

[0569] A "travel support application" is software designed to support a user's travel activities.

[0570] An "emotion analysis engine" is a technology that analyzes user input data to identify their emotional state.

[0571] An "external data storage device" is a database that stores information such as the results of user sentiment analysis and allows it to be retrieved as needed.

[0572] A "travel plan" is a proposed plan that includes schedules and activities related to a trip.

[0573] A "multilingual translation function" is a technology that translates between different languages, enabling users to communicate with speakers of other languages.

[0574] A "service platform" is a foundational technology that provides various services related to travel and can be integrated with other systems.

[0575] "Cultural description information" refers to information that provides users with an overview and details about the culture and customs of their destination.

[0576] "Local emergency services" refer to locally provided support organizations and facilities that can be used in the event of an emergency while traveling.

[0577] This invention is a system for providing travelers with personalized travel experiences. Users run a travel support application using a mobile communication terminal. This application is software developed to support the user's travels.

[0578] The device is equipped with an emotion analysis engine that can analyze voice and text input from the user in real time to identify their emotional state. The emotion analysis engine utilizes natural language processing technology to analyze the user's emotional data in detail and transmit the results to an external data storage device.

[0579] The server generates an optimal travel plan for the user based on the received emotional data. The generated travel plan takes into account the user's emotions and preferences, and the suggested activities are tailored to the user's state of mind. For example, if the user wants to relax, it can suggest quiet tourist destinations and relaxation spots.

[0580] Furthermore, the device features multilingual translation capabilities, facilitating smooth communication when users speak foreign languages. This translation function not only ensures that cultural information and interactions with locals at travel destinations are understood, but also enhances cultural descriptions based on sentiment data. As a result, users can minimize cultural misunderstandings during their travels.

[0581] This system also integrates with other service platforms, allowing users to easily access various local and payment information. This enables a quick response to emergencies during travel.

[0582] For example, if a user enters "I want to relax, tell me a quiet place" into the device, the emotion analysis engine analyzes the user's desire to relax, and the server generates suggestions accordingly. As a result, the user can visit a suitable tourist destination.

[0583] An example of a prompt to a generative AI model might be, "Based on the user's sentiment data, suggest activities for this weekend's trip." This prompt allows the system to provide information optimized for the user.

[0584] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0585] Step 1:

[0586] The user launches a travel assistance application on their mobile communication terminal. The terminal first obtains authentication information from the user. Specifically, it receives a user ID and password, or biometric authentication data such as fingerprint or facial recognition, as input. The terminal uses this data to perform the authentication process and, if successful, establishes a connection with the server. The output is the result of whether the authentication was successful or unsuccessful.

[0587] Step 2:

[0588] After the terminal connects to the server, it requests individual travel data. The server uses the user's profile data and past travel history as input to generate travel information tailored to the user. This data processing based on the profile yields individual travel information as output, which is then sent to the terminal. The user can then view this information on the terminal's display.

[0589] Step 3:

[0590] The user inputs emotional information into the terminal in voice or text format. The terminal uses an emotion analysis engine to analyze this input data and identify the user's current emotional state. Specifically, natural language processing techniques are applied to analyze the meaning of the text and the tone of the voice. As a result, an output representing the user's emotional state is obtained and sent to the server.

[0591] Step 4:

[0592] The server generates a travel plan optimized for the user based on the emotion analysis results. The server receives emotion data and user profile data as input and processes the data according to the emotional state. As output, for example, suggestions for relaxing tourist spots and activities are generated and sent to the terminal.

[0593] Step 5:

[0594] The device displays travel plans and activity suggestions received from the server to the user. The user can review them and choose whether to include the suggested activities in their schedule. The user's selection is sent back to the server as feedback, and the plan is adjusted as needed.

[0595] Step 6:

[0596] Because a user's emotional state may change during their trip, the device continuously performs emotion analysis. It analyzes any new emotional input from the user and sends the results to the server. Based on this new input, the server updates information and services in real time and provides them to the device. This continuously optimizes the user's travel experience.

[0597] (Application Example 2)

[0598] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0599] In today's restaurant and food delivery industries, it is difficult to suggest optimal food choices that are tailored to the individual emotional state and mood of each customer. Furthermore, there is a lack of means to provide appropriate information based on the customer's emotional state and to create sophisticated experiences through collaboration with other services.

[0600] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0601] In this invention, the server includes means for executing a support program that operates on a mobile communication device, means for analyzing the user's emotional state using an emotion analysis engine and suggesting food based on the analysis results, and means for coordinating with multiple service platforms to provide local information and payment information. This makes it possible to provide appropriate food and services according to the user's emotions.

[0602] A "mobile communication device" refers to a terminal that is portable by the user and has communication capabilities. Typical examples include smartphones and tablet devices.

[0603] A "support program" refers to software that provides information and service suggestions based on user requests. This program runs on the device and supports the diverse needs of users.

[0604] An "emotion analysis engine" refers to a technology that analyzes a user's emotional state based on their input data. This engine accurately determines the user's emotions by utilizing voice, text, and other data.

[0605] "Suggesting food" refers to the process of selecting and providing the most suitable dishes and dining locations to the user based on their emotional state and other parameters.

[0606] A "service platform" refers to a foundation for integrating multiple different services and providing information and functionality to users. This includes payment services and information provision services.

[0607] "Local information" refers to data related to a specific geographical area. This includes information on tourist attractions, shops, and transportation.

[0608] "Payment information" refers to payment-related data necessary for conducting transactions. This includes the user's payment method and transaction amount.

[0609] The system that implements this application consists of a server program that works in conjunction with a mobile communication device. The mobile communication device has a support program installed and incorporates an emotion analysis engine that analyzes the user's emotional state. Specifically, it can use emotion analysis technologies such as the Google Cloud Natural Language API.

[0610] When a user enters information, the device's emotion analysis engine processes the data and analyzes their emotional state. The analysis results are sent to the server. The server receives the analyzed emotion data and begins integrating with multiple service platforms. For example, it uses APIs to retrieve necessary information from external databases such as local information and payment information. In this case, to suggest food products, it might be possible to integrate with data service APIs for the food service industry (e.g., the Uber Eats API).

[0611] The server suggests services and food items suitable for the user based on the acquired information and the results of sentiment analysis. The suggested information and options are sent back to the terminal and displayed to the user. As a result, the user can obtain an optimal experience that matches their emotional state.

[0612] For example, if a user enters "I'm tired today," the emotion analysis engine recognizes their stress level and suggests a relaxing meal. The suggested dishes might be something like "warm soup" or "Japanese food," and these are selected based on the user's emotional state.

[0613] An example of a prompt message would be: "User's emotion: stressed, type of food suggested: warm and relaxing, e.g., ramen or hot pot."

[0614] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0615] Step 1:

[0616] The terminal receives user input data. The user provides emotional information to the terminal via voice or text. This input consists of the user's voice messages or text messages. The input information is processed as text data by an emotion analysis engine.

[0617] Step 2:

[0618] The device uses an emotion analysis engine to analyze the user's emotional state from input data. A generative AI model is used for the analysis, specifically the Google Cloud Natural Language API. Text data is the target of the analysis, and an indicator of the emotional state is obtained as output. The analysis results are represented as integer values ​​or category labels, clearly indicating the user's current emotions.

[0619] Step 3:

[0620] The device sends the analysis results to the server. Based on the received emotional data, the server creates prompt messages to collect information relevant to the user. These prompt messages include recommendations based on the emotional state. For example, "User's emotion: stressed, desired type of food: warm and relaxing, e.g., ramen or hot pot."

[0621] Step 4:

[0622] The server uses prompts to call external APIs and retrieve the information needed for food recommendations. These APIs include a food delivery service API and a local information service API. The input is a request based on prompts, and the output is a list of foods corresponding to the user's emotional state. The server analyzes the responses from the APIs and extracts the appropriate data.

[0623] Step 5:

[0624] The server organizes suggestion information based on the data it has acquired and sends it to the terminal. After receiving the suggestion information, the terminal displays food and service options to the user. The user can choose what they want from the presented options, thereby providing an optimal experience tailored to their emotional state. The output is suggestions presented to the user visually or audibly.

[0625] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0626] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0628] [Fourth Embodiment]

[0629] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0630] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0631] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0633] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0635] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0636] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0637] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0638] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0641] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] This invention utilizes a travel support program that operates on a mobile communication terminal to enable foreign travelers to overcome language and cultural barriers in different cultural spheres and to quickly and easily obtain local information.

[0643] The user first launches a travel support program pre-installed on their mobile communication terminal. The terminal accepts the user's login and sends authentication information to the server, initiating the management of travel data associated with the individual. The server collects data based on the user's current location, past usage history, and interests, and suggests recommended travel plans to the user.

[0644] When a user selects a plan, the device requests related details from the server, displaying information such as tourist attractions, restaurants, and transportation. The server provides this information to the device in an up-to-date state, helping the user navigate smoothly.

[0645] In real-time communication support, the terminal receives voice input through its built-in microphone and sends it to the server. The server translates the voice data using its multilingual translation function and sends it back to the terminal. The terminal then provides the user with this translation result in voice or text, facilitating smooth communication with local residents.

[0646] Furthermore, through this program, users can utilize the linked service platform to handle local reservations and payments. This allows users to enjoy their stay in Japan comfortably without experiencing language or cultural barriers.

[0647] In emergencies, the device uses GPS to pinpoint the user's current location and provides information on the nearest emergency services based on that location. Users can quickly access these services via the app and receive appropriate assistance.

[0648] By utilizing the above functions, travelers can enjoy a safe and comfortable trip to Japan using mobile communication terminals and travel support programs. This invention significantly enhances the individual travel experience and will be a particularly beneficial infrastructure for foreign tourists visiting Japan.

[0649] The following describes the processing flow.

[0650] Step 1:

[0651] The terminal launches the travel support program and displays an authentication screen to first-time users. The user enters their authentication information and sends it from the terminal to the server.

[0652] Step 2:

[0653] The server receives authentication information and verifies the user's account. If authentication is successful, it retrieves a personalized travel plan from the database based on the user's past history and current location, and sends it to the device.

[0654] Step 3:

[0655] The device displays travel plans received from the server on its home screen. The user selects a plan of interest and sends a request for more details from the device to the server.

[0656] Step 4:

[0657] The server receives the request and extracts detailed information about the selected plan from the database. This information covers a wide range of topics, including tourist attractions, restaurants, and activities, and is then sent to the terminal.

[0658] Step 5:

[0659] The device visually presents the received details to the user, allowing the user to further review and select desired activities.

[0660] Step 6:

[0661] When a user needs to communicate locally, they can use the device's voice input to send their speech to the server.

[0662] Step 7:

[0663] The server analyzes the received audio data and translates it into the selected language using its multilingual translation function. The translated information is then sent back to the terminal.

[0664] Step 8:

[0665] The device outputs the translation results to the user in either voice or text, facilitating smooth communication with local residents.

[0666] Step 9:

[0667] Through the app, users can utilize integrated service platforms to complete transportation reservations and payments for local services.

[0668] Step 10:

[0669] In the event of an emergency, the device uses GPS to pinpoint the user's current location and displays information about the nearest emergency services. This allows the user to respond quickly.

[0670] (Example 1)

[0671] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0672] While many travel support systems using mobile communication devices exist, they are insufficient to overcome language and cultural barriers in different cultural spheres and to enable foreign travelers to smoothly carry out their travel plans. Furthermore, there is a lack of means to respond comfortably and quickly in emergencies, so improvements in safety are needed.

[0673] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0674] In this invention, the server includes means for executing travel support software that operates on a mobile communication device, means for processing travel-related requests from users and obtaining information based on those requests from external sources, and means for translating input voice into different languages ​​in real time to assist communication between the user and others. This enables travelers to overcome language and cultural barriers in different cultural spheres and to have a travel experience that allows for quick and safe responses even in emergencies.

[0675] A "mobile communication device" is a portable electronic device that can send and receive information through a communication network.

[0676] "Travel support software" is a program that has functions to help travelers smoothly execute their travel plans.

[0677] "External information sources" refer to external databases and information services that provide data for travel support.

[0678] A "travel plan" is a set of information and schedules for travel, accommodation, and sightseeing activities that are optimized according to the traveler's purpose.

[0679] "Translating into different languages" is the process of converting content entered in one language into another language.

[0680] A "user" is an individual who uses this system and receives travel assistance.

[0681] "Other service platforms" refer to other online services and systems that the travel support software integrates with.

[0682] "Positioning technology" refers to technologies that can determine precise location information using technologies such as GPS.

[0683] A "different cultural sphere" refers to a region with a cultural background and language different from one's own country.

[0684] This invention is designed to enable users to travel smoothly in different cultural regions by using travel support software that operates on a mobile communication device. The following describes a specific embodiment of the system.

[0685] The user first launches the travel assistance software pre-installed on their mobile communication device. The device receives the user's login information and sends it to the server for authentication. The server then verifies the user's authentication information and grants access to their individual travel data. This process ensures data security.

[0686] The server continuously retrieves the latest travel-related information from external sources, generates an optimal travel plan based on the user's current location, past travel history, and interests, and provides it to the user's device. This allows the user to obtain information on tourist attractions, restaurants, and other destinations based on the suggested travel plan.

[0687] The device also uses its built-in microphone to collect voice information and sends it to the server. The server translates the input voice into different languages ​​through its multilingual translation function and sends the result back to the device. Through this function, users can communicate smoothly with people from different cultural backgrounds.

[0688] Furthermore, users can use their devices to connect with multiple service platforms and make reservations and payments locally. This connectivity feature allows users to arrange services in advance and enjoy their trip without worry.

[0689] For example, if a user visits Kyoto, a popular tourist destination in Japan, and books a tea ceremony experience, they can use their device to send their current location and past interest history to a server. The server can then suggest recommended tea ceremony facilities. Based on this information, the user can complete the booking and payment through the service platform.

[0690] As an example of a prompt, entering "Please suggest a travel plan that supports sightseeing and a tea ceremony experience in Kyoto" will cause the AI ​​model to provide the optimal plan. This system allows users to enjoy travel experiences in different cultural spheres more safely and comfortably.

[0691] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0692] Step 1:

[0693] The user launches travel support software on their mobile communication terminal. The terminal displays a login screen and receives authentication information (e.g., user ID, password) from the user. This input is sent to the server as authentication data. The server verifies the authentication data and outputs whether authentication was successful or not. If authentication is successful, the user gains access to their unique travel data. The specific actions include pressing the login button and the subsequent display of an authentication success message.

[0694] Step 2:

[0695] The server receives the user's current location data and data indicating past usage history and interests as input. Based on this data, it collects relevant travel information from the database and generates an optimal travel plan. As a result, the generated travel plan is output and sent to the terminal. Specifically, it displays a list of travel plans on the screen and provides an interface for the user to select one.

[0696] Step 3:

[0697] The user selects a desired travel plan from those displayed on their device. This selection is sent from the device to the server, which retrieves detailed information related to that plan from external sources. As output, detailed information on tourist attractions, restaurants, and transportation is displayed on the user's device. Specifically, the system displays detailed information along with maps and photos and provides navigation functionality.

[0698] Step 4:

[0699] To facilitate communication with locals, users input voice using the device's built-in microphone. The device sends this voice data to a server. The server uses a multilingual translation function to translate the voice data into the specified language and send it back to the device. The output is provided to the user visually or audibly as translated audio or text. Specific actions include the translation results being scrolled as text or played back as audio.

[0700] Step 5:

[0701] Users access the service platform within the travel support software via their device to make reservations and payments at their destination. The server receives the relevant reservation data and payment information as input and processes it. The output is a reservation confirmation and payment completion notification, displayed on the device. Specific actions include receiving a reservation confirmation email and completing payment using a QR code.

[0702] Step 6:

[0703] In an emergency, the device's GPS function automatically determines the user's current location. This location information is sent to the server as input, and the server searches for the nearest emergency service information based on the location. The output displays the nearest emergency contact information and access methods on the device. Specifically, pressing the emergency contact button sends a phone call or message.

[0704] (Application Example 1)

[0705] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] When travelers are in a different cultural environment, they need to overcome challenges such as language barriers, communication difficulties due to cultural differences, difficulty accessing local food culture, and the need for quick responses in emergencies. In particular, ordering food locally can be confusing due to the language of the menu or differences in currency when paying. It is essential to overcome these problems and provide an environment where travelers can smoothly enjoy their local experiences.

[0707] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0708] In this invention, the server includes means for executing a travel support application that runs on a computer terminal, means for processing travel-related requests from users and obtaining information based on those requests from an external database, and means for taking pictures of menus and translating them into the user's native language through a translation function. This makes it possible for travelers to easily access local information and food culture, overcoming language barriers in different cultural spheres, and to enjoy a comfortable travel experience.

[0709] A "computer terminal" is an electronic device that performs information processing; it is a hardware device that can install software and run applications.

[0710] A "travel support application" is software designed to help travelers obtain local information and facilitate smooth travel and communication.

[0711] A "user" is someone who operates the travel support application and utilizes the service.

[0712] "Processing a request" is the process of receiving input from a user, analyzing its content, and providing the necessary information.

[0713] An "external database" is a collection of data that cannot be accessed directly but whose information can be obtained via a network.

[0714] "Taking a picture of a menu" refers to the action of using a camera to capture an image of a menu from a restaurant or similar establishment.

[0715] A "translation function" is a technology that converts input text or audio into another language.

[0716] "Cultural information" refers to data about traditions, customs, and social background in a particular region or country.

[0717] "Emergency services" refer to organizations and systems that provide the rapid support needed when users face a critical situation.

[0718] "Converting prices" refers to the process of converting the price of a product into a different currency unit and displaying it.

[0719] The system for implementing this invention consists of a computer terminal equipped with a travel support application and a server that works in the backend. The terminal provides various information according to the user's requests, and the server is responsible for data processing and communication with external databases.

[0720] The computer terminal includes hardware such as a camera, microphone, and GPS, which are used to collect and process information. Software includes image recognition APIs, translation APIs, and speech recognition engines. For example, a menu can be scanned using the Google Vision API and translated using the Google Translate API. Additionally, user voice input is converted to text using the Google Speech-to-Text API and then translated into multiple languages.

[0721] The server uses the user's past activity history and current location information to suggest appropriate travel plans and local cuisine. It also uses a fixed exchange rate to convert prices in different currencies in real time.

[0722] As a concrete example, there is a function that automatically translates the image and displays the price in USD when a user scans a menu using their smartphone camera at a restaurant overseas. In this process, users can enjoy their meal more efficiently by using prompts such as "Take a picture of the menu and translate it" and "Convert the price to dollars and display it."

[0723] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0724] Step 1:

[0725] When the device boots up, the user launches the travel assistance application. The application authenticates the user, connects to the server to retrieve the current location and past history. Based on this data, the available functions are displayed on the screen.

[0726] Step 2:

[0727] The user takes a picture of a menu at a local restaurant using their camera. The device then uses an image recognition API to convert the menu into text data. The input is a camera image, and the output is text information.

[0728] Step 3:

[0729] After the text information is generated, the device calls a translation API to translate the text data into the user's native language. In this process, the input is text information, and the output is the translated text.

[0730] Step 4:

[0731] Translated menu information is displayed on the screen, and the user makes a meal selection. Based on the user's selection, the terminal retrieves local dish recommendations from the server, taking into account past history and allergy information. The input is the selected menu item, and the output is the recommendation information.

[0732] Step 5:

[0733] The terminal checks the price of the dish selected by the user and uses a currency conversion function to convert the local currency to the user's home currency. The terminal retrieves exchange rate information from the server, with the input being the local price and the output being the converted amount.

[0734] Step 6:

[0735] The user places an order using voice commands. The terminal uses a speech recognition engine to convert the speech into text and a translation API to translate it into the target language. The input is voice data, and the output is the translated order text.

[0736] Step 7:

[0737] Once the user confirms their order, the terminal sends the order information to the restaurant via the server. The server then verifies the order and sends a confirmation message to the user. The input is the order text, and the output is the order confirmation.

[0738] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0739] This invention aims to make the travel experience more personalized and comfortable for users by using a travel support program and emotion engine that operate on a mobile communication terminal. The emotion engine aims to analyze the user's emotional state and provide appropriate information and services based on the results.

[0740] The user first launches a travel support program installed on their mobile communication terminal. The terminal completes the authentication process and connects to the server to receive individual travel data. When suggesting travel plans, the server considers the user's emotional state as a parameter. An emotion engine analyzes the user's emotions through their input (voice and text data) and sends that information to the server.

[0741] The server generates travel plans and activity suggestions that reflect emotional data and sends them back to the device. For example, if the analysis indicates that the user is feeling stressed, it will suggest relaxing places and activities. Conversely, if feelings of excitement or joy are recognized, it will recommend more active activities and tourist destinations.

[0742] Real-time communication features can also utilize emotion recognition. For example, if a user is feeling anxious, the emotion engine prioritizes providing reassuring translations and local information. Furthermore, explanations of cultural backgrounds based on emotion data are enhanced to help users avoid misunderstandings in interpersonal relationships.

[0743] This system enriches the travel experience by understanding the user's emotions at any given moment through interaction with the device and supporting their actions and choices. Furthermore, users can connect with other service platforms through this program and access essential services in various situations while abroad.

[0744] Thus, the present invention provides more comfortable and personalized travel support through a travel assistance program equipped with an emotion engine, realizing services that meet the user's needs and emotions.

[0745] The following describes the processing flow.

[0746] Step 1:

[0747] The user launches the travel support program on their mobile communication terminal. The terminal displays the user's login information on an input screen, and the user enters the necessary information.

[0748] Step 2:

[0749] The terminal sends the user's login information to the server for user authentication. The server verifies the authentication information and retrieves the user's personal profile from the database.

[0750] Step 3:

[0751] The server generates the optimal travel plan for the user based on their past travel history and current location. In addition, it utilizes the results of an emotion engine analysis to take into account the user's emotional information.

[0752] Step 4:

[0753] The device uses a microphone and camera to collect user voice and text input, and sends that data to a server and an emotion engine.

[0754] Step 5:

[0755] The emotion engine analyzes the received user input data and determines the user's current emotional state. This information is sent to the server as emotion parameters.

[0756] Step 6:

[0757] The server adjusts travel plans and suggestions based on emotional data received from the emotion engine. For example, if a user is feeling the need to relax, it will prioritize suggesting calm tourist destinations and relaxation activities.

[0758] Step 7:

[0759] The device visually presents users with customized travel plans and activity suggestions. Users can refer to these and select places they want to go and activities they want to participate in.

[0760] Step 8:

[0761] When users need to communicate locally, they can use the device's translation function to translate voice input into the local language. Furthermore, emotion-based translations and advice are provided.

[0762] Step 9:

[0763] Depending on the user's emotions and choices, the device will also interact with other service platforms to assist with processing related information, ticket reservations, payments, and other tasks.

[0764] Step 10:

[0765] In emergencies, the device will immediately provide necessary emergency service information based on the analysis results of the emotion engine and the user's current location, helping the user take swift action.

[0766] (Example 2)

[0767] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0768] Providing travelers with a travel experience tailored to their individual emotional states presents challenges, including the difficulty of facilitating smooth multilingual communication and providing insufficient information based on the cultural background of their destination. Furthermore, it is necessary to ensure that there are means to respond quickly and appropriately to emergencies during travel.

[0769] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0770] In this invention, the server includes means for executing a travel support application that operates on a mobile communication device, means for analyzing the user's emotional data using an emotion analysis engine and acquiring information based on the analysis results from an external data storage device, and means for proposing a travel plan tailored to the user based on the acquired information and the emotion analysis results. As a result, the user can enjoy a travel experience that matches their emotions, communicate smoothly in multiple languages, and respond appropriately to emergencies during travel.

[0771] A "mobile communication device" is an electronic device that a user can carry and that has communication capabilities.

[0772] A "travel support application" is software designed to support a user's travel activities.

[0773] An "emotion analysis engine" is a technology that analyzes user input data to identify their emotional state.

[0774] An "external data storage device" is a database that stores information such as the results of user sentiment analysis and allows it to be retrieved as needed.

[0775] A "travel plan" is a proposed plan that includes schedules and activities related to a trip.

[0776] A "multilingual translation function" is a technology that translates between different languages, enabling users to communicate with speakers of other languages.

[0777] A "service platform" is a foundational technology that provides various services related to travel and can be integrated with other systems.

[0778] "Cultural description information" refers to information that provides users with an overview and details about the culture and customs of their destination.

[0779] "Local emergency services" refer to locally provided support organizations and facilities that can be used in the event of an emergency while traveling.

[0780] This invention is a system for providing travelers with personalized travel experiences. Users run a travel support application using a mobile communication terminal. This application is software developed to support the user's travels.

[0781] The device is equipped with an emotion analysis engine that can analyze voice and text input from the user in real time to identify their emotional state. The emotion analysis engine utilizes natural language processing technology to analyze the user's emotional data in detail and transmit the results to an external data storage device.

[0782] The server generates an optimal travel plan for the user based on the received emotional data. The generated travel plan takes into account the user's emotions and preferences, and the suggested activities are tailored to the user's state of mind. For example, if the user wants to relax, it can suggest quiet tourist destinations and relaxation spots.

[0783] Furthermore, the device features multilingual translation capabilities, facilitating smooth communication when users speak foreign languages. This translation function not only ensures that cultural information and interactions with locals at travel destinations are understood, but also enhances cultural descriptions based on sentiment data. As a result, users can minimize cultural misunderstandings during their travels.

[0784] This system also integrates with other service platforms, allowing users to easily access various local and payment information. This enables a quick response to emergencies during travel.

[0785] For example, if a user enters "I want to relax, tell me a quiet place" into the device, the emotion analysis engine analyzes the user's desire to relax, and the server generates suggestions accordingly. As a result, the user can visit a suitable tourist destination.

[0786] An example of a prompt to a generative AI model might be, "Based on the user's sentiment data, suggest activities for this weekend's trip." This prompt allows the system to provide information optimized for the user.

[0787] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0788] Step 1:

[0789] The user launches a travel assistance application on their mobile communication terminal. The terminal first obtains authentication information from the user. Specifically, it receives a user ID and password, or biometric authentication data such as fingerprint or facial recognition, as input. The terminal uses this data to perform the authentication process and, if successful, establishes a connection with the server. The output is the result of whether the authentication was successful or unsuccessful.

[0790] Step 2:

[0791] After the terminal connects to the server, it requests individual travel data. The server uses the user's profile data and past travel history as input to generate travel information tailored to the user. This data processing based on the profile yields individual travel information as output, which is then sent to the terminal. The user can then view this information on the terminal's display.

[0792] Step 3:

[0793] The user inputs emotional information into the terminal in voice or text format. The terminal uses an emotion analysis engine to analyze this input data and identify the user's current emotional state. Specifically, natural language processing techniques are applied to analyze the meaning of the text and the tone of the voice. As a result, an output representing the user's emotional state is obtained and sent to the server.

[0794] Step 4:

[0795] The server generates a travel plan optimized for the user based on the emotion analysis results. The server receives emotion data and user profile data as input and processes the data according to the emotional state. As output, for example, suggestions for relaxing tourist spots and activities are generated and sent to the terminal.

[0796] Step 5:

[0797] The device displays travel plans and activity suggestions received from the server to the user. The user can review them and choose whether to include the suggested activities in their schedule. The user's selection is sent back to the server as feedback, and the plan is adjusted as needed.

[0798] Step 6:

[0799] Because a user's emotional state may change during their trip, the device continuously performs emotion analysis. It analyzes any new emotional input from the user and sends the results to the server. Based on this new input, the server updates information and services in real time and provides them to the device. This continuously optimizes the user's travel experience.

[0800] (Application Example 2)

[0801] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0802] In today's restaurant and food delivery industries, it is difficult to suggest optimal food choices that are tailored to the individual emotional state and mood of each customer. Furthermore, there is a lack of means to provide appropriate information based on the customer's emotional state and to create sophisticated experiences through collaboration with other services.

[0803] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0804] In this invention, the server includes means for executing a support program that operates on a mobile communication device, means for analyzing the user's emotional state using an emotion analysis engine and suggesting food based on the analysis results, and means for coordinating with multiple service platforms to provide local information and payment information. This makes it possible to provide appropriate food and services according to the user's emotions.

[0805] A "mobile communication device" refers to a terminal that is portable by the user and has communication capabilities. Typical examples include smartphones and tablet devices.

[0806] A "support program" refers to software that provides information and service suggestions based on user requests. This program runs on the device and supports the diverse needs of users.

[0807] An "emotion analysis engine" refers to a technology that analyzes a user's emotional state based on their input data. This engine accurately determines the user's emotions by utilizing voice, text, and other data.

[0808] "Suggesting food" refers to the process of selecting and providing the most suitable dishes and dining locations to the user based on their emotional state and other parameters.

[0809] A "service platform" refers to a foundation for integrating multiple different services and providing information and functionality to users. This includes payment services and information provision services.

[0810] "Local information" refers to data related to a specific geographical area. This includes information on tourist attractions, shops, and transportation.

[0811] "Payment information" refers to payment-related data necessary for conducting transactions. This includes the user's payment method and transaction amount.

[0812] The system that implements this application consists of a server program that works in conjunction with a mobile communication device. The mobile communication device has a support program installed and incorporates an emotion analysis engine that analyzes the user's emotional state. Specifically, it can use emotion analysis technologies such as the Google Cloud Natural Language API.

[0813] When a user enters information, the device's emotion analysis engine processes the data and analyzes their emotional state. The analysis results are sent to the server. The server receives the analyzed emotion data and begins integrating with multiple service platforms. For example, it uses APIs to retrieve necessary information from external databases such as local information and payment information. In this case, to suggest food products, it might be possible to integrate with data service APIs for the food service industry (e.g., the Uber Eats API).

[0814] The server suggests services and food items suitable for the user based on the acquired information and the results of sentiment analysis. The suggested information and options are sent back to the terminal and displayed to the user. As a result, the user can obtain an optimal experience that matches their emotional state.

[0815] For example, if a user enters "I'm tired today," the emotion analysis engine recognizes their stress level and suggests a relaxing meal. The suggested dishes might be something like "warm soup" or "Japanese food," and these are selected based on the user's emotional state.

[0816] An example of a prompt message would be: "User's emotion: stressed, type of food suggested: warm and relaxing, e.g., ramen or hot pot."

[0817] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0818] Step 1:

[0819] The terminal receives user input data. The user provides emotional information to the terminal via voice or text. This input consists of the user's voice messages or text messages. The input information is processed as text data by an emotion analysis engine.

[0820] Step 2:

[0821] The device uses an emotion analysis engine to analyze the user's emotional state from input data. A generative AI model is used for the analysis, specifically the Google Cloud Natural Language API. Text data is the target of the analysis, and an indicator of the emotional state is obtained as output. The analysis results are represented as integer values ​​or category labels, clearly indicating the user's current emotions.

[0822] Step 3:

[0823] The device sends the analysis results to the server. Based on the received emotional data, the server creates prompt messages to collect information relevant to the user. These prompt messages include recommendations based on the emotional state. For example, "User's emotion: stressed, desired type of food: warm and relaxing, e.g., ramen or hot pot."

[0824] Step 4:

[0825] The server uses prompts to call external APIs and retrieve the information needed for food recommendations. These APIs include a food delivery service API and a local information service API. The input is a request based on prompts, and the output is a list of foods corresponding to the user's emotional state. The server analyzes the responses from the APIs and extracts the appropriate data.

[0826] Step 5:

[0827] The server organizes suggestion information based on the data it has acquired and sends it to the terminal. After receiving the suggestion information, the terminal displays food and service options to the user. The user can choose what they want from the presented options, thereby providing an optimal experience tailored to their emotional state. The output is suggestions presented to the user visually or audibly.

[0828] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0829] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0830] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0831] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0832] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0833] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0834] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0835] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0836] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0837] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0838] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0839] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0840] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0841] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0842] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0843] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0844] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0845] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0846] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0847] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0848] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0849] The following is further disclosed regarding the embodiments described above.

[0850] (Claim 1)

[0851] A means of executing a travel support program that operates on a mobile communication terminal,

[0852] A means for processing travel requests from users and retrieving information based on those requests from an external database,

[0853] A means of suggesting a travel plan suitable for the user based on the acquired information,

[0854] A means of translating input audio using a multilingual translation function to support communication between the user and third parties,

[0855] A system that integrates with other service platforms and includes means for providing local information and payment information.

[0856] (Claim 2)

[0857] The system according to claim 1, which provides cultural information based on the user's native language and target culture.

[0858] (Claim 3)

[0859] The system according to claim 1, which provides a user with a convenient means of accessing local emergency services in an emergency.

[0860] "Example 1"

[0861] (Claim 1)

[0862] A means for executing travel assistance software that operates on a mobile communication device,

[0863] A means for processing travel-related requests from users and obtaining information based on those requests from external sources,

[0864] A means of generating the optimal travel plan for the user based on the acquired information,

[0865] A means of translating input speech in real time into different languages ​​to assist communication between the user and others,

[0866] By collaborating with other service platforms, a means of providing local knowledge and payment information,

[0867] A means of providing personalized travel plans based on the user's past behavioral history and interests,

[0868] A system that includes means for quickly and accurately determining the user's location using the device's positioning technology in an emergency, and providing information on the nearest emergency response service.

[0869] (Claim 2)

[0870] The system according to claim 1, which provides cultural information based on the user's native language and the culture in question.

[0871] (Claim 3)

[0872] The system according to claim 1, which provides users with an easy means of accessing local emergency services in an emergency.

[0873] "Application Example 1"

[0874] (Claim 1)

[0875] A means of running a travel support application on a computer terminal,

[0876] A means for processing travel-related requests from users and retrieving information based on those requests from an external database,

[0877] A means of proposing a travel plan suitable for the user based on the information obtained,

[0878] A means of supporting communication between the user and others by translating input audio using a multilingual translation function,

[0879] A means of providing local information and payment information by collaborating with other platforms,

[0880] A method for taking a picture of the menu and translating it into the user's native language using a translation function,

[0881] A method of suggesting local dishes based on past dining history,

[0882] A system that includes means for converting and displaying prices in a specified currency.

[0883] (Claim 2)

[0884] The system according to claim 1, which provides cultural information based on the user's language and target culture.

[0885] (Claim 3)

[0886] The system according to claim 1, which provides users with a convenient means of accessing local emergency services in an emergency.

[0887] "Example 2 of combining an emotion engine"

[0888] (Claim 1)

[0889] A means for running a travel support application that operates on a mobile communication device,

[0890] A means for analyzing user emotional data using an emotion analysis engine and obtaining information based on the analysis results from an external data storage device,

[0891] Based on the acquired information and sentiment analysis results, a means of proposing a travel plan tailored to the user,

[0892] A means of supporting communication between the user and others by converting input language sounds using a multilingual translation function,

[0893] A system that integrates with other service platforms and includes means for providing local information and payment information.

[0894] (Claim 2)

[0895] The system according to claim 1, which provides cultural descriptive information based on the user's native language and the culture of the destination.

[0896] (Claim 3)

[0897] The system according to claim 1, which provides a user with an easy means of accessing local emergency services in an emergency situation.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A means for executing a support program that operates on a mobile communication device,

[0901] A means of processing requests from users regarding cultural activities and obtaining information based on those requests from external sources,

[0902] A means of proposing an activity plan suitable for the user based on the acquired information,

[0903] A means of translating input natural language using a translation function to support dialogue between the user and others,

[0904] A means of providing local information and payment information by collaborating with multiple service platforms,

[0905] A system that includes a means of analyzing a user's emotional state using an emotion analysis engine and suggesting food products based on the analysis results.

[0906] (Claim 2)

[0907] The system according to claim 1, which provides cultural content based on the user's native language and the target culture.

[0908] (Claim 3)

[0909] The system according to claim 1, which provides users with an easy means of connecting to local emergency services in an emergency. [Explanation of Symbols]

[0910] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of executing a travel support program that operates on a mobile communication terminal, A means for processing travel requests from users and retrieving information based on those requests from an external database, A means of suggesting a travel plan suitable for the user based on the acquired information, A means of translating input audio using a multilingual translation function to support communication between the user and third parties, A system that integrates with other service platforms and includes means for providing local information and payment information.

2. The system according to claim 1, which provides cultural information based on the user's native language and target culture.

3. The system according to claim 1, which provides a user with a convenient means of accessing local emergency services in an emergency.

Images