system

Abstract

We provide the system. [Solution] Means of obtaining weather information for the area to be visited, A means of determining whether planned travel activities will be affected by weather based on acquired weather information, A means of generating alternative activity plans in case of bad weather, Means that allow travelers to choose alternative activities, A means of automatically arranging reservations for selected alternative activities, A means of generating and presenting route guidance to the location of the reserved alternative activity, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a foreign tourist visiting Japan has their travel plan suddenly cancelled due to bad weather during their trip, it is difficult to quickly find an alternative plan due to language barriers and lack of information. This problem not only wastes precious time during the trip but also reduces the satisfaction of the traveler. Therefore, it is required to enable travelers to have a fulfilling travel experience even on days with bad weather.

Means for Solving the Problems

[0005] This invention provides a system that acquires weather information for a planned travel area and determines whether planned activities will be affected by the weather based on that information. If bad weather is expected, it generates and makes available alternative indoor activity plans for travelers. Furthermore, it automatically arranges reservations for the selected alternative activities and provides route guidance to the activity locations, thereby eliminating obstacles caused by language barriers and lack of information. As a result, travelers can quickly restructure their plans without being aware of language barriers.

[0006] "Planned destination" refers to the geographical location or area that the traveler plans to visit.

[0007] "Weather information" refers to data such as temperature, precipitation, and wind speed that show the current weather conditions and forecasts.

[0008] "Planned travel activities" refer to specific activities or events that travelers plan to do during their trip.

[0009] "Bad weather" refers to unfavorable weather conditions that may disrupt planned travel activities.

[0010] An "alternative activity plan" refers to new activities or events proposed if the activities originally planned for the trip cannot be carried out.

[0011] "Making something selectable" refers to a state where a user can choose from multiple options presented to them according to their own will.

[0012] "Automatically arranging reservations" refers to a process where the system makes reservations and arrangements without requiring human intervention.

[0013] "Route guidance" refers to providing travelers with information on directions and transportation options to reach their destination. [Brief explanation of the drawing]

[0014] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system for implementing this invention consists of a travel management server and a user's terminal, enabling visitors to Japan to quickly respond to changes in their travel plans due to weather conditions.

[0036] First, the server retrieves weather data for the destination based on the traveler's planned visit information. Data can be obtained from weather APIs or public weather databases, and the information collected includes precipitation, wind speed, and temperature. Using this information, the server determines whether the planned travel activity will be affected by the weather.

[0037] If bad weather is expected, the server searches for alternative activities that might interest travelers, such as indoor leisure or cultural experiences. Based on user profiles and past preferences, it can provide optimized options. The device then presents these alternative plans to the user, and an automatic translation function ensures that the details are understood in the user's native language.

[0038] The user selects an activity of interest from the alternative plans presented. The server automatically checks the availability of the selected activity location and makes the reservation, so the user does not need to communicate in the local language.

[0039] Once the reservation is complete, the device displays confirmation information to the user. At this time, the device also provides detailed route guidance to the destination and quickly displays services including public transport timetables and directions.

[0040] For example, if a user is visiting Kyoto and a typhoon is approaching, making their planned temple visits difficult, this system can be used to quickly plan and execute alternative indoor activities such as a tea ceremony or a traditional pottery class. In this way, the present invention helps travelers enjoy a rich and diverse range of experiences even in bad weather.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The server retrieves the traveler's planned visit information and accesses weather APIs and public weather databases to collect future weather forecasts for the planned visit area. This data includes specific information such as temperature, precipitation, and wind speed.

[0044] Step 2:

[0045] The server analyzes the collected weather data to determine whether the planned travel activity will be affected by bad weather. The criteria for this determination may include, for example, whether the amount of rainfall exceeds a certain threshold.

[0046] Step 3:

[0047] If it is determined that bad weather is likely, the server searches for alternative activity plans based on the user's interests and past preferences. It extracts about 10 types of information on indoor leisure and cultural experiences from the database.

[0048] Step 4:

[0049] The terminal displays alternative plans received from the server to the user. The plans are presented in the user's native language through an automatic translation function, making them easier to understand.

[0050] Step 5:

[0051] Users select activities that interest them from the provided options. Detailed information can be viewed when making a selection.

[0052] Step 6:

[0053] The server initiates the reservation process based on the user's selection. The automated process completes the reservation on the Japanese website. This reservation information is then communicated to the user via their device.

[0054] Step 7:

[0055] The device generates and presents route guidance to the user for their booked alternative activity. This includes public transport schedules and route information to ensure the user can reach their destination smoothly.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] A challenge exists when visitors face unexpected weather changes during their trip, making it difficult to quickly and efficiently adjust their travel plans. In particular, when there are language and cultural barriers, it is not easy for travelers to choose and arrange new activities locally.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for acquiring environmental information of the area to be visited, means for determining whether the activities of the travel plan will be affected by the environment based on the acquired environmental information, and means for generating alternative cultural experiences. This enables travelers to flexibly respond to changes in weather, quickly select new cultural experiences, and automatically adjust their plans.

[0061] "Environmental information" refers to weather data and other information about natural conditions related to the area you plan to visit.

[0062] A "travel plan" is detailed planning information that includes the activities and itinerary that a traveler plans to do in the area they intend to visit.

[0063] "Affected" means that, based on environmental information, the activities in your travel plan become impossible to carry out or require modification.

[0064] "Alternative cultural experiences" are new indoor activity options offered when planned activities are affected by environmental factors.

[0065] "Automating arrangements" refers to automatically handling reservations and scheduling adjustments for alternative experiences selected by the user.

[0066] "Route information" refers to detailed guidance information regarding the means of transportation and directions necessary for users to reach their desired activity location.

[0067] "Language conversion functionality" is a technology that translates information into the user's native language to make it easier to understand.

[0068] To implement this invention, a system will be built to manage travelers' visit schedules and provide flexible travel plans tailored to environmental conditions. The system will primarily consist of servers and terminals, supporting users in making the most of their travel experience.

[0069] The server receives the user's travel plan and retrieves relevant environmental information based on it. This environmental information is accessible from weather APIs and publicly available weather databases and includes details such as wind speed, precipitation, and temperature. The server has scripts implemented using programming languages ​​such as Python and Java® to retrieve data from the API. If the server determines that the weather will affect the travel plan, it refers to the user's profile data and past preferences to select the most suitable alternative activity to provide an alternative cultural experience.

[0070] The terminal receives information from the server and displays it to the user. It has an automatic translation function that uses the Google® Translate API to translate information into the user's native language. As a result, users can understand the information smoothly without experiencing language barriers.

[0071] Users select activities of interest from alternative plans provided through their device. The server automatically arranges and completes the booking for the selected activities. This allows for convenient travel planning without the need for communication in the local language.

[0072] As a concrete example, when a user visits an area prone to natural disasters, the server detects weather data in real time and suggests changes to the plan. In this case, by inputting a prompt such as "Please suggest alternative activities in case of bad weather in Kyoto" into the AI ​​model, the system can quickly create a new plan.

[0073] This allows travelers to respond flexibly to unforeseen circumstances, ensuring their planned trips are safe and fulfilling.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The server receives user travel plan information as input and stores details such as destination and itinerary in a database. This process prepares the basic data for managing basic information about travel destinations. Specifically, it accesses reservation management systems and calendar applications via APIs.

[0077] Step 2:

[0078] The server sends a request to a weather API to obtain environmental information for the target area based on the stored travel plan information. The input includes geographical information of the travel destination. The data obtained is meteorological information such as wind speed, precipitation, and temperature. Specifically, a Python script is used to retrieve the information from the external API in JSON format.

[0079] Step 3:

[0080] The server analyzes the acquired environmental information to determine whether the travel plan will be affected. The input includes weather data, and the output is a determination of whether or not there is an impact. This step uses a conditional branching algorithm to process the data based on defined criteria.

[0081] Step 4:

[0082] If the server determines that weather will affect the plan, it inputs a prompt message into an AI model that generates alternative cultural experiences based on user profile data. The input uses the user's past preferences and profile information, and the output generates proposed alternative plans. A machine learning model is used to provide appropriate alternatives.

[0083] Step 5:

[0084] The terminal receives an alternative plan provided by the server and presents it to the user. The input here is the alternative plan, and the output is the user's visually displayed information. It has an automatic translation function, specifically using the Google Translate API to translate the information into the user's native language.

[0085] Step 6:

[0086] The user selects an activity of interest from the presented alternative plans and sends this information to the server via their device. The input includes user preference data, and the output is information about the selected activity. Based on this selection information, the next procedure begins.

[0087] Step 7:

[0088] The server checks the availability of the selected activity and automatically arranges the reservation. The input is the selected activity information, and the output is the reservation confirmation information. The specific operation includes checking availability and confirming the reservation using the reservation management API.

[0089] Step 8:

[0090] The terminal displays reservation confirmation information to the user and generates route guidance to the destination. Inputs here are reservation and geographical information, while output is route guidance information. Using the Google Maps API, detailed directions are provided using map data and public transport timetables.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] When tourists visiting Japan are forced to change their sightseeing plans due to weather changes during their trip, they often lack the ability to quickly find suitable alternatives, which can negatively impact the quality of their travel experience. Furthermore, tourists unfamiliar with the local language and culture face the challenge of having to expend considerable effort finding and booking alternative activities.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for acquiring weather information for the planned visit area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, means for generating an alternative activity plan in case of bad weather, means for enabling the traveler to select an alternative activity, means for automatically arranging a reservation for the selected alternative activity, means for generating and presenting route guidance to the location of the reserved alternative activity, and means for presenting alternative activity information in real time using a visual display device. This enables travelers to quickly find a suitable alternative activity even in bad weather and to make reservations and travel smoothly, overcoming language barriers.

[0096] "Weather information for the planned visit area" refers to data about the current weather conditions in the area that the traveler plans to visit.

[0097] "Providing real-time alternative activity information" means showing users information about activities currently available in a timely manner.

[0098] A "visual display device" is an electronic device used to convey information to users visually.

[0099] "Generating alternative activity plans" means proposing alternative activities when the originally planned activities are not feasible.

[0100] "Generating and presenting route guidance" means creating a detailed travel route to a destination and guiding the user visually or audibly.

[0101] The system implementing this invention consists of a server, a terminal, and a visual display device. The server obtains data from weather APIs and public weather databases to acquire weather information for the area to be visited. This data includes weather information such as precipitation, wind speed, and temperature. Based on the acquired weather information, the server executes an algorithm to determine whether the traveler's activities will be affected by the weather.

[0102] If bad weather is predicted, the server generates an alternative activity plan. This process utilizes the user's past preferences and profile data. The generated alternative activity information is presented to the user in real time via a visual display. The user can then select an activity that interests them from the presented options.

[0103] Subsequently, the server automatically makes a reservation for the alternative activity selected by the user. Once the reservation is complete, the terminal displays the reservation information to the user and generates and presents route guidance to the destination. Specifically, it uses the Google Maps API to provide travel routes and transportation schedules to the destination. At this time, the visual display device also enables real-time navigation to the location where the selected activity will take place.

[0104] For example, if a traveler cancels their planned temple visit in Kyoto due to an approaching typhoon, a reservation for a tea ceremony experience will be made instead. This information will be displayed on the user's smart glasses, clearly showing how to use public transportation and the estimated travel time.

[0105] An example of a prompt message would be: "There is a forecast of heavy rain in Tokyo in 3 hours, so please suggest changing from an outdoor activity to an indoor activity. Also, please assist with booking tickets for that indoor activity."

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The server obtains weather information for the planned visit area from weather APIs and public weather databases. The input is information about the planned visit area, and the output is weather data. This data acquisition involves using APIs to obtain precipitation, wind speed, temperature, and other data in real time.

[0109] Step 2:

[0110] The server determines, based on weather information, whether the planned travel activities will be affected by the weather. The input is the acquired weather data and travel plan information, and the output is the determination result indicating whether or not there will be an impact. This determination uses conditional branching logic to identify cases where bad weather is predicted.

[0111] Step 3:

[0112] When bad weather is expected, the server generates an alternative activity plan, taking into account the user profile and past preferences. The input is user profile information and current weather conditions, and the output is a list of recommended alternative activities. The generation process involves querying a database and using a recommendation algorithm.

[0113] Step 4:

[0114] The terminal presents alternative activity information to the user in real time via a visual display device. The input is a list of recommended alternative activities, and the output is the user's selection. The presentation is an action that allows the user to easily view the information through the user interface.

[0115] Step 5:

[0116] The server automatically makes reservations for the alternative activity selected by the user. The input is the user's selected alternative activity, and the output is reservation confirmation information. The reservation process includes integration with a reservation management system and automated processing.

[0117] Step 6:

[0118] The terminal displays reservation information and generates and presents route guidance for the user for the alternative activity for which the reservation has been completed. Inputs are reservation completion information and destination information, and output is the displayed navigation information. The presentation includes using the Google Maps API to generate the route and display the route map.

[0119] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0120] The system for implementing the present invention consists of a travel management server, a user terminal, and an emotion engine, enabling visitors to Japan to respond flexibly to changes in their travel plans due to weather conditions, in accordance with their emotional state.

[0121] First, the server collects weather data for the destination based on the traveler's planned visit information. Based on this data, it determines whether the planned travel activities will be affected by bad weather and prepares suggestions for alternative activities if necessary. In parallel, the device uses an emotion engine to analyze the user's emotional state based on the user's facial expressions, words, and device operation.

[0122] The server receives the results from the emotion engine and searches and selects the most appropriate alternative activity from the database based on the user's emotional state. In this process, it prioritizes presenting relaxation activities that can reduce stress, or entertainment plans that will make the user feel comfortable.

[0123] The device displays these alternative plans to the user and prompts them to make a selection. The plans feature automatic translation, making them easily understandable in the user's native language and providing detailed information about the options. Once the user selects an activity of interest, the server automates the booking process for the selected activity, completing it quickly.

[0124] Once selections and reservations are complete, the device will display guidance tailored to the user's needs to provide a more comfortable experience. Comprehensive support for public transport timetables and transportation options is also provided, allowing users to confidently move on to their next plan.

[0125] For example, if rain is expected while a user is visiting Osaka and it is determined that planned outdoor sightseeing is difficult, this system uses an emotion engine to analyze the user's emotions and proposes indoor sightseeing plans that can help the user change their mood, such as visiting an indoor art gallery or a relaxation spa. In this way, the present invention provides an optimal travel experience by taking into account the emotional state of the traveler.

[0126] The following describes the processing flow.

[0127] Step 1:

[0128] The server retrieves the traveler's planned visit information and uses weather APIs and public weather databases to collect future weather forecast data for the planned visit area. This includes collecting detailed weather information such as temperature, precipitation, and wind speed.

[0129] Step 2:

[0130] The server analyzes the collected weather data to determine whether planned travel activities will be affected by the weather. This analysis provides criteria for determining whether itinerary changes are necessary.

[0131] Step 3:

[0132] The device activates an emotion engine to monitor the user's emotional state. During this process, it uses sensors and microphones to analyze emotions based on facial expressions, voice, and operation patterns.

[0133] Step 4:

[0134] The server receives user emotion data obtained by the emotion engine and uses this information to search for alternative activities that are suitable for both the effects of the weather and the user's emotional state. It selects activities from the database that are suitable for stress reduction and mood improvement.

[0135] Step 5:

[0136] The device presents the user with alternative plans received from the server. To ensure user understanding, the details are displayed in the user's native language using an automatic translation function. Each plan's description and conditions are clearly stated, allowing the user to select activities that interest them.

[0137] Step 6:

[0138] Users select activities that interest them from the presented plans. This selection process is designed with an intuitive UI / UX.

[0139] Step 7:

[0140] The server automatically executes the necessary reservation procedures based on the user's selection. It processes reservations seamlessly even on Japanese reservation websites and provides feedback to the user's device.

[0141] Step 8:

[0142] The terminal displays reservation confirmation information to the user, generates route guidance to the selected alternative activity location, and displays detailed information. It provides reliable information, including transportation options and estimated travel time.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Travelers may find their travel plans disrupted by environmental conditions at their intended destination. In particular, when planned activities become difficult due to external conditions such as weather, it is not easy for travelers to find suitable alternative plans on the spot. Furthermore, if suggestions are made that disregard the traveler's emotional state, it becomes difficult to provide a satisfying travel experience. These issues pose a risk of decreased traveler satisfaction.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for acquiring environmental data of the planned destination; means for determining whether the planned activity will be affected by external conditions based on the acquired environmental data; means for generating an alternative plan if the external conditions are unfavorable; means for analyzing the emotional state using an information processing device and proposing activities appropriate to the user's state; means for enabling the user to select an alternative activity; means for automating the arrangement of the selected alternative activity; and means for generating and presenting route information to the booked activity. As a result, travelers can receive an optimal alternative plan that suits their emotional state even under the influence of external conditions, enabling a smooth and satisfying trip.

[0148] "Planned destinations" refer to places that travelers have planned to visit in advance, and are a concept that includes geographical locations that serve as a basis for obtaining information about those areas.

[0149] "Environmental data" refers to data about external conditions that may affect travelers' activities, such as weather information and traffic conditions related to their planned destination.

[0150] "External conditions" refer to external factors that may affect travelers' planning activities, such as weather, traffic, and natural disasters.

[0151] An "alternative plan" refers to new activities or schedules proposed to complement or replace travel plans affected by unforeseen external conditions.

[0152] An "information processing device" is a general term for devices and software used to analyze a user's emotional state, and in particular, it performs data analysis based on the user's facial expressions, voice, and operation history.

[0153] "Emotional state" refers to the user's current psychological or emotional state and is a factor that should be considered in order to improve the traveler's satisfaction and comfort.

[0154] "Means of automating activity arrangements" refers to a system where the system automatically handles booking and registration procedures for alternative activities selected by the user, without human intervention.

[0155] "Route information" includes detailed information about the route and means of transportation necessary to travel to the location where the booked activity will take place.

[0156] The embodiment of the invention consists of a system designed to facilitate travelers' planning. This system includes a server, a terminal, and an emotion analysis device. Each component plays a specific role and works together, sharing data with the others.

[0157] First, the server uses online databases to collect environmental data for a region based on information about the traveler's planned destinations. Specifically, weather information is obtained using a weather API. Next, the program analyzes the collected data to determine how external conditions will affect the traveler's plans. Based on this determination, it may generate alternative activity plans that are suitable for the external conditions.

[0158] The terminal uses an emotion analysis device to analyze the traveler's emotional state in real time. Sensors collect the user's facial expressions, voice tone, and device operation history, and the emotion engine performs the analysis. Based on this emotional data, the server selects the most appropriate alternative plan for the user's state and provides it to the terminal.

[0159] For example, if a user is planning a visit to Osaka Castle, and rain is forecast, the program will proactively suggest an indoor activity plan. This plan might include visits to relaxation facilities to improve the user's condition, or indoor art galleries at tourist attractions. In addition, the server automates the booking process for these activities and provides route information to the user via the terminal.

[0160] An example of a prompt in a generative AI model might be: "When the user needs to change their destination due to rain, suggest indoor activities while considering the user's emotional data." By using this prompt, the AI ​​generates optimal suggestions, providing the user with a more fulfilling travel experience.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The server receives information about the traveler's planned destinations as input. Specifically, the input data includes the name of the city and the itinerary of the trip. Based on this, the server uses an open weather data API to obtain weather information for the planned destinations. As output, it generates weather forecast data for the relevant region. The weather data includes temperature, probability of precipitation, wind speed, etc.

[0164] Step 2:

[0165] The server analyzes the acquired weather information to determine whether the user's planned activities will be affected by external conditions. The analysis uses a method that references weather information for the planned date and the user's planned activities from a database. As a result, it may output a judgment such as, "Outdoor activities are difficult to carry out due to the rain forecast for the planned date." Based on this judgment, the server proceeds with preparing to generate alternative plans.

[0166] Step 3:

[0167] The device receives the user's facial expressions, voice tone, and operation history as input, and uses an emotion engine to diagnose their emotional state. Specifically, it employs facial recognition technology using a camera and voice analysis technology using a microphone. Through this analysis, emotional data such as "the user's stress level is high" is generated. This emotional data is sent to a server and used as a basis for decisions in the next step.

[0168] Step 4:

[0169] The server uses a generative AI model to generate prompts that suggest alternative activities, taking into account emotional state data and weather conditions. An example of a prompt might be, "Please prioritize indoor activities to reduce stress." This prompt is input into the AI ​​model, and activity suggestions appropriate to the user's emotional state are generated as output.

[0170] Step 5:

[0171] The device displays generated alternative activity suggestions and prompts the user to make a selection. These options are displayed in the user's native language via an automatic translation function. For example, options might include "Visit an indoor garden" or "Use a relaxation facility." The user selects their desired activity through the device's operation, and then proceeds to the next step.

[0172] Step 6:

[0173] The server automates the booking process based on the alternative activity selected by the user. This process involves coordinating with stakeholders via an external booking system API to complete the booking. As a result of the process, a confirmation message stating "Activity booking complete" is output and sent to the terminal.

[0174] Step 7:

[0175] The device provides detailed route information to the selected alternative activity location along with the completed reservation information. Using map services such as the Google Maps API, it calculates and presents the optimal mode of transport and timetable from the origin to the destination. This allows the user to confidently move on to their next plan.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] In recent years, with the advancement of autonomous driving technology and the personalization of travel, there has been a growing need to appropriately respond to unexpected weather changes and itinerary alterations, and to provide entertainment activities that take into account the emotional state of travelers. However, conventional methods have faced challenges such as the deterioration of the quality of the trip due to bad weather and the difficulty in quickly providing content that adapts to the emotions of users.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes means for acquiring weather information for the planned destination area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, and means for generating alternative activity plans in case of bad weather, allowing the traveler to choose an alternative activity. This enables the traveler to respond quickly to unexpected weather changes, and further enables the analysis of the emotional state of the mobile vehicle user using an emotion analysis engine, and suggests entertainment activities available on the mobile vehicle based on the results. This makes it possible to improve the traveler's experience and increase their satisfaction with the trip.

[0181] "Means for obtaining weather information for a planned visit area" refers to a mechanism for collecting weather data related to the area a traveler plans to visit and incorporating that information into the system.

[0182] "Means for determining whether planned travel activities will be affected by weather based on acquired weather information" refers to a mechanism that analyzes collected weather data and uses that data to determine whether planned travel activities are feasible.

[0183] "Means for generating alternative activity plans in case of bad weather" refers to a mechanism for automatically creating a new activity plan to replace the original travel plan when weather conditions worsen.

[0184] "Means that enable travelers to choose alternative activities" refers to an interface that allows travelers to select from several alternative activity plans presented according to their preferences.

[0185] "Means for automatically arranging reservations for selected alternative activities" refers to a mechanism that automatically handles reservations and arrangements for alternative activities selected by travelers, enabling them to carry out these activities smoothly.

[0186] "Means for generating and presenting route guidance to the location of a booked alternative activity" refers to a mechanism for calculating the route to the location where the selected activity will take place and presenting it to the traveler visually or audibly.

[0187] "A means of analyzing the emotional state of mobile vehicle users using an emotion analysis engine" refers to a system designed to analyze the emotional state of users based on their facial expressions, behavioral data, and other factors.

[0188] "A means of suggesting entertainment activities available in a mobile vehicle based on analyzed emotional states" refers to a mechanism that, based on the results of emotional analysis, supports the selection of entertainment best suited to the user's mental state and suggests activities that can be enjoyed inside the vehicle.

[0189] This system has the function of obtaining weather information for the planned destination area and determining how the planned travel activities will be affected by the weather based on the obtained information. The server uses a weather information API to collect weather data and analyze whether the planned travel activities are feasible.

[0190] If bad weather is expected, the server generates an alternative activity plan. The smart devices used by travelers have an emotion analysis engine built in, which analyzes the user's current emotional state using facial expression data, voice tone, and device operation status as input. This process uses Python and emotion analysis libraries (e.g., OpenFace, DeepFace).

[0191] The next step is to suggest alternative activities best suited to the user's emotional state. Based on the analysis results, the server presents entertainment activities that can be performed on the in-car entertainment system (e.g., music playlists, relaxing videos, VR experiences, etc.). At this time, the entertainment activity options are generated in real time and presented to the user via an interface.

[0192] As a concrete example, suppose a self-driving vehicle used by a traveler is en route to a resort area when heavy snow is expected. Based on the heavy snow data, the server determines that outdoor activities will be difficult and uses an emotion analysis engine to identify that the traveler needs relaxation. Based on this, it suggests a VR tour of a hot spring inn that can be experienced inside the vehicle, as well as relaxation music.

[0193] Examples of prompts to input into a generative AI model are as follows:

[0194] "Based on current weather conditions and user sentiment data, please suggest the optimal in-car entertainment experience."

[0195] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0196] Step 1:

[0197] The server obtains weather information for the planned travel area via a weather information API. The input is information about the area the traveler plans to visit, and the output is detailed weather data for that area. Based on the obtained data, the server starts a process to determine whether the planned travel activities will be affected by the weather.

[0198] Step 2:

[0199] The terminal uses the traveler's smart device's camera and microphone to collect user facial and voice data. The input is real-time emotional data obtained through the device. Using emotion analysis libraries (OpenFace, DeepFace), this data is analyzed to identify the traveler's current emotional state. The output is data representing the user's emotional state.

[0200] Step 3:

[0201] The server searches a database for and selects alternative activities suitable for the traveler's emotional state, based on acquired weather data and sentiment analysis results. The inputs are weather data and sentiment data, and the output is the suggested alternative activities. This is processed by a generative AI model, which generates individual prompt statements and then formulates recommended activities based on them.

[0202] Step 4:

[0203] The terminal visually presents the selected alternative activity to the user and prompts them to make a choice. The input is the alternative activity data received from the server. The output is an option list displayed in the user interface. The user proceeds to the next step by selecting an activity of interest.

[0204] Step 5:

[0205] The server automates and completes the booking process for the alternative activity selected by the user. The input is the selected alternative activity, and the output is booking confirmation information. The process includes data communication with the booking system.

[0206] Step 6:

[0207] After a reservation is complete, the terminal displays entertainment activities available within the vehicle and provides further customized instructions based on sentiment analysis results. Inputs include information on content available in the vehicle and the user's sentiment data. Outputs include entertainment options and route guidance optimized for the user's current situation.

[0208] This series of steps creates a system that provides travelers with the best possible experience in response to changing weather conditions.

[0209] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0210] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0211] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0212] [Second Embodiment]

[0213] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0214] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0215] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0216] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0217] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0219] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0220] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0221] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0222] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0223] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0224] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0225] The system for implementing this invention consists of a travel management server and a user's terminal, enabling visitors to Japan to quickly respond to changes in their travel plans due to weather conditions.

[0226] First, the server retrieves weather data for the destination based on the traveler's planned visit information. Data can be obtained from weather APIs or public weather databases, and the information collected includes precipitation, wind speed, and temperature. Using this information, the server determines whether the planned travel activity will be affected by the weather.

[0227] If bad weather is expected, the server searches for alternative activities that might interest travelers, such as indoor leisure or cultural experiences. Based on user profiles and past preferences, it can provide optimized options. The device then presents these alternative plans to the user, and an automatic translation function ensures that the details are understood in the user's native language.

[0228] The user selects an activity of interest from the alternative plans presented. The server automatically checks the availability of the selected activity location and makes the reservation, so the user does not need to communicate in the local language.

[0229] Once the reservation is complete, the device displays confirmation information to the user. At this time, the device also provides detailed route guidance to the destination and quickly displays services including public transport timetables and directions.

[0230] For example, if a user is visiting Kyoto and a typhoon is approaching, making their planned temple visits difficult, this system can be used to quickly plan and execute alternative indoor activities such as a tea ceremony or a traditional pottery class. In this way, the present invention helps travelers enjoy a rich and diverse range of experiences even in bad weather.

[0231] The following describes the processing flow.

[0232] Step 1:

[0233] The server retrieves the traveler's planned visit information and accesses weather APIs and public weather databases to collect future weather forecasts for the planned visit area. This data includes specific information such as temperature, precipitation, and wind speed.

[0234] Step 2:

[0235] The server analyzes the collected weather data to determine whether the planned travel activity will be affected by bad weather. The criteria for this determination may include, for example, whether the amount of rainfall exceeds a certain threshold.

[0236] Step 3:

[0237] If it is determined that bad weather is likely, the server searches for alternative activity plans based on the user's interests and past preferences. It extracts about 10 types of information on indoor leisure and cultural experiences from the database.

[0238] Step 4:

[0239] The terminal displays alternative plans received from the server to the user. The plans are presented in the user's native language through an automatic translation function, making them easier to understand.

[0240] Step 5:

[0241] Users select activities that interest them from the provided options. Detailed information can be viewed when making a selection.

[0242] Step 6:

[0243] The server initiates the reservation process based on the user's selection. The automated process completes the reservation on the Japanese website. This reservation information is then communicated to the user via their device.

[0244] Step 7:

[0245] The device generates and presents route guidance to the user for their booked alternative activity. This includes public transport schedules and route information to ensure the user can reach their destination smoothly.

[0246] (Example 1)

[0247] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0248] A challenge exists when visitors face unexpected weather changes during their trip, making it difficult to quickly and efficiently adjust their travel plans. In particular, when there are language and cultural barriers, it is not easy for travelers to choose and arrange new activities locally.

[0249] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0250] In this invention, the server includes means for acquiring environmental information of the area to be visited, means for determining whether the activities of the travel plan will be affected by the environment based on the acquired environmental information, and means for generating alternative cultural experiences. This enables travelers to flexibly respond to changes in weather, quickly select new cultural experiences, and automatically adjust their plans.

[0251] "Environmental information" refers to weather data and other information about natural conditions related to the area you plan to visit.

[0252] A "travel plan" is detailed planning information that includes the activities and itinerary that a traveler plans to do in the area they intend to visit.

[0253] "Affected" means that, based on environmental information, the activities in your travel plan become impossible to carry out or require modification.

[0254] "Alternative cultural experiences" are new indoor activity options offered when planned activities are affected by environmental factors.

[0255] "Automating arrangements" refers to automatically handling reservations and scheduling adjustments for alternative experiences selected by the user.

[0256] "Route information" refers to detailed guidance information regarding the means of transportation and directions necessary for users to reach their desired activity location.

[0257] "Language conversion functionality" is a technology that translates information into the user's native language to make it easier to understand.

[0258] To implement this invention, a system will be built to manage travelers' visit schedules and provide flexible travel plans tailored to environmental conditions. The system will primarily consist of servers and terminals, supporting users in making the most of their travel experience.

[0259] The server receives the user's travel plan and retrieves relevant environmental information based on it. This environmental information is accessible from weather APIs and publicly available weather databases and includes details such as wind speed, precipitation, and temperature. The server has scripts implemented using programming languages ​​such as Python and Java to retrieve data from the API. If it is determined that the weather will affect the travel plan, the server refers to the user's profile data and past preferences to select the most suitable alternative activity in order to provide an alternative cultural experience.

[0260] The terminal receives information from the server and displays it to the user. It has an automatic translation function that uses the Google Translate API to translate information into the user's native language. As a result, users can understand the information smoothly without experiencing language barriers.

[0261] Users select activities of interest from alternative plans provided through their device. The server automatically arranges and completes the booking for the selected activities. This allows for convenient travel planning without the need for communication in the local language.

[0262] As a concrete example, when a user visits an area prone to natural disasters, the server detects weather data in real time and suggests changes to the plan. In this case, by inputting a prompt such as "Please suggest alternative activities in case of bad weather in Kyoto" into the AI ​​model, the system can quickly create a new plan.

[0263] This allows travelers to respond flexibly to unforeseen circumstances, ensuring their planned trips are safe and fulfilling.

[0264] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0265] Step 1:

[0266] The server receives user travel plan information as input and stores details such as destination and itinerary in a database. This process prepares the basic data for managing basic information about travel destinations. Specifically, it accesses reservation management systems and calendar applications via APIs.

[0267] Step 2:

[0268] The server sends a request to a weather API to obtain environmental information for the target area based on the stored travel plan information. The input includes geographical information of the travel destination. The data obtained is meteorological information such as wind speed, precipitation, and temperature. Specifically, a Python script is used to retrieve the information from the external API in JSON format.

[0269] Step 3:

[0270] The server analyzes the acquired environmental information to determine whether the travel plan will be affected. The input includes weather data, and the output is a determination of whether or not there is an impact. This step uses a conditional branching algorithm to process the data based on defined criteria.

[0271] Step 4:

[0272] If the server determines that weather will affect the plan, it inputs a prompt message into an AI model that generates alternative cultural experiences based on user profile data. The input uses the user's past preferences and profile information, and the output generates proposed alternative plans. A machine learning model is used to provide appropriate alternatives.

[0273] Step 5:

[0274] The terminal receives an alternative plan provided by the server and presents it to the user. The input here is the alternative plan, and the output is the user's visually displayed information. It has an automatic translation function, specifically using the Google Translate API to translate the information into the user's native language.

[0275] Step 6:

[0276] The user selects an activity of interest from the presented alternative plans and sends this information to the server via their device. The input includes user preference data, and the output is information about the selected activity. Based on this selection information, the next procedure begins.

[0277] Step 7:

[0278] The server checks the availability of the selected activity and automatically arranges the reservation. The input is the selected activity information, and the output is the reservation confirmation information. The specific operation includes checking availability and confirming the reservation using the reservation management API.

[0279] Step 8:

[0280] The terminal displays reservation confirmation information to the user and generates a route guide to the destination. The input here is reservation and geographical information, and the output is route guide information. Using the Google Maps API, detailed guidance is provided using map data and transportation schedules.

[0281] (Application Example 1)

[0282] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0283] When a traveler visiting Japan has to change their sightseeing plan due to a change in weather during the trip, there is a problem that they cannot obtain a quick and appropriate alternative plan, resulting in a decline in the quality of the travel experience. In addition, there is a problem that travelers who are not familiar with the local language and culture require a lot of effort in the process of finding and reserving alternative activities.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server includes means for acquiring weather information of the planned visit area, means for determining whether the activities planned for the trip will be affected by the weather based on the acquired weather information, means for generating an alternative activity plan in case of bad weather, means for enabling the traveler to select an alternative activity, means for automatically arranging the reservation of the selected alternative activity, means for generating and presenting a route guide to the location of the reserved alternative activity, and means for presenting alternative activity information in real time using a visual display device. As a result, even in bad weather, travelers can quickly find appropriate alternative activities, overcome the language barrier, and smoothly make reservations and move.

[0286] The "weather information of the planned visit area" is data on the current weather conditions in the area where the traveler plans to visit.

[0287] "Providing real-time alternative activity information" means showing users information about activities currently available in a timely manner.

[0288] A "visual display device" is an electronic device used to convey information to users visually.

[0289] "Generating alternative activity plans" means proposing alternative activities when the originally planned activities are not feasible.

[0290] "Generating and presenting route guidance" means creating a detailed travel route to a destination and guiding the user visually or audibly.

[0291] The system implementing this invention consists of a server, a terminal, and a visual display device. The server obtains data from weather APIs and public weather databases to acquire weather information for the area to be visited. This data includes weather information such as precipitation, wind speed, and temperature. Based on the acquired weather information, the server executes an algorithm to determine whether the traveler's activities will be affected by the weather.

[0292] If bad weather is predicted, the server generates an alternative activity plan. This process utilizes the user's past preferences and profile data. The generated alternative activity information is presented to the user in real time via a visual display. The user can then select an activity that interests them from the presented options.

[0293] Subsequently, the server automatically makes a reservation for the alternative activity selected by the user. Once the reservation is complete, the terminal displays the reservation information to the user and generates and presents route guidance to the destination. Specifically, it uses the Google Maps API to provide travel routes and transportation schedules to the destination. At this time, the visual display device also enables real-time navigation to the location where the selected activity will take place.

[0294] For example, if a traveler cancels their planned temple visit in Kyoto due to an approaching typhoon, a reservation for a tea ceremony experience will be made instead. This information will be displayed on the user's smart glasses, clearly showing how to use public transportation and the estimated travel time.

[0295] An example of a prompt message would be: "There is a forecast of heavy rain in Tokyo in 3 hours, so please suggest changing from an outdoor activity to an indoor activity. Also, please assist with booking tickets for that indoor activity."

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The server obtains weather information for the planned visit area from weather APIs and public weather databases. The input is information about the planned visit area, and the output is weather data. This data acquisition involves using APIs to obtain precipitation, wind speed, temperature, and other data in real time.

[0299] Step 2:

[0300] The server determines, based on weather information, whether the planned travel activities will be affected by the weather. The input is the acquired weather data and travel plan information, and the output is the determination result indicating whether or not there will be an impact. This determination uses conditional branching logic to identify cases where bad weather is predicted.

[0301] Step 3:

[0302] When bad weather is expected, the server generates an alternative activity plan, taking into account the user profile and past preferences. The input is user profile information and current weather conditions, and the output is a list of recommended alternative activities. The generation process involves querying a database and using a recommendation algorithm.

[0303] Step 4:

[0304] The terminal presents alternative activity information to the user in real time through a visual display device. The input is a list of recommended alternative activities, and the output is the user's selection. The presentation is an operation that enables the user to easily view information through the user interface.

[0305] Step 5:

[0306] The server automatically makes a reservation arrangement for the alternative activity selected by the user. The input is the alternative activity selected by the user, and the output is reservation completion information. The reservation includes operations for linking to and automatically processing the reservation management system.

[0307] Step 6:

[0308] The terminal displays reservation information to the user for the alternative activity for which the reservation has been completed, generates and presents a route guide. The input is the reservation completion information and the destination information, and the output is the displayed navigation information. The presentation includes operations for generating a route and displaying a route map using the Google Maps API.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0310] The system for implementing the present invention is composed of a travel management server, a terminal owned by the user, and an emotion engine, and enables flexible responses according to the emotional state to changes in travel plans due to the weather for travelers visiting Japan.

[0311] First, the server collects weather data for the destination based on the traveler's planned visit information. Based on this data, it determines whether the planned travel activities will be affected by bad weather and prepares suggestions for alternative activities if necessary. In parallel, the device uses an emotion engine to analyze the user's emotional state based on the user's facial expressions, words, and device operation.

[0312] The server receives the results from the emotion engine and searches and selects the most appropriate alternative activity from the database based on the user's emotional state. In this process, it prioritizes presenting relaxation activities that can reduce stress, or entertainment plans that will make the user feel comfortable.

[0313] The device displays these alternative plans to the user and prompts them to make a selection. The plans feature automatic translation, making them easily understandable in the user's native language and providing detailed information about the options. Once the user selects an activity of interest, the server automates the booking process for the selected activity, completing it quickly.

[0314] Once selections and reservations are complete, the device will display guidance tailored to the user's needs to provide a more comfortable experience. Comprehensive support for public transport timetables and transportation options is also provided, allowing users to confidently move on to their next plan.

[0315] For example, if rain is expected while a user is visiting Osaka and it is determined that planned outdoor sightseeing is difficult, this system uses an emotion engine to analyze the user's emotions and proposes indoor sightseeing plans that can help the user change their mood, such as visiting an indoor art gallery or a relaxation spa. In this way, the present invention provides an optimal travel experience by taking into account the emotional state of the traveler.

[0316] The following describes the processing flow.

[0317] Step 1:

[0318] The server retrieves the traveler's planned visit information and uses weather APIs and public weather databases to collect future weather forecast data for the planned visit area. This includes collecting detailed weather information such as temperature, precipitation, and wind speed.

[0319] Step 2:

[0320] The server analyzes the collected weather data to determine whether planned travel activities will be affected by the weather. This analysis provides criteria for determining whether itinerary changes are necessary.

[0321] Step 3:

[0322] The device activates an emotion engine to monitor the user's emotional state. During this process, it uses sensors and microphones to analyze emotions based on facial expressions, voice, and operation patterns.

[0323] Step 4:

[0324] The server receives user emotion data obtained by the emotion engine and uses this information to search for alternative activities that are suitable for both the effects of the weather and the user's emotional state. It selects activities from the database that are suitable for stress reduction and mood improvement.

[0325] Step 5:

[0326] The device presents the user with alternative plans received from the server. To ensure user understanding, the details are displayed in the user's native language using an automatic translation function. Each plan's description and conditions are clearly stated, allowing the user to select activities that interest them.

[0327] Step 6:

[0328] Users select activities that interest them from the presented plans. This selection process is designed with an intuitive UI / UX.

[0329] Step 7:

[0330] The server automatically executes the necessary reservation procedures based on the user's selection. It processes reservations seamlessly even on Japanese reservation websites and provides feedback to the user's device.

[0331] Step 8:

[0332] The terminal displays reservation confirmation information to the user, generates route guidance to the selected alternative activity location, and displays detailed information. It provides reliable information, including transportation options and estimated travel time.

[0333] (Example 2)

[0334] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0335] Travelers may find their travel plans disrupted by environmental conditions at their intended destination. In particular, when planned activities become difficult due to external conditions such as weather, it is not easy for travelers to find suitable alternative plans on the spot. Furthermore, if suggestions are made that disregard the traveler's emotional state, it becomes difficult to provide a satisfying travel experience. These issues pose a risk of decreased traveler satisfaction.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0337] In this invention, the server includes means for acquiring environmental data of the planned destination; means for determining whether the planned activity will be affected by external conditions based on the acquired environmental data; means for generating an alternative plan if the external conditions are unfavorable; means for analyzing the emotional state using an information processing device and proposing activities appropriate to the user's state; means for enabling the user to select an alternative activity; means for automating the arrangement of the selected alternative activity; and means for generating and presenting route information to the booked activity. As a result, travelers can receive an optimal alternative plan that suits their emotional state even under the influence of external conditions, enabling a smooth and satisfying trip.

[0338] "Planned destinations" refer to places that travelers have planned to visit in advance, and are a concept that includes geographical locations that serve as a basis for obtaining information about those areas.

[0339] "Environmental data" refers to data about external conditions that may affect travelers' activities, such as weather information and traffic conditions related to their planned destination.

[0340] "External conditions" refer to external factors that may affect travelers' planning activities, such as weather, traffic, and natural disasters.

[0341] An "alternative plan" refers to new activities or schedules proposed to complement or replace travel plans affected by unforeseen external conditions.

[0342] An "information processing device" is a general term for devices and software used to analyze a user's emotional state, and in particular, it performs data analysis based on the user's facial expressions, voice, and operation history.

[0343] "Emotional state" refers to the user's current psychological or emotional state and is a factor that should be considered in order to improve the traveler's satisfaction and comfort.

[0344] "Means of automating activity arrangements" refers to a system where the system automatically handles booking and registration procedures for alternative activities selected by the user, without human intervention.

[0345] "Route information" includes detailed information about the route and means of transportation necessary to travel to the location where the booked activity will take place.

[0346] The embodiment of the invention consists of a system designed to facilitate travelers' planning. This system includes a server, a terminal, and an emotion analysis device. Each component plays a specific role and works together, sharing data with the others.

[0347] First, the server uses online databases to collect environmental data for a region based on information about the traveler's planned destinations. Specifically, weather information is obtained using a weather API. Next, the program analyzes the collected data to determine how external conditions will affect the traveler's plans. Based on this determination, it may generate alternative activity plans that are suitable for the external conditions.

[0348] The terminal uses an emotion analysis device to analyze the traveler's emotional state in real time. Sensors collect the user's facial expressions, voice tone, and device operation history, and the emotion engine performs the analysis. Based on this emotional data, the server selects the most appropriate alternative plan for the user's state and provides it to the terminal.

[0349] For example, if a user is planning a visit to Osaka Castle, and rain is forecast, the program will proactively suggest an indoor activity plan. This plan might include visits to relaxation facilities to improve the user's condition, or indoor art galleries at tourist attractions. In addition, the server automates the booking process for these activities and provides route information to the user via the terminal.

[0350] An example of a prompt in a generative AI model might be: "When the user needs to change their destination due to rain, suggest indoor activities while considering the user's emotional data." By using this prompt, the AI ​​generates optimal suggestions, providing the user with a more fulfilling travel experience.

[0351] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0352] Step 1:

[0353] The server receives information about the traveler's planned destinations as input. Specifically, the input data includes the name of the city and the itinerary of the trip. Based on this, the server uses an open weather data API to obtain weather information for the planned destinations. As output, it generates weather forecast data for the relevant region. The weather data includes temperature, probability of precipitation, wind speed, etc.

[0354] Step 2:

[0355] The server analyzes the acquired weather information to determine whether the user's planned activities will be affected by external conditions. The analysis uses a method that references weather information for the planned date and the user's planned activities from a database. As a result, it may output a judgment such as, "Outdoor activities are difficult to carry out due to the rain forecast for the planned date." Based on this judgment, the server proceeds with preparing to generate alternative plans.

[0356] Step 3:

[0357] The device receives the user's facial expressions, voice tone, and operation history as input, and uses an emotion engine to diagnose their emotional state. Specifically, it employs facial recognition technology using a camera and voice analysis technology using a microphone. Through this analysis, emotional data such as "the user's stress level is high" is generated. This emotional data is sent to a server and used as a basis for decisions in the next step.

[0358] Step 4:

[0359] The server uses a generative AI model to generate prompts that suggest alternative activities, taking into account emotional state data and weather conditions. An example of a prompt might be, "Please prioritize indoor activities to reduce stress." This prompt is input into the AI ​​model, and activity suggestions appropriate to the user's emotional state are generated as output.

[0360] Step 5:

[0361] The device displays generated alternative activity suggestions and prompts the user to make a selection. These options are displayed in the user's native language via an automatic translation function. For example, options might include "Visit an indoor garden" or "Use a relaxation facility." The user selects their desired activity through the device's operation, and then proceeds to the next step.

[0362] Step 6:

[0363] The server automates the booking process based on the alternative activity selected by the user. This process involves coordinating with stakeholders via an external booking system API to complete the booking. As a result of the process, a confirmation message stating "Activity booking complete" is output and sent to the terminal.

[0364] Step 7:

[0365] The device provides detailed route information to the selected alternative activity location along with the completed reservation information. Using map services such as the Google Maps API, it calculates and presents the optimal mode of transport and timetable from the origin to the destination. This allows the user to confidently move on to their next plan.

[0366] (Application Example 2)

[0367] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0368] In recent years, with the advancement of autonomous driving technology and the personalization of travel, there has been a growing need to appropriately respond to unexpected weather changes and itinerary alterations, and to provide entertainment activities that take into account the emotional state of travelers. However, conventional methods have faced challenges such as the deterioration of the quality of the trip due to bad weather and the difficulty in quickly providing content that adapts to the emotions of users.

[0369] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0370] In this invention, the server includes means for acquiring weather information for the planned destination area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, and means for generating alternative activity plans in case of bad weather, allowing the traveler to choose an alternative activity. This enables the traveler to respond quickly to unexpected weather changes, and further enables the analysis of the emotional state of the mobile vehicle user using an emotion analysis engine, and suggests entertainment activities available on the mobile vehicle based on the results. This makes it possible to improve the traveler's experience and increase their satisfaction with the trip.

[0371] "Means for obtaining weather information for a planned visit area" refers to a mechanism for collecting weather data related to the area a traveler plans to visit and incorporating that information into the system.

[0372] "Means for determining whether planned travel activities will be affected by weather based on acquired weather information" refers to a mechanism that analyzes collected weather data and uses that data to determine whether planned travel activities are feasible.

[0373] "Means for generating alternative activity plans in case of bad weather" refers to a mechanism for automatically creating a new activity plan to replace the original travel plan when weather deteriorates.

[0374] "Means that enable travelers to choose alternative activities" refers to an interface that allows travelers to select from several alternative activity plans presented according to their preferences.

[0375] "Means for automatically arranging reservations for selected alternative activities" refers to a mechanism that automatically handles reservations and arrangements for alternative activities selected by travelers, enabling them to carry out these activities smoothly.

[0376] "Means for generating and presenting route guidance to the location of a booked alternative activity" refers to a mechanism for calculating the route to the location where the selected activity will take place and presenting it to the traveler visually or audibly.

[0377] "A means of analyzing the emotional state of mobile vehicle users using an emotion analysis engine" refers to a system designed to analyze the emotional state of users based on their facial expressions, behavioral data, and other factors.

[0378] "A means of suggesting entertainment activities available in a mobile vehicle based on analyzed emotional states" refers to a mechanism that, based on the results of emotional analysis, supports the selection of entertainment best suited to the user's mental state and suggests activities that can be enjoyed inside the vehicle.

[0379] This system has the function of obtaining weather information for the planned destination area and determining how the planned travel activities will be affected by the weather based on the obtained information. The server uses a weather information API to collect weather data and analyze whether the planned travel activities are feasible.

[0380] If bad weather is expected, the server generates an alternative activity plan. The smart devices used by travelers have an emotion analysis engine built in, which analyzes the user's current emotional state using facial expression data, voice tone, and device operation status as input. This process uses Python and emotion analysis libraries (e.g., OpenFace, DeepFace).

[0381] The next step is to suggest alternative activities best suited to the user's emotional state. Based on the analysis results, the server presents entertainment activities that can be performed on the in-car entertainment system (e.g., music playlists, relaxing videos, VR experiences, etc.). At this time, the entertainment activity options are generated in real time and presented to the user via an interface.

[0382] As a concrete example, suppose a self-driving vehicle used by a traveler is en route to a resort area when heavy snow is expected. Based on the heavy snow data, the server determines that outdoor activities will be difficult and uses an emotion analysis engine to identify that the traveler needs relaxation. Based on this, it suggests a VR tour of a hot spring inn that can be experienced inside the vehicle, as well as relaxation music.

[0383] Examples of prompts to input into a generative AI model are as follows:

[0384] "Based on current weather conditions and user sentiment data, please suggest the optimal in-car entertainment experience."

[0385] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0386] Step 1:

[0387] The server obtains weather information for the planned visit area via a weather information API. The input is information about the area the traveler plans to visit, and the output is detailed weather data for that area. Based on the obtained data, the server starts a process to determine whether the planned travel activities will be affected by the weather.

[0388] Step 2:

[0389] The terminal uses the traveler's smart device's camera and microphone to collect user facial and voice data. The input is real-time emotional data obtained through the device. Using emotion analysis libraries (OpenFace, DeepFace), this data is analyzed to identify the traveler's current emotional state. The output is data representing the user's emotional state.

[0390] Step 3:

[0391] The server searches a database for and selects alternative activities suitable for the traveler's emotional state, based on acquired weather data and sentiment analysis results. The inputs are weather data and sentiment data, and the output is the suggested alternative activities. This is processed by a generative AI model, which generates individual prompt statements and then formulates recommended activities based on them.

[0392] Step 4:

[0393] The terminal visually presents the selected alternative activity to the user and prompts them to make a choice. The input is alternative activity data received from the server. The output is an option list displayed in the user interface. The user proceeds to the next step by selecting an activity of interest.

[0394] Step 5:

[0395] The server automates and completes the booking process for the alternative activity selected by the user. The input is the selected alternative activity, and the output is booking confirmation information. The process includes data communication with the booking system.

[0396] Step 6:

[0397] After a reservation is complete, the terminal displays entertainment activities available within the vehicle and provides further customized instructions based on sentiment analysis results. Inputs include information on content available in the vehicle and the user's sentiment data. Outputs include entertainment options and route guidance optimized for the user's current situation.

[0398] This series of steps creates a system that provides travelers with the best possible experience in response to changing weather conditions.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0400] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0402] [Third Embodiment]

[0403] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0404] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0405] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0407] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0409] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0410] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0411] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0414] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0415] The system for implementing this invention consists of a travel management server and a user's terminal, enabling visitors to Japan to quickly respond to changes in their travel plans due to weather conditions.

[0416] First, the server retrieves weather data for the destination based on the traveler's planned visit information. Data can be obtained from weather APIs or public weather databases, and the information collected includes precipitation, wind speed, and temperature. Using this information, the server determines whether the planned travel activity will be affected by the weather.

[0417] If bad weather is expected, the server searches for alternative activities that might interest travelers, such as indoor leisure or cultural experiences. Based on user profiles and past preferences, it can provide optimized options. The device then presents these alternative plans to the user, and an automatic translation function ensures that the details are understood in the user's native language.

[0418] The user selects an activity of interest from the alternative plans presented. The server automatically checks the availability of the selected activity location and makes the reservation, so the user does not need to communicate in the local language.

[0419] Once the reservation is complete, the device displays confirmation information to the user. At this time, the device also provides detailed route guidance to the destination and quickly displays services including public transport timetables and directions.

[0420] For example, if a user is visiting Kyoto and a typhoon is approaching, making their planned temple visits difficult, this system can be used to quickly plan and execute alternative indoor activities such as a tea ceremony or a traditional pottery class. In this way, the present invention helps travelers enjoy a rich and diverse range of experiences even in bad weather.

[0421] The following describes the processing flow.

[0422] Step 1:

[0423] The server retrieves the traveler's planned visit information and accesses weather APIs and public weather databases to collect future weather forecasts for the planned visit area. This data includes specific information such as temperature, precipitation, and wind speed.

[0424] Step 2:

[0425] The server analyzes the collected weather data to determine whether the planned travel activity will be affected by bad weather. The criteria for this determination may include, for example, whether the amount of rainfall exceeds a certain threshold.

[0426] Step 3:

[0427] If it is determined that bad weather is likely, the server searches for alternative activity plans based on the user's interests and past preferences. It extracts about 10 types of information on indoor leisure and cultural experiences from the database.

[0428] Step 4:

[0429] The terminal displays alternative plans received from the server to the user. The plans are presented in the user's native language through an automatic translation function, making them easier to understand.

[0430] Step 5:

[0431] Users select activities that interest them from the provided options. Detailed information can be viewed when making a selection.

[0432] Step 6:

[0433] The server initiates the reservation process based on the user's selection. The automated process completes the reservation on the Japanese website. This reservation information is then communicated to the user via their device.

[0434] Step 7:

[0435] The device generates and presents route guidance to the user for their booked alternative activity. This includes public transport schedules and route information to ensure the user can reach their destination smoothly.

[0436] (Example 1)

[0437] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0438] A challenge exists when visitors face unexpected weather changes during their trip, making it difficult to quickly and efficiently adjust their travel plans. In particular, when there are language and cultural barriers, it is not easy for travelers to choose and arrange new activities locally.

[0439] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0440] In this invention, the server includes means for acquiring environmental information of the area to be visited, means for determining whether the activities of the travel plan will be affected by the environment based on the acquired environmental information, and means for generating alternative cultural experiences. This enables travelers to flexibly respond to changes in weather, quickly select new cultural experiences, and automatically adjust their plans.

[0441] "Environmental information" refers to weather data and other information about natural conditions related to the area you plan to visit.

[0442] A "travel plan" is detailed planning information that includes the activities and itinerary that a traveler plans to do in the area they intend to visit.

[0443] "Affected" means that, based on environmental information, the activities in your travel plan become impossible to carry out or require modification.

[0444] "Alternative cultural experiences" are new indoor activity options offered when planned activities are affected by environmental factors.

[0445] "Automating arrangements" refers to automatically handling reservations and scheduling adjustments for alternative experiences selected by the user.

[0446] "Route information" refers to detailed guidance information regarding the means of transportation and directions necessary for users to reach their desired activity location.

[0447] "Language conversion functionality" is a technology that translates information into the user's native language to make it easier to understand.

[0448] To implement this invention, a system will be built to manage travelers' visit schedules and provide flexible travel plans tailored to environmental conditions. The system will primarily consist of servers and terminals, supporting users in making the most of their travel experience.

[0449] The server receives the user's travel plan and retrieves relevant environmental information based on it. This environmental information is accessible from weather APIs and publicly available weather databases and includes details such as wind speed, precipitation, and temperature. The server has scripts implemented using programming languages ​​such as Python and Java to retrieve data from the API. If it is determined that the weather will affect the travel plan, the server refers to the user's profile data and past preferences to select the most suitable alternative activity in order to provide an alternative cultural experience.

[0450] The terminal receives information from the server and displays it to the user. It has an automatic translation function that uses the Google Translate API to translate information into the user's native language. As a result, users can understand the information smoothly without experiencing language barriers.

[0451] Users select activities of interest from alternative plans provided through their device. The server automatically arranges and completes the booking for the selected activities. This allows for convenient travel planning without the need for communication in the local language.

[0452] As a concrete example, when a user visits an area prone to natural disasters, the server detects weather data in real time and suggests changes to the plan. In this case, by inputting a prompt such as "Please suggest alternative activities in case of bad weather in Kyoto" into the AI ​​model, the system can quickly create a new plan.

[0453] This allows travelers to respond flexibly to unforeseen circumstances, ensuring their planned trips are safe and fulfilling.

[0454] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0455] Step 1:

[0456] The server receives user travel plan information as input and stores details such as destination and itinerary in a database. This process prepares the basic data for managing basic information about travel destinations. Specifically, it accesses reservation management systems and calendar applications via APIs.

[0457] Step 2:

[0458] The server sends a request to a weather API to obtain environmental information for the target area based on the stored travel plan information. The input includes geographical information of the travel destination. The data obtained is meteorological information such as wind speed, precipitation, and temperature. Specifically, a Python script is used to retrieve the information from the external API in JSON format.

[0459] Step 3:

[0460] The server analyzes the acquired environmental information to determine whether the travel plan will be affected. The input includes weather data, and the output is a determination of whether or not there is an impact. This step uses a conditional branching algorithm to process the data based on defined criteria.

[0461] Step 4:

[0462] If the server determines that weather will affect the plan, it inputs a prompt message into an AI model that generates alternative cultural experiences based on user profile data. The input uses the user's past preferences and profile information, and the output generates proposed alternative plans. A machine learning model is used to provide appropriate alternatives.

[0463] Step 5:

[0464] The terminal receives an alternative plan provided by the server and presents it to the user. The input here is the alternative plan, and the output is the user's visually displayed information. It has an automatic translation function, specifically using the Google Translate API to translate the information into the user's native language.

[0465] Step 6:

[0466] The user selects an activity of interest from the presented alternative plans and sends this information to the server via their device. The input includes user preference data, and the output is information about the selected activity. Based on this selection information, the next procedure begins.

[0467] Step 7:

[0468] The server checks the availability of the selected activity and automatically arranges the reservation. The input is the selected activity information, and the output is the reservation confirmation information. The specific operation includes checking availability and confirming the reservation using the reservation management API.

[0469] Step 8:

[0470] The terminal displays reservation confirmation information to the user and generates route guidance to the destination. Inputs here are reservation and geographical information, while output is route guidance information. Using the Google Maps API, detailed directions are provided using map data and public transport timetables.

[0471] (Application Example 1)

[0472] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] When tourists visiting Japan are forced to change their sightseeing plans due to weather changes during their trip, they often lack the ability to quickly find suitable alternatives, which can negatively impact the quality of their travel experience. Furthermore, tourists unfamiliar with the local language and culture face the challenge of having to expend considerable effort finding and booking alternative activities.

[0474] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0475] In this invention, the server includes means for acquiring weather information for the planned visit area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, means for generating an alternative activity plan in case of bad weather, means for enabling the traveler to select an alternative activity, means for automatically arranging a reservation for the selected alternative activity, means for generating and presenting route guidance to the location of the reserved alternative activity, and means for presenting alternative activity information in real time using a visual display device. This enables travelers to quickly find a suitable alternative activity even in bad weather and to make reservations and travel smoothly, overcoming language barriers.

[0476] "Weather information for the planned visit area" refers to data about the current weather conditions in the area that the traveler plans to visit.

[0477] "Providing real-time alternative activity information" means showing users information about activities currently available in a timely manner.

[0478] A "visual display device" is an electronic device used to convey information to users visually.

[0479] "Generating alternative activity plans" means proposing alternative activities when the originally planned activities are not feasible.

[0480] "Generating and presenting route guidance" means creating a detailed travel route to a destination and guiding the user visually or audibly.

[0481] The system implementing this invention consists of a server, a terminal, and a visual display device. The server obtains data from weather APIs and public weather databases to acquire weather information for the area to be visited. This data includes weather information such as precipitation, wind speed, and temperature. Based on the acquired weather information, the server executes an algorithm to determine whether the traveler's activities will be affected by the weather.

[0482] If bad weather is predicted, the server generates an alternative activity plan. This process utilizes the user's past preferences and profile data. The generated alternative activity information is presented to the user in real time via a visual display. The user can then select an activity that interests them from the presented options.

[0483] Subsequently, the server automatically makes a reservation for the alternative activity selected by the user. Once the reservation is complete, the terminal displays the reservation information to the user and generates and presents route guidance to the destination. Specifically, it uses the Google Maps API to provide travel routes and transportation schedules to the destination. At this time, the visual display device also enables real-time navigation to the location where the selected activity will take place.

[0484] For example, if a traveler cancels their planned temple visit in Kyoto due to an approaching typhoon, a reservation for a tea ceremony experience will be made instead. This information will be displayed on the user's smart glasses, clearly showing how to use public transportation and the estimated travel time.

[0485] An example of a prompt message would be: "There is a forecast of heavy rain in Tokyo in 3 hours, so please suggest changing from an outdoor activity to an indoor activity. Also, please assist with booking tickets for that indoor activity."

[0486] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0487] Step 1:

[0488] The server obtains weather information for the planned visit area from weather APIs and public weather databases. The input is information about the planned visit area, and the output is weather data. This data acquisition involves using APIs to obtain precipitation, wind speed, temperature, and other data in real time.

[0489] Step 2:

[0490] The server determines, based on weather information, whether the planned travel activities will be affected by the weather. The input is the acquired weather data and travel plan information, and the output is the determination result indicating whether or not there will be an impact. This determination uses conditional branching logic to identify cases where bad weather is predicted.

[0491] Step 3:

[0492] When bad weather is expected, the server generates an alternative activity plan, taking into account the user profile and past preferences. The input is user profile information and current weather conditions, and the output is a list of recommended alternative activities. The generation process involves querying a database and using a recommendation algorithm.

[0493] Step 4:

[0494] The terminal presents alternative activity information to the user in real time via a visual display device. The input is a list of recommended alternative activities, and the output is the user's selection. The presentation is an action that allows the user to easily view the information through the user interface.

[0495] Step 5:

[0496] The server automatically makes reservations for the alternative activity selected by the user. The input is the user's selected alternative activity, and the output is reservation confirmation information. The reservation process includes integration with a reservation management system and automated processing.

[0497] Step 6:

[0498] The terminal displays reservation information and generates and presents route guidance for the user for the alternative activity for which the reservation has been completed. Inputs are reservation completion information and destination information, and output is the displayed navigation information. The presentation includes using the Google Maps API to generate the route and display the route map.

[0499] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0500] The system for implementing the present invention consists of a travel management server, a user terminal, and an emotion engine, enabling visitors to Japan to respond flexibly to changes in their travel plans due to weather conditions, in accordance with their emotional state.

[0501] First, the server collects weather data for the destination based on the traveler's planned visit information. Based on this data, it determines whether the planned travel activities will be affected by bad weather and prepares suggestions for alternative activities if necessary. In parallel, the device uses an emotion engine to analyze the user's emotional state based on the user's facial expressions, words, and device operation.

[0502] The server receives the results from the emotion engine and searches and selects the most appropriate alternative activity from the database based on the user's emotional state. In this process, it prioritizes presenting relaxation activities that can reduce stress, or entertainment plans that will make the user feel comfortable.

[0503] The device displays these alternative plans to the user and prompts them to make a selection. The plans feature automatic translation, making them easily understandable in the user's native language and providing detailed information about the options. Once the user selects an activity of interest, the server automates the booking process for the selected activity, completing it quickly.

[0504] Once selections and reservations are complete, the device will display guidance tailored to the user's needs to provide a more comfortable experience. Comprehensive support for public transport timetables and transportation options is also provided, allowing users to confidently move on to their next plan.

[0505] For example, if rain is expected while a user is visiting Osaka and it is determined that planned outdoor sightseeing is difficult, this system uses an emotion engine to analyze the user's emotions and proposes indoor sightseeing plans that can help the user change their mood, such as visiting an indoor art gallery or a relaxation spa. In this way, the present invention provides an optimal travel experience by taking into account the emotional state of the traveler.

[0506] The following describes the processing flow.

[0507] Step 1:

[0508] The server retrieves the traveler's planned visit information and uses weather APIs and public weather databases to collect future weather forecast data for the planned visit area. This includes collecting detailed weather information such as temperature, precipitation, and wind speed.

[0509] Step 2:

[0510] The server analyzes the collected weather data to determine whether planned travel activities will be affected by the weather. This analysis provides criteria for determining whether itinerary changes are necessary.

[0511] Step 3:

[0512] The device activates an emotion engine to monitor the user's emotional state. During this process, it uses sensors and microphones to analyze emotions based on facial expressions, voice, and operation patterns.

[0513] Step 4:

[0514] The server receives user emotion data obtained by the emotion engine and uses this information to search for alternative activities that are suitable for both the effects of the weather and the user's emotional state. It selects activities from the database that are suitable for stress reduction and mood improvement.

[0515] Step 5:

[0516] The device presents the user with alternative plans received from the server. To ensure user understanding, the details are displayed in the user's native language using an automatic translation function. Each plan's description and conditions are clearly stated, allowing the user to select activities that interest them.

[0517] Step 6:

[0518] Users select activities that interest them from the presented plans. This selection process is designed with an intuitive UI / UX.

[0519] Step 7:

[0520] The server automatically executes the necessary reservation procedures based on the user's selection. It processes reservations seamlessly even on Japanese reservation websites and provides feedback to the user's device.

[0521] Step 8:

[0522] The terminal displays reservation confirmation information to the user, generates route guidance to the selected alternative activity location, and displays detailed information. It provides reliable information, including transportation options and estimated travel time.

[0523] (Example 2)

[0524] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] Travelers may find their travel plans disrupted by environmental conditions at their intended destination. In particular, when planned activities become difficult due to external conditions such as weather, it is not easy for travelers to find suitable alternative plans on the spot. Furthermore, if suggestions are made that disregard the traveler's emotional state, it becomes difficult to provide a satisfying travel experience. These issues pose a risk of decreased traveler satisfaction.

[0526] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0527] In this invention, the server includes means for acquiring environmental data of the planned destination; means for determining whether the planned activity will be affected by external conditions based on the acquired environmental data; means for generating an alternative plan if the external conditions are unfavorable; means for analyzing the emotional state using an information processing device and proposing activities appropriate to the user's state; means for enabling the user to select an alternative activity; means for automating the arrangement of the selected alternative activity; and means for generating and presenting route information to the booked activity. As a result, travelers can receive an optimal alternative plan that suits their emotional state even under the influence of external conditions, enabling a smooth and satisfying trip.

[0528] "Planned destinations" refer to places that travelers have planned to visit in advance, and are a concept that includes geographical locations that serve as a basis for obtaining information about those areas.

[0529] "Environmental data" refers to data about external conditions that may affect travelers' activities, such as weather information and traffic conditions related to their planned destination.

[0530] "External conditions" refer to external factors that may affect travelers' planning activities, such as weather, traffic, and natural disasters.

[0531] An "alternative plan" refers to new activities or schedules proposed to complement or replace travel plans affected by unforeseen external conditions.

[0532] An "information processing device" is a general term for devices and software used to analyze a user's emotional state, and in particular, it performs data analysis based on the user's facial expressions, voice, and operation history.

[0533] "Emotional state" refers to the user's current psychological or emotional state and is a factor that should be considered in order to improve the traveler's satisfaction and comfort.

[0534] "Means of automating activity arrangements" refers to a system where the system automatically handles booking and registration procedures for alternative activities selected by the user, without human intervention.

[0535] "Route information" includes detailed information about the route and means of transportation necessary to travel to the location where the booked activity will take place.

[0536] The embodiment of the invention consists of a system designed to facilitate travelers' planning. This system includes a server, a terminal, and an emotion analysis device. Each component plays a specific role and works together, sharing data with the others.

[0537] First, the server uses online databases to collect environmental data for a region based on information about the traveler's planned destinations. Specifically, weather information is obtained using a weather API. Next, the program analyzes the collected data to determine how external conditions will affect the traveler's plans. Based on this determination, it may generate alternative activity plans that are suitable for the external conditions.

[0538] The terminal uses an emotion analysis device to analyze the traveler's emotional state in real time. Sensors collect the user's facial expressions, voice tone, and device operation history, and the emotion engine performs the analysis. Based on this emotional data, the server selects the most appropriate alternative plan for the user's state and provides it to the terminal.

[0539] For example, if a user is planning a visit to Osaka Castle, and rain is forecast, the program will proactively suggest an indoor activity plan. This plan might include visits to relaxation facilities to improve the user's condition, or indoor art galleries at tourist attractions. In addition, the server automates the booking process for these activities and provides route information to the user via the terminal.

[0540] An example of a prompt in a generative AI model might be: "When the user needs to change their destination due to rain, suggest indoor activities while considering the user's emotional data." By using this prompt, the AI ​​generates optimal suggestions, providing the user with a more fulfilling travel experience.

[0541] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0542] Step 1:

[0543] The server receives information about the traveler's planned destinations as input. Specifically, the input data includes the name of the city and the itinerary of the trip. Based on this, the server uses an open weather data API to obtain weather information for the planned destinations. As output, it generates weather forecast data for the relevant region. The weather data includes temperature, probability of precipitation, wind speed, etc.

[0544] Step 2:

[0545] The server analyzes the acquired weather information to determine whether the user's planned activities will be affected by external conditions. The analysis uses a method that references weather information for the planned date and the user's planned activities from a database. As a result, it may output a judgment such as, "Outdoor activities are difficult to carry out due to the rain forecast for the planned date." Based on this judgment, the server proceeds with preparing to generate alternative plans.

[0546] Step 3:

[0547] The device receives the user's facial expressions, voice tone, and operation history as input, and uses an emotion engine to diagnose their emotional state. Specifically, it employs facial recognition technology using a camera and voice analysis technology using a microphone. Through this analysis, emotional data such as "the user's stress level is high" is generated. This emotional data is sent to a server and used as a basis for decisions in the next step.

[0548] Step 4:

[0549] The server uses a generative AI model to generate prompts that suggest alternative activities, taking into account emotional state data and weather conditions. An example of a prompt might be, "Please prioritize indoor activities to reduce stress." This prompt is input into the AI ​​model, and activity suggestions appropriate to the user's emotional state are generated as output.

[0550] Step 5:

[0551] The device displays generated alternative activity suggestions and prompts the user to make a selection. These options are displayed in the user's native language via an automatic translation function. For example, options might include "Visit an indoor garden" or "Use a relaxation facility." The user selects their desired activity through the device's operation, and then proceeds to the next step.

[0552] Step 6:

[0553] The server automates the booking process based on the alternative activity selected by the user. This process involves coordinating with stakeholders via an external booking system API to complete the booking. As a result of the process, a confirmation message stating "Activity booking complete" is output and sent to the terminal.

[0554] Step 7:

[0555] The device provides detailed route information to the selected alternative activity location along with the completed reservation information. Using map services such as the Google Maps API, it calculates and presents the optimal mode of transport and timetable from the origin to the destination. This allows the user to confidently move on to their next plan.

[0556] (Application Example 2)

[0557] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0558] In recent years, with the advancement of autonomous driving technology and the personalization of travel, there has been a growing need to appropriately respond to unexpected weather changes and itinerary alterations, and to provide entertainment activities that take into account the emotional state of travelers. However, conventional methods have faced challenges such as the deterioration of the quality of the trip due to bad weather and the difficulty in quickly providing content that adapts to the emotions of users.

[0559] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0560] In this invention, the server includes means for acquiring weather information for the planned destination area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, and means for generating alternative activity plans in case of bad weather, allowing the traveler to choose an alternative activity. This enables the traveler to respond quickly to unexpected weather changes, and further enables the analysis of the emotional state of the mobile vehicle user using an emotion analysis engine, and suggests entertainment activities available on the mobile vehicle based on the results. This makes it possible to improve the traveler's experience and increase their satisfaction with the trip.

[0561] "Means for obtaining weather information for a planned visit area" refers to a mechanism for collecting weather data related to the area a traveler plans to visit and incorporating that information into the system.

[0562] "Means for determining whether planned travel activities will be affected by weather based on acquired weather information" refers to a mechanism that analyzes collected weather data and uses that data to determine whether planned travel activities are feasible.

[0563] "Means for generating alternative activity plans in case of bad weather" refers to a mechanism for automatically creating a new activity plan to replace the original travel plan when weather deteriorates.

[0564] "Means that enable travelers to choose alternative activities" refers to an interface that allows travelers to select from several alternative activity plans presented according to their preferences.

[0565] "Means for automatically arranging reservations for selected alternative activities" refers to a mechanism that automatically handles reservations and arrangements for alternative activities selected by travelers, enabling them to carry out these activities smoothly.

[0566] "Means for generating and presenting route guidance to the location of a booked alternative activity" refers to a mechanism for calculating the route to the location where the selected activity will take place and presenting it to the traveler visually or audibly.

[0567] "A means of analyzing the emotional state of mobile vehicle users using an emotion analysis engine" refers to a system designed to analyze the emotional state of users based on their facial expressions, behavioral data, and other factors.

[0568] "A means of suggesting entertainment activities available in a mobile vehicle based on analyzed emotional states" refers to a mechanism that, based on the results of emotional analysis, supports the selection of entertainment best suited to the user's mental state and suggests activities that can be enjoyed inside the vehicle.

[0569] This system has the function of obtaining weather information for the planned destination area and determining how the planned travel activities will be affected by the weather based on the obtained information. The server uses a weather information API to collect weather data and analyze whether the planned travel activities are feasible.

[0570] If bad weather is expected, the server generates an alternative activity plan. The smart devices used by travelers have an emotion analysis engine built in, which analyzes the user's current emotional state using facial expression data, voice tone, and device operation status as input. This process uses Python and emotion analysis libraries (e.g., OpenFace, DeepFace).

[0571] The next step is to suggest alternative activities best suited to the user's emotional state. Based on the analysis results, the server presents entertainment activities that can be performed on the in-car entertainment system (e.g., music playlists, relaxing videos, VR experiences, etc.). At this time, the entertainment activity options are generated in real time and presented to the user via an interface.

[0572] As a concrete example, suppose a self-driving vehicle used by a traveler is en route to a resort area when heavy snow is expected. Based on the heavy snow data, the server determines that outdoor activities will be difficult and uses an emotion analysis engine to identify that the traveler needs relaxation. Based on this, it suggests a VR tour of a hot spring inn that can be experienced inside the vehicle, as well as relaxation music.

[0573] Examples of prompts to input into a generative AI model are as follows:

[0574] "Based on current weather conditions and user sentiment data, please suggest the optimal in-car entertainment experience."

[0575] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0576] Step 1:

[0577] The server obtains weather information for the planned visit area via a weather information API. The input is information about the area the traveler plans to visit, and the output is detailed weather data for that area. Based on the obtained data, the server starts a process to determine whether the planned travel activities will be affected by the weather.

[0578] Step 2:

[0579] The terminal uses the traveler's smart device's camera and microphone to collect user facial and voice data. The input is real-time emotional data obtained through the device. Using emotion analysis libraries (OpenFace, DeepFace), this data is analyzed to identify the traveler's current emotional state. The output is data representing the user's emotional state.

[0580] Step 3:

[0581] The server searches a database for and selects alternative activities suitable for the traveler's emotional state, based on acquired weather data and sentiment analysis results. The inputs are weather data and sentiment data, and the output is the suggested alternative activities. This is processed by a generative AI model, which generates individual prompt statements and then formulates recommended activities based on them.

[0582] Step 4:

[0583] The terminal visually presents the selected alternative activity to the user and prompts them to make a choice. The input is alternative activity data received from the server. The output is an option list displayed in the user interface. The user proceeds to the next step by selecting an activity of interest.

[0584] Step 5:

[0585] The server automates and completes the booking process for the alternative activity selected by the user. The input is the selected alternative activity, and the output is booking confirmation information. The process includes data communication with the booking system.

[0586] Step 6:

[0587] After a reservation is complete, the terminal displays entertainment activities available within the vehicle and provides further customized instructions based on sentiment analysis results. Inputs include information on content available in the vehicle and the user's sentiment data. Outputs include entertainment options and route guidance optimized for the user's current situation.

[0588] This series of steps creates a system that provides travelers with the best possible experience in response to changing weather conditions.

[0589] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0590] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0591] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0592] [Fourth Embodiment]

[0593] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0594] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0595] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0596] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0597] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0598] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0599] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0600] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0601] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0602] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0603] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0604] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0605] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0606] The system for implementing this invention consists of a travel management server and a user's terminal, enabling visitors to Japan to quickly respond to changes in their travel plans due to weather conditions.

[0607] First, the server retrieves weather data for the destination based on the traveler's planned visit information. Data can be obtained from weather APIs or public weather databases, and the information collected includes precipitation, wind speed, and temperature. Using this information, the server determines whether the planned travel activity will be affected by the weather.

[0608] If bad weather is expected, the server searches for alternative activities that might interest travelers, such as indoor leisure or cultural experiences. Based on user profiles and past preferences, it can provide optimized options. The device then presents these alternative plans to the user, and an automatic translation function ensures that the details are understood in the user's native language.

[0609] The user selects an activity of interest from the alternative plans presented. The server automatically checks the availability of the selected activity location and makes the reservation, so the user does not need to communicate in the local language.

[0610] Once the reservation is complete, the device displays confirmation information to the user. At this time, the device also provides detailed route guidance to the destination and quickly displays services including public transport timetables and directions.

[0611] For example, if a user is visiting Kyoto and a typhoon is approaching, making their planned temple visits difficult, this system can be used to quickly plan and execute alternative indoor activities such as a tea ceremony or a traditional pottery class. In this way, the present invention helps travelers enjoy a rich and diverse range of experiences even in bad weather.

[0612] The following describes the processing flow.

[0613] Step 1:

[0614] The server retrieves the traveler's planned visit information and accesses weather APIs and public weather databases to collect future weather forecasts for the planned visit area. This data includes specific information such as temperature, precipitation, and wind speed.

[0615] Step 2:

[0616] The server analyzes the collected weather data to determine whether the planned travel activity will be affected by bad weather. The criteria for this determination may include, for example, whether the amount of rainfall exceeds a certain threshold.

[0617] Step 3:

[0618] If it is determined that bad weather is likely, the server searches for alternative activity plans based on the user's interests and past preferences. It extracts about 10 types of information on indoor leisure and cultural experiences from the database.

[0619] Step 4:

[0620] The terminal displays alternative plans received from the server to the user. The plans are presented in the user's native language through an automatic translation function, making them easier to understand.

[0621] Step 5:

[0622] Users select activities that interest them from the provided options. Detailed information can be viewed when making a selection.

[0623] Step 6:

[0624] The server initiates the reservation process based on the user's selection. The automated process completes the reservation on the Japanese website. This reservation information is then communicated to the user via their device.

[0625] Step 7:

[0626] The device generates and presents route guidance to the user for their booked alternative activity. This includes public transport schedules and route information to ensure the user can reach their destination smoothly.

[0627] (Example 1)

[0628] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0629] A challenge exists when visitors face unexpected weather changes during their trip, making it difficult to quickly and efficiently adjust their travel plans. In particular, when there are language and cultural barriers, it is not easy for travelers to choose and arrange new activities locally.

[0630] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0631] In this invention, the server includes means for acquiring environmental information of the area to be visited, means for determining whether the activities of the travel plan will be affected by the environment based on the acquired environmental information, and means for generating alternative cultural experiences. This enables travelers to flexibly respond to changes in weather, quickly select new cultural experiences, and automatically adjust their plans.

[0632] "Environmental information" refers to weather data and other information about natural conditions related to the area you plan to visit.

[0633] A "travel plan" is detailed planning information that includes the activities and itinerary that a traveler plans to do in the area they intend to visit.

[0634] "Affected" means that, based on environmental information, the activities in your travel plan become impossible to carry out or require modification.

[0635] "Alternative cultural experiences" are new indoor activity options offered when planned activities are affected by environmental factors.

[0636] "Automating arrangements" refers to automatically handling reservations and scheduling adjustments for alternative experiences selected by the user.

[0637] "Route information" refers to detailed guidance information regarding the means of transportation and directions necessary for users to reach their desired activity location.

[0638] "Language conversion functionality" is a technology that translates information into the user's native language to make it easier to understand.

[0639] To implement this invention, a system will be built to manage travelers' visit schedules and provide flexible travel plans tailored to environmental conditions. The system will primarily consist of servers and terminals, supporting users in making the most of their travel experience.

[0640] The server receives the user's travel plan and retrieves relevant environmental information based on it. This environmental information is accessible from weather APIs and publicly available weather databases and includes details such as wind speed, precipitation, and temperature. The server has scripts implemented using programming languages ​​such as Python and Java to retrieve data from the API. If it is determined that the weather will affect the travel plan, the server refers to the user's profile data and past preferences to select the most suitable alternative activity in order to provide an alternative cultural experience.

[0641] The terminal receives information from the server and displays it to the user. It has an automatic translation function that uses the Google Translate API to translate information into the user's native language. As a result, users can understand the information smoothly without experiencing language barriers.

[0642] Users select activities of interest from alternative plans provided through their device. The server automatically arranges and completes the booking for the selected activities. This allows for convenient travel planning without the need for communication in the local language.

[0643] As a concrete example, when a user visits an area prone to natural disasters, the server detects weather data in real time and suggests changes to the plan. In this case, by inputting a prompt such as "Please suggest alternative activities in case of bad weather in Kyoto" into the AI ​​model, the system can quickly create a new plan.

[0644] This allows travelers to respond flexibly to unforeseen circumstances, ensuring their planned trips are safe and fulfilling.

[0645] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0646] Step 1:

[0647] The server receives user travel plan information as input and stores details such as destination and itinerary in a database. This process prepares the basic data for managing basic information about travel destinations. Specifically, it accesses reservation management systems and calendar applications via APIs.

[0648] Step 2:

[0649] The server sends a request to a weather API to obtain environmental information for the target area based on the stored travel plan information. The input includes geographical information of the travel destination. The data obtained is meteorological information such as wind speed, precipitation, and temperature. Specifically, a Python script is used to retrieve the information from the external API in JSON format.

[0650] Step 3:

[0651] The server analyzes the acquired environmental information to determine whether the travel plan will be affected. The input includes weather data, and the output is a determination of whether or not there is an impact. This step uses a conditional branching algorithm to process the data based on defined criteria.

[0652] Step 4:

[0653] If the server determines that weather will affect the plan, it inputs a prompt message into an AI model that generates alternative cultural experiences based on user profile data. The input uses the user's past preferences and profile information, and the output generates proposed alternative plans. A machine learning model is used to provide appropriate alternatives.

[0654] Step 5:

[0655] The terminal receives an alternative plan provided by the server and presents it to the user. The input here is the alternative plan, and the output is the user's visually displayed information. It has an automatic translation function, specifically using the Google Translate API to translate the information into the user's native language.

[0656] Step 6:

[0657] The user selects an activity of interest from the presented alternative plans and sends this information to the server via their device. The input includes user preference data, and the output is information about the selected activity. Based on this selection information, the next procedure begins.

[0658] Step 7:

[0659] The server checks the availability of the selected activity and automatically arranges the reservation. The input is the selected activity information, and the output is the reservation confirmation information. The specific operation includes checking availability and confirming the reservation using the reservation management API.

[0660] Step 8:

[0661] The terminal displays reservation confirmation information to the user and generates route guidance to the destination. Inputs here are reservation and geographical information, while output is route guidance information. Using the Google Maps API, detailed directions are provided using map data and public transport timetables.

[0662] (Application Example 1)

[0663] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0664] When tourists visiting Japan are forced to change their sightseeing plans due to weather changes during their trip, they often lack the ability to quickly find suitable alternatives, which can negatively impact the quality of their travel experience. Furthermore, tourists unfamiliar with the local language and culture face the challenge of having to expend considerable effort finding and booking alternative activities.

[0665] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0666] In this invention, the server includes means for acquiring weather information for the planned visit area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, means for generating an alternative activity plan in case of bad weather, means for enabling the traveler to select an alternative activity, means for automatically arranging a reservation for the selected alternative activity, means for generating and presenting route guidance to the location of the reserved alternative activity, and means for presenting alternative activity information in real time using a visual display device. This enables travelers to quickly find a suitable alternative activity even in bad weather and to make reservations and travel smoothly, overcoming language barriers.

[0667] "Weather information for the planned visit area" refers to data about the current weather conditions in the area that the traveler plans to visit.

[0668] "Providing real-time alternative activity information" means showing users information about activities currently available in a timely manner.

[0669] A "visual display device" is an electronic device used to convey information to users visually.

[0670] "Generating alternative activity plans" means proposing alternative activities when the originally planned activities are not feasible.

[0671] "Generating and presenting route guidance" means creating a detailed travel route to a destination and guiding the user visually or audibly.

[0672] The system implementing this invention consists of a server, a terminal, and a visual display device. The server obtains data from weather APIs and public weather databases to acquire weather information for the area to be visited. This data includes weather information such as precipitation, wind speed, and temperature. Based on the acquired weather information, the server executes an algorithm to determine whether the traveler's activities will be affected by the weather.

[0673] If bad weather is predicted, the server generates an alternative activity plan. This process utilizes the user's past preferences and profile data. The generated alternative activity information is presented to the user in real time via a visual display. The user can then select an activity that interests them from the presented options.

[0674] Subsequently, the server automatically makes a reservation for the alternative activity selected by the user. Once the reservation is complete, the terminal displays the reservation information to the user and generates and presents route guidance to the destination. Specifically, it uses the Google Maps API to provide travel routes and transportation schedules to the destination. At this time, the visual display device also enables real-time navigation to the location where the selected activity will take place.

[0675] For example, if a traveler cancels their planned temple visit in Kyoto due to an approaching typhoon, a reservation for a tea ceremony experience will be made instead. This information will be displayed on the user's smart glasses, clearly showing how to use public transportation and the estimated travel time.

[0676] An example of a prompt message would be: "There is a forecast of heavy rain in Tokyo in 3 hours, so please suggest changing from an outdoor activity to an indoor activity. Also, please assist with booking tickets for that indoor activity."

[0677] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0678] Step 1:

[0679] The server obtains weather information for the planned visit area from weather APIs and public weather databases. The input is information about the planned visit area, and the output is weather data. This data acquisition involves using APIs to obtain precipitation, wind speed, temperature, and other data in real time.

[0680] Step 2:

[0681] The server determines, based on weather information, whether the planned travel activities will be affected by the weather. The input is the acquired weather data and travel plan information, and the output is the determination result indicating whether or not there will be an impact. This determination uses conditional branching logic to identify cases where bad weather is predicted.

[0682] Step 3:

[0683] When bad weather is expected, the server generates an alternative activity plan, taking into account the user profile and past preferences. The input is user profile information and current weather conditions, and the output is a list of recommended alternative activities. The generation process involves querying a database and using a recommendation algorithm.

[0684] Step 4:

[0685] The terminal presents alternative activity information to the user in real time via a visual display device. The input is a list of recommended alternative activities, and the output is the user's selection. The presentation is an action that allows the user to easily view the information through the user interface.

[0686] Step 5:

[0687] The server automatically makes reservations for the alternative activity selected by the user. The input is the user's selected alternative activity, and the output is reservation confirmation information. The reservation process includes integration with a reservation management system and automated processing.

[0688] Step 6:

[0689] The terminal displays reservation information and generates and presents route guidance for the user for the alternative activity for which the reservation has been completed. Inputs are reservation completion information and destination information, and output is the displayed navigation information. The presentation includes using the Google Maps API to generate the route and display the route map.

[0690] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0691] The system for implementing the present invention consists of a travel management server, a user terminal, and an emotion engine, enabling visitors to Japan to respond flexibly to changes in their travel plans due to weather conditions, in accordance with their emotional state.

[0692] First, the server collects weather data for the destination based on the traveler's planned visit information. Based on this data, it determines whether the planned travel activities will be affected by bad weather and prepares suggestions for alternative activities if necessary. In parallel, the device uses an emotion engine to analyze the user's emotional state based on the user's facial expressions, words, and device operation.

[0693] The server receives the results from the emotion engine and searches and selects the most appropriate alternative activity from the database based on the user's emotional state. In this process, it prioritizes presenting relaxation activities that can reduce stress, or entertainment plans that will make the user feel comfortable.

[0694] The device displays these alternative plans to the user and prompts them to make a selection. The plans feature automatic translation, making them easily understandable in the user's native language and providing detailed information about the options. Once the user selects an activity of interest, the server automates the booking process for the selected activity, completing it quickly.

[0695] Once selections and reservations are complete, the device will display guidance tailored to the user's needs to provide a more comfortable experience. Comprehensive support for public transport timetables and transportation options is also provided, allowing users to confidently move on to their next plan.

[0696] For example, if rain is expected while a user is visiting Osaka and it is determined that planned outdoor sightseeing is difficult, this system uses an emotion engine to analyze the user's emotions and proposes indoor sightseeing plans that can help the user change their mood, such as visiting an indoor art gallery or a relaxation spa. In this way, the present invention provides an optimal travel experience by taking into account the emotional state of the traveler.

[0697] The following describes the processing flow.

[0698] Step 1:

[0699] The server retrieves the traveler's planned visit information and uses weather APIs and public weather databases to collect future weather forecast data for the planned visit area. This includes collecting detailed weather information such as temperature, precipitation, and wind speed.

[0700] Step 2:

[0701] The server analyzes the collected weather data to determine whether planned travel activities will be affected by the weather. This analysis provides criteria for determining whether itinerary changes are necessary.

[0702] Step 3:

[0703] The device activates an emotion engine to monitor the user's emotional state. During this process, it uses sensors and microphones to analyze emotions based on facial expressions, voice, and operation patterns.

[0704] Step 4:

[0705] The server receives user emotion data obtained by the emotion engine and uses this information to search for alternative activities that are suitable for both the effects of the weather and the user's emotional state. It selects activities from the database that are suitable for stress reduction and mood improvement.

[0706] Step 5:

[0707] The device presents the user with alternative plans received from the server. To ensure user understanding, the details are displayed in the user's native language using an automatic translation function. Each plan's description and conditions are clearly stated, allowing the user to select activities that interest them.

[0708] Step 6:

[0709] Users select activities that interest them from the presented plans. This selection process is designed with an intuitive UI / UX.

[0710] Step 7:

[0711] The server automatically executes the necessary reservation procedures based on the user's selection. It processes reservations seamlessly even on Japanese reservation websites and provides feedback to the user's device.

[0712] Step 8:

[0713] The terminal displays reservation confirmation information to the user, generates route guidance to the selected alternative activity location, and displays detailed information. It provides reliable information, including transportation options and estimated travel time.

[0714] (Example 2)

[0715] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0716] Travelers may find their travel plans disrupted by environmental conditions at their intended destination. In particular, when planned activities become difficult due to external conditions such as weather, it is not easy for travelers to find suitable alternative plans on the spot. Furthermore, if suggestions are made that disregard the traveler's emotional state, it becomes difficult to provide a satisfying travel experience. These issues pose a risk of decreased traveler satisfaction.

[0717] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0718] In this invention, the server includes means for acquiring environmental data of the planned destination; means for determining whether the planned activity will be affected by external conditions based on the acquired environmental data; means for generating an alternative plan if the external conditions are unfavorable; means for analyzing the emotional state using an information processing device and proposing activities appropriate to the user's state; means for enabling the user to select an alternative activity; means for automating the arrangement of the selected alternative activity; and means for generating and presenting route information to the booked activity. As a result, travelers can receive an optimal alternative plan that suits their emotional state even under the influence of external conditions, enabling a smooth and satisfying trip.

[0719] "Planned destinations" refer to places that travelers have planned to visit in advance, and are a concept that includes geographical locations that serve as a basis for obtaining information about those areas.

[0720] "Environmental data" refers to data about external conditions that may affect travelers' activities, such as weather information and traffic conditions related to their planned destination.

[0721] "External conditions" refer to external factors that may affect travelers' planning activities, such as weather, traffic, and natural disasters.

[0722] An "alternative plan" refers to new activities or schedules proposed to complement or replace travel plans affected by unforeseen external conditions.

[0723] An "information processing device" is a general term for devices and software used to analyze a user's emotional state, and in particular, it performs data analysis based on the user's facial expressions, voice, and operation history.

[0724] "Emotional state" refers to the user's current psychological or emotional state and is a factor that should be considered in order to improve the traveler's satisfaction and comfort.

[0725] "Means of automating activity arrangements" refers to a system where the system automatically handles booking and registration procedures for alternative activities selected by the user, without human intervention.

[0726] "Route information" includes detailed information about the route and means of transportation necessary to travel to the location where the booked activity will take place.

[0727] The embodiment of the invention consists of a system designed to facilitate travelers' planning. This system includes a server, a terminal, and an emotion analysis device. Each component plays a specific role and works together, sharing data with the others.

[0728] First, the server uses online databases to collect environmental data for a region based on information about the traveler's planned destinations. Specifically, weather information is obtained using a weather API. Next, the program analyzes the collected data to determine how external conditions will affect the traveler's plans. Based on this determination, it may generate alternative activity plans that are suitable for the external conditions.

[0729] The terminal uses an emotion analysis device to analyze the traveler's emotional state in real time. Sensors collect the user's facial expressions, voice tone, and device operation history, and the emotion engine performs the analysis. Based on this emotional data, the server selects the most appropriate alternative plan for the user's state and provides it to the terminal.

[0730] For example, if a user is planning a visit to Osaka Castle, and rain is forecast, the program will proactively suggest an indoor activity plan. This plan might include visits to relaxation facilities to improve the user's condition, or indoor art galleries at tourist attractions. In addition, the server automates the booking process for these activities and provides route information to the user via the terminal.

[0731] An example of a prompt in a generative AI model might be: "When the user needs to change their destination due to rain, suggest indoor activities while considering the user's emotional data." By using this prompt, the AI ​​generates optimal suggestions, providing the user with a more fulfilling travel experience.

[0732] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0733] Step 1:

[0734] The server receives information about the traveler's planned destinations as input. Specifically, the input data includes the name of the city and the itinerary of the trip. Based on this, the server uses an open weather data API to obtain weather information for the planned destinations. As output, it generates weather forecast data for the relevant region. The weather data includes temperature, probability of precipitation, wind speed, etc.

[0735] Step 2:

[0736] The server analyzes the acquired weather information to determine whether the user's planned activities will be affected by external conditions. The analysis uses a method that references weather information for the planned date and the user's planned activities from a database. As a result, it may output a judgment such as, "Outdoor activities are difficult to carry out due to the rain forecast for the planned date." Based on this judgment, the server proceeds with preparing to generate alternative plans.

[0737] Step 3:

[0738] The device receives the user's facial expressions, voice tone, and operation history as input, and uses an emotion engine to diagnose their emotional state. Specifically, it employs facial recognition technology using a camera and voice analysis technology using a microphone. Through this analysis, emotional data such as "the user's stress level is high" is generated. This emotional data is sent to a server and used as a basis for decisions in the next step.

[0739] Step 4:

[0740] The server uses a generative AI model to generate prompts that suggest alternative activities, taking into account emotional state data and weather conditions. An example of a prompt might be, "Please prioritize indoor activities to reduce stress." This prompt is input into the AI ​​model, and activity suggestions appropriate to the user's emotional state are generated as output.

[0741] Step 5:

[0742] The device displays generated alternative activity suggestions and prompts the user to make a selection. These options are displayed in the user's native language via an automatic translation function. For example, options might include "Visit an indoor garden" or "Use a relaxation facility." The user selects their desired activity through the device's operation, and then proceeds to the next step.

[0743] Step 6:

[0744] The server automates the booking process based on the alternative activity selected by the user. This process involves coordinating with stakeholders via an external booking system API to complete the booking. As a result of the process, a confirmation message stating "Activity booking complete" is output and sent to the terminal.

[0745] Step 7:

[0746] The device provides detailed route information to the selected alternative activity location along with the completed reservation information. Using map services such as the Google Maps API, it calculates and presents the optimal mode of transport and timetable from the origin to the destination. This allows the user to confidently move on to their next plan.

[0747] (Application Example 2)

[0748] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] In recent years, with the advancement of autonomous driving technology and the personalization of travel, there has been a growing need to appropriately respond to unexpected weather changes and itinerary alterations, and to provide entertainment activities that take into account the emotional state of travelers. However, conventional methods have faced challenges such as the deterioration of the quality of the trip due to bad weather and the difficulty in quickly providing content that adapts to the emotions of users.

[0750] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0751] In this invention, the server includes means for acquiring weather information for the planned destination area, means for determining whether the planned travel activities will be affected by the weather based on the acquired weather information, and means for generating alternative activity plans in case of bad weather, allowing the traveler to choose an alternative activity. This enables the traveler to respond quickly to unexpected weather changes, and further enables the analysis of the emotional state of the mobile vehicle user using an emotion analysis engine, and suggests entertainment activities available on the mobile vehicle based on the results. This makes it possible to improve the traveler's experience and increase their satisfaction with the trip.

[0752] "Means for obtaining weather information for a planned visit area" refers to a mechanism for collecting weather data related to the area a traveler plans to visit and incorporating that information into the system.

[0753] "Means for determining whether planned travel activities will be affected by weather based on acquired weather information" refers to a mechanism that analyzes collected weather data and uses that data to determine whether planned travel activities are feasible.

[0754] "Means for generating alternative activity plans in case of bad weather" refers to a mechanism for automatically creating a new activity plan to replace the original travel plan when weather deteriorates.

[0755] "Means that enable travelers to choose alternative activities" refers to an interface that allows travelers to select from several alternative activity plans presented according to their preferences.

[0756] "Means for automatically arranging reservations for selected alternative activities" refers to a mechanism that automatically handles reservations and arrangements for alternative activities selected by travelers, enabling them to carry out these activities smoothly.

[0757] "Means for generating and presenting route guidance to the location of a booked alternative activity" refers to a mechanism for calculating the route to the location where the selected activity will take place and presenting it to the traveler visually or audibly.

[0758] "A means of analyzing the emotional state of mobile vehicle users using an emotion analysis engine" refers to a system designed to analyze the emotional state of users based on their facial expressions, behavioral data, and other factors.

[0759] "A means of suggesting entertainment activities available in a mobile vehicle based on analyzed emotional states" refers to a mechanism that, based on the results of emotional analysis, supports the selection of entertainment best suited to the user's mental state and suggests activities that can be enjoyed inside the vehicle.

[0760] This system has the function of obtaining weather information for the planned destination area and determining how the planned travel activities will be affected by the weather based on the obtained information. The server uses a weather information API to collect weather data and analyze whether the planned travel activities are feasible.

[0761] If bad weather is expected, the server generates an alternative activity plan. The smart devices used by travelers have an emotion analysis engine built in, which analyzes the user's current emotional state using facial expression data, voice tone, and device operation status as input. This process uses Python and emotion analysis libraries (e.g., OpenFace, DeepFace).

[0762] The next step is to suggest alternative activities best suited to the user's emotional state. Based on the analysis results, the server presents entertainment activities that can be performed on the in-car entertainment system (e.g., music playlists, relaxing videos, VR experiences, etc.). At this time, the entertainment activity options are generated in real time and presented to the user via an interface.

[0763] As a concrete example, suppose a self-driving vehicle used by a traveler is en route to a resort area when heavy snow is expected. Based on the heavy snow data, the server determines that outdoor activities will be difficult and uses an emotion analysis engine to identify that the traveler needs relaxation. Based on this, it suggests a VR tour of a hot spring inn that can be experienced inside the vehicle, as well as relaxation music.

[0764] Examples of prompts to input into a generative AI model are as follows:

[0765] "Based on current weather conditions and user sentiment data, please suggest the optimal in-car entertainment experience."

[0766] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0767] Step 1:

[0768] The server obtains weather information for the planned visit area via a weather information API. The input is information about the area the traveler plans to visit, and the output is detailed weather data for that area. Based on the obtained data, the server starts a process to determine whether the planned travel activities will be affected by the weather.

[0769] Step 2:

[0770] The terminal uses the traveler's smart device's camera and microphone to collect user facial and voice data. The input is real-time emotional data obtained through the device. Using emotion analysis libraries (OpenFace, DeepFace), this data is analyzed to identify the traveler's current emotional state. The output is data representing the user's emotional state.

[0771] Step 3:

[0772] The server searches a database for and selects alternative activities suitable for the traveler's emotional state, based on acquired weather data and sentiment analysis results. The inputs are weather data and sentiment data, and the output is the suggested alternative activities. This is processed by a generative AI model, which generates individual prompt statements and then formulates recommended activities based on them.

[0773] Step 4:

[0774] The terminal visually presents the selected alternative activity to the user and prompts them to make a choice. The input is alternative activity data received from the server. The output is an option list displayed in the user interface. The user proceeds to the next step by selecting an activity of interest.

[0775] Step 5:

[0776] The server automates and completes the booking process for the alternative activity selected by the user. The input is the selected alternative activity, and the output is booking confirmation information. The process includes data communication with the booking system.

[0777] Step 6:

[0778] After a reservation is complete, the terminal displays entertainment activities available within the vehicle and provides further customized instructions based on sentiment analysis results. Inputs include information on content available in the vehicle and the user's sentiment data. Outputs include entertainment options and route guidance optimized for the user's current situation.

[0779] This series of steps creates a system that provides travelers with the best possible experience in response to changing weather conditions.

[0780] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0781] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0782] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0783] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0784] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0785] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0786] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0787] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0788] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0789] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0790] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0791] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0792] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0793] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0794] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0795] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0796] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0797] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0798] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0799] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0800] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0801] The following is further disclosed regarding the embodiments described above.

[0802] (Claim 1)

[0803] Means of obtaining weather information for the area to be visited,

[0804] A means of determining whether planned travel activities will be affected by weather based on acquired weather information,

[0805] A means of generating alternative activity plans in case of bad weather,

[0806] Means that allow travelers to choose alternative activities,

[0807] A means of automatically arranging reservations for selected alternative activities,

[0808] A means of generating and presenting route guidance to the location of the reserved alternative activity,

[0809] A system that includes this.

[0810] (Claim 2)

[0811] The system according to claim 1, which proposes multiple indoor activities suitable for bad weather.

[0812] (Claim 3)

[0813] The system according to claim 1, which incorporates a language translation function during the reservation and arrangement process.

[0814] "Example 1"

[0815] (Claim 1)

[0816] Means of obtaining environmental information for the area to be visited,

[0817] A means of determining whether the activities in a travel plan will be affected by environmental impacts based on acquired environmental information,

[0818] Means for generating alternative cultural experiences,

[0819] A means to allow users to choose an alternative experience,

[0820] A means to automate the arrangement of selected alternative experiences,

[0821] A means of generating and providing route information to the location of an automated alternative experience,

[0822] A system that includes this.

[0823] (Claim 2)

[0824] The system according to claim 1, which proposes multiple indoor activities suitable for adverse events.

[0825] (Claim 3)

[0826] The system according to claim 1, which incorporates a language conversion function in the process of arrangement and management.

[0827] "Application Example 1"

[0828] (Claim 1)

[0829] Means of obtaining weather information for the area to be visited,

[0830] A means of determining whether planned travel activities will be affected by weather based on acquired weather information,

[0831] A means of generating alternative activity plans in case of bad weather,

[0832] Means that allow travelers to choose alternative activities,

[0833] A means of automatically arranging reservations for selected alternative activities,

[0834] A means of generating and presenting route guidance to the location of the reserved alternative activity,

[0835] A means of presenting alternative activity information in real time using a visual display device,

[0836] A system that includes this.

[0837] (Claim 2)

[0838] The system according to claim 1, which proposes multiple indoor activities suitable for bad weather.

[0839] (Claim 3)

[0840] The system according to claim 1, which incorporates a language translation function during the reservation and arrangement process.

[0841] "Example 2 of combining an emotion engine"

[0842] (Claim 1)

[0843] A means of obtaining environmental data for the planned visit location,

[0844] A means for determining whether planning activities are affected by external conditions based on acquired environmental data,

[0845] Means for generating alternative plans when external conditions are unfavorable,

[0846] A means of analyzing emotional states using an information processing device and proposing activities appropriate to the user's state,

[0847] Means that allow users to choose alternative activities,

[0848] A means to automate the arrangement of selected alternative activities,

[0849] A means for generating and presenting route information to a reserved activity,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which proposes a plurality of internal activities suitable for unfavorable external conditions.

[0853] (Claim 3)

[0854] The system according to claim 1, which incorporates a language conversion function during the reservation and arrangement process.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] Means of obtaining weather information for the area to be visited,

[0858] A means of determining whether planned travel activities will be affected by weather based on acquired weather information,

[0859] A means of generating alternative activity plans in case of bad weather,

[0860] Means that allow travelers to choose alternative activities,

[0861] A means of automatically arranging reservations for selected alternative activities,

[0862] A means of generating and presenting route guidance to the location of the reserved alternative activity,

[0863] A means for analyzing the emotional state of mobile vehicle users using an emotion analysis engine,

[0864] A means of suggesting entertainment activities available within a mobile vehicle based on analyzed emotional states,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, which proposes multiple indoor activities suitable for bad weather.

[0868] (Claim 3)

[0869] The system according to claim 1, which incorporates a language translation function during the reservation and arrangement process. [Explanation of symbols]

[0870] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of obtaining weather information for the area to be visited, A means of determining whether planned travel activities will be affected by weather based on acquired weather information, A means of generating alternative activity plans in case of bad weather, Means that allow travelers to choose alternative activities, A means of automatically arranging reservations for selected alternative activities, A means of generating and presenting route guidance to the location of the reserved alternative activity, A system that includes this.

2. The system according to claim 1, which proposes multiple indoor activities suitable for bad weather.

3. The system according to claim 1, which incorporates a language translation function during the reservation and arrangement process.

Images