system

Abstract

We provide the system. [Solution] A means for receiving image data containing information and extracting location information and time information from said data, A means for organizing the places to visit and the order of visits based on the extracted location and time information, A means for automatically generating a travel record based on the aforementioned places visited and the order of visits, A means of suggesting new places to visit and travel routes based on other publicly available evaluation information, The proposed travel route is evaluated by other users, and a means is provided to calculate a reward based on the evaluation. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need to effectively organize a plurality of image data taken by a traveler after a trip and easily create a record of visited places. However, in the conventional method, manually organizing data and creating a travel record requires labor and time. Also, it is not easy to incorporate information on highly rated tourist attractions and restaurants that could not be visited during the trip into the next travel plan. Furthermore, there is a demand for a system in which travelers share travel plans with each other and those plans are evaluated by others, but such a mechanism is not sufficiently developed at present. The present invention aims to solve these problems and provide a new experience creation for travelers and a travel plan sharing environment.

Means for Solving the Problems

[0005] This invention provides means for receiving image data containing information and extracting location and time information from that data. It also provides means for organizing visited places and their order of visit based on the extracted location and time information. Furthermore, it provides means for automatically generating a travel record based on the organized information, making it easily accessible to the user in both visual and text-based formats. It also includes means for acquiring publicly available rating information for highly-rated tourist destinations and restaurants, and for suggesting new places to visit and travel routes based on that information. This allows travelers to incorporate new elements into their next travel plans. In addition, by including means for calculating and providing rewards based on ratings from other users of the suggested travel routes, the invention aims to promote interaction among travelers and enhance the value of shared travel experiences.

[0006] "Image data" refers to a collection of information, including photographs taken and their metadata.

[0007] "Location information" refers to data that indicates the geographical location where an image was taken, and is usually expressed as latitude and longitude.

[0008] "Time information" refers to data indicating the time the image was taken, and is usually recorded in timestamp format.

[0009] "Place visited" refers to a specific geographical location where a traveler stays or visits.

[0010] "Visiting order" refers to the chronological order in which a traveler visited places.

[0011] A "travel record" is a document that describes and visualizes the places and order in which a traveler visited, and may include text and images.

[0012] "Evaluation information" refers to data that shows third-party evaluations of tourist destinations and restaurants.

[0013] A "travel route" refers to the recommended route for travelers to take when moving between a series of destinations.

[0014] "Reward" refers to the valuable thing provided as a result of the proposed movement route being evaluated by others.

[0015] "User" refers to travelers and related persons who use this system.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system that effectively utilizes image data taken by travelers during their trips, automatically generates travel records, and proposes new travel plans. The core of this system consists of a series of operations involving a server, terminals, and users.

[0038] The server receives image data uploaded by the user. This includes typical digital images and associated metadata. The server extracts location and time information from the image files and uses it to organize which places were visited and in what order. This process creates a list of places the traveler visited during their trip.

[0039] The terminal is a device that provides the user with an interface for uploading image data. This typically includes a smartphone or computer. The user selects photos taken during their trip and sends them to the server via the terminal. Once the upload is complete, the user can view the analysis results on the server. The terminal also serves as a tool for displaying the generated travel record (travel diary), allowing the user to view this information and add additional information as needed.

[0040] Users can use the system to automatically create their own travel records and obtain information to help plan their next trip. By uploading photos taken during their trip to the server via their device, users can digitize and organize their travel experiences. In this process, users can review the generated travel diary and add comments about the places they visited. They can also check new travel routes suggested by the server and use them to help plan their next trip.

[0041] As a concrete example, suppose a user takes photos in Tokyo, Kyoto, and Osaka during a visit to Japan. The server retrieves location and time information from these photos and automatically determines the overall travel route. Based on this process, it generates a detailed travel record for each city visited and provides it to the user in map and text format. Furthermore, it suggests a new travel route to the user that incorporates unvisited landmarks and highly-rated places (for example, unvisited temples in Kyoto or restaurants in Osaka). This allows the user to create a travel plan that offers new experiences on their next visit.

[0042] Through this system, travelers can obtain attractive future travel plans while minimizing the effort required to organize their travel history. In addition, travel plans are valued for being helpful to others, promoting interaction within the community.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user uses their device to select image data taken during their trip. They can select multiple photos at once and prepare them for upload.

[0046] Step 2:

[0047] The device uploads the selected image data to the server. Once the upload begins, the device sends the image data to the server one by one and displays the progress to the user.

[0048] Step 3:

[0049] The server analyzes metadata from the received image data to extract location information (GPS coordinates) and time information (timestamp). This identifies the location and time each photograph was taken, and the data is stored in a database.

[0050] Step 4:

[0051] Based on the location and time information extracted by the server, the user's visited locations and their order are organized. This organization includes creating routes based on the chronological order and geographical relationships of the visited spots.

[0052] Step 5:

[0053] Based on the information compiled by the server, a travel log (travel diary) is automatically generated. This log includes a list of places visited, the route on a map, and story text corresponding to the time spent at each location.

[0054] Step 6:

[0055] The device displays the generated travel record to the user. The user reviews it and can edit comments and add information as needed.

[0056] Step 7:

[0057] The server retrieves publicly available rating information from external sources and, based on the user's travel records, proposes travel routes that include new highly-rated spots. This proposed route is then sent to the user.

[0058] Step 8:

[0059] Users can review new travel routes and use them to plan their next trip. Additionally, if a suggested route is rated by other users, those ratings are compiled by the server.

[0060] Step 9:

[0061] The server calculates a reward for the creator of the proposed travel route based on the evaluation results and notifies the user of the details. The reward is provided as points or a monetary incentive.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] Travelers often take a large number of photos during their trips, and organizing them later to create a travel log is time-consuming and laborious. Furthermore, it can be difficult to find new destinations or routes when planning future trips. Additionally, it's challenging to know how one's own travel plans are perceived by others.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and the order of visits based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and the order of visits. This makes it possible for users to easily create a travel record and use it to help plan their next trip.

[0067] "Image data containing information" refers to digital image files that include metadata such as location and time information.

[0068] "Extraction method" refers to the software or algorithm used to extract necessary metadata from image data.

[0069] "Means for organizing visited locations and order of visits" refers to a function that arranges visited locations according to the order in which the traveler visited them, based on extracted location and time information.

[0070] "Methods for automatically generating travel records" refer to methods for automatically creating a record of a trip's progress and content as text or graphics, based on organized visit data.

[0071] A "generative AI model" is an artificial intelligence system that uses machine learning techniques to generate new text and images.

[0072] A "prompt" is a text of instructions or questions that serves as the starting point for a generative AI model when generating information.

[0073] "Publicly available evaluation information" refers to evaluation data of tourist destinations and facilities provided by third parties that is generally accessible to the public.

[0074] "Means for suggesting new destinations and travel routes" refers to a function that suggests new destinations and routes for the next trip based on existing travel data and evaluation information.

[0075] A "terminal" refers to an electronic device that provides an interface for tasks such as uploading image data and displaying travel records.

[0076] "Means for calculating rewards" refers to a calculation function that calculates rewards based on evaluations made by other users when a proposed travel route is evaluated by them.

[0077] This invention is a system that automatically generates travel records and suggests new travel plans by utilizing image data taken by travelers during their trips. This system consists of server, terminal, and user components.

[0078] The server receives image data sent by the user. This image data contains metadata, including location and time information, as a digital file. The server uses Python libraries such as ExifRead to extract the necessary metadata from the images. Based on this data, it identifies the locations where the images were taken and their order, and builds a list of visited places. Next, the server uses a generative AI model to generate this information as a travel record in text format. Furthermore, it suggests new places to visit and travel routes to the user using prompts. Using prompts such as "Suggest new sights to visit on my next trip to Kyoto," it is possible to enrich travel planning.

[0079] The terminal provides an interface for users to upload image data taken by travelers. The terminal can be a smartphone or personal computer, where users select images and upload them to the server. The terminal also functions as a tool to display the generated travel record to the user, allowing them to review the record and add or modify information as needed.

[0080] This system allows users to easily create their own travel records and receive suggestions to help plan their next trip. By uploading images taken during their trip to the server via their device, they can obtain a record of the places they visited. Users can add their own comments to the generated travel records and further enrich their future travels by referring to new travel plan suggestions from the server.

[0081] For example, if a user has photos taken during a trip in Japan, the server extracts location and time information from these photos and creates a travel record based on the order in which the visits were made. For instance, it can suggest a next travel plan that includes landmarks the user hasn't yet visited, thus broadening the scope of their travel experience.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The user selects image files taken during their trip on their device. As input, the user provides image data taken with a digital camera or smartphone to the device's upload interface. The device sends the selected images and their metadata (location and time information) to the server using the HTTPS protocol. This completes the secure transfer of the images.

[0085] Step 2:

[0086] The server analyzes the image data received from the user. Image files and metadata are provided as input. The server uses the Python ExifRead library to extract GPS information and timestamps from the images. During this process, the server constructs a data frame that identifies the locations where the images were taken and their time order. An initial list of visited locations is generated as output of this process.

[0087] Step 3:

[0088] The server generates travel records based on visit data. It uses the list of visited locations constructed in step 2 as input. A generative AI model is employed to generate detailed text about each visited location. Data processing in this process includes text generation using a natural language generation model. This allows the user to obtain detailed travel records for each visited location.

[0089] Step 4:

[0090] The terminal displays the generated travel record to the user. It receives the travel record in text format from the server as input. The user can view and edit the travel record through the terminal's interface. This interface is often provided as a web page using HTML or JavaScript (registered trademark). The user can add comments about visited locations and save the information back to the server.

[0091] Step 5:

[0092] The server suggests new travel plans based on the user's travel history and publicly available ratings. It uses a database of places the user has visited and their ratings as input, and generates suggestions using a generative AI model and prompts. The output could be something like, "Suggest new sights to visit on my next trip to Kyoto." This allows the user to gain ideas for creating new itineraries.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] There is a growing need for systems that efficiently utilize photos taken by travelers during their trips to automatically generate travel records and suggest new travel plans. However, conventional systems have problems such as difficulty in organizing visited locations and the inability to receive real-time suggestions that are useful for future travel planning. Furthermore, it has been difficult to provide optimal travel routes in tourist destinations tailored to individual travelers, and incentives based on the social reputation of visited areas have not been realized.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and their order based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and their order. This allows travelers to easily generate travel records based on photographs they have taken and receive suggestions for new places to visit. Furthermore, by linking with urban infrastructure, it becomes possible to provide optimal routes in real time, enriching the travel experience.

[0098] "Image data containing information" refers to photographs and visual records that have location and time information embedded in them.

[0099] "Location information" refers to geographical coordinate data, which indicates where an image was taken.

[0100] "Time information" refers to data that indicates the date and time when a specific event or action occurred.

[0101] "Places visited and order of visits" refers to the places a traveler visited and the order in which they were visited.

[0102] A "travel log" is a record of places visited during a trip, the order in which they were visited, and events that occurred, organized and documented in that way.

[0103] The "means of suggestion" refers to a mechanism that generates and presents new destinations and routes based on user data.

[0104] "Integration with urban infrastructure" refers to the process by which systems share information with urban transportation, tourist facilities, public services, and other related elements.

[0105] "Providing information in real time" means presenting information immediately in response to the user's current situation and requests.

[0106] "Being evaluated and having compensation calculated based on that evaluation" means quantifying the value of a proposal based on feedback from other users and calculating compensation accordingly.

[0107] "User's portable device" refers to an electronic device that a user can carry with them and that allows for the input and output of information, such as a smartphone or tablet.

[0108] A "third-party information source" refers to a source of information provided from an external party other than the user or developer.

[0109] "Past activity history" refers to data about the user's past activities and visit records.

[0110] The system for implementing this invention primarily involves three elements: a server, a terminal, and a user. The server receives image data containing information captured by the user and extracts location and time information from the data. This makes it possible to identify and organize where and when the image data was taken. The server further lists the visited locations and their order based on the extracted information and automatically generates a travel record based on this list.

[0111] The terminal is a user's portable device, such as a smartphone or tablet. This terminal acts as a link between the user and the server, providing an interface for uploading captured images. On the terminal, users can view their generated travel records, and new places to visit and travel routes are suggested based on publicly available evaluation information.

[0112] Users send images they take using the system to the server via their device. This allows them to view automatically generated travel records and check suggested routes that have been rated by others. Furthermore, integration with urban infrastructure enables real-time route suggestions, making actual travel smoother and more fulfilling.

[0113] As a concrete example, when a user uploads photos taken at tourist destinations during their trip to the app, the system analyzes the time and location information obtained from the images and suggests places to visit next based on their visit history. In doing so, it takes into account the city's transportation network and the congestion levels of tourist spots to present the optimal travel route in real time.

[0114] An example of a prompt message to use a generative AI model to suggest new routes is: "A traveler who has already visited Tokyo and Osaka should suggest places to visit on their next trip. Please include tourist destinations in Hokkaido in the new plan."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The device receives an image taken by the user. This image contains Exif data, including location and time information. The input data is a photo file, and the output remains unchanged but is ready to be sent to the next processing step.

[0118] Step 2:

[0119] The terminal sends the received image data to the server. The server receives this data and begins data analysis. The input is an image file, and the output is the state in which the server is ready for data analysis.

[0120] Step 3:

[0121] The server extracts Exif information from image data to obtain location and time information. Based on this information, it identifies the visited locations and their order. The input is image data, and the output is an organized list of visited locations. The server uses software such as a geolocation API to analyze the location information.

[0122] Step 4:

[0123] The server automatically generates travel records based on extracted information about visited locations. The input is an organized list of visited locations, and the output is the generated travel record. The generated record is presented in text and map formats and used for further analysis.

[0124] Step 5:

[0125] The server uses a generative AI model to suggest new destinations and travel routes suitable for the user based on publicly available evaluation information. Inputs are travel records and publicly available evaluation information, and output is a proposed new travel route. The AI ​​model generates suggestions based on prompt statements.

[0126] Step 6:

[0127] The server works in conjunction with urban infrastructure to send optimized suggestions to the user's terminal in real time. The input is the suggestion data from the previous step, and the output is real-time updated travel route information. Dynamic data such as traffic conditions are also taken into consideration during this process.

[0128] Step 7:

[0129] Users can receive suggested plans on their devices and use them to plan their next trip. The input is suggested data from the server, and the output is a new travel plan for the user. This allows travelers to enjoy a richer and more efficient travel experience.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention is a system that utilizes image data taken by travelers during their trips to recognize the user's emotions from the information within the images, and then automatically generates travel records or suggests new travel plans based on those emotions. By combining this system with an emotion engine, it is possible to provide new value based on the user's emotions in addition to the usual visit history.

[0132] The server receives image data uploaded by the user and first extracts location and time information. This provides data that organizes where and how the traveler visited. Next, an emotion engine is used to analyze the facial expressions and context of people in the images to determine the user's emotional state. This information serves as additional information indicating what emotions the traveler felt at each location.

[0133] The device is used for uploading image data and also functions as a tool for reviewing travel records that include emotional data. Users use the device to send photos they've taken to the server and then visually review the generated travel records. The provided travel records include estimated emotional information for each visited location, vividly reflecting the richness of the travel experience.

[0134] Through this system, users can re-examine their own travel experiences and plan trips with a fresh perspective. For example, based on a user's enjoyable experience at a particular tourist destination, the server can suggest other highly-rated spots with a similar atmosphere, based on emotional data. By considering past emotions when selecting a user's next travel destination, they can plan a more satisfying trip.

[0135] For example, by analyzing facial expressions in photos taken by users in various locations, if there are a large number of smiles in a particular city, the system will determine that the user had a highly positive experience in that city. The emotion engine will then suggest other locations with a similar atmosphere, allowing users to incorporate new emotional experiences into their next trip. In addition, this data can be used to provide travel guidance to other users and also functions as emotion-based word-of-mouth information.

[0136] In this way, this system not only uses users' visit history but also subjective data such as emotions to create new travel value and contribute to deepening communication among travelers.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] The user uses their device to select image data taken during their trip. The user chooses multiple image files from their smartphone or computer and prepares to complete the upload to the system.

[0140] Step 2:

[0141] The device uploads the selected image data to the server. The device sequentially sends the image files to the server via the internet, and the user is shown the progress in real time.

[0142] Step 3:

[0143] The server analyzes metadata from the received image data to extract location and time information. This allows it to identify which locations were visited and when, and then organize and store this information in a database.

[0144] Step 4:

[0145] The server uses an emotion engine to analyze the facial expressions and elements of the people in each image and estimate the user's emotions. For example, images with many smiles are recorded as a positive experience.

[0146] Step 5:

[0147] The server automatically generates a travel log based on location information, time information, and estimated sentiment data. The travel log is compiled as the order of visited places, the user's sentiment state, and comments for each visited place.

[0148] Step 6:

[0149] The device displays the generated travel record to the user. The user can review the displayed record and edit any additional comments or notes as needed.

[0150] Step 7:

[0151] Based on sentiment data collected by the server, the system suggests new travel routes that include highly-rated spots the user hasn't visited yet. These routes are selected based on the user's past positive emotional experiences.

[0152] Step 8:

[0153] Users can review suggested new travel routes and use them to plan their next trip. They can also refer to ratings from other users to determine which spots offer the highest emotional satisfaction.

[0154] Step 9:

[0155] The server compiles evaluations of the proposed new travel routes based on feedback provided by the user and calculates a reward based on those evaluations. The server then notifies the user of the details of the reward.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0158] While there are many ways for travelers to record the places they visit, there is a lack of systems that capture the emotions and experiences of those visits in detail. Existing travel recording systems focus on physical location and time information, and do not reflect the subjective value of how users felt at a place. This results in a challenge in improving travel satisfaction and suggesting new travel plans.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for analyzing the extracted image data and recognizing the user's emotions using an emotion engine; and means for suggesting similar places to visit and travel routes based on the emotion recognition data of the generated travel record. This makes it possible to generate detailed travel records based on emotional information during travel and to suggest new travel plans.

[0161] "Image data" refers to data containing digital visual information captured by a user.

[0162] "Location information" refers to geographical data that indicates the location where an image was taken, as contained within the image data.

[0163] "Time information" refers to data included in image data that indicates the date and time of shooting.

[0164] An "emotion engine" is a technology equipped with an algorithm that analyzes facial expressions and context within an image to determine the user's emotions.

[0165] A "travel record" is data that represents an individual travel experience, generated by integrating the user's visited locations, the order of visits, and emotion recognition data.

[0166] "Similar destinations" are new places to visit that are suggested based on the user's sentiment data from places they have visited in the past, suggesting that they have a similar atmosphere and experience.

[0167] "Travel route" refers to information that shows the path a user will take during their trip and provides directions for moving to their next destination.

[0168] "Emotion recognition data" refers to data that quantifies or categorizes the emotional state of a user, as analyzed using an emotion engine.

[0169] This invention is a system that analyzes image data taken by travelers, recognizes the location information, time information, and user emotions contained within it, and generates and suggests travel records.

[0170] The server receives image data uploaded from the user's device and automatically extracts location and time information from that data. This is done using a database management system and image processing libraries. For example, the server obtains location information by analyzing EXIF ​​data and records when a traveler visited a specific location.

[0171] Next, the server uses an emotion engine to analyze the image data and recognize the user's emotions. This process utilizes deep learning technology and image analysis software to determine emotions such as "joy" and "surprise" from the facial expressions and background of the people in the image.

[0172] This information is integrated and provided to the user via their device as a travel log. The travel log includes estimated emotional information for each place visited, allowing the user to see in chronological order what emotions they felt at each location.

[0173] Users can refer to their generated travel records and have the server suggest new travel plans based on their emotional data. For example, if a user inputs, "Tell me about new tourist destinations that have scenery similar to cities I've enjoyed in the past," the system will refer to the emotional data and use a generative AI model to suggest places with a similar atmosphere. This allows users to create personalized travel plans based on their emotions.

[0174] An example of a prompt would be, "For a traveler who smiled a lot in Paris, what would you recommend as their next travel destination?" Through this prompt, the system can provide suggestions that take into account the user's past emotional data.

[0175] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0176] Step 1:

[0177] The terminal selects image data captured by the user and uploads it to the server. At this time, the terminal displays a file selection dialog, allowing the user to choose the image to send, and then transmits the data to the server via the internet. The input is the image file selected by the user, and the output is the transfer of the image data to the server.

[0178] Step 2:

[0179] The server extracts location and time information from the received image data. Using a dedicated image processing library, the server analyzes the EXIF ​​data within the image file to obtain the shooting location (e.g., GPS coordinates) and the date and time of shooting. The input is the uploaded image data, and the output is a dataset of location and time information.

[0180] Step 3:

[0181] The server uses an emotion engine to recognize the emotions of people in an image. The server runs a deep learning-based facial expression analysis model and assigns emotion labels such as "joy" or "surprise" to the image. The input is the image to be analyzed, and the output is the emotion label and its confidence level.

[0182] Step 4:

[0183] The server integrates the extracted location, time, and emotion labels to generate a travel record. The server then organizes this into a time-series data set, creating a travel record that details the emotions the user experienced at different locations. The input consists of the aforementioned location, time, and emotion labels, while the output is a dataset of travel records.

[0184] Step 5:

[0185] The terminal presents the generated travel record to the user. The terminal uses a visually appealing interface to plot the travel record on a map, allowing for visual confirmation. The input is travel record data received from the server, and the output is a visual presentation of the travel record to the user.

[0186] Step 6:

[0187] The server proposes a new travel plan based on travel records and user sentiment data. Using a generative AI model, the server searches for places to visit that offer similar emotional experiences and presents recommended travel destinations to the user. Input is the generated travel record and prompt text, and output is the new travel plan.

[0188] (Application Example 2)

[0189] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0190] In today's information-driven society, there is a need to appropriately understand individual emotions and preferences in travel experiences and dining services, and to provide new experiences and suggestions based on that understanding. However, conventional systems are limited to visit history and publicly available evaluation information, and have been unable to utilize users' subjective emotions. Therefore, technology is needed to provide more personalized suggestions.

[0191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0192] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and order of visits based on the extracted location information and time information; means for automatically generating a travel record based on the visited places and order of visits; means for analyzing the image data and determining the emotional state; means for suggesting candidates for the next visit based on the emotional state; and means for having the suggested candidates evaluated by other users and improving the recommendation content based on that evaluation. This enables new and personalized suggestions based on the user's emotions.

[0193] "Information-containing image data" refers to visual data associated with location and time information, and is used to analyze user activity and preferences.

[0194] "Location information" is data that indicates the geographical location where an image was taken, and it is an element that makes up a user's browsing history.

[0195] "Time information" refers to data indicating the date and time an image was taken, and is used to understand the timeline of a user's actions.

[0196] "Means for organizing visited locations and order of visits" refers to an algorithm that has the function of arranging user actions chronologically based on location and time information.

[0197] "Means for automatically generating travel records" refers to a system that records and visualizes the flow of a trip based on the user's visited locations and order.

[0198] A "means for determining emotional state" refers to an algorithm that uses analytical techniques to identify emotions from a person's facial expressions and situation within an image.

[0199] The "method for suggesting potential next visit locations" is a feature that suggests new places that might interest the user based on their past emotional data.

[0200] "Methods for improving recommendations based on evaluation" refers to the process of collecting feedback from other users and using that feedback to improve the accuracy of future recommendations.

[0201] The core of this system is a server. The server receives image data containing information uploaded by the user from their terminal. This image data is processed using an image analysis engine, and location and time information is extracted. This data is used to organize the user's visited locations and their order, and to automatically generate a travel record.

[0202] The server also operates an emotion analysis engine, which analyzes the facial expressions and backgrounds of people in image data to determine their emotional state. This emotion data is used to enrich the user's experience. For example, if many "happy" emotions are detected in photos taken by a user at a particular place they visited, it is presumed that the place is highly rated by the user. Based on this data, the server generates potential destinations for the next visit within the suggestion environment system, based on past emotion data.

[0203] Based on feedback from other users, the server collects evaluation information for the evaluated proposal candidates, and the system's algorithm continuously improves the recommendations.

[0204] This entire process utilizes Python programs running on a cloud server. OpenCV is used for image analysis, and TENSORFLOW® is used for sentiment analysis. Furthermore, the user terminal can easily be a common computing device such as a smartphone or tablet.

[0205] As a concrete example, a user takes photos with their smartphone while traveling and uploads them to the server from the terminal app. By using a prompt message such as, "Please upload a photo expressing your feelings while eating pasta. We will suggest a recommended menu item for your next visit," users can easily provide data through the user interface.

[0206] In this way, it becomes possible to provide new user experiences based on emotions throughout the entire system.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The user's device collects image data. The user takes photos using their smartphone while traveling or eating, and saves those images to the application. The input is the image captured by the camera. The output is the image file saved in the application.

[0210] Step 2:

[0211] The device uploads image data to the server. The device sends the image selected by the user to the server. The input is the image file stored on the device. The output is the image data sent to the server.

[0212] Step 3:

[0213] The server extracts location and time information. The server obtains the shooting location and date / time by analyzing Exif information from the received image data. The input is the image data sent to the server. The output is metadata including location and time information.

[0214] Step 4:

[0215] The server performs emotion analysis. Using TensorFlow, the server identifies faces in an image and estimates emotions from facial expressions. The input is image data sent to the server. The output is numerical data representing the estimated emotional state.

[0216] Step 5:

[0217] The server generates a travel record. The server combines extracted location, time, and sentiment data to create a travel record summarizing the user's visit history. Inputs are location, time, and sentiment data. Output is the generated travel record.

[0218] Step 6:

[0219] The server suggests potential destinations for your next visit. Using an algorithm based on past sentiment data, the server generates new suggestions similar to places the user has highly rated. The input is the generated travel records. The output is a list of suggested destinations.

[0220] Step 7:

[0221] The server collects and improves evaluations of candidate sites. The server also collects feedback from other users and strengthens the proposed algorithm based on that data. The input is evaluation data from other users. The output is the updated proposed algorithm.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention is a system that effectively utilizes image data taken by travelers during their trips, automatically generates travel records, and proposes new travel plans. The core of this system consists of a series of operations involving a server, terminals, and users.

[0239] The server receives image data uploaded by the user. This includes typical digital images and associated metadata. The server extracts location and time information from the image files and uses it to organize which places were visited and in what order. This process creates a list of places the traveler visited during their trip.

[0240] The terminal is a device that provides the user with an interface for uploading image data. This typically includes a smartphone or computer. The user selects photos taken during their trip and sends them to the server via the terminal. Once the upload is complete, the user can view the analysis results on the server. The terminal also serves as a tool for displaying the generated travel record (travel diary), allowing the user to view this information and add additional information as needed.

[0241] Users can use the system to automatically create their own travel records and obtain information to help plan their next trip. By uploading photos taken during their trip to the server via their device, users can digitize and organize their travel experiences. In this process, users can review the generated travel diary and add comments about the places they visited. They can also check new travel routes suggested by the server and use them to help plan their next trip.

[0242] As a concrete example, suppose a user takes photos in Tokyo, Kyoto, and Osaka during a visit to Japan. The server retrieves location and time information from these photos and automatically determines the overall travel route. Based on this process, it generates a detailed travel record for each city visited and provides it to the user in map and text format. Furthermore, it suggests a new travel route to the user that incorporates unvisited landmarks and highly-rated places (for example, unvisited temples in Kyoto or restaurants in Osaka). This allows the user to create a travel plan that offers new experiences on their next visit.

[0243] Through this system, travelers can obtain attractive future travel plans while minimizing the effort required to organize their travel history. In addition, travel plans are valued for being helpful to others, promoting interaction within the community.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The user uses their device to select image data taken during their trip. They can select multiple photos at once and prepare them for upload.

[0247] Step 2:

[0248] The device uploads the selected image data to the server. Once the upload begins, the device sends the image data to the server one by one and displays the progress to the user.

[0249] Step 3:

[0250] The server analyzes metadata from the received image data to extract location information (GPS coordinates) and time information (timestamp). This identifies the location and time each photograph was taken, and the data is stored in a database.

[0251] Step 4:

[0252] Based on the location and time information extracted by the server, the user's visited locations and their order are organized. This organization includes creating routes based on the chronological order and geographical relationships of the visited spots.

[0253] Step 5:

[0254] Based on the information compiled by the server, a travel log (travel diary) is automatically generated. This log includes a list of places visited, the route on a map, and story text corresponding to the time spent at each location.

[0255] Step 6:

[0256] The device displays the generated travel record to the user. The user reviews it and can edit comments and add information as needed.

[0257] Step 7:

[0258] The server retrieves publicly available rating information from external sources and, based on the user's travel records, proposes travel routes that include new highly-rated spots. This proposed route is then sent to the user.

[0259] Step 8:

[0260] Users can review new travel routes and use them to plan their next trip. Additionally, if a suggested route is rated by other users, those ratings are compiled by the server.

[0261] Step 9:

[0262] The server calculates a reward for the creator of the proposed travel route based on the evaluation results and notifies the user of the details. The reward is provided as points or a monetary incentive.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] Travelers often take a large number of photos during their trips, and organizing them later to create a travel log is time-consuming and laborious. Furthermore, it can be difficult to find new destinations or routes when planning future trips. Additionally, it's challenging to know how one's own travel plans are perceived by others.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and the order of visits based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and the order of visits. This makes it possible for users to easily create a travel record and use it to help plan their next trip.

[0268] "Image data containing information" refers to digital image files that include metadata such as location and time information.

[0269] "Extraction method" refers to the software or algorithm used to extract necessary metadata from image data.

[0270] "Means for organizing visited locations and order of visits" refers to a function that arranges visited locations according to the order in which the traveler visited them, based on extracted location and time information.

[0271] "Methods for automatically generating travel records" refer to methods for automatically creating a record of a trip's progress and content as text or graphics, based on organized visit data.

[0272] A "generative AI model" is an artificial intelligence system that uses machine learning techniques to generate new text and images.

[0273] A "prompt" is a text of instructions or questions that serves as the starting point for a generative AI model when generating information.

[0274] "Publicly available evaluation information" refers to evaluation data of tourist destinations and facilities provided by third parties that is generally accessible to the public.

[0275] "Means for suggesting new destinations and travel routes" refers to a function that suggests new destinations and routes for the next trip based on existing travel data and evaluation information.

[0276] A "terminal" refers to an electronic device that provides an interface for tasks such as uploading image data and displaying travel records.

[0277] "Means for calculating rewards" refers to a calculation function that calculates rewards based on evaluations made by other users when a proposed travel route is evaluated by them.

[0278] This invention is a system that automatically generates travel records and suggests new travel plans by utilizing image data taken by travelers during their trips. This system consists of server, terminal, and user components.

[0279] The server receives image data sent by the user. This image data contains metadata, including location and time information, as a digital file. The server uses Python libraries such as ExifRead to extract the necessary metadata from the images. Based on this data, it identifies the locations where the images were taken and their order, and builds a list of visited places. Next, the server uses a generative AI model to generate this information as a travel record in text format. Furthermore, it suggests new places to visit and travel routes to the user using prompts. Using prompts such as "Suggest new sights to visit on my next trip to Kyoto," it is possible to enrich travel planning.

[0280] The terminal provides an interface for users to upload image data taken by travelers. The terminal can be a smartphone or personal computer, where users select images and upload them to the server. The terminal also functions as a tool to display the generated travel record to the user, allowing them to review the record and add or modify information as needed.

[0281] By using this system, users can easily create their own travel records and receive suggestions that are useful for their next travel plans. By uploading the images taken during the trip to the server via the terminal, records of the visited locations can be obtained. Users can add their own comments to the generated travel records and further enrich future trips by referring to the suggestions for new travel plans from the server.

[0282] As a specific example, if a user has photos taken during a trip in Japan, the server extracts the location information and time information from these photos and creates a travel record based on the order of visit. For example, it can propose a next travel plan that includes famous places the user has not visited yet, thus expanding the breadth of the travel experience.

[0283] The flow of the specific process in Example 1 will be described using FIG. 11.

[0284] Step 1:

[0285] The user selects the image file taken during the trip on the terminal. As input, the user provides the image data taken with a digital camera or smartphone to the upload interface of the terminal. The terminal sends the selected image and its metadata (location information and time information) to the server using the HTTPS protocol. This completes the secure transfer of the image.

[0286] Step 2:

[0287] The server analyzes the image data received from the user. As input, the image file and metadata are provided. The server uses the ExifRead library in Python to extract the GPS information and timestamp from the image. In this process, the server constructs a data frame that identifies the locations where the photos were taken and their chronological order. As the output of this process, an initial list of visited locations is generated.

[0288] Step 3:

[0289] The server generates travel records based on visit data. It uses the list of visited locations constructed in step 2 as input. A generative AI model is employed to generate detailed text about each visited location. Data processing in this process includes text generation using a natural language generation model. This allows the user to obtain detailed travel records for each visited location.

[0290] Step 4:

[0291] The terminal displays the generated travel record to the user. It receives the travel record in text format from the server as input. The user can view and edit the travel record through the terminal's interface. This interface is often provided as a web page using HTML or JavaScript. The user can add comments about visited locations and save the information back to the server.

[0292] Step 5:

[0293] The server suggests new travel plans based on the user's travel history and publicly available ratings. It uses a database of places the user has visited and their ratings as input, and generates suggestions using a generative AI model and prompts. The output could be something like, "Suggest new sights to visit on my next trip to Kyoto." This allows the user to gain ideas for creating new itineraries.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] There is a growing need for systems that efficiently utilize photos taken by travelers during their trips to automatically generate travel records and suggest new travel plans. However, conventional systems have problems such as difficulty in organizing visited locations and the inability to receive real-time suggestions that are useful for future travel planning. Furthermore, it has been difficult to provide optimal travel routes in tourist destinations tailored to individual travelers, and incentives based on the social reputation of visited areas have not been realized.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and their order based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and their order. This allows travelers to easily generate travel records based on photographs they have taken and receive suggestions for new places to visit. Furthermore, by linking with urban infrastructure, it becomes possible to provide optimal routes in real time, enriching the travel experience.

[0299] "Image data containing information" refers to photographs and visual records that have location and time information embedded in them.

[0300] "Location information" refers to geographical coordinate data, which indicates where an image was taken.

[0301] "Time information" refers to data that indicates the date and time when a specific event or action occurred.

[0302] "Places visited and order of visits" refers to the places a traveler visited and the order in which they were visited.

[0303] A "travel log" is a record of places visited during a trip, the order in which they were visited, and events that occurred, organized and documented in that way.

[0304] The "proposed means" is a mechanism that generates and presents new destinations and routes based on user data.

[0305] "Cooperation with urban infrastructure" is a process in which the system shares information with urban transportation, tourist facilities, public services, etc.

[0306] "Provided in real time" means presenting information immediately in response to the user's current situation and requirements.

[0307] "Evaluated and rewarded based on the evaluation" means quantifying the value of the proposal based on feedback from other users and calculating the corresponding reward.

[0308] The "user's mobile device" is an electronic device such as a smartphone or tablet that the user can carry and can input and output information.

[0309] The "third-party information source" is the source of information provided from outside, other than the user and the developer.

[0310] The "past behavior history" is data related to the activities and visit records that the user has carried out previously.

[0311] For the system for implementing this invention, mainly three elements, namely the server, the terminal, and the user, are involved. The server receives image data including information photographed by the user, and extracts location information and time information from the data. Thereby, it becomes possible to identify and organize where and when the image data was photographed. The server further lists the visited locations and the order of visits based on the extracted information, and automatically generates a travel record based on this.

[0312] The terminal is a user's portable device, such as a smartphone or tablet. This terminal acts as a link between the user and the server, providing an interface for uploading captured images. On the terminal, users can view their generated travel records, and new places to visit and travel routes are suggested based on publicly available evaluation information.

[0313] Users send images they take using the system to the server via their device. This allows them to view automatically generated travel records and check suggested routes that have been rated by others. Furthermore, integration with urban infrastructure enables real-time route suggestions, making actual travel smoother and more fulfilling.

[0314] As a concrete example, when a user uploads photos taken at tourist destinations during their trip to the app, the system analyzes the time and location information obtained from the images and suggests places to visit next based on their visit history. In doing so, it takes into account the city's transportation network and the congestion levels of tourist spots to present the optimal travel route in real time.

[0315] An example of a prompt message to use a generative AI model to suggest new routes is: "A traveler who has already visited Tokyo and Osaka should suggest places to visit on their next trip. Please include tourist destinations in Hokkaido in the new plan."

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The device receives an image taken by the user. This image contains Exif data, including location and time information. The input data is a photo file, and the output remains unchanged but is ready to be sent to the next processing step.

[0319] Step 2:

[0320] The terminal sends the received image data to the server. The server receives this data and begins data analysis. The input is an image file, and the output is the state in which the server is ready for data analysis.

[0321] Step 3:

[0322] The server extracts Exif information from image data to obtain location and time information. Based on this information, it identifies the visited locations and their order. The input is image data, and the output is an organized list of visited locations. The server uses software such as a geolocation API to analyze the location information.

[0323] Step 4:

[0324] The server automatically generates travel records based on extracted information about visited locations. The input is an organized list of visited locations, and the output is the generated travel record. The generated record is presented in text and map formats and used for further analysis.

[0325] Step 5:

[0326] The server uses a generative AI model to suggest new destinations and travel routes suitable for the user based on publicly available evaluation information. Inputs are travel records and publicly available evaluation information, and output is a proposed new travel route. The AI ​​model generates suggestions based on prompt statements.

[0327] Step 6:

[0328] The server works in conjunction with urban infrastructure to send optimized suggestions to the user's terminal in real time. The input is the suggestion data from the previous step, and the output is real-time updated travel route information. Dynamic data such as traffic conditions are also taken into consideration during this process.

[0329] Step 7:

[0330] Users can receive suggested plans on their devices and use them to plan their next trip. The input is suggested data from the server, and the output is a new travel plan for the user. This allows travelers to enjoy a richer and more efficient travel experience.

[0331] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0332] This invention is a system that utilizes image data taken by travelers during their trips to recognize the user's emotions from the information within the images, and then automatically generates travel records or suggests new travel plans based on those emotions. By combining this system with an emotion engine, it is possible to provide new value based on the user's emotions in addition to the usual visit history.

[0333] The server receives image data uploaded by the user and first extracts location and time information. This provides data that organizes where and how the traveler visited. Next, an emotion engine is used to analyze the facial expressions and context of people in the images to determine the user's emotional state. This information serves as additional information indicating what emotions the traveler felt at each location.

[0334] The device is used for uploading image data and also functions as a tool for reviewing travel records that include emotional data. Users use the device to send photos they've taken to the server and then visually review the generated travel records. The provided travel records include estimated emotional information for each visited location, vividly reflecting the richness of the travel experience.

[0335] Through this system, users can re-examine their own travel experiences and plan trips with a fresh perspective. For example, based on a user's enjoyable experience at a particular tourist destination, the server can suggest other highly-rated spots with a similar atmosphere, based on emotional data. By considering past emotions when selecting a user's next travel destination, they can plan a more satisfying trip.

[0336] For example, by analyzing facial expressions in photos taken by users in various locations, if there are a large number of smiles in a particular city, the system will determine that the user had a highly positive experience in that city. The emotion engine will then suggest other locations with a similar atmosphere, allowing users to incorporate new emotional experiences into their next trip. In addition, this data can be used to provide travel guidance to other users and also functions as emotion-based word-of-mouth information.

[0337] In this way, this system not only uses users' visit history but also subjective data such as emotions to create new travel value and contribute to deepening communication among travelers.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The user uses their device to select image data taken during their trip. The user chooses multiple image files from their smartphone or computer and prepares to complete the upload to the system.

[0341] Step 2:

[0342] The device uploads the selected image data to the server. The device sequentially sends the image files to the server via the internet, and the user is shown the progress in real time.

[0343] Step 3:

[0344] The server analyzes metadata from the received image data to extract location and time information. This allows it to identify which locations were visited and when, and then organize and store this information in a database.

[0345] Step 4:

[0346] The server uses an emotion engine to analyze the facial expressions and elements of the people in each image and estimate the user's emotions. For example, images with many smiles are recorded as a positive experience.

[0347] Step 5:

[0348] The server automatically generates a travel log based on location information, time information, and estimated sentiment data. The travel log is compiled as the order of visited places, the user's sentiment state, and comments for each visited place.

[0349] Step 6:

[0350] The device displays the generated travel record to the user. The user can review the displayed record and edit any additional comments or notes as needed.

[0351] Step 7:

[0352] Based on sentiment data collected by the server, the system suggests new travel routes that include highly-rated spots the user hasn't visited yet. These routes are selected based on the user's past positive emotional experiences.

[0353] Step 8:

[0354] Users can review suggested new travel routes and use them to plan their next trip. They can also refer to ratings from other users to determine which spots offer the highest emotional satisfaction.

[0355] Step 9:

[0356] The server compiles evaluations of the proposed new travel routes based on feedback provided by the user and calculates a reward based on those evaluations. The server then notifies the user of the details of the reward.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] While there are many ways for travelers to record the places they visit, there is a lack of systems that capture the emotions and experiences of those visits in detail. Existing travel recording systems focus on physical location and time information, and do not reflect the subjective value of how users felt at a place. This results in a challenge in improving travel satisfaction and suggesting new travel plans.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for analyzing the extracted image data and recognizing the user's emotions using an emotion engine; and means for suggesting similar places to visit and travel routes based on the emotion recognition data of the generated travel record. This makes it possible to generate detailed travel records based on emotional information during travel and to suggest new travel plans.

[0362] "Image data" refers to data containing digital visual information captured by a user.

[0363] "Location information" refers to geographical data that indicates the location where an image was taken, as contained within the image data.

[0364] "Time information" refers to data included in image data that indicates the date and time of shooting.

[0365] An "emotion engine" is a technology equipped with an algorithm that analyzes facial expressions and context within an image to determine the user's emotions.

[0366] A "travel record" is data that represents an individual travel experience, generated by integrating the user's visited locations, the order of visits, and emotion recognition data.

[0367] "Similar destinations" are new places to visit that are suggested based on the user's sentiment data from places they have visited in the past, suggesting that they have a similar atmosphere and experience.

[0368] "Travel route" refers to information that shows the path a user will take during their trip and provides directions for moving to their next destination.

[0369] "Emotion recognition data" refers to data that quantifies or categorizes the emotional state of a user, as analyzed using an emotion engine.

[0370] This invention is a system that analyzes image data taken by travelers, recognizes the location information, time information, and user emotions contained within it, and generates and suggests travel records.

[0371] The server receives image data uploaded from the user's device and automatically extracts location and time information from that data. This is done using a database management system and image processing libraries. For example, the server obtains location information by analyzing EXIF ​​data and records when a traveler visited a specific location.

[0372] Next, the server uses an emotion engine to analyze the image data and recognize the user's emotions. This process utilizes deep learning technology and image analysis software to determine emotions such as "joy" and "surprise" from the facial expressions and background of the people in the image.

[0373] This information is integrated and provided to the user via their device as a travel log. The travel log includes estimated emotional information for each place visited, allowing the user to see in chronological order what emotions they felt at each location.

[0374] Users can refer to their generated travel records and have the server suggest new travel plans based on their emotional data. For example, if a user inputs, "Tell me about new tourist destinations that have scenery similar to cities I've enjoyed in the past," the system will refer to the emotional data and use a generative AI model to suggest places with a similar atmosphere. This allows users to create personalized travel plans based on their emotions.

[0375] An example of a prompt would be, "For a traveler who smiled a lot in Paris, what would you recommend as their next travel destination?" Through this prompt, the system can provide suggestions that take into account the user's past emotional data.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The terminal selects image data captured by the user and uploads it to the server. At this time, the terminal displays a file selection dialog, allowing the user to choose the image to send, and then transmits the data to the server via the internet. The input is the image file selected by the user, and the output is the transfer of the image data to the server.

[0379] Step 2:

[0380] The server extracts location and time information from the received image data. Using a dedicated image processing library, the server analyzes the EXIF ​​data within the image file to obtain the shooting location (e.g., GPS coordinates) and the date and time of shooting. The input is the uploaded image data, and the output is a dataset of location and time information.

[0381] Step 3:

[0382] The server uses an emotion engine to recognize the emotions of people in an image. The server runs a deep learning-based facial expression analysis model and assigns emotion labels such as "joy" or "surprise" to the image. The input is the image to be analyzed, and the output is the emotion label and its confidence level.

[0383] Step 4:

[0384] The server integrates the extracted location, time, and emotion labels to generate a travel record. The server then organizes this into a time-series data set, creating a travel record that details the emotions the user experienced at different locations. The input consists of the aforementioned location, time, and emotion labels, while the output is a dataset of travel records.

[0385] Step 5:

[0386] The terminal presents the generated travel record to the user. The terminal uses a visually appealing interface to plot the travel record on a map, allowing for visual confirmation. The input is travel record data received from the server, and the output is a visual presentation of the travel record to the user.

[0387] Step 6:

[0388] The server proposes a new travel plan based on travel records and user sentiment data. Using a generative AI model, the server searches for places to visit that offer similar emotional experiences and presents recommended travel destinations to the user. Input is the generated travel record and prompt text, and output is the new travel plan.

[0389] (Application Example 2)

[0390] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0391] In today's information-driven society, there is a need to appropriately understand individual emotions and preferences in travel experiences and dining services, and to provide new experiences and suggestions based on that understanding. However, conventional systems are limited to visit history and publicly available evaluation information, and have been unable to utilize users' subjective emotions. Therefore, technology is needed to provide more personalized suggestions.

[0392] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0393] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and order of visits based on the extracted location information and time information; means for automatically generating a travel record based on the visited places and order of visits; means for analyzing the image data and determining the emotional state; means for suggesting candidates for the next visit based on the emotional state; and means for having the suggested candidates evaluated by other users and improving the recommendation content based on that evaluation. This enables new and personalized suggestions based on the user's emotions.

[0394] "Information-containing image data" refers to visual data associated with location and time information, and is used to analyze user activity and preferences.

[0395] "Location information" is data that indicates the geographical location where an image was taken, and it is an element that makes up a user's browsing history.

[0396] "Time information" refers to data indicating the date and time an image was taken, and is used to understand the timeline of a user's actions.

[0397] "Means for organizing visited locations and order of visits" refers to an algorithm that has the function of arranging user actions chronologically based on location and time information.

[0398] "Means for automatically generating travel records" refers to a system that records and visualizes the flow of a trip based on the user's visited locations and order.

[0399] A "means for determining emotional state" refers to an algorithm that uses analytical techniques to identify emotions from a person's facial expressions and situation within an image.

[0400] The "method for suggesting potential next visit locations" is a feature that suggests new places that might interest the user based on their past emotional data.

[0401] "Methods for improving recommendations based on evaluation" refers to the process of collecting feedback from other users and using that feedback to improve the accuracy of future recommendations.

[0402] The core of this system is a server. The server receives image data containing information uploaded by the user from their terminal. This image data is processed using an image analysis engine, and location and time information is extracted. This data is used to organize the user's visited locations and their order, and to automatically generate a travel record.

[0403] The server also operates an emotion analysis engine, which analyzes the facial expressions and backgrounds of people in image data to determine their emotional state. This emotion data is used to enrich the user's experience. For example, if many "happy" emotions are detected in photos taken by a user at a particular place they visited, it is presumed that the place is highly rated by the user. Based on this data, the server generates potential destinations for the next visit within the suggestion environment system, based on past emotion data.

[0404] Based on feedback from other users, the server collects evaluation information for the evaluated proposal candidates, and the system's algorithm continuously improves the recommendations.

[0405] This entire process utilizes Python programs running on a cloud server. OpenCV is used for image analysis, and TensorFlow for sentiment analysis. Furthermore, user devices such as smartphones and tablets are perfectly adequate.

[0406] As a concrete example, a user takes photos with their smartphone while traveling and uploads them to the server from the terminal app. By using a prompt message such as, "Please upload a photo expressing your feelings while eating pasta. We will suggest a recommended menu item for your next visit," users can easily provide data through the user interface.

[0407] In this way, it becomes possible to provide new user experiences based on emotions throughout the entire system.

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The user's device collects image data. The user takes photos using their smartphone while traveling or eating, and saves those images to the application. The input is the image captured by the camera. The output is the image file saved in the application.

[0411] Step 2:

[0412] The device uploads image data to the server. The device sends the image selected by the user to the server. The input is the image file stored on the device. The output is the image data sent to the server.

[0413] Step 3:

[0414] The server extracts location and time information. The server obtains the shooting location and date / time by analyzing Exif information from the received image data. The input is the image data sent to the server. The output is metadata including location and time information.

[0415] Step 4:

[0416] The server performs emotion analysis. Using TensorFlow, the server identifies faces in an image and estimates emotions from facial expressions. The input is image data sent to the server. The output is numerical data representing the estimated emotional state.

[0417] Step 5:

[0418] The server generates a travel record. The server combines extracted location, time, and sentiment data to create a travel record summarizing the user's visit history. Inputs are location, time, and sentiment data. Output is the generated travel record.

[0419] Step 6:

[0420] The server suggests potential destinations for your next visit. Using an algorithm based on past sentiment data, the server generates new suggestions similar to places the user has highly rated. The input is the generated travel records. The output is a list of suggested destinations.

[0421] Step 7:

[0422] The server collects and improves evaluations of candidate sites. The server also collects feedback from other users and strengthens the proposed algorithm based on that data. The input is evaluation data from other users. The output is the updated proposed algorithm.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] This invention is a system that effectively utilizes image data taken by travelers during their trips, automatically generates travel records, and proposes new travel plans. The core of this system consists of a series of operations involving a server, terminals, and users.

[0440] The server receives image data uploaded by the user. This includes typical digital images and associated metadata. The server extracts location and time information from the image files and uses it to organize which places were visited and in what order. This process creates a list of places the traveler visited during their trip.

[0441] The terminal is a device that provides the user with an interface for uploading image data. This typically includes a smartphone or computer. The user selects photos taken during their trip and sends them to the server via the terminal. Once the upload is complete, the user can view the analysis results on the server. The terminal also serves as a tool for displaying the generated travel record (travel diary), allowing the user to view this information and add additional information as needed.

[0442] Users can use the system to automatically create their own travel records and obtain information to help plan their next trip. By uploading photos taken during their trip to the server via their device, users can digitize and organize their travel experiences. In this process, users can review the generated travel diary and add comments about the places they visited. They can also check new travel routes suggested by the server and use them to help plan their next trip.

[0443] As a concrete example, suppose a user takes photos in Tokyo, Kyoto, and Osaka during a visit to Japan. The server retrieves location and time information from these photos and automatically determines the overall travel route. Based on this process, it generates a detailed travel record for each city visited and provides it to the user in map and text format. Furthermore, it suggests a new travel route to the user that incorporates unvisited landmarks and highly-rated places (for example, unvisited temples in Kyoto or restaurants in Osaka). This allows the user to create a travel plan that offers new experiences on their next visit.

[0444] Through this system, travelers can obtain attractive future travel plans while minimizing the effort required to organize their travel history. In addition, travel plans are valued for being helpful to others, promoting interaction within the community.

[0445] The following describes the processing flow.

[0446] Step 1:

[0447] The user uses their device to select image data taken during their trip. They can select multiple photos at once and prepare them for upload.

[0448] Step 2:

[0449] The device uploads the selected image data to the server. Once the upload begins, the device sends the image data to the server one by one and displays the progress to the user.

[0450] Step 3:

[0451] The server analyzes metadata from the received image data to extract location information (GPS coordinates) and time information (timestamp). This identifies the location and time each photograph was taken, and the data is stored in a database.

[0452] Step 4:

[0453] Based on the location and time information extracted by the server, the user's visited locations and their order are organized. This organization includes creating routes based on the chronological order and geographical relationships of the visited spots.

[0454] Step 5:

[0455] Based on the information compiled by the server, a travel log (travel diary) is automatically generated. This log includes a list of places visited, the route on a map, and story text corresponding to the time spent at each location.

[0456] Step 6:

[0457] The device displays the generated travel record to the user. The user reviews it and can edit comments and add information as needed.

[0458] Step 7:

[0459] The server retrieves publicly available rating information from external sources and, based on the user's travel records, proposes travel routes that include new highly-rated spots. This proposed route is then sent to the user.

[0460] Step 8:

[0461] Users can review new travel routes and use them to plan their next trip. Additionally, if a suggested route is rated by other users, those ratings are compiled by the server.

[0462] Step 9:

[0463] The server calculates a reward for the creator of the proposed travel route based on the evaluation results and notifies the user of the details. The reward is provided as points or a monetary incentive.

[0464] (Example 1)

[0465] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] Travelers often take a large number of photos during their trips, and organizing them later to create a travel log is time-consuming and laborious. Furthermore, it can be difficult to find new destinations or routes when planning future trips. Additionally, it's challenging to know how one's own travel plans are perceived by others.

[0467] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0468] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and the order of visits based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and the order of visits. This makes it possible for users to easily create a travel record and use it to help plan their next trip.

[0469] "Image data containing information" refers to digital image files that include metadata such as location and time information.

[0470] "Extraction method" refers to the software or algorithm used to extract necessary metadata from image data.

[0471] "Means for organizing visited locations and order of visits" refers to a function that arranges visited locations according to the order in which the traveler visited them, based on extracted location and time information.

[0472] "Methods for automatically generating travel records" refer to methods for automatically creating a record of a trip's progress and content as text or graphics, based on organized visit data.

[0473] A "generative AI model" is an artificial intelligence system that uses machine learning techniques to generate new text and images.

[0474] A "prompt" is a text of instructions or questions that serves as the starting point for a generative AI model when generating information.

[0475] "Publicly available evaluation information" refers to evaluation data of tourist destinations and facilities provided by third parties that is generally accessible to the public.

[0476] "Means for suggesting new destinations and travel routes" refers to a function that suggests new destinations and routes for the next trip based on existing travel data and evaluation information.

[0477] A "terminal" refers to an electronic device that provides an interface for tasks such as uploading image data and displaying travel records.

[0478] "Means for calculating rewards" refers to a calculation function that calculates rewards based on evaluations made by other users when a proposed travel route is evaluated by them.

[0479] This invention is a system that automatically generates travel records and suggests new travel plans by utilizing image data taken by travelers during their trips. This system consists of server, terminal, and user components.

[0480] The server receives image data sent by the user. This image data contains metadata, including location and time information, as a digital file. The server uses Python libraries such as ExifRead to extract the necessary metadata from the images. Based on this data, it identifies the locations where the images were taken and their order, and builds a list of visited places. Next, the server uses a generative AI model to generate this information as a travel record in text format. Furthermore, it suggests new places to visit and travel routes to the user using prompts. Using prompts such as "Suggest new sights to visit on my next trip to Kyoto," it is possible to enrich travel planning.

[0481] The terminal provides an interface for users to upload image data taken by travelers. The terminal can be a smartphone or personal computer, where users select images and upload them to the server. The terminal also functions as a tool to display the generated travel record to the user, allowing them to review the record and add or modify information as needed.

[0482] This system allows users to easily create their own travel records and receive suggestions to help plan their next trip. By uploading images taken during their trip to the server via their device, they can obtain a record of the places they visited. Users can add their own comments to the generated travel records and further enrich their future travels by referring to new travel plan suggestions from the server.

[0483] For example, if a user has photos taken during a trip in Japan, the server extracts location and time information from these photos and creates a travel record based on the order in which the visits were made. For instance, it can suggest a next travel plan that includes landmarks the user hasn't yet visited, thus broadening the scope of their travel experience.

[0484] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0485] Step 1:

[0486] The user selects image files taken during their trip on their device. As input, the user provides image data taken with a digital camera or smartphone to the device's upload interface. The device sends the selected images and their metadata (location and time information) to the server using the HTTPS protocol. This completes the secure transfer of the images.

[0487] Step 2:

[0488] The server analyzes the image data received from the user. Image files and metadata are provided as input. The server uses the Python ExifRead library to extract GPS information and timestamps from the images. During this process, the server constructs a data frame that identifies the locations where the images were taken and their time order. An initial list of visited locations is generated as output of this process.

[0489] Step 3:

[0490] The server generates travel records based on visit data. It uses the list of visited locations constructed in step 2 as input. A generative AI model is employed to generate detailed text about each visited location. Data processing in this process includes text generation using a natural language generation model. This allows the user to obtain detailed travel records for each visited location.

[0491] Step 4:

[0492] The terminal displays the generated travel record to the user. It receives the travel record in text format from the server as input. The user can view and edit the travel record through the terminal's interface. This interface is often provided as a web page using HTML or JavaScript. The user can add comments about visited locations and save the information back to the server.

[0493] Step 5:

[0494] The server suggests new travel plans based on the user's travel history and publicly available ratings. It uses a database of places the user has visited and their ratings as input, and generates suggestions using a generative AI model and prompts. The output could be something like, "Suggest new sights to visit on my next trip to Kyoto." This allows the user to gain ideas for creating new itineraries.

[0495] (Application Example 1)

[0496] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0497] There is a growing need for systems that efficiently utilize photos taken by travelers during their trips to automatically generate travel records and suggest new travel plans. However, conventional systems have problems such as difficulty in organizing visited locations and the inability to receive real-time suggestions that are useful for future travel planning. Furthermore, it has been difficult to provide optimal travel routes in tourist destinations tailored to individual travelers, and incentives based on the social reputation of visited areas have not been realized.

[0498] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0499] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and their order based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and their order. This allows travelers to easily generate travel records based on photographs they have taken and receive suggestions for new places to visit. Furthermore, by linking with urban infrastructure, it becomes possible to provide optimal routes in real time, enriching the travel experience.

[0500] "Image data containing information" refers to photographs and visual records that have location and time information embedded in them.

[0501] "Location information" refers to geographical coordinate data, which indicates where an image was taken.

[0502] "Time information" refers to data that indicates the date and time when a specific event or action occurred.

[0503] "Places visited and order of visits" refers to the places a traveler visited and the order in which they were visited.

[0504] A "travel log" is a record of places visited during a trip, the order in which they were visited, and events that occurred, organized and documented in that way.

[0505] The "means of suggestion" refers to a mechanism that generates and presents new destinations and routes based on user data.

[0506] "Integration with urban infrastructure" refers to the process by which systems share information with urban transportation, tourist facilities, public services, and other related elements.

[0507] "Providing information in real time" means presenting information immediately in response to the user's current situation and requests.

[0508] "Being evaluated and having compensation calculated based on that evaluation" means quantifying the value of a proposal based on feedback from other users and calculating compensation accordingly.

[0509] "User's portable device" refers to an electronic device that a user can carry with them and that allows for the input and output of information, such as a smartphone or tablet.

[0510] A "third-party information source" refers to a source of information provided from an external party other than the user or developer.

[0511] "Past activity history" refers to data about the user's past activities and visit records.

[0512] The system for implementing this invention primarily involves three elements: a server, a terminal, and a user. The server receives image data containing information captured by the user and extracts location and time information from the data. This makes it possible to identify and organize where and when the image data was taken. The server further lists the visited locations and their order based on the extracted information and automatically generates a travel record based on this list.

[0513] The terminal is a user's portable device, such as a smartphone or tablet. This terminal acts as a link between the user and the server, providing an interface for uploading captured images. On the terminal, users can view their generated travel records, and new places to visit and travel routes are suggested based on publicly available evaluation information.

[0514] Users send images they take using the system to the server via their device. This allows them to view automatically generated travel records and check suggested routes that have been rated by others. Furthermore, integration with urban infrastructure enables real-time route suggestions, making actual travel smoother and more fulfilling.

[0515] As a concrete example, when a user uploads photos taken at tourist destinations during their trip to the app, the system analyzes the time and location information obtained from the images and suggests places to visit next based on their visit history. In doing so, it takes into account the city's transportation network and the congestion levels of tourist spots to present the optimal travel route in real time.

[0516] An example of a prompt message to use a generative AI model to suggest new routes is: "A traveler who has already visited Tokyo and Osaka should suggest places to visit on their next trip. Please include tourist destinations in Hokkaido in the new plan."

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The device receives an image taken by the user. This image contains Exif data, including location and time information. The input data is a photo file, and the output remains unchanged but is ready to be sent to the next processing step.

[0520] Step 2:

[0521] The terminal sends the received image data to the server. The server receives this data and begins data analysis. The input is an image file, and the output is the state in which the server is ready for data analysis.

[0522] Step 3:

[0523] The server extracts Exif information from image data to obtain location and time information. Based on this information, it identifies the visited locations and their order. The input is image data, and the output is an organized list of visited locations. The server uses software such as a geolocation API to analyze the location information.

[0524] Step 4:

[0525] The server automatically generates travel records based on extracted information about visited locations. The input is an organized list of visited locations, and the output is the generated travel record. The generated record is presented in text and map formats and used for further analysis.

[0526] Step 5:

[0527] The server uses a generative AI model to suggest new destinations and travel routes suitable for the user based on publicly available evaluation information. Inputs are travel records and publicly available evaluation information, and output is a proposed new travel route. The AI ​​model generates suggestions based on prompt statements.

[0528] Step 6:

[0529] The server works in conjunction with urban infrastructure to send optimized suggestions to the user's terminal in real time. The input is the suggestion data from the previous step, and the output is real-time updated travel route information. Dynamic data such as traffic conditions are also taken into consideration during this process.

[0530] Step 7:

[0531] Users can receive suggested plans on their devices and use them to plan their next trip. The input is suggested data from the server, and the output is a new travel plan for the user. This allows travelers to enjoy a richer and more efficient travel experience.

[0532] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0533] This invention is a system that utilizes image data taken by travelers during their trips to recognize the user's emotions from the information within the images, and then automatically generates travel records or suggests new travel plans based on those emotions. By combining this system with an emotion engine, it is possible to provide new value based on the user's emotions in addition to the usual visit history.

[0534] The server receives image data uploaded by the user and first extracts location and time information. This provides data that organizes where and how the traveler visited. Next, an emotion engine is used to analyze the facial expressions and context of people in the images to determine the user's emotional state. This information serves as additional information indicating what emotions the traveler felt at each location.

[0535] The device is used for uploading image data and also functions as a tool for reviewing travel records that include emotional data. Users use the device to send photos they've taken to the server and then visually review the generated travel records. The provided travel records include estimated emotional information for each visited location, vividly reflecting the richness of the travel experience.

[0536] Through this system, users can re-examine their own travel experiences and plan trips with a fresh perspective. For example, based on a user's enjoyable experience at a particular tourist destination, the server can suggest other highly-rated spots with a similar atmosphere, based on emotional data. By considering past emotions when selecting a user's next travel destination, they can plan a more satisfying trip.

[0537] For example, by analyzing facial expressions in photos taken by users in various locations, if there are a large number of smiles in a particular city, the system will determine that the user had a highly positive experience in that city. The emotion engine will then suggest other locations with a similar atmosphere, allowing users to incorporate new emotional experiences into their next trip. In addition, this data can be used to provide travel guidance to other users and also functions as emotion-based word-of-mouth information.

[0538] In this way, this system not only uses users' visit history but also subjective data such as emotions to create new travel value and contribute to deepening communication among travelers.

[0539] The following describes the processing flow.

[0540] Step 1:

[0541] The user uses their device to select image data taken during their trip. The user chooses multiple image files from their smartphone or computer and prepares to complete the upload to the system.

[0542] Step 2:

[0543] The device uploads the selected image data to the server. The device sequentially sends the image files to the server via the internet, and the user is shown the progress in real time.

[0544] Step 3:

[0545] The server analyzes metadata from the received image data to extract location and time information. This allows it to identify which locations were visited and when, and then organize and store this information in a database.

[0546] Step 4:

[0547] The server uses an emotion engine to analyze the facial expressions and elements of the people in each image and estimate the user's emotions. For example, images with many smiles are recorded as a positive experience.

[0548] Step 5:

[0549] The server automatically generates a travel log based on location information, time information, and estimated sentiment data. The travel log is compiled as the order of visited places, the user's sentiment state, and comments for each visited place.

[0550] Step 6:

[0551] The device displays the generated travel record to the user. The user can review the displayed record and edit any additional comments or notes as needed.

[0552] Step 7:

[0553] Based on sentiment data collected by the server, the system suggests new travel routes that include highly-rated spots the user hasn't visited yet. These routes are selected based on the user's past positive emotional experiences.

[0554] Step 8:

[0555] Users can review suggested new travel routes and use them to plan their next trip. They can also refer to ratings from other users to determine which spots offer the highest emotional satisfaction.

[0556] Step 9:

[0557] The server compiles evaluations of the proposed new travel routes based on feedback provided by the user and calculates a reward based on those evaluations. The server then notifies the user of the details of the reward.

[0558] (Example 2)

[0559] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] While there are many ways for travelers to record the places they visit, there is a lack of systems that capture the emotions and experiences of those visits in detail. Existing travel recording systems focus on physical location and time information, and do not reflect the subjective value of how users felt at a place. This results in a challenge in improving travel satisfaction and suggesting new travel plans.

[0561] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0562] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for analyzing the extracted image data and recognizing the user's emotions using an emotion engine; and means for suggesting similar places to visit and travel routes based on the emotion recognition data of the generated travel record. This makes it possible to generate detailed travel records based on emotional information during travel and to suggest new travel plans.

[0563] "Image data" refers to data containing digital visual information captured by a user.

[0564] "Location information" refers to geographical data that indicates the location where an image was taken, as contained within the image data.

[0565] "Time information" refers to data included in image data that indicates the date and time of shooting.

[0566] An "emotion engine" is a technology equipped with an algorithm that analyzes facial expressions and context within an image to determine the user's emotions.

[0567] A "travel record" is data that represents an individual travel experience, generated by integrating the user's visited locations, the order of visits, and emotion recognition data.

[0568] "Similar destinations" are new places to visit that are suggested based on the user's sentiment data from places they have visited in the past, suggesting that they have a similar atmosphere and experience.

[0569] "Travel route" refers to information that shows the path a user will take during their trip and provides directions for moving to their next destination.

[0570] "Emotion recognition data" refers to data that quantifies or categorizes the emotional state of a user, as analyzed using an emotion engine.

[0571] This invention is a system that analyzes image data taken by travelers, recognizes the location information, time information, and user emotions contained within it, and generates and suggests travel records.

[0572] The server receives image data uploaded from the user's device and automatically extracts location and time information from that data. This is done using a database management system and image processing libraries. For example, the server obtains location information by analyzing EXIF ​​data and records when a traveler visited a specific location.

[0573] Next, the server uses an emotion engine to analyze the image data and recognize the user's emotions. This process utilizes deep learning technology and image analysis software to determine emotions such as "joy" and "surprise" from the facial expressions and background of the people in the image.

[0574] This information is integrated and provided to the user via their device as a travel log. The travel log includes estimated emotional information for each place visited, allowing the user to see in chronological order what emotions they felt at each location.

[0575] Users can refer to their generated travel records and have the server suggest new travel plans based on their emotional data. For example, if a user inputs, "Tell me about new tourist destinations that have scenery similar to cities I've enjoyed in the past," the system will refer to the emotional data and use a generative AI model to suggest places with a similar atmosphere. This allows users to create personalized travel plans based on their emotions.

[0576] An example of a prompt would be, "For a traveler who smiled a lot in Paris, what would you recommend as their next travel destination?" Through this prompt, the system can provide suggestions that take into account the user's past emotional data.

[0577] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0578] Step 1:

[0579] The terminal selects image data captured by the user and uploads it to the server. At this time, the terminal displays a file selection dialog, allowing the user to choose the image to send, and then transmits the data to the server via the internet. The input is the image file selected by the user, and the output is the transfer of the image data to the server.

[0580] Step 2:

[0581] The server extracts location and time information from the received image data. Using a dedicated image processing library, the server analyzes the EXIF ​​data within the image file to obtain the shooting location (e.g., GPS coordinates) and the date and time of shooting. The input is the uploaded image data, and the output is a dataset of location and time information.

[0582] Step 3:

[0583] The server uses an emotion engine to recognize the emotions of people in an image. The server runs a deep learning-based facial expression analysis model and assigns emotion labels such as "joy" or "surprise" to the image. The input is the image to be analyzed, and the output is the emotion label and its confidence level.

[0584] Step 4:

[0585] The server integrates the extracted location, time, and emotion labels to generate a travel record. The server then organizes this into a time-series data set, creating a travel record that details the emotions the user experienced at different locations. The input consists of the aforementioned location, time, and emotion labels, while the output is a dataset of travel records.

[0586] Step 5:

[0587] The terminal presents the generated travel record to the user. The terminal uses a visually appealing interface to plot the travel record on a map, allowing for visual confirmation. The input is travel record data received from the server, and the output is a visual presentation of the travel record to the user.

[0588] Step 6:

[0589] The server proposes a new travel plan based on travel records and user sentiment data. Using a generative AI model, the server searches for places to visit that offer similar emotional experiences and presents recommended travel destinations to the user. Input is the generated travel record and prompt text, and output is the new travel plan.

[0590] (Application Example 2)

[0591] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0592] In today's information-driven society, there is a need to appropriately understand individual emotions and preferences in travel experiences and dining services, and to provide new experiences and suggestions based on that understanding. However, conventional systems are limited to visit history and publicly available evaluation information, and have been unable to utilize users' subjective emotions. Therefore, technology is needed to provide more personalized suggestions.

[0593] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0594] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and order of visits based on the extracted location information and time information; means for automatically generating a travel record based on the visited places and order of visits; means for analyzing the image data and determining the emotional state; means for suggesting candidates for the next visit based on the emotional state; and means for having the suggested candidates evaluated by other users and improving the recommendation content based on that evaluation. This enables new and personalized suggestions based on the user's emotions.

[0595] "Information-containing image data" refers to visual data associated with location and time information, and is used to analyze user activity and preferences.

[0596] "Location information" is data that indicates the geographical location where an image was taken, and it is an element that makes up a user's browsing history.

[0597] "Time information" refers to data indicating the date and time an image was taken, and is used to understand the timeline of a user's actions.

[0598] "Means for organizing visited locations and order of visits" refers to an algorithm that has the function of arranging user actions chronologically based on location and time information.

[0599] "Means for automatically generating travel records" refers to a system that records and visualizes the flow of a trip based on the user's visited locations and order.

[0600] A "means for determining emotional state" refers to an algorithm that uses analytical techniques to identify emotions from a person's facial expressions and situation within an image.

[0601] The "method for suggesting potential next visit locations" is a feature that suggests new places that might interest the user based on their past emotional data.

[0602] "Methods for improving recommendations based on evaluation" refers to the process of collecting feedback from other users and using that feedback to improve the accuracy of future recommendations.

[0603] The core of this system is a server. The server receives image data containing information uploaded by the user from their terminal. This image data is processed using an image analysis engine, and location and time information is extracted. This data is used to organize the user's visited locations and their order, and to automatically generate a travel record.

[0604] The server also operates an emotion analysis engine, which analyzes the facial expressions and backgrounds of people in image data to determine their emotional state. This emotion data is used to enrich the user's experience. For example, if many "happy" emotions are detected in photos taken by a user at a particular place they visited, it is presumed that the place is highly rated by the user. Based on this data, the server generates potential destinations for the next visit within the suggestion environment system, based on past emotion data.

[0605] Based on feedback from other users, the server collects evaluation information for the evaluated proposal candidates, and the system's algorithm continuously improves the recommendations.

[0606] This entire process utilizes Python programs running on a cloud server. OpenCV is used for image analysis, and TensorFlow for sentiment analysis. Furthermore, user devices such as smartphones and tablets are perfectly adequate.

[0607] As a concrete example, a user takes photos with their smartphone while traveling and uploads them to the server from the terminal app. By using a prompt message such as, "Please upload a photo expressing your feelings while eating pasta. We will suggest a recommended menu item for your next visit," users can easily provide data through the user interface.

[0608] In this way, it becomes possible to provide new user experiences based on emotions throughout the entire system.

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The user's device collects image data. The user takes photos using their smartphone while traveling or eating, and saves those images to the application. The input is the image captured by the camera. The output is the image file saved in the application.

[0612] Step 2:

[0613] The device uploads image data to the server. The device sends the image selected by the user to the server. The input is the image file stored on the device. The output is the image data sent to the server.

[0614] Step 3:

[0615] The server extracts location and time information. The server obtains the shooting location and date / time by analyzing Exif information from the received image data. The input is the image data sent to the server. The output is metadata including location and time information.

[0616] Step 4:

[0617] The server performs emotion analysis. Using TensorFlow, the server identifies faces in an image and estimates emotions from facial expressions. The input is image data sent to the server. The output is numerical data representing the estimated emotional state.

[0618] Step 5:

[0619] The server generates a travel record. The server combines extracted location, time, and sentiment data to create a travel record summarizing the user's visit history. Inputs are location, time, and sentiment data. Output is the generated travel record.

[0620] Step 6:

[0621] The server suggests potential destinations for your next visit. Using an algorithm based on past sentiment data, the server generates new suggestions similar to places the user has highly rated. The input is the generated travel records. The output is a list of suggested destinations.

[0622] Step 7:

[0623] The server collects and improves evaluations of candidate sites. The server also collects feedback from other users and strengthens the proposed algorithm based on that data. The input is evaluation data from other users. The output is the updated proposed algorithm.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] This invention is a system that effectively utilizes image data taken by travelers during their trips, automatically generates travel records, and proposes new travel plans. The core of this system consists of a series of operations involving a server, terminals, and users.

[0642] The server receives image data uploaded by the user. This includes typical digital images and associated metadata. The server extracts location and time information from the image files and uses it to organize which places were visited and in what order. This process creates a list of places the traveler visited during their trip.

[0643] The terminal is a device that provides the user with an interface for uploading image data. This typically includes a smartphone or computer. The user selects photos taken during their trip and sends them to the server via the terminal. Once the upload is complete, the user can view the analysis results on the server. The terminal also serves as a tool for displaying the generated travel record (travel diary), allowing the user to view this information and add additional information as needed.

[0644] Users can use the system to automatically create their own travel records and obtain information to help plan their next trip. By uploading photos taken during their trip to the server via their device, users can digitize and organize their travel experiences. In this process, users can review the generated travel diary and add comments about the places they visited. They can also check new travel routes suggested by the server and use them to help plan their next trip.

[0645] As a concrete example, suppose a user takes photos in Tokyo, Kyoto, and Osaka during a visit to Japan. The server retrieves location and time information from these photos and automatically determines the overall travel route. Based on this process, it generates a detailed travel record for each city visited and provides it to the user in map and text format. Furthermore, it suggests a new travel route to the user that incorporates unvisited landmarks and highly-rated places (for example, unvisited temples in Kyoto or restaurants in Osaka). This allows the user to create a travel plan that offers new experiences on their next visit.

[0646] Through this system, travelers can obtain attractive future travel plans while minimizing the effort required to organize their travel history. In addition, travel plans are valued for being helpful to others, promoting interaction within the community.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The user uses their device to select image data taken during their trip. They can select multiple photos at once and prepare them for upload.

[0650] Step 2:

[0651] The device uploads the selected image data to the server. Once the upload begins, the device sends the image data to the server one by one and displays the progress to the user.

[0652] Step 3:

[0653] The server analyzes metadata from the received image data to extract location information (GPS coordinates) and time information (timestamp). This identifies the location and time each photograph was taken, and the data is stored in a database.

[0654] Step 4:

[0655] Based on the location and time information extracted by the server, the user's visited locations and their order are organized. This organization includes creating routes based on the chronological order and geographical relationships of the visited spots.

[0656] Step 5:

[0657] Based on the information compiled by the server, a travel record (travel diary) is automatically generated. This record includes a list of places visited, the route on a map, and story text corresponding to the time spent at each location.

[0658] Step 6:

[0659] The device displays the generated travel record to the user. The user reviews it and can edit it with comments or additional information as needed.

[0660] Step 7:

[0661] The server retrieves publicly available rating information from external sources and, based on the user's travel records, proposes travel routes that include new highly-rated spots. This proposed route is then sent to the user.

[0662] Step 8:

[0663] Users can review new travel routes and use them to plan their next trip. Additionally, if suggested routes are rated by other users, those ratings are compiled by the server.

[0664] Step 9:

[0665] The server calculates a reward for the creator of the proposed travel route based on the evaluation results and notifies the user of the details. The reward is provided as points or a monetary incentive.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] Travelers often take a large number of photos during their trips, and organizing them later to create a travel log is time-consuming and laborious. Furthermore, it can be difficult to find new destinations or routes when planning future trips. Additionally, it's challenging to know how one's own travel plans are perceived by others.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and the order of visits based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and the order of visits. This makes it possible for users to easily create a travel record and use it to help plan their next trip.

[0671] "Image data containing information" refers to digital image files that include metadata such as location and time information.

[0672] "Extraction method" refers to the software or algorithm used to extract necessary metadata from image data.

[0673] "Means for organizing visited locations and order of visits" refers to a function that arranges visited locations according to the order in which the traveler visited them, based on extracted location and time information.

[0674] "Methods for automatically generating travel records" refer to methods for automatically creating a record of a trip's progress and content as text or graphics, based on organized visit data.

[0675] A "generative AI model" is an artificial intelligence system that uses machine learning techniques to generate new text and images.

[0676] A "prompt" is a text of instructions or questions that serves as the starting point for a generative AI model when generating information.

[0677] "Publicly available evaluation information" refers to evaluation data of tourist destinations and facilities provided by third parties that is generally accessible to the public.

[0678] "Means for suggesting new destinations and travel routes" refers to a function that, based on existing travel data and evaluation information, suggests new destinations and routes for the next trip.

[0679] A "terminal" refers to an electronic device that provides an interface for tasks such as uploading image data and displaying travel records.

[0680] "Means for calculating rewards" refers to a calculation function that calculates rewards based on evaluations made by other users when a proposed travel route is evaluated by them.

[0681] This invention is a system that automatically generates travel records and suggests new travel plans by utilizing image data taken by travelers during their trips. This system consists of server, terminal, and user components.

[0682] The server receives image data sent by the user. This image data contains metadata, including location and time information, as a digital file. The server uses Python libraries such as ExifRead to extract the necessary metadata from the images. Based on this data, it identifies the locations where the images were taken and their order, and builds a list of visited places. Next, the server uses a generative AI model to generate this information as a travel record in text format. Furthermore, it suggests new places to visit and travel routes to the user using prompts. Using prompts such as "Suggest new sights to visit on my next trip to Kyoto," it is possible to enrich travel planning.

[0683] The terminal provides an interface for users to upload image data taken by travelers. The terminal can be a smartphone or personal computer, where users select images and upload them to the server. The terminal also functions as a tool to display the generated travel record to the user, allowing them to review the record and add or modify information as needed.

[0684] This system allows users to easily create their own travel records and receive suggestions to help plan their next trip. By uploading images taken during their trip to the server via their device, they can obtain a record of the places they visited. Users can add their own comments to the generated travel records and further enrich their future travels by referring to new travel plan suggestions from the server.

[0685] For example, if a user has photos taken during a trip in Japan, the server extracts location and time information from these photos and creates a travel record based on the order in which the visits were made. For instance, it can suggest a next travel plan that includes landmarks the user hasn't yet visited, thus broadening the scope of their travel experience.

[0686] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0687] Step 1:

[0688] The user selects image files taken during their trip on their device. As input, the user provides image data taken with a digital camera or smartphone to the device's upload interface. The device sends the selected images and their metadata (location and time information) to the server using the HTTPS protocol. This completes the secure transfer of the images.

[0689] Step 2:

[0690] The server analyzes the image data received from the user. Image files and metadata are provided as input. The server uses the Python ExifRead library to extract GPS information and timestamps from the images. During this process, the server constructs a data frame that identifies the locations where the images were taken and their time order. An initial list of visited locations is generated as output of this process.

[0691] Step 3:

[0692] The server generates travel records based on visit data. It uses the list of visited locations constructed in step 2 as input. A generative AI model is employed to generate detailed text about each visited location. Data processing in this process includes text generation using a natural language generation model. This allows the user to obtain detailed travel records for each visited location.

[0693] Step 4:

[0694] The terminal displays the generated travel record to the user. It receives the travel record in text format from the server as input. The user can view and edit the travel record through the terminal's interface. This interface is often provided as a web page using HTML or JavaScript. The user can add comments about visited locations and save the information back to the server.

[0695] Step 5:

[0696] The server suggests new travel plans based on the user's travel history and publicly available ratings. It uses a database of places the user has visited and their ratings as input, and generates suggestions using a generative AI model and prompts. The output could be something like, "Suggest new sights to visit on my next trip to Kyoto." This allows the user to gain ideas for creating new itineraries.

[0697] (Application Example 1)

[0698] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0699] There is a growing need for systems that efficiently utilize photos taken by travelers during their trips to automatically generate travel records and suggest new travel plans. However, conventional systems have problems such as difficulty in organizing visited locations and the inability to receive real-time suggestions that are useful for future travel planning. Furthermore, it has been difficult to provide optimal travel routes in tourist destinations tailored to individual travelers, and incentives based on the social reputation of visited areas have not been realized.

[0700] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0701] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and their order based on the extracted location information and time information; and means for automatically generating a travel record based on the visited places and their order. This allows travelers to easily generate travel records based on photographs they have taken and receive suggestions for new places to visit. Furthermore, by linking with urban infrastructure, it becomes possible to provide optimal routes in real time, enriching the travel experience.

[0702] "Image data containing information" refers to photographs and visual records that have location and time information embedded in them.

[0703] "Location information" refers to geographical coordinate data, which indicates where an image was taken.

[0704] "Time information" refers to data that indicates the date and time when a specific event or action occurred.

[0705] "Places visited and order of visits" refers to the places a traveler visited and the order in which they were visited.

[0706] A "travel log" is a record of places visited during a trip, the order in which they were visited, and events that occurred, organized and documented in that way.

[0707] The "means of suggestion" refers to a mechanism that generates and presents new destinations and routes based on user data.

[0708] "Integration with urban infrastructure" refers to the process by which systems share information with urban transportation, tourist facilities, public services, and other related elements.

[0709] "Providing information in real time" means presenting information immediately in response to the user's current situation and requests.

[0710] "Being evaluated and having compensation calculated based on that evaluation" means quantifying the value of a proposal based on feedback from other users and calculating compensation accordingly.

[0711] "User's portable device" refers to an electronic device that a user can carry with them and that allows for the input and output of information, such as a smartphone or tablet.

[0712] A "third-party information source" refers to a source of information provided from an external party other than the user or developer.

[0713] "Past activity history" refers to data about the user's past activities and visit records.

[0714] The system for implementing this invention primarily involves three elements: a server, a terminal, and a user. The server receives image data containing information captured by the user and extracts location and time information from the data. This makes it possible to identify and organize where and when the image data was taken. The server further lists the visited locations and their order based on the extracted information and automatically generates a travel record based on this list.

[0715] The terminal is a user's portable device, such as a smartphone or tablet. This terminal acts as a link between the user and the server, providing an interface for uploading captured images. On the terminal, users can view their generated travel records, and new places to visit and travel routes are suggested based on publicly available evaluation information.

[0716] Users send images they take using the system to the server via their device. This allows them to view automatically generated travel records and check suggested routes that have been rated by others. Furthermore, integration with urban infrastructure enables real-time route suggestions, making actual travel smoother and more fulfilling.

[0717] As a concrete example, when a user uploads photos taken at tourist destinations during their trip to the app, the system analyzes the time and location information obtained from the images and suggests places to visit next based on their visit history. In doing so, it takes into account the city's transportation network and the congestion levels of tourist spots to present the optimal travel route in real time.

[0718] An example of a prompt message to use a generative AI model to suggest new routes is: "A traveler who has already visited Tokyo and Osaka should suggest places to visit on their next trip. Please include tourist destinations in Hokkaido in the new plan."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The device receives an image taken by the user. This image contains Exif data, including location and time information. The input data is a photo file, and the output remains unchanged but is ready to be sent to the next processing step.

[0722] Step 2:

[0723] The terminal sends the received image data to the server. The server receives this data and begins data analysis. The input is an image file, and the output is the state in which the server is ready for data analysis.

[0724] Step 3:

[0725] The server extracts Exif information from image data to obtain location and time information. Based on this information, it identifies the visited locations and their order. The input is image data, and the output is an organized list of visited locations. The server uses software such as a geolocation API to analyze the location information.

[0726] Step 4:

[0727] The server automatically generates travel records based on extracted information about visited locations. The input is an organized list of visited locations, and the output is the generated travel record. The generated record is presented in text and map formats and used for further analysis.

[0728] Step 5:

[0729] The server uses a generative AI model to suggest new destinations and travel routes suitable for the user based on publicly available evaluation information. Inputs are travel records and publicly available evaluation information, and output is a proposed new travel route. The AI ​​model generates suggestions based on prompt statements.

[0730] Step 6:

[0731] The server works in conjunction with urban infrastructure to send optimized suggestions to the user's terminal in real time. The input is the suggestion data from the previous step, and the output is real-time updated travel route information. Dynamic data such as traffic conditions are also taken into consideration during this process.

[0732] Step 7:

[0733] Users can receive suggested plans on their devices and use them to plan their next trip. The input is suggested data from the server, and the output is a new travel plan for the user. This allows travelers to enjoy a richer and more efficient travel experience.

[0734] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0735] This invention is a system that utilizes image data taken by travelers during their trips to recognize the user's emotions from the information within the images, and then automatically generates travel records or suggests new travel plans based on those emotions. By combining this system with an emotion engine, it is possible to provide new value based on the user's emotions in addition to the usual visit history.

[0736] The server receives image data uploaded by the user and first extracts location and time information. This provides data that organizes where and how the traveler visited. Next, an emotion engine is used to analyze the facial expressions and context of people in the images to determine the user's emotional state. This information serves as additional information indicating what emotions the traveler felt at each location.

[0737] The device is used for uploading image data and also functions as a tool for reviewing travel records that include emotional data. Users use the device to send photos they've taken to the server and then visually review the generated travel records. The provided travel records include estimated emotional information for each visited location, vividly reflecting the richness of the travel experience.

[0738] Through this system, users can re-examine their own travel experiences and plan trips with a fresh perspective. For example, based on a user's enjoyable experience at a particular tourist destination, the server can suggest other highly-rated spots with a similar atmosphere, based on emotional data. By considering past emotions when selecting a user's next travel destination, they can plan a more satisfying trip.

[0739] For example, by analyzing facial expressions in photos taken by users in various locations, if there are a large number of smiles in a particular city, the system will determine that the user had a highly positive experience in that city. The emotion engine will then suggest other locations with a similar atmosphere, allowing users to incorporate new emotional experiences into their next trip. In addition, this data can be used to provide travel guidance to other users and also functions as emotion-based word-of-mouth information.

[0740] In this way, this system not only uses users' visit history but also subjective data such as emotions to create new travel value and contribute to deepening communication among travelers.

[0741] The following describes the processing flow.

[0742] Step 1:

[0743] The user uses their device to select image data taken during their trip. The user chooses multiple image files from their smartphone or computer and prepares to complete the upload to the system.

[0744] Step 2:

[0745] The device uploads the selected image data to the server. The device sequentially sends the image files to the server via the internet, and the user is shown the progress in real time.

[0746] Step 3:

[0747] The server analyzes metadata from the received image data to extract location and time information. This allows it to identify which locations were visited and when, and then organize and store this information in a database.

[0748] Step 4:

[0749] The server uses an emotion engine to analyze the facial expressions and elements of the people in each image and estimate the user's emotions. For example, images with many smiles are recorded as a positive experience.

[0750] Step 5:

[0751] The server automatically generates a travel log based on location information, time information, and estimated sentiment data. The travel log is compiled as the order of visited places, the user's sentiment state, and comments for each visited place.

[0752] Step 6:

[0753] The device displays the generated travel record to the user. The user can review the displayed record and edit any additional comments or notes as needed.

[0754] Step 7:

[0755] Based on sentiment data collected by the server, the system suggests new travel routes that include highly-rated spots the user hasn't visited yet. These routes are selected based on the user's past positive emotional experiences.

[0756] Step 8:

[0757] Users can review suggested new travel routes and use them to plan their next trip. They can also refer to ratings from other users to determine which spots offer the highest emotional satisfaction.

[0758] Step 9:

[0759] The server compiles evaluations of the proposed new travel routes based on feedback provided by the user and calculates a reward based on those evaluations. The server then notifies the user of the details of the reward.

[0760] (Example 2)

[0761] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] While there are many ways for travelers to record the places they visit, there is a lack of systems that capture the emotions and experiences of those visits in detail. Existing travel recording systems focus on physical location and time information, and do not reflect the subjective value of how users felt at a place. This results in a challenge in improving travel satisfaction and suggesting new travel plans.

[0763] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0764] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for analyzing the extracted image data and recognizing the user's emotions using an emotion engine; and means for suggesting similar places to visit and travel routes based on the emotion recognition data of the generated travel record. This makes it possible to generate detailed travel records based on emotional information during travel and to suggest new travel plans.

[0765] "Image data" refers to data containing digital visual information captured by a user.

[0766] "Location information" refers to geographical data that indicates the location where an image was taken, as contained within the image data.

[0767] "Time information" refers to data included in image data that indicates the date and time of shooting.

[0768] An "emotion engine" is a technology equipped with an algorithm that analyzes facial expressions and context within an image to determine the user's emotions.

[0769] A "travel record" is data that represents an individual travel experience, generated by integrating the user's visited locations, the order of visits, and emotion recognition data.

[0770] "Similar destinations" are new places to visit that are suggested based on the user's sentiment data from places they have visited in the past, suggesting that they have a similar atmosphere and experience.

[0771] "Travel route" refers to information that shows the path a user will take during their trip and provides directions for moving to their next destination.

[0772] "Emotion recognition data" refers to data that quantifies or categorizes the emotional state of a user, as analyzed using an emotion engine.

[0773] This invention is a system that analyzes image data taken by travelers, recognizes the location information, time information, and user emotions contained within it, and generates and suggests travel records.

[0774] The server receives image data uploaded from the user's device and automatically extracts location and time information from that data. This is done using a database management system and image processing libraries. For example, the server obtains location information by analyzing EXIF ​​data and records when a traveler visited a specific location.

[0775] Next, the server uses an emotion engine to analyze the image data and recognize the user's emotions. This process utilizes deep learning technology and image analysis software to determine emotions such as "joy" and "surprise" from the facial expressions and background of the people in the image.

[0776] This information is integrated and provided to the user via their device as a travel log. The travel log includes estimated emotional information for each place visited, allowing the user to see in chronological order what emotions they felt at each location.

[0777] Users can refer to their generated travel records and have the server suggest new travel plans based on their emotional data. For example, if a user inputs, "Tell me about new tourist destinations that have scenery similar to cities I've enjoyed in the past," the system will refer to the emotional data and use a generative AI model to suggest places with a similar atmosphere. This allows users to create personalized travel plans based on their emotions.

[0778] An example of a prompt would be, "For a traveler who smiled a lot in Paris, what would you recommend as their next travel destination?" Through this prompt, the system can provide suggestions that take into account the user's past emotional data.

[0779] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0780] Step 1:

[0781] The terminal selects image data captured by the user and uploads it to the server. At this time, the terminal displays a file selection dialog, allowing the user to choose the image to send, and then transmits the data to the server via the internet. The input is the image file selected by the user, and the output is the transfer of the image data to the server.

[0782] Step 2:

[0783] The server extracts location and time information from the received image data. Using a dedicated image processing library, the server analyzes the EXIF ​​data within the image file to obtain the shooting location (e.g., GPS coordinates) and the date and time of shooting. The input is the uploaded image data, and the output is a dataset of location and time information.

[0784] Step 3:

[0785] The server uses an emotion engine to recognize the emotions of people in an image. The server runs a deep learning-based facial expression analysis model and assigns emotion labels such as "joy" or "surprise" to the image. The input is the image to be analyzed, and the output is the emotion label and its confidence level.

[0786] Step 4:

[0787] The server integrates the extracted location, time, and emotion labels to generate a travel record. The server then organizes this into a time-series data set, creating a travel record that details the emotions the user experienced at different locations. The input consists of the aforementioned location, time, and emotion labels, while the output is a dataset of travel records.

[0788] Step 5:

[0789] The terminal presents the generated travel record to the user. The terminal uses a visually appealing interface to plot the travel record on a map, allowing for visual confirmation. The input is travel record data received from the server, and the output is a visual presentation of the travel record to the user.

[0790] Step 6:

[0791] The server proposes a new travel plan based on travel records and user sentiment data. Using a generative AI model, the server searches for places to visit that offer similar emotional experiences and presents recommended travel destinations to the user. Input is the generated travel record and prompt text, and output is the new travel plan.

[0792] (Application Example 2)

[0793] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0794] In today's information-driven society, there is a need to appropriately understand individual emotions and preferences in travel experiences and dining services, and to provide new experiences and suggestions based on that understanding. However, conventional systems are limited to visit history and publicly available evaluation information, and have been unable to utilize users' subjective emotions. Therefore, technology is needed to provide more personalized suggestions.

[0795] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0796] In this invention, the server includes means for receiving image data containing information and extracting location information and time information from the data; means for organizing visited places and order of visits based on the extracted location information and time information; means for automatically generating a travel record based on the visited places and order of visits; means for analyzing the image data and determining the emotional state; means for suggesting candidates for the next visit based on the emotional state; and means for having the suggested candidates evaluated by other users and improving the recommendation content based on that evaluation. This enables new and personalized suggestions based on the user's emotions.

[0797] "Information-containing image data" refers to visual data associated with location and time information, and is used to analyze user activity and preferences.

[0798] "Location information" is data that indicates the geographical location where an image was taken, and it is an element that makes up a user's browsing history.

[0799] "Time information" refers to data indicating the date and time an image was taken, and is used to understand the timeline of a user's actions.

[0800] "Means for organizing visited locations and order of visits" refers to an algorithm that has the function of arranging user actions chronologically based on location and time information.

[0801] "Means for automatically generating travel records" refers to a system that records and visualizes the flow of a trip based on the user's visited locations and order.

[0802] A "means for determining emotional state" refers to an algorithm that uses analytical techniques to identify emotions from a person's facial expressions and situation within an image.

[0803] The "method for suggesting potential next visit locations" is a feature that suggests new places that might interest the user based on their past emotional data.

[0804] "Methods for improving recommendations based on evaluation" refers to the process of collecting feedback from other users and using that feedback to improve the accuracy of future recommendations.

[0805] The core of this system is a server. The server receives image data containing information uploaded by the user from their terminal. This image data is processed using an image analysis engine, and location and time information is extracted. This data is used to organize the user's visited locations and their order, and to automatically generate a travel record.

[0806] The server also operates an emotion analysis engine, which analyzes the facial expressions and backgrounds of people in image data to determine their emotional state. This emotion data is used to enrich the user's experience. For example, if many "happy" emotions are detected in photos taken by a user at a particular place they visited, it is presumed that the place is highly rated by the user. Based on this data, the server generates potential destinations for the next visit within the suggestion environment system, based on past emotion data.

[0807] Based on feedback from other users, the server collects evaluation information for the evaluated proposal candidates, and the system's algorithm continuously improves the recommendations.

[0808] This entire process utilizes Python programs running on a cloud server. OpenCV is used for image analysis, and TensorFlow for sentiment analysis. Furthermore, user devices such as smartphones and tablets are perfectly adequate.

[0809] As a concrete example, a user takes photos with their smartphone while traveling and uploads them to the server from the terminal app. By using a prompt message such as, "Please upload a photo expressing your feelings while eating pasta. We will suggest a recommended menu item for your next visit," users can easily provide data through the user interface.

[0810] In this way, it becomes possible to provide new user experiences based on emotions throughout the entire system.

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The user's device collects image data. The user takes photos using their smartphone while traveling or eating, and saves those images to the application. The input is the image captured by the camera. The output is the image file saved in the application.

[0814] Step 2:

[0815] The device uploads image data to the server. The device sends the image selected by the user to the server. The input is the image file stored on the device. The output is the image data sent to the server.

[0816] Step 3:

[0817] The server extracts location and time information. The server obtains the shooting location and date / time by analyzing Exif information from the received image data. The input is the image data sent to the server. The output is metadata including location and time information.

[0818] Step 4:

[0819] The server performs emotion analysis. Using TensorFlow, the server identifies faces in an image and estimates emotions from facial expressions. The input is image data sent to the server. The output is numerical data representing the estimated emotional state.

[0820] Step 5:

[0821] The server generates a travel record. The server combines extracted location, time, and sentiment data to create a travel record summarizing the user's visit history. Inputs are location, time, and sentiment data. Output is the generated travel record.

[0822] Step 6:

[0823] The server suggests potential destinations for your next visit. Using an algorithm based on past sentiment data, the server generates new suggestions similar to places the user has highly rated. The input is the generated travel records. The output is a list of suggested destinations.

[0824] Step 7:

[0825] The server collects and improves evaluations of candidate sites. The server also collects feedback from other users and strengthens the proposed algorithm based on that data. The input is evaluation data from other users. The output is the updated proposed algorithm.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] A means for receiving image data containing information and extracting location information and time information from said data,

[0850] A means for organizing the places to visit and the order of visits based on the extracted location and time information,

[0851] A means for automatically generating a travel record based on the aforementioned places visited and the order of visits,

[0852] A means of suggesting new places to visit and travel routes based on other publicly available evaluation information,

[0853] The proposed travel route is evaluated by other users, and a means is provided to calculate a reward based on the evaluation.

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, wherein the image data is uploaded from the user's terminal.

[0857] (Claim 3)

[0858] The system according to claim 1, wherein the aforementioned evaluation information is obtained from a third-party source.

[0859] "Example 1"

[0860] (Claim 1)

[0861] A means for receiving image data containing information and extracting location information and time information from said data,

[0862] A means for organizing the places to visit and the order of visits based on the extracted location and time information,

[0863] A means for automatically generating a travel record based on the aforementioned places visited and the order of visits,

[0864] A means to display the generated travel record to the user and allow them to input additional information about the places visited,

[0865] A means of proposing new places to visit and travel routes based on other publicly available evaluation information and generative AI models,

[0866] The proposed travel route is evaluated by other users, and a means is provided to calculate a reward based on the evaluation.

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, wherein the aforementioned image data is uploaded from the user's terminal.

[0870] (Claim 3)

[0871] The system according to claim 1, wherein the aforementioned evaluation information is obtained from a third-party source.

[0872] "Application Example 1"

[0873] (Claim 1)

[0874] A means for receiving image data containing information and extracting location information and time information from said data,

[0875] A means for organizing the places to visit and the order of visits based on the extracted location and time information,

[0876] A means for automatically generating a travel record based on the aforementioned places visited and the order of visits,

[0877] A means of suggesting new places to visit and travel routes based on other publicly available evaluation information,

[0878] By integrating with urban infrastructure, this provides a means to offer tourists appropriate routes in real time,

[0879] The proposed travel route is evaluated by other users, and a means is provided to calculate a reward based on the evaluation.

[0880] A system that includes this.

[0881] (Claim 2)

[0882] The system according to claim 1, wherein the aforementioned image data is uploaded from the user's mobile device.

[0883] (Claim 3)

[0884] The system according to claim 1, wherein the evaluation information is obtained from a third-party source, and the generated tourist route is adjusted based on the user's past behavioral history.

[0885] "Example 2 of combining an emotion engine"

[0886] (Claim 1)

[0887] A means for receiving image data containing information and extracting location information and time information from said data,

[0888] A means for organizing the places to visit and the order of visits based on the extracted location and time information,

[0889] The extracted image data is analyzed and a means for recognizing the user's emotions using an emotion engine.

[0890] A means for automatically generating a travel record based on the aforementioned visited locations, order of visits, and emotion recognition data,

[0891] A means for suggesting similar places to visit and travel routes based on emotion recognition data from generated travel records,

[0892] The proposed travel route is evaluated, and a means is used to calculate a reward based on the evaluation.

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, wherein the aforementioned image data is uploaded from the user's terminal.

[0896] (Claim 3)

[0897] The system according to claim 1, wherein the aforementioned evaluation information is obtained from other sources.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A means for receiving image data containing information and extracting location information and time information from said data,

[0901] A means for organizing the places to visit and the order of visits based on the extracted location and time information,

[0902] A means for automatically generating a travel record based on the aforementioned places visited and the order of visits,

[0903] A means for analyzing the aforementioned image data and determining the emotional state,

[0904] A means of suggesting potential destinations for the next visit based on the aforementioned emotional state,

[0905] The proposed candidates are evaluated by other users, and the recommendations are improved based on that evaluation.

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, wherein the aforementioned image data is uploaded from the user's terminal.

[0909] (Claim 3)

[0910] The system according to claim 1, wherein the evaluation information is obtained from a third-party information source or from the sentiment analysis results of past users. [Explanation of Symbols]

[0911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving image data containing information and extracting location information and time information from said data, A means for organizing the places to visit and the order of visits based on the extracted location and time information, A means for automatically generating a travel record based on the aforementioned places visited and the order of visits, A means of suggesting new places to visit and travel routes based on other publicly available evaluation information, The proposed travel route is evaluated by other users, and a means is provided to calculate a reward based on the evaluation. A system that includes this.

2. The system according to claim 1, wherein the image data is uploaded from the user's terminal.

3. The system according to claim 1, wherein the aforementioned evaluation information is obtained from a third-party information source.

Images