system

Abstract

Provide a system. [Solution] means for receiving user input of interest and preference information; A means for generating an optimal sightseeing tour plan based on the received interest and desire information using a generative AI; means for presenting the generated sightseeing tour plan to a user; a means for displaying real-world guidance as augmented reality; a means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional sightseeing tours and stamp rallies have few elements that can be enjoyed again once the experience is complete, making it difficult to attract repeat visitors. Furthermore, because they lack the customizability to fully meet users' interests and desires, individual sightseeing experiences tend to be uniform. Furthermore, on-site navigation and tourist information provision can be insufficient, making it difficult for users to efficiently enjoy sightseeing. There is a need for a system that can solve these issues and provide optimal sightseeing experiences that meet the diverse needs of users. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving user-inputted interest and preference information; means for generating an optimal sightseeing tour plan based on the received interest and preference information using a generative AI; means for presenting the generated sightseeing tour plan to the user; means for displaying real-world guidance as augmented reality; and means for regenerating the sightseeing tour plan based on the new conditions using the generative AI when the user changes the plan during the tour. This system provides a customized sightseeing experience tailored to the user's individual needs, making it easier to attract repeat visitors. Furthermore, by utilizing augmented reality technology, it is possible to provide on-site navigation and tourist information in real time, allowing users to enjoy sightseeing efficiently. Furthermore, by providing a means for users to input feedback after the tour and store that feedback, the quality of the tour plan can be continuously improved.

[0006] 1. "User" means an individual or organization that uses the system to create and experience a sightseeing tour plan.

[0007] 2. "Interests and desired information" is information entered by the user indicating desired conditions such as areas of interest, budget, and date of visit regarding tourism.

[0008] 3. "Means for receiving" refers to the system functions and devices for receiving information input by the user.

[0009] 4. "Generative AI" is a technology that uses artificial intelligence to generate optimal sightseeing tour plans based on user input information.

[0010] 5. "Sightseeing tour plan" is a plan that includes the order and schedule of visiting tourist spots, generated based on the user's interests and wishes.

[0011] 6. "Presentation means" refers to the system functions and devices that visually display the generated sightseeing tour plan to the user.

[0012] 7. "Real-world guidance" refers to the information and directions that users see on-site during the tour.

[0013] 8. "Augmented reality (AR)" is a technology that overlays digital information onto real-world scenery, adding virtual guidance displays to the user's field of vision.

[0014] 9. "Means for regeneration" refers to the system functions and devices that regenerate sightseeing tour plans based on new conditions when the user changes the tour plan midway.

[0015] 10. "Feedback" refers to information indicating the evaluation and opinions of the tour that the user enters after completing the tour. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for providing users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Specific embodiments of the system of the present invention will be described below.

[0038] User Registration and Login

[0039] User

[0040] Users access the "UTour" app through their device (smartphone, tablet, etc.) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[0041] server

[0042] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0043] Enter your interests and preferences

[0044] User

[0045] Users go to the tour plan creation page of the "UTour" app and enter information such as their areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[0046] Terminal

[0047] The terminal transmits the inputted interest and desired information to the server.

[0048] server

[0049] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[0050] Tour plan generation and provision

[0051] server

[0052] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[0053] Terminal

[0054] The tour plan sent from the server is sent back to the terminal and presented to the user, who can then check the presented plan and make any necessary adjustments.

[0055] AR guidance

[0056] User

[0057] Once users start the tour, they can enable AR navigation on their device to receive on-site guidance.

[0058] Terminal

[0059] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real world. For example, instructions like "The next spot is 500 meters away, turn right" are displayed in real time.

[0060] Changing plans mid-way

[0061] User

[0062] If an unexpected change becomes necessary during the tour, the user re-enters the desired conditions and transmits them to the system.

[0063] server

[0064] The server then re-queues the generative AI based on the new conditions to generate a new sightseeing tour plan. The updated plan is sent to the device and provided to the user in real time.

[0065] Feedback and Ratings

[0066] User

[0067] After completing the tour, users enter feedback and ratings based on their experience into the UTour app.

[0068] Terminal

[0069] The terminal transmits the input feedback information to the server.

[0070] server

[0071] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system.

[0072] Specific examples

[0073] Example 1: One-day tour of historical sites

[0074] User A wishes to visit multiple historical sites in one day, with a budget of 5,000 yen or less. User A logs into "UTour" and enters his / her area of ​​interest as "historical sites," his / her budget as "under 5,000 yen," and the date of visit as "tomorrow." The server then has the generative AI generate a plan. The generated plan includes historical sites A, B, and C, and route guidance is displayed in AR. User A is satisfied with the presented plan and begins the tour.

[0075] Example 2: Changing the plan for an anime pilgrimage tour

[0076] While planning a tour of anime holy sites, User B unexpectedly needs to skip one of the stops. User B enters the new desired information through the app and sends it to the server. The server then requests the generative AI to generate a new plan. The regenerated plan is provided to User B in real time, and the AR navigation is also updated.

[0077] This allows users to always receive the best possible sightseeing experience.

[0078] The system provides a customized sightseeing experience tailored to the user's individual needs and circumstances, and its real-time navigation and information display enable efficient and fulfilling sightseeing.

[0079] The processing flow will be explained below.

[0080] User Registration and Login

[0081] Step 1:

[0082] The user launches the "UTour" app on their device and accesses the new registration page.

[0083] Step 2:

[0084] The user enters the required information such as name, email address, and password.

[0085] Step 3:

[0086] The terminal transmits the input information to the server.

[0087] Step 4:

[0088] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[0089] Step 5:

[0090] The user enters their email address and password on the login page of the "UTour" app.

[0091] Step 6:

[0092] The terminal transmits the login information to the server.

[0093] Step 7:

[0094] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[0095] Enter your interests and preferences

[0096] Step 1:

[0097] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[0098] Step 2:

[0099] The terminal transmits the input desired information to the server.

[0100] Step 3:

[0101] The server temporarily stores the received desired information in a database.

[0102] Tour plan generation and provision

[0103] Step 1:

[0104] The server sends the saved desired information to the generative AI.

[0105] Step 2:

[0106] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[0107] Step 3:

[0108] The generative AI returns the generated tour plan to the server.

[0109] Step 4:

[0110] The server transmits the generated tour plan to the terminal.

[0111] Step 5:

[0112] The terminal presents the received tour plan to the user.

[0113] Step 6:

[0114] The user can review the proposed plan and make fine adjustments as needed.

[0115] AR guidance

[0116] Step 1:

[0117] The user starts the tour and enables AR navigation on the device.

[0118] Step 2:

[0119] The device activates the camera function and prepares for AR display.

[0120] Step 3:

[0121] The server receives the current location information and transmits route information to the next destination.

[0122] Step 4:

[0123] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[0124] Changing plans mid-way

[0125] Step 1:

[0126] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[0127] Step 2:

[0128] The terminal sends the new conditions to the server.

[0129] Step 3:

[0130] The server will then make a new request to the generative AI based on the new conditions.

[0131] Step 4:

[0132] The generative AI generates a revised tour plan based on the new conditions.

[0133] Step 5:

[0134] The generative AI sends the revised tour plan back to the server.

[0135] Step 6:

[0136] The server sends the modified plan to the terminal.

[0137] Step 7:

[0138] The device displays the new tour plan to the user and updates the AR navigation.

[0139] Feedback and Ratings

[0140] Step 1:

[0141] After completing the tour, users can enter their tour rating and comments on the feedback page.

[0142] Step 2:

[0143] The terminal transmits the input feedback to the server.

[0144] Step 3:

[0145] The server stores the feedback in a database and uses it for analysis.

[0146] The above are the specific processing steps of the "UTour" system.

[0147] Example 1

[0148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] Conventional sightseeing tour systems have had difficulty quickly providing optimal plans tailored to individual users' interests and wishes. Furthermore, they lacked the ability to change plans mid-tour and real-time guidance, resulting in an unsatisfactory user experience. The present invention aims to solve these problems and provide customized sightseeing tour plans that satisfy users.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0151] In this invention, the server includes means for receiving information on interests and desires input by a user, means for temporarily storing the received information in a database, means for generating an optimal sightseeing tour plan based on the received information on interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, and means for regenerating the sightseeing tour plan based on the new conditions using the generative AI when the user changes the plan during the tour, thereby enabling users to receive an optimal sightseeing experience tailored to their individual interests and desires in real time.

[0152] "Interests and desired information input by the user" refers to information about the user's individual areas of interest and desired conditions for the sightseeing tour (for example, tourist spots, budget, planned dates of visit, etc.).

[0153] The "receiving means" is a mechanism that has the function of acquiring information entered by the user from the terminal and transmitting it to the server.

[0154] The "means for temporarily storing in a database" is a system for temporarily storing received information about the user's interests and preferences, and can be, for example, a NoSQL database or an SQL database.

[0155] "Generative AI" is an artificial intelligence model that generates specific outputs based on input conditions, such as generating sightseeing tour plans using natural language processing.

[0156] The "means for generating optimal sightseeing tour plans" is an algorithm that uses generative AI to construct the order in which tourist spots should be visited and a schedule based on information input by the user.

[0157] The "means for presenting the generated sightseeing tour plan to the user" is a system for displaying the generated tour plan information on the terminal in a format that is easy for the user to understand.

[0158] "Means for displaying real-world guidance as augmented reality" refers to augmented reality technology that uses camera functions to display route guidance and tourist information overlaid on real-world scenery.

[0159] "Means for regenerating sightseeing tour plans based on new conditions when the plan is changed" refers to a mechanism that, when a user inputs new conditions during a tour, requests the generative AI to create a new sightseeing tour plan again.

[0160] The "means for saving feedback" is a system for saving the feedback entered by the user after the tour has ended in a database.

[0161] "Means for user authentication" refers to the process of verifying the identity of a user when logging into a system, for example by verifying an email address and password.

[0162] "Means for securely storing authenticated user data" refers to technology that protects the personal information and interest / desired information of authenticated users while safely storing them in a database.

[0163] This invention is a system that provides users with optimized sightseeing tour plans, automatically generating plans based on the user's individual interests and wishes, and providing real-time navigation and information. A specific embodiment of this system will be described.

[0164] User Registration and Login

[0165] User

[0166] Users access the "Tourism Planning App" using a device such as a smartphone or tablet and register or log in. When registering, users enter their name, email address, and password, which are then sent to the system. If they already have an account, they enter their email address and password for authentication.

[0167] Terminal

[0168] The terminal sends the input information to the server using the HTTPS protocol. Specifically, it makes a POST request to the API endpoint (e.g., / api / login) on the server.

[0169] server

[0170] The server receives the transmitted information and verifies it against the user information stored in a database (e.g., MySQL (registered trademark)). If the verification is successful, it generates an authenticated session ID, stores the session information in an in-memory database such as Redis, and sends it back to the terminal.

[0171] Enter your interests and preferences

[0172] User

[0173] Users go to the tour plan creation page of the "tourism planning app" and enter information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and planned date of visit.

[0174] Terminal

[0175] The terminal parses the input information into JSON format and sends it to the server. Specifically, it makes a POST request to the / api / planrequest endpoint.

[0176] server

[0177] The server temporarily stores the received information in a NoSQL database (e.g., MongoDB).

[0178] Tour plan generation and provision

[0179] server

[0180] The server generates a prompt based on the stored information and requests it from a generative AI model (e.g., GPT-4 (registered trademark)). An example of a prompt is: "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan that matches these."

[0181] Based on these prompts, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[0182] Terminal

[0183] The tour plan sent from the server is received and presented to the user, who can then check the plan and make any necessary adjustments.

[0184] AR guidance

[0185] User

[0186] When users start the tour, they enable AR navigation on their device and receive on-site guidance.

[0187] Terminal

[0188] The device utilizes its camera function to display route guidance and tourist information superimposed on the real landscape as augmented reality (AR). For example, instructions such as "The next spot is 500 meters away, turn right" are displayed in real time.

[0189] Changing plans mid-way

[0190] User

[0191] If an unplanned change occurs during the tour, the user enters new desired conditions and submits them to the system.

[0192] server

[0193] The server then requests the generative AI model again based on the new conditions to generate a new sightseeing tour plan. The updated plan is then sent back to the device and provided to the user in real time.

[0194] Feedback and Ratings

[0195] User

[0196] After completing the tour, users enter feedback and ratings based on their experience into the "tourism planning app."

[0197] Terminal

[0198] The terminal transmits the input feedback information to the server.

[0199] server

[0200] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system's accuracy and user satisfaction.

[0201] This will allow users to enjoy a customized sightseeing experience in real time that is tailored to their individual interests and desires.

[0202] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0203] Step 1: User registers or logs in

[0204] User

[0205] Users access the "Tourism Planning App" from their device and register an account or log in. If registering for the first time, they enter information such as their name, email address, and password, and submit. If they already have an account, they enter their email address and password for authentication.

[0206] Terminal

[0207] The device sends the entered information to the server using HTTPS communication, by sending a POST request to an API endpoint such as / api / login.

[0208] server

[0209] The server compares the received information with a database (e.g., MySQL) and saves it in the database in the case of a new registration. In the case of a login, the authentication information is checked, and if it matches, a session ID is generated, encrypted, and sent back to the terminal. The session information is stored in an in-memory database such as Redis.

[0210] Input and Output

[0211] Input: Registration and login information from the user

[0212] Output: Authenticated session ID

[0213] Step 2: User enters their interests and preferences

[0214] User

[0215] The user proceeds to the tour plan creation page and enters information such as areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[0216] Terminal

[0217] The terminal converts the input information into JSON format and sends it to the server. Specifically, it makes a POST request to / api / planrequest.

[0218] server

[0219] The server temporarily stores the received information in a database, such as MongoDB.

[0220] Input and Output

[0221] Input: User's interests and preferences

[0222] Output: Information ID stored in the database

[0223] Step 3: Request to generate a tour plan

[0224] server

[0225] The server generates prompts based on the stored information and sends them to a generative AI model (e.g., GPT-4). For example, it generates a prompt such as, "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan based on this information."

[0226] Input and Output

[0227] Input: Saved user information

[0228] Output: Prompt for the generative AI

[0229] Step 4: Plan generation using a generative AI model

[0230] Generative AI

[0231] Based on the prompts received, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule.

[0232] Input and Output

[0233] Input: Prompt sentence for generative AI

[0234] Output: Optimal sightseeing tour plan

[0235] Step 5: Present your tour plan

[0236] server

[0237] The generated tour plan is sent to the server, and the server retransmits the result to the terminal.

[0238] Terminal

[0239] The terminal presents the received tour plan to the user, who can then review the plan and make minor adjustments as necessary.

[0240] Input and Output

[0241] Input: Generated tour plan

[0242] Output: The tour plan displayed to the user

[0243] Step 6: Enabling AR Navigation

[0244] User

[0245] The user enables the AR navigation function on their device when starting the tour.

[0246] Terminal

[0247] The device uses its camera to display route guidance to tourist spots in augmented reality (AR) overlaid on the real-world scenery, with real-time guidance such as "Next destination 500m, turn right."

[0248] Input and Output

[0249] Input: Location information based on tour plan

[0250] Output: AR display overlaid on real scenery

[0251] Step 7: Change your plan mid-way

[0252] User

[0253] If an unplanned change is required during the tour, the user enters new requirements and submits them to the system.

[0254] Terminal

[0255] The terminal transmits the new desired conditions to the server.

[0256] server

[0257] The server then resubmits the request to the generative AI model based on the new conditions to generate a new tour plan, which is then sent back to the device and provided to the user in real time.

[0258] Input and Output

[0259] Input: User's new preferences

[0260] Output: New sightseeing tour plan

[0261] Step 8: Enter and save your feedback

[0262] User

[0263] After completing the tour, users provide feedback and a rating based on their experience.

[0264] Terminal

[0265] The terminal transmits the input feedback information to the server.

[0266] server

[0267] The server stores the collected feedback information in a database and uses it as reference information when generating subsequent tour plans.

[0268] Input and Output

[0269] Input: User feedback information

[0270] Output: Feedback information stored in a database

[0271] Through this series of processes, users can enjoy sightseeing tour plans optimized in real time to suit their individual interests and desires.

[0272] (Application example 1)

[0273] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] Conventional sightseeing tour planning systems do not take into account the user's consumption preferences or in-store route guidance, which makes it difficult to achieve efficient shopping in stores. Furthermore, guidance using augmented reality (AR) technology is limited, making it difficult for users to quickly and intuitively obtain the information they need in-store. This often results in an unsatisfactory shopping experience for users.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0276] In this invention, the server includes means for receiving information on interests and desires input by a user, means for generating an optimal sightseeing tour plan and in-store route based on the received information on interests and desires using a generative AI, means for presenting the generated plan and route to the user, means for displaying real-world guidance as augmented reality, means for regenerating the plan based on new conditions using the generative AI when the user changes the plan during the tour or shopping, and means for providing in-store product locations as augmented reality, thereby enabling users to enjoy optimal sightseeing and shopping experiences based on their individual interests and consumption preferences.

[0277] "Interest and desired information" is a general term for information that a user inputs about their areas of interest and the experiences they are looking for.

[0278] "Generative AI" is an artificial intelligence technology that automatically generates optimal plans and proposals based on input information.

[0279] A "tourist tour plan" refers to a travel plan that includes the order and schedule of visiting tourist spots.

[0280] Augmented reality (AR) is a technology that displays digital information overlaid on real-world scenery.

[0281] "In-store route" refers to an optimal route plan for a user to move efficiently within the store.

[0282] "Product suggestions" are information about products recommended based on the user's consumption preferences.

[0283] "User authentication" is the process of identifying an individual user and verifying their access rights.

[0284] "Consumption preferences" refers to trends in a user's purchasing behavior and preferences.

[0285] "Feedback" refers to the evaluations and opinions provided by users after using the service.

[0286] "Plan regeneration" is the process of regenerating an existing plan based on new conditions.

[0287] The present invention is a system that provides users with optimized sightseeing tour plans and in-store routes. This system generates plans based on the user's interests and desired information, and provides guide information using augmented reality (AR) technology. Specific embodiments of the present invention are described below.

[0288] User Registration and Login

[0289] User

[0290] Users access a dedicated application through their device (such as a smartphone or tablet) to register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register with the system. When logging in with an existing account, users enter their email address and password for authentication.

[0291] server

[0292] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0293] Enter your interests and preferences

[0294] User

[0295] The user goes to the planning page of the dedicated application and enters information such as their areas of interest (e.g., history, fashion, food, etc.), budget, and planned dates of visit.

[0296] Terminal

[0297] The terminal transmits the inputted interest and desired information to the server.

[0298] server

[0299] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal plan.

[0300] Generate and deliver plans

[0301] server

[0302] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which tourist spots should be visited, a schedule, in-store routes, and product suggestions, and sends the results back to the server.

[0303] Terminal

[0304] The plan sent from the server is sent back to the terminal and presented to the user, who can then check the plan and make any necessary adjustments.

[0305] AR guidance

[0306] User

[0307] Once the user executes the plan, they can enable AR navigation on their device to receive on-site guidance.

[0308] Terminal

[0309] The device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. For example, instructions such as "The next stop is 500 meters away, turn right" or "Recommended products are on this shelf" are displayed in real time.

[0310] Changing plans mid-way

[0311] User

[0312] If an unexpected change is required during the plan, the user re-enters the desired conditions and sends them to the system.

[0313] server

[0314] The server then re-submits the request to the AI ​​based on the new conditions to generate a new plan. The updated plan is then sent to the device and provided to the user in real time.

[0315] Feedback and Ratings

[0316] User

[0317] After completing the plan, users enter feedback and ratings based on their experience into a dedicated application.

[0318] Terminal

[0319] The terminal transmits the input feedback information to the server.

[0320] server

[0321] The server stores the collected feedback in a database and uses it as reference information when generating future plans, thereby improving the system.

[0322] Specific prompt examples

[0323] "Generate the optimal shopping plan based on the following user preferences and wishes: Preferences: Apparel, Budget: Under 10,000 yen, Season: Spring"

[0324] This system enables users to efficiently enjoy optimal sightseeing and shopping experiences based on their interests and consumption preferences.

[0325] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0326] Step 1:

[0327] A user accesses a dedicated application using a terminal to register or log in. As input, they provide their name, email address, and password. The server receives this information and compares it with the user information stored in a database. As output, it generates an authenticated session and sends it back to the terminal.

[0328] Step 2:

[0329] The user enters information such as their areas of interest, budget, and planned visit date on the plan creation page of the dedicated application. The input information includes interests and wishes. The device sends this information to the server, which then temporarily stores the received information in a database.

[0330] Step 3:

[0331] The server sends the stored information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which to visit tourist spots, a schedule, in-store routes, and product suggestions. The optimal plan is then returned to the server as output.

[0332] Step 4:

[0333] The server sends the plan returned by the generative AI to the device and presents it to the user. The user can then review the plan and make any necessary adjustments. At this stage, the user can also enable AR navigation for when they arrive at the destination.

[0334] Step 5:

[0335] When the user executes the plan, the device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. The input includes the user's current location and the generated plan. The output is the route instructions and product location information displayed on the real world.

[0336] Step 6:

[0337] If an unexpected change is required midway through the plan, the user re-enters their desired conditions and sends them to the system. The new desired conditions are included as input. The server then requests the generative AI again based on the new conditions to generate a new plan. As output, the updated plan is sent to the device and provided to the user in real time.

[0338] Step 7:

[0339] After completing the plan, the user enters feedback and evaluation based on their experience into a dedicated application. The input includes the user's feedback. The device sends this feedback information to the server, which stores it in a database. The output provides reference information for use in improving the system.

[0340] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0341] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[0342] User Registration and Login

[0343] User

[0344] Users access the "UTour" app through their device (smartphone or tablet) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[0345] server

[0346] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0347] Enter your interests and preferences

[0348] User

[0349] The user goes to the "Tour Plan Creation" page on the terminal and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[0350] Terminal

[0351] The terminal transmits the input information to the server.

[0352] server

[0353] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[0354] Tour plan generation and provision

[0355] server

[0356] The server sends the stored information to a generative AI, which determines tourist spots and the order in which to visit them based on the user's interests and preferences, and generates a tour plan. The generated plan is then sent back to the server, which then sends it to the device.

[0357] Terminal

[0358] The terminal presents the received tour plan to the user, who can then check the plan and make fine adjustments to it as necessary.

[0359] Introducing the Emotion Engine

[0360] User

[0361] During the tour, the emotion engine collects data such as facial expressions and voice in real time, and this emotional information reflects how the user feels throughout the tour experience.

[0362] Terminal

[0363] The terminal transmits the collected emotion information to the server.

[0364] server

[0365] The server analyzes the emotional information sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI to adjust the tour plan. For example, if the user feels tired, it adds rest stops and sends the adjusted plan back to the device.

[0366] AR guidance and plan changes

[0367] User

[0368] Once a user starts a tour, they can activate AR navigation on their device to receive on-site guidance. If an unexpected change is needed during the tour, they can enter their new preferences and submit them to the system.

[0369] Terminal

[0370] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real landscape. When the user inputs new desired conditions, the device sends them to the server.

[0371] server

[0372] The server then requests the generative AI to generate a new sightseeing tour plan based on the new conditions. The updated plan is sent to the device and provided to the user in real time.

[0373] Feedback and Ratings

[0374] User

[0375] After the tour, users can access the feedback page in the UTour app and enter their tour rating and comments.

[0376] Terminal

[0377] The terminal transmits the input feedback information to the server.

[0378] server

[0379] The server stores the feedback in a database and uses it for analysis. Emotion data collected by the emotion engine is also stored as feedback information and taken into account when generating the next plan.

[0380] Specific examples

[0381] Example 1: Tailoring tours to match the emotions of history buffs

[0382] User A plans a one-day tour of historical sites, with his / her area of ​​interest being "historical sites," his / her budget being 5,000 yen, and his / her visit date being the following day. The system uses generative AI to create an optimal tour plan and guides him / her with AR navigation. During the tour, the emotion engine recognizes that User A is tired and sends this to the server. The server then generates a new plan with appropriate rest spots added and sends it to the device.

[0383] Example 2: Anime pilgrimage with changes and emotional feedback

[0384] During a tour of anime holy sites, User B must skip one of the stops due to unplanned circumstances. The emotion engine detects stress and anxiety and reports this to the server. The server regenerates the tour plan based on the new conditions and provides it to the user. After the tour ends, User B inputs feedback and emotion data, and the emotion engine provides new insights for the next tour.

[0385] By incorporating an emotion engine, the present invention makes it possible to provide flexible tour plans that correspond to the user's emotional state, thereby realizing a more advanced and personalized sightseeing experience.

[0386] The processing flow will be explained below.

[0387] User Registration and Login

[0388] Step 1:

[0389] The user launches the "UTour" app on their device and accesses the new registration page.

[0390] Step 2:

[0391] The user enters the required information such as name, email address, and password.

[0392] Step 3:

[0393] The terminal transmits the input information to the server.

[0394] Step 4:

[0395] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[0396] Step 5:

[0397] The user enters their email address and password on the login page of the "UTour" app.

[0398] Step 6:

[0399] The terminal transmits the login information to the server.

[0400] Step 7:

[0401] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[0402] Enter your interests and preferences

[0403] Step 1:

[0404] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[0405] Step 2:

[0406] The terminal transmits the input desired information to the server.

[0407] Step 3:

[0408] The server temporarily stores the received desired information in a database.

[0409] Tour plan generation and provision

[0410] Step 1:

[0411] The server sends the saved desired information to the generative AI.

[0412] Step 2:

[0413] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[0414] Step 3:

[0415] The generative AI returns the generated tour plan to the server.

[0416] Step 4:

[0417] The server transmits the generated tour plan to the terminal.

[0418] Step 5:

[0419] The terminal presents the received tour plan to the user.

[0420] Step 6:

[0421] The user can review the proposed plan and make fine adjustments as needed.

[0422] Introducing the Emotion Engine

[0423] Step 1:

[0424] During the tour, users use the device's camera and microphone to input their facial expressions and voices into the emotion engine.

[0425] Step 2:

[0426] The terminal recognizes the user's emotions in real time using an emotion engine and transmits the data to the server.

[0427] Step 3:

[0428] The server analyzes the emotion information sent from the emotion engine, and if the user's emotion indicates discomfort or stress, it determines that the tour plan needs to be adjusted.

[0429] Step 4:

[0430] The server requests generative AI to adjust the tour plan based on the emotional information.

[0431] Step 5:

[0432] The generative AI generates a revised tour plan based on the new conditions.

[0433] Step 6:

[0434] The generative AI sends the revised tour plan back to the server.

[0435] Step 7:

[0436] The server sends the modified plan to the terminal.

[0437] Step 8:

[0438] The device displays the new tour plan to the user and updates the AR navigation.

[0439] AR guidance and plan changes

[0440] Step 1:

[0441] The user starts the tour and enables AR navigation on the device.

[0442] Step 2:

[0443] The device activates the camera function and prepares for AR display.

[0444] Step 3:

[0445] The server receives the current location information and transmits route information to the next destination.

[0446] Step 4:

[0447] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[0448] Step 5:

[0449] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[0450] Step 6:

[0451] The terminal sends the new conditions to the server.

[0452] Step 7:

[0453] The server will then make a new request to the generative AI based on the new conditions.

[0454] Step 8:

[0455] The generative AI generates a revised tour plan based on the new conditions.

[0456] Step 9:

[0457] The generative AI sends the revised tour plan back to the server.

[0458] Step 10:

[0459] The server sends the modified plan to the terminal.

[0460] Step 11:

[0461] The device displays the new tour plan to the user and updates the AR navigation.

[0462] Feedback and Ratings

[0463] Step 1:

[0464] After completing the tour, users can enter their tour rating and comments on the feedback page.

[0465] Step 2:

[0466] The terminal transmits the input feedback to the server.

[0467] Step 3:

[0468] The server stores the feedback in a database and uses it for analysis. It also stores the emotion data collected by the emotion engine as feedback information.

[0469] The above are the specific processing steps of the "UTour" system, which incorporates an emotion engine.

[0470] Example 2

[0471] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] Conventional tourist tour plan generation systems provide tour plans based on the user's interests and preferences, but they are unable to consider information about the user's emotions and psychological state. Furthermore, they lack the means to dynamically adjust tour plans in real time, making it difficult to maximize user satisfaction. Therefore, there is a need for a system that can flexibly adjust tour plans based on the user's emotions and provide a personalized tourist experience.

[0473] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving interest and preference information input by a user; means for generating an optimal sightseeing tour plan based on the received interest and preference information using a generative AI model; means for presenting the generated sightseeing tour plan to the user; means for displaying real-world guidance as augmented reality; means for regenerating the sightseeing tour plan based on new conditions using the generative AI model when the user changes the plan during the tour; means having an emotion engine for collecting the user's facial expressions and voice in real time; and means for analyzing the collected emotion data and adjusting the tour plan according to the emotion. This makes it possible to provide flexible tour plans that correspond to the user's emotional state, resulting in a more advanced and personalized sightseeing experience.

[0474] A "user" is a person who uses the system to create a sightseeing tour plan and enjoy a personalized sightseeing experience.

[0475] "Interest and desired information" is information input by the user such as areas of interest, budget, and dates of visit related to tourism.

[0476] A "generative AI model" is an artificial intelligence model that generates optimal sightseeing tour plans based on the user's interests and desired information.

[0477] A "tour plan" is a tour plan that includes specific tourist spots and the order in which they are to be visited, created by a generative AI model based on the user's interests and preferences.

[0478] "Augmented reality" is a technology that displays digital information overlaid on a real-world environment.

[0479] The "emotion engine" is part of a system that collects and analyzes emotions from the user's facial expressions and voice in real time.

[0480] "Emotion data" refers to data such as facial expressions and voice collected by the user's emotion engine.

[0481] "Feedback" refers to information such as ratings and comments entered by the user after the tour has ended.

[0482] "User authentication" is the process of verifying a user's identity when accessing a system.

[0483] "Generate" means that the generative AI model creates a sightseeing tour plan based on the user's interests and desired information.

[0484] "Adjustment" means making changes or corrections to existing sightseeing tour plans based on the results of analyzing emotional data.

[0485] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[0486] 1. User Registration and Login

[0487] User

[0488] Users access the dedicated application using a device such as a smartphone or tablet. When using for the first time, they create a new account by entering their name, email address, and password on the new registration screen. Existing users can authenticate by entering their email address and password on the login screen.

[0489] Terminal

[0490] The terminal transmits the registration information or login information input by the user to the server.

[0491] server

[0492] The server checks the entered information against the user information in the database. If the check is successful, it creates an authenticated session and returns it to the terminal. If the check is unsuccessful, it creates an error message and returns it to the terminal.

[0493] 2. Enter your interests and preferences

[0494] User

[0495] The user goes to the "Create a Tour Plan" page and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[0496] Terminal

[0497] The terminal transmits the inputted interest and desired information to the server.

[0498] server

[0499] The server stores the received information in a database, and uses a generative AI model to generate prompts based on that information to create optimal sightseeing tour plans.

[0500] 3. Creating and providing tour plans

[0501] server

[0502] The server sends prompts to the generative AI model, which then generates a tour plan based on the user's interests, budget, visit dates, and other information, including the selection of tourist spots and the order in which to visit them.

[0503] Terminal

[0504] The tour plan received from the server is presented to the user, who can then review the plan and make any necessary adjustments.

[0505] 4. Introducing the Emotion Engine

[0506] User

[0507] During the tour, the emotion engine collects the user's facial expressions and voice data in real time, and this emotion information reflects the user's emotional state throughout the tour experience.

[0508] Terminal

[0509] The device transmits the collected emotion data to the server in real time.

[0510] server

[0511] The server analyzes the emotion data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to readjust the tour plan. For example, if the user shows signs of fatigue, it will regenerate the plan by adding rest stops and send it to the device.

[0512] 5. AR Guidance and Plan Changes

[0513] User

[0514] Once the tour begins, users can activate AR navigation on their device to receive on-site guidance. If unplanned changes occur, users can enter new preferences and submit them to the system.

[0515] Terminal

[0516] The device displays route guidance and tourist information superimposed on the real landscape as AR, and also transmits new desired conditions to the server.

[0517] server

[0518] The server then requests the generative AI model to generate a new sightseeing tour plan based on the new conditions, and the plan is provided to the device in real time.

[0519] 6. Feedback and Ratings

[0520] User

[0521] After the tour is over, the user inputs feedback and sends the feedback and emotion data to the server.

[0522] Terminal

[0523] The terminal transmits the feedback information to the server.

[0524] server

[0525] The server stores the feedback and emotion data in a database and takes them into account when generating the next plan.

[0526] Specific examples

[0527] Example 1: Tour plan for history buffs

[0528] User A selects "historical sites" as an area of ​​interest, budgets 5,000 yen, and the date of visit for the following day. The server generates an appropriate tour plan using a generative AI model and sends it to the device. If User A shows signs of fatigue during the tour, the server incorporates rest spots based on information from the emotion engine and resends the adjusted plan.

[0529] Example 2: Anime pilgrimages that require changes mid-trip

[0530] While touring the anime holy sites, User B needs to skip one of the stops. The emotion engine detects the stress and reports it to the server. The server regenerates the tour plan taking the new conditions into account and provides it to User B. After the tour ends, the feedback and emotion data are used to generate the next plan.

[0531] Example prompt sentence:

[0532] "Generate a tour plan for historical sites with a budget of 5000 yen. The visit date is tomorrow."

[0533] The above is an embodiment of the sightseeing tour plan creation system of the present invention. By using this system, it is possible to provide flexible tour plans that correspond to the emotional state of the user, realizing a more advanced and personalized sightseeing experience.

[0534] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0535] Step 1: User Registration

[0536] Users access the application using a smartphone or tablet and enter their name, email address, and password on the new registration screen (input: user information).

[0537] The terminal transmits the input user information to the server (output: user information).

[0538] The server stores the received user information in a database, generates a registration completion message and an authentication token, and returns them to the terminal (data processing: store user information in database, data calculation: generate token) (output: registration completion message, authentication token).

[0539] The terminal displays a registration completion message to the user (action: display message).

[0540] Step 2: User Login

[0541] The user enters their email address and password on the login screen (input: login information).

[0542] The terminal sends the login information to the server (output: login information).

[0543] The server checks the entered login information against the database, and if it matches, it creates an authenticated session; if it doesn't, it generates an error message and sends it to the terminal (data processing: checking login information, data calculation: session generation) (output: authenticated session or error message).

[0544] The terminal maintains the authenticated session and displays a login success message or an error message to the user (action: display message).

[0545] Step 3: Enter your interests and preferences

[0546] The user enters the area of ​​interest, budget, visit date, etc. on the "Create Tour Plan" page and taps the submit button (input: area of ​​interest, budget, visit date).

[0547] The terminal sends the input information to the server (output: areas of interest, budget, date of visit).

[0548] The server stores the received information in a database and generates a prompt statement based on that information (data processing: storing in a database, data calculation: generating a prompt statement) (output: prompt statement).

[0549] Step 4: Generate a tour plan

[0550] The server sends the generated prompt sentence to the generative AI model (input: prompt sentence) (output: generation request).

[0551] The generative AI model generates a tour plan including tourist attractions and the order in which to visit them based on the prompt sentence (Data calculation: Generate tour plan) (Output: Tour plan).

[0552] The server receives the generated tour plan and transmits it to the terminal (output: tour plan).

[0553] Step 5: Present your tour plan

[0554] The terminal displays the received tour plan to the user (operation: display plan).

[0555] The user checks the presented plan and makes fine adjustments as necessary (input: fine adjustment information).

[0556] Step 6: Collect emotion data

[0557] During the tour, the emotion engine collects the user's facial expression and voice data in real time (input: facial expression data, voice data).

[0558] The terminal transmits the collected emotion data to the server in real time (output: emotion data).

[0559] Step 7: Analyze emotional data and adjust your plan

[0560] The server analyzes the emotional data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to adjust a new tour plan (data calculation: analysis of emotional data, regeneration of plan) (output: readjustment request).

[0561] The generative AI model generates a new tour plan with additional rest stops added as needed and sends it to the server (output: adjusted tour plan).

[0562] The server transmits the adjusted tour plan to the terminal (output: adjusted tour plan).

[0563] The terminal presents the adjusted tour plan to the user (action: show plan).

[0564] Step 8: Providing AR Navigation

[0565] When the user starts the tour, he or she uses the AR navigation of the device (action: start AR navigation).

[0566] The device displays route guidance and tourist information superimposed on real-world scenery (operation: composite display of the physical world and digital information).

[0567] Step 9: Feedback and Rating

[0568] After completing the tour, the user enters a rating and comments on the feedback page (input: feedback information).

[0569] The terminal transmits the feedback information to the server (output: feedback information).

[0570] The server stores the feedback and emotion data in a database and takes them into consideration when generating the next plan (data processing: save, data calculation: take into consideration when generating the next plan).

[0571] Through these steps, the system can flexibly adjust the tour plan according to the user's emotional state and provide a personalized sightseeing experience.

[0572] (Application example 2)

[0573] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0574] Conventional sightseeing tour planning systems can generate plans based on the user's interests and preferences, but they cannot adjust the plan to accommodate real-time emotional changes during the tour. Furthermore, no systems exist that can personalize the in-store shopping experience or promote sales based on the user's emotional state. This makes it difficult to provide an optimal and comfortable experience for users.

[0575] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information about interests and desires input by a user, means for generating an optimal sightseeing tour plan based on the received information about interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, means for regenerating the sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour, and means for detecting the user's emotions and adjusting the plan based on the emotions. This makes it possible to adjust the plan in response to real-time changes in the user's emotions, providing a more personalized and comfortable experience.

[0576] The "means for receiving information about interests and desires entered by the user" refers to a device such as a smartphone or tablet that allows the user to enter information about their interests and desires, which is then received by the system.

[0577] "Means for generating optimal sightseeing tour plans based on the received information on interests and desires using generative AI" refers to a means for automatically generating optimal sightseeing tour plans using artificial intelligence technology based on the received information on the user's interests and desires.

[0578] The "means for presenting the generated sightseeing tour plan to the user" is a means for presenting the generated sightseeing tour plan to the user in an easy-to-view manner, allowing the user to confirm and adjust the plan.

[0579] "Means for displaying real-world guidance as augmented reality" refers to the display of virtual information superimposed on real-world scenery, making it easier for users to visually understand the information.

[0580] "Means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour" refers to a means for the generative AI to regenerate a new sightseeing tour plan in response to changes in the user's wishes or situation during the tour.

[0581] The "means for detecting a user's emotions and adjusting the plan based on the emotions" is for detecting a user's real-time emotional state and adjusting the tour plan or shopping experience according to the emotions.

[0582] The present invention is a system that provides users with optimized sightseeing tour plans and in-store shopping plans, and also detects the user's emotions in real time and adjusts the plans based on the emotions.

[0583] Hardware and Software Configuration

[0584] Hardware

[0585] Smart Glasses: Devices with built-in camera and display capabilities (e.g., Google® Glass®, Vuzix Blade).

[0586] Server: A high-performance server device that runs a database and generative AI.

[0587] software

[0588] Emotion engine: A software module that uses libraries such as OpenCV to analyze the user's facial expressions and detect emotions.

[0589] Generative AI: A module that uses generative AI models such as GPT-3 (registered trademark) to generate and regenerate optimal plans based on the user's interests, desired information, and emotional data.

[0590] AR module: A software module that displays virtual guidance information overlaid on the real-world scenery.

[0591] System Operation

[0592] User Registration and Login

[0593] The user puts on the smart glasses and accesses the app to register or log in. The server receives the user's registration information or login information and compares it with the user information stored in the database.

[0594] Enter your interests and preferences

[0595] Through the smart glasses, users input information such as their interests, budget, and visit date, which is then sent to the server.

[0596] Plan generation and presentation

[0597] The server then sends the received information about interests and preferences to the generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the information. The generated plans are then sent from the server to the smart glasses and presented to the user.

[0598] Real-time emotion detection and plan adjustment

[0599] As the user tours or shops, the smart glasses' camera captures the user's facial expressions, and the emotion engine detects their emotions in real time. The emotion information is sent to the server, which analyzes the information from the emotion engine. If the user feels stressed or tired, the generative AI regenerates the plan based on the new conditions.

[0600] Augmented reality (AR) guidance

[0601] The smart glasses overlay virtual guide information on real-world scenery, providing users with visually easy-to-understand navigation and tourist information.

[0602] Specific examples

[0603] Scenario 1: If a user feels tired while shopping, the emotion engine detects this. The generative AI recommends a simple rest area and refreshing items (e.g., drinks). The in-store AR display also guides the user to the shortest route.

[0604] Scenario 2: When a user makes an unexpected change during a sightseeing tour, they input their new interests and preferences and submit them to the system. The generative AI then regenerates the optimal tour plan based on the new conditions and displays it on the smart glasses.

[0605] Example prompt sentence:

[0606] The user's interests are "fashion" and "technology," their budget is 10,000 yen, and their current emotion is "tired." Please suggest places to rest and products that will refresh them.

[0607] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0608] Step 1:

[0609] A user puts on smart glasses and accesses the app to register or log in. The input is the user's registration information (name, email address, password) or login information (email address, password). The server receives this information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session and sends it back to the device. The output is the authenticated session information.

[0610] Step 2:

[0611] The user inputs information such as their interests, budget, and visit date through the smart glasses. The input information is sent from the device to the server. The server temporarily stores the received interest and preference information in a database. The input is the interest and preference information entered by the user, and the output is the saved interest and preference information.

[0612] Step 3:

[0613] The server sends the stored interests and preferences to a generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the interests and preferences. The input is the interests and preferences, and the output is the generated plan.

[0614] Step 4:

[0615] The generated plan is sent from the server to the smart glasses. The device presents the generated plan to the user. The input is the generated plan, and the output is the presented plan. The user can review the plan and make fine adjustments if necessary.

[0616] Step 5:

[0617] As the user tours or goes shopping, the camera in the smart glasses continues to capture the user's facial expressions. Based on the captured frames, the device uses an emotion engine to detect the user's emotions in real time. The input is the camera frames, and the output is the detected emotion data.

[0618] Step 6:

[0619] The detected emotional information is sent from the device to the server. The server analyzes the information from the emotion engine, and if the user is feeling stressed or tired, the generative AI regenerates a plan based on the new conditions. The input is emotional data, and the output is the regenerated plan.

[0620] Step 7:

[0621] The smart glasses receive the regenerated plan and overlay virtual guide information on the real-world scenery, providing the user with a visually easy-to-understand navigation or shopping guide. The input is the regenerated plan, and the output is the guide information displayed in AR.

[0622] Step 8:

[0623] After completing the tour or shopping, the user accesses the feedback page and enters their experience rating and comments. The device sends the entered feedback information to the server, which stores the feedback in a database and uses it for analysis. The input is the user's feedback, and the output is the stored feedback information.

[0624] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0625] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0627] [Second embodiment]

[0628] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0629] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0630] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0632] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0634] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0635] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0636] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0637] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0638] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0639] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0640] The present invention is a system for providing users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Specific embodiments of the system of the present invention will be described below.

[0641] User Registration and Login

[0642] User

[0643] Users access the "UTour" app through their device (smartphone, tablet, etc.) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[0644] server

[0645] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0646] Enter your interests and preferences

[0647] User

[0648] Users go to the tour plan creation page of the "UTour" app and enter information such as their areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[0649] Terminal

[0650] The terminal transmits the inputted interest and desired information to the server.

[0651] server

[0652] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[0653] Tour plan generation and provision

[0654] server

[0655] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[0656] Terminal

[0657] The tour plan sent from the server is sent back to the terminal and presented to the user, who can then check the presented plan and make any necessary adjustments.

[0658] AR guidance

[0659] User

[0660] Once users start the tour, they can enable AR navigation on their device to receive on-site guidance.

[0661] Terminal

[0662] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real world. For example, instructions like "The next spot is 500 meters away, turn right" are displayed in real time.

[0663] Changing plans mid-way

[0664] User

[0665] If an unexpected change becomes necessary during the tour, the user re-enters the desired conditions and transmits them to the system.

[0666] server

[0667] The server then re-queues the generative AI based on the new conditions to generate a new sightseeing tour plan. The updated plan is sent to the device and provided to the user in real time.

[0668] Feedback and Ratings

[0669] User

[0670] After completing the tour, users enter feedback and ratings based on their experience into the UTour app.

[0671] Terminal

[0672] The terminal transmits the input feedback information to the server.

[0673] server

[0674] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system.

[0675] Specific examples

[0676] Example 1: One-day tour of historical sites

[0677] User A wishes to visit multiple historical sites in one day, with a budget of 5,000 yen or less. User A logs into "UTour" and enters his / her area of ​​interest as "historical sites," his / her budget as "under 5,000 yen," and the date of visit as "tomorrow." The server then has the generative AI generate a plan. The generated plan includes historical sites A, B, and C, and route guidance is displayed in AR. User A is satisfied with the presented plan and begins the tour.

[0678] Example 2: Changing the plan for an anime pilgrimage tour

[0679] While planning a tour of anime holy sites, User B unexpectedly needs to skip one of the stops. User B enters the new desired information through the app and sends it to the server. The server then requests the generative AI to generate a new plan. The regenerated plan is provided to User B in real time, and the AR navigation is also updated.

[0680] This allows users to always receive the best possible sightseeing experience.

[0681] The system provides a customized sightseeing experience tailored to the user's individual needs and circumstances, and its real-time navigation and information display enable efficient and fulfilling sightseeing.

[0682] The processing flow will be explained below.

[0683] User Registration and Login

[0684] Step 1:

[0685] The user launches the "UTour" app on their device and accesses the new registration page.

[0686] Step 2:

[0687] The user enters the required information such as name, email address, and password.

[0688] Step 3:

[0689] The terminal transmits the input information to the server.

[0690] Step 4:

[0691] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[0692] Step 5:

[0693] The user enters their email address and password on the login page of the "UTour" app.

[0694] Step 6:

[0695] The terminal transmits the login information to the server.

[0696] Step 7:

[0697] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[0698] Enter your interests and preferences

[0699] Step 1:

[0700] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[0701] Step 2:

[0702] The terminal transmits the input desired information to the server.

[0703] Step 3:

[0704] The server temporarily stores the received desired information in a database.

[0705] Tour plan generation and provision

[0706] Step 1:

[0707] The server sends the saved desired information to the generative AI.

[0708] Step 2:

[0709] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[0710] Step 3:

[0711] The generative AI returns the generated tour plan to the server.

[0712] Step 4:

[0713] The server transmits the generated tour plan to the terminal.

[0714] Step 5:

[0715] The terminal presents the received tour plan to the user.

[0716] Step 6:

[0717] The user can review the proposed plan and make fine adjustments as needed.

[0718] AR guidance

[0719] Step 1:

[0720] The user starts the tour and enables AR navigation on the device.

[0721] Step 2:

[0722] The device activates the camera function and prepares for AR display.

[0723] Step 3:

[0724] The server receives the current location information and transmits route information to the next destination.

[0725] Step 4:

[0726] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[0727] Changing plans mid-way

[0728] Step 1:

[0729] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[0730] Step 2:

[0731] The terminal sends the new conditions to the server.

[0732] Step 3:

[0733] The server will then make a new request to the generative AI based on the new conditions.

[0734] Step 4:

[0735] The generative AI generates a revised tour plan based on the new conditions.

[0736] Step 5:

[0737] The generative AI sends the revised tour plan back to the server.

[0738] Step 6:

[0739] The server sends the modified plan to the terminal.

[0740] Step 7:

[0741] The device displays the new tour plan to the user and updates the AR navigation.

[0742] Feedback and Ratings

[0743] Step 1:

[0744] After completing the tour, users can enter their tour rating and comments on the feedback page.

[0745] Step 2:

[0746] The terminal transmits the input feedback to the server.

[0747] Step 3:

[0748] The server stores the feedback in a database and uses it for analysis.

[0749] The above are the specific processing steps of the "UTour" system.

[0750] Example 1

[0751] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Conventional sightseeing tour systems have had difficulty quickly providing optimal plans tailored to individual users' interests and wishes. Furthermore, they lacked the ability to change plans mid-tour and real-time guidance, resulting in an unsatisfactory user experience. The present invention aims to solve these problems and provide customized sightseeing tour plans that satisfy users.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0754] In this invention, the server includes means for receiving information on interests and desires input by a user, means for temporarily storing the received information in a database, means for generating an optimal sightseeing tour plan based on the received information on interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, and means for regenerating the sightseeing tour plan based on the new conditions using the generative AI when the user changes the plan during the tour, thereby enabling users to receive an optimal sightseeing experience tailored to their individual interests and desires in real time.

[0755] "Interests and desired information input by the user" refers to information about the user's individual areas of interest and desired conditions for the sightseeing tour (for example, tourist spots, budget, planned dates of visit, etc.).

[0756] The "receiving means" is a mechanism that has the function of acquiring information entered by the user from the terminal and transmitting it to the server.

[0757] The "means for temporarily storing in a database" is a system for temporarily storing received information about the user's interests and preferences, and can be, for example, a NoSQL database or an SQL database.

[0758] "Generative AI" is an artificial intelligence model that generates specific outputs based on input conditions, such as generating sightseeing tour plans using natural language processing.

[0759] The "means for generating optimal sightseeing tour plans" is an algorithm that uses generative AI to construct the order in which tourist spots should be visited and a schedule based on information input by the user.

[0760] The "means for presenting the generated sightseeing tour plan to the user" is a system for displaying the generated tour plan information on the terminal in a format that is easy for the user to understand.

[0761] "Means for displaying real-world guidance as augmented reality" refers to augmented reality technology that uses camera functions to display route guidance and tourist information overlaid on real-world scenery.

[0762] "Means for regenerating sightseeing tour plans based on new conditions when the plan is changed" refers to a mechanism that, when a user inputs new conditions during a tour, requests the generative AI to create a new sightseeing tour plan again.

[0763] The "means for saving feedback" is a system for saving the feedback entered by the user after the tour has ended in a database.

[0764] "Means for user authentication" refers to the process of verifying the identity of a user when logging into a system, for example by verifying an email address and password.

[0765] "Means for securely storing authenticated user data" refers to technology that protects the personal information and interest / desired information of authenticated users while safely storing them in a database.

[0766] This invention is a system that provides users with optimized sightseeing tour plans, automatically generating plans based on the user's individual interests and wishes, and providing real-time navigation and information. A specific embodiment of this system will be described.

[0767] User Registration and Login

[0768] User

[0769] Users access the "Tourism Planning App" using a device such as a smartphone or tablet and register or log in. When registering, users enter their name, email address, and password, which are then sent to the system. If they already have an account, they enter their email address and password for authentication.

[0770] Terminal

[0771] The terminal sends the input information to the server using the HTTPS protocol. Specifically, it makes a POST request to the API endpoint (e.g., / api / login) on the server.

[0772] server

[0773] The server receives the submitted information and checks it against the user information stored in a database (e.g., MySQL). If the check is successful, it generates an authenticated session ID, stores the session information in an in-memory database such as Redis, and sends it back to the device.

[0774] Enter your interests and preferences

[0775] User

[0776] Users go to the tour plan creation page of the "tourism planning app" and enter information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and planned date of visit.

[0777] Terminal

[0778] The terminal parses the input information into JSON format and sends it to the server. Specifically, it makes a POST request to the / api / planrequest endpoint.

[0779] server

[0780] The server temporarily stores the received information in a NoSQL database (e.g., MongoDB).

[0781] Tour plan generation and provision

[0782] server

[0783] The server generates a prompt based on the stored information and requests it from a generative AI model (e.g., GPT-4). An example prompt is: "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan that matches these."

[0784] Based on these prompts, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[0785] Terminal

[0786] The tour plan sent from the server is received and presented to the user, who can then check the plan and make any necessary adjustments.

[0787] AR guidance

[0788] User

[0789] When users start the tour, they enable AR navigation on their device and receive on-site guidance.

[0790] Terminal

[0791] The device utilizes its camera function to display route guidance and tourist information superimposed on the real landscape as augmented reality (AR). For example, instructions such as "The next spot is 500 meters away, turn right" are displayed in real time.

[0792] Changing plans mid-way

[0793] User

[0794] If an unplanned change occurs during the tour, the user enters new desired conditions and submits them to the system.

[0795] server

[0796] The server then requests the generative AI model again based on the new conditions to generate a new sightseeing tour plan. The updated plan is then sent back to the device and provided to the user in real time.

[0797] Feedback and Ratings

[0798] User

[0799] After completing the tour, users enter feedback and ratings based on their experience into the "tourism planning app."

[0800] Terminal

[0801] The terminal transmits the input feedback information to the server.

[0802] server

[0803] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system's accuracy and user satisfaction.

[0804] This will allow users to enjoy a customized sightseeing experience in real time that is tailored to their individual interests and desires.

[0805] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0806] Step 1: User registers or logs in

[0807] User

[0808] Users access the "Tourism Planning App" from their device and register an account or log in. If registering for the first time, they enter information such as their name, email address, and password, and submit. If they already have an account, they enter their email address and password for authentication.

[0809] Terminal

[0810] The device sends the entered information to the server using HTTPS communication, by sending a POST request to an API endpoint such as / api / login.

[0811] server

[0812] The server compares the received information with a database (e.g., MySQL) and saves it in the database in the case of a new registration. In the case of a login, the authentication information is checked, and if it matches, a session ID is generated, encrypted, and sent back to the terminal. The session information is stored in an in-memory database such as Redis.

[0813] Input and Output

[0814] Input: Registration and login information from the user

[0815] Output: Authenticated session ID

[0816] Step 2: User enters their interests and preferences

[0817] User

[0818] The user proceeds to the tour plan creation page and enters information such as areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[0819] Terminal

[0820] The terminal converts the input information into JSON format and sends it to the server. Specifically, it makes a POST request to / api / planrequest.

[0821] server

[0822] The server temporarily stores the received information in a database, such as MongoDB.

[0823] Input and Output

[0824] Input: User's interests and preferences

[0825] Output: Information ID stored in the database

[0826] Step 3: Request to generate a tour plan

[0827] server

[0828] The server generates prompts based on the stored information and sends them to a generative AI model (e.g., GPT-4). For example, it generates a prompt such as, "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan based on this information."

[0829] Input and Output

[0830] Input: Saved user information

[0831] Output: Prompt for the generative AI

[0832] Step 4: Plan generation using a generative AI model

[0833] Generative AI

[0834] Based on the prompts received, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule.

[0835] Input and Output

[0836] Input: Prompt sentence for generative AI

[0837] Output: Optimal sightseeing tour plan

[0838] Step 5: Present your tour plan

[0839] server

[0840] The generated tour plan is sent to the server, and the server retransmits the result to the terminal.

[0841] Terminal

[0842] The terminal presents the received tour plan to the user, who can then review the plan and make minor adjustments as necessary.

[0843] Input and Output

[0844] Input: Generated tour plan

[0845] Output: The tour plan displayed to the user

[0846] Step 6: Enabling AR Navigation

[0847] User

[0848] The user enables the AR navigation function on their device when starting the tour.

[0849] Terminal

[0850] The device uses its camera to display route guidance to tourist spots in augmented reality (AR) overlaid on the real-world scenery, with real-time guidance such as "Next destination 500m, turn right."

[0851] Input and Output

[0852] Input: Location information based on tour plan

[0853] Output: AR display overlaid on real scenery

[0854] Step 7: Change your plan mid-way

[0855] User

[0856] If an unplanned change is required during the tour, the user enters new requirements and submits them to the system.

[0857] Terminal

[0858] The terminal transmits the new desired conditions to the server.

[0859] server

[0860] The server then resubmits the request to the generative AI model based on the new conditions to generate a new tour plan, which is then sent back to the device and provided to the user in real time.

[0861] Input and Output

[0862] Input: User's new preferences

[0863] Output: New sightseeing tour plan

[0864] Step 8: Enter and save your feedback

[0865] User

[0866] After completing the tour, users provide feedback and a rating based on their experience.

[0867] Terminal

[0868] The terminal transmits the input feedback information to the server.

[0869] server

[0870] The server stores the collected feedback information in a database and uses it as reference information when generating subsequent tour plans.

[0871] Input and Output

[0872] Input: User feedback information

[0873] Output: Feedback information stored in a database

[0874] Through this series of processes, users can enjoy sightseeing tour plans optimized in real time to suit their individual interests and desires.

[0875] (Application example 1)

[0876] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0877] Conventional sightseeing tour planning systems do not take into account the user's consumption preferences or in-store route guidance, which makes it difficult to achieve efficient shopping in stores. Furthermore, guidance using augmented reality (AR) technology is limited, making it difficult for users to quickly and intuitively obtain the information they need in-store. This often results in an unsatisfactory shopping experience for users.

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0879] In this invention, the server includes means for receiving information on interests and desires input by a user, means for generating an optimal sightseeing tour plan and in-store route based on the received information on interests and desires using a generative AI, means for presenting the generated plan and route to the user, means for displaying real-world guidance as augmented reality, means for regenerating the plan based on new conditions using the generative AI when the user changes the plan during the tour or shopping, and means for providing in-store product locations as augmented reality, thereby enabling users to enjoy optimal sightseeing and shopping experiences based on their individual interests and consumption preferences.

[0880] "Interest and desired information" is a general term for information that a user inputs about their areas of interest and the experiences they are looking for.

[0881] "Generative AI" is an artificial intelligence technology that automatically generates optimal plans and proposals based on input information.

[0882] A "tourist tour plan" refers to a travel plan that includes the order and schedule of visiting tourist spots.

[0883] Augmented reality (AR) is a technology that displays digital information overlaid on real-world scenery.

[0884] "In-store route" refers to an optimal route plan for a user to move efficiently within the store.

[0885] "Product suggestions" are information about products recommended based on the user's consumption preferences.

[0886] "User authentication" is the process of identifying an individual user and verifying their access rights.

[0887] "Consumption preferences" refers to trends in a user's purchasing behavior and preferences.

[0888] "Feedback" refers to the evaluations and opinions provided by users after using the service.

[0889] "Plan regeneration" is the process of regenerating an existing plan based on new conditions.

[0890] The present invention is a system that provides users with optimized sightseeing tour plans and in-store routes. This system generates plans based on the user's interests and desired information, and provides guide information using augmented reality (AR) technology. Specific embodiments of the present invention are described below.

[0891] User Registration and Login

[0892] User

[0893] Users access a dedicated application through their device (such as a smartphone or tablet) to register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register with the system. When logging in with an existing account, users enter their email address and password for authentication.

[0894] server

[0895] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0896] Enter your interests and preferences

[0897] User

[0898] The user goes to the planning page of the dedicated application and enters information such as their areas of interest (e.g., history, fashion, food, etc.), budget, and planned dates of visit.

[0899] Terminal

[0900] The terminal transmits the inputted interest and desired information to the server.

[0901] server

[0902] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal plan.

[0903] Generate and deliver plans

[0904] server

[0905] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which tourist spots should be visited, a schedule, in-store routes, and product suggestions, and sends the results back to the server.

[0906] Terminal

[0907] The plan sent from the server is sent back to the terminal and presented to the user, who can then check the plan and make any necessary adjustments.

[0908] AR guidance

[0909] User

[0910] Once the user executes the plan, they can enable AR navigation on their device to receive on-site guidance.

[0911] Terminal

[0912] The device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. For example, instructions such as "The next stop is 500 meters away, turn right" or "Recommended products are on this shelf" are displayed in real time.

[0913] Changing plans mid-way

[0914] User

[0915] If an unexpected change is required during the plan, the user re-enters the desired conditions and sends them to the system.

[0916] server

[0917] The server then re-submits the request to the AI ​​based on the new conditions to generate a new plan. The updated plan is then sent to the device and provided to the user in real time.

[0918] Feedback and Ratings

[0919] User

[0920] After completing the plan, users enter feedback and ratings based on their experience into a dedicated application.

[0921] Terminal

[0922] The terminal transmits the input feedback information to the server.

[0923] server

[0924] The server stores the collected feedback in a database and uses it as reference information when generating future plans, thereby improving the system.

[0925] Specific prompt examples

[0926] "Generate the optimal shopping plan based on the following user preferences and wishes: Preferences: Apparel, Budget: Under 10,000 yen, Season: Spring"

[0927] This system enables users to efficiently enjoy optimal sightseeing and shopping experiences based on their interests and consumption preferences.

[0928] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0929] Step 1:

[0930] A user accesses a dedicated application using a terminal to register or log in. As input, they provide their name, email address, and password. The server receives this information and compares it with the user information stored in a database. As output, it generates an authenticated session and sends it back to the terminal.

[0931] Step 2:

[0932] The user enters information such as their areas of interest, budget, and planned visit date on the plan creation page of the dedicated application. The input information includes interests and wishes. The device sends this information to the server, which then temporarily stores the received information in a database.

[0933] Step 3:

[0934] The server sends the stored information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which to visit tourist spots, a schedule, in-store routes, and product suggestions. The optimal plan is then returned to the server as output.

[0935] Step 4:

[0936] The server sends the plan returned by the generative AI to the device and presents it to the user. The user can then review the plan and make any necessary adjustments. At this stage, the user can also enable AR navigation for when they arrive at the destination.

[0937] Step 5:

[0938] When the user executes the plan, the device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. The input includes the user's current location and the generated plan. The output is the route instructions and product location information displayed on the real world.

[0939] Step 6:

[0940] If an unexpected change is required midway through the plan, the user re-enters their desired conditions and sends them to the system. The new desired conditions are included as input. The server then requests the generative AI again based on the new conditions to generate a new plan. As output, the updated plan is sent to the device and provided to the user in real time.

[0941] Step 7:

[0942] After completing the plan, the user enters feedback and evaluation based on their experience into a dedicated application. The input includes the user's feedback. The device sends this feedback information to the server, which stores it in a database. The output provides reference information for use in improving the system.

[0943] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0944] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[0945] User Registration and Login

[0946] User

[0947] Users access the "UTour" app through their device (smartphone or tablet) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[0948] server

[0949] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[0950] Enter your interests and preferences

[0951] User

[0952] The user goes to the "Tour Plan Creation" page on the terminal and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[0953] Terminal

[0954] The terminal transmits the input information to the server.

[0955] server

[0956] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[0957] Tour plan generation and provision

[0958] server

[0959] The server sends the stored information to a generative AI, which determines tourist spots and the order in which to visit them based on the user's interests and preferences, and generates a tour plan. The generated plan is then sent back to the server, which then sends it to the device.

[0960] Terminal

[0961] The terminal presents the received tour plan to the user, who can then check the plan and make fine adjustments to it as necessary.

[0962] Introducing the Emotion Engine

[0963] User

[0964] During the tour, the emotion engine collects data such as facial expressions and voice in real time, and this emotional information reflects how the user feels throughout the tour experience.

[0965] Terminal

[0966] The terminal transmits the collected emotion information to the server.

[0967] server

[0968] The server analyzes the emotional information sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI to adjust the tour plan. For example, if the user feels tired, it adds rest stops and sends the adjusted plan back to the device.

[0969] AR guidance and plan changes

[0970] User

[0971] Once a user starts a tour, they can activate AR navigation on their device to receive on-site guidance. If an unexpected change is needed during the tour, they can enter their new preferences and submit them to the system.

[0972] Terminal

[0973] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real landscape. When the user inputs new desired conditions, the device sends them to the server.

[0974] server

[0975] The server then requests the generative AI to generate a new sightseeing tour plan based on the new conditions. The updated plan is sent to the device and provided to the user in real time.

[0976] Feedback and Ratings

[0977] User

[0978] After the tour, users can access the feedback page in the UTour app and enter their tour rating and comments.

[0979] Terminal

[0980] The terminal transmits the input feedback information to the server.

[0981] server

[0982] The server stores the feedback in a database and uses it for analysis. Emotion data collected by the emotion engine is also stored as feedback information and taken into account when generating the next plan.

[0983] Specific examples

[0984] Example 1: Tailoring tours to match the emotions of history buffs

[0985] User A plans a one-day tour of historical sites, with his / her area of ​​interest being "historical sites," his / her budget being 5,000 yen, and his / her visit date being the following day. The system uses generative AI to create an optimal tour plan and guides him / her with AR navigation. During the tour, the emotion engine recognizes that User A is tired and sends this to the server. The server then generates a new plan with appropriate rest spots added and sends it to the device.

[0986] Example 2: Anime pilgrimage with changes and emotional feedback

[0987] During a tour of anime holy sites, User B must skip one of the stops due to unplanned circumstances. The emotion engine detects stress and anxiety and reports this to the server. The server regenerates the tour plan based on the new conditions and provides it to the user. After the tour ends, User B inputs feedback and emotion data, and the emotion engine provides new insights for the next tour.

[0988] By incorporating an emotion engine, the present invention makes it possible to provide flexible tour plans that correspond to the user's emotional state, thereby realizing a more advanced and personalized sightseeing experience.

[0989] The processing flow will be explained below.

[0990] User Registration and Login

[0991] Step 1:

[0992] The user launches the "UTour" app on their device and accesses the new registration page.

[0993] Step 2:

[0994] The user enters the required information such as name, email address, and password.

[0995] Step 3:

[0996] The terminal transmits the input information to the server.

[0997] Step 4:

[0998] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[0999] Step 5:

[1000] The user enters their email address and password on the login page of the "UTour" app.

[1001] Step 6:

[1002] The terminal transmits the login information to the server.

[1003] Step 7:

[1004] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[1005] Enter your interests and preferences

[1006] Step 1:

[1007] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[1008] Step 2:

[1009] The terminal transmits the input desired information to the server.

[1010] Step 3:

[1011] The server temporarily stores the received desired information in a database.

[1012] Tour plan generation and provision

[1013] Step 1:

[1014] The server sends the saved desired information to the generative AI.

[1015] Step 2:

[1016] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[1017] Step 3:

[1018] The generative AI returns the generated tour plan to the server.

[1019] Step 4:

[1020] The server transmits the generated tour plan to the terminal.

[1021] Step 5:

[1022] The terminal presents the received tour plan to the user.

[1023] Step 6:

[1024] The user can review the proposed plan and make fine adjustments as needed.

[1025] Introducing the Emotion Engine

[1026] Step 1:

[1027] During the tour, users use the device's camera and microphone to input their facial expressions and voices into the emotion engine.

[1028] Step 2:

[1029] The terminal recognizes the user's emotions in real time using an emotion engine and transmits the data to the server.

[1030] Step 3:

[1031] The server analyzes the emotion information sent from the emotion engine, and if the user's emotion indicates discomfort or stress, it determines that the tour plan needs to be adjusted.

[1032] Step 4:

[1033] The server requests generative AI to adjust the tour plan based on the emotional information.

[1034] Step 5:

[1035] The generative AI generates a revised tour plan based on the new conditions.

[1036] Step 6:

[1037] The generative AI sends the revised tour plan back to the server.

[1038] Step 7:

[1039] The server sends the modified plan to the terminal.

[1040] Step 8:

[1041] The device displays the new tour plan to the user and updates the AR navigation.

[1042] AR guidance and plan changes

[1043] Step 1:

[1044] The user starts the tour and enables AR navigation on the device.

[1045] Step 2:

[1046] The device activates the camera function and prepares for AR display.

[1047] Step 3:

[1048] The server receives the current location information and transmits route information to the next destination.

[1049] Step 4:

[1050] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[1051] Step 5:

[1052] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[1053] Step 6:

[1054] The terminal sends the new conditions to the server.

[1055] Step 7:

[1056] The server will then make a new request to the generative AI based on the new conditions.

[1057] Step 8:

[1058] The generative AI generates a revised tour plan based on the new conditions.

[1059] Step 9:

[1060] The generative AI sends the revised tour plan back to the server.

[1061] Step 10:

[1062] The server sends the modified plan to the terminal.

[1063] Step 11:

[1064] The device displays the new tour plan to the user and updates the AR navigation.

[1065] Feedback and Ratings

[1066] Step 1:

[1067] After completing the tour, users can enter their tour rating and comments on the feedback page.

[1068] Step 2:

[1069] The terminal transmits the input feedback to the server.

[1070] Step 3:

[1071] The server stores the feedback in a database and uses it for analysis. It also stores the emotion data collected by the emotion engine as feedback information.

[1072] The above are the specific processing steps of the "UTour" system, which incorporates an emotion engine.

[1073] Example 2

[1074] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1075] Conventional tourist tour plan generation systems provide tour plans based on the user's interests and preferences, but they are unable to consider information about the user's emotions and psychological state. Furthermore, they lack the means to dynamically adjust tour plans in real time, making it difficult to maximize user satisfaction. Therefore, there is a need for a system that can flexibly adjust tour plans based on the user's emotions and provide a personalized tourist experience.

[1076] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving interest and preference information input by a user; means for generating an optimal sightseeing tour plan based on the received interest and preference information using a generative AI model; means for presenting the generated sightseeing tour plan to the user; means for displaying real-world guidance as augmented reality; means for regenerating the sightseeing tour plan based on new conditions using the generative AI model when the user changes the plan during the tour; means having an emotion engine for collecting the user's facial expressions and voice in real time; and means for analyzing the collected emotion data and adjusting the tour plan according to the emotion. This makes it possible to provide flexible tour plans that correspond to the user's emotional state, resulting in a more advanced and personalized sightseeing experience.

[1077] A "user" is a person who uses the system to create a sightseeing tour plan and enjoy a personalized sightseeing experience.

[1078] "Interest and desired information" is information input by the user such as areas of interest, budget, and dates of visit related to tourism.

[1079] A "generative AI model" is an artificial intelligence model that generates optimal sightseeing tour plans based on the user's interests and desired information.

[1080] A "tour plan" is a tour plan that includes specific tourist spots and the order in which they are to be visited, created by a generative AI model based on the user's interests and preferences.

[1081] "Augmented reality" is a technology that displays digital information overlaid on a real-world environment.

[1082] The "emotion engine" is part of a system that collects and analyzes emotions from the user's facial expressions and voice in real time.

[1083] "Emotion data" refers to data such as facial expressions and voice collected by the user's emotion engine.

[1084] "Feedback" refers to information such as ratings and comments entered by the user after the tour has ended.

[1085] "User authentication" is the process of verifying a user's identity when accessing a system.

[1086] "Generate" means that the generative AI model creates a sightseeing tour plan based on the user's interests and desired information.

[1087] "Adjustment" means making changes or corrections to existing sightseeing tour plans based on the results of analyzing emotional data.

[1088] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[1089] 1. User Registration and Login

[1090] User

[1091] Users access the dedicated application using a device such as a smartphone or tablet. When using for the first time, they create a new account by entering their name, email address, and password on the new registration screen. Existing users can authenticate by entering their email address and password on the login screen.

[1092] Terminal

[1093] The terminal transmits the registration information or login information input by the user to the server.

[1094] server

[1095] The server checks the entered information against the user information in the database. If the check is successful, it creates an authenticated session and returns it to the terminal. If the check is unsuccessful, it creates an error message and returns it to the terminal.

[1096] 2. Enter your interests and preferences

[1097] User

[1098] The user goes to the "Create a Tour Plan" page and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[1099] Terminal

[1100] The terminal transmits the inputted interest and desired information to the server.

[1101] server

[1102] The server stores the received information in a database, and uses a generative AI model to generate prompts based on that information to create optimal sightseeing tour plans.

[1103] 3. Creating and providing tour plans

[1104] server

[1105] The server sends prompts to the generative AI model, which then generates a tour plan based on the user's interests, budget, visit dates, and other information, including the selection of tourist spots and the order in which to visit them.

[1106] Terminal

[1107] The tour plan received from the server is presented to the user, who can then review the plan and make any necessary adjustments.

[1108] 4. Introducing the Emotion Engine

[1109] User

[1110] During the tour, the emotion engine collects the user's facial expressions and voice data in real time, and this emotion information reflects the user's emotional state throughout the tour experience.

[1111] Terminal

[1112] The device transmits the collected emotion data to the server in real time.

[1113] server

[1114] The server analyzes the emotion data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to readjust the tour plan. For example, if the user shows signs of fatigue, it will regenerate the plan by adding rest stops and send it to the device.

[1115] 5. AR Guidance and Plan Changes

[1116] User

[1117] Once the tour begins, users can activate AR navigation on their device to receive on-site guidance. If unplanned changes occur, users can enter new preferences and submit them to the system.

[1118] Terminal

[1119] The device displays route guidance and tourist information superimposed on the real landscape as AR, and also transmits new desired conditions to the server.

[1120] server

[1121] The server then requests the generative AI model to generate a new sightseeing tour plan based on the new conditions, and the plan is provided to the device in real time.

[1122] 6. Feedback and Ratings

[1123] User

[1124] After the tour is over, the user inputs feedback and sends the feedback and emotion data to the server.

[1125] Terminal

[1126] The terminal transmits the feedback information to the server.

[1127] server

[1128] The server stores the feedback and emotion data in a database and takes them into account when generating the next plan.

[1129] Specific examples

[1130] Example 1: Tour plan for history buffs

[1131] User A selects "historical sites" as an area of ​​interest, budgets 5,000 yen, and the date of visit for the following day. The server generates an appropriate tour plan using a generative AI model and sends it to the device. If User A shows signs of fatigue during the tour, the server incorporates rest spots based on information from the emotion engine and resends the adjusted plan.

[1132] Example 2: Anime pilgrimages that require changes mid-trip

[1133] While touring the anime holy sites, User B needs to skip one of the stops. The emotion engine detects the stress and reports it to the server. The server regenerates the tour plan taking the new conditions into account and provides it to User B. After the tour ends, the feedback and emotion data are used to generate the next plan.

[1134] Example prompt sentence:

[1135] "Generate a tour plan for historical sites with a budget of 5000 yen. The visit date is tomorrow."

[1136] The above is an embodiment of the sightseeing tour plan creation system of the present invention. By using this system, it is possible to provide flexible tour plans that correspond to the emotional state of the user, realizing a more advanced and personalized sightseeing experience.

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1: User Registration

[1139] Users access the application using a smartphone or tablet and enter their name, email address, and password on the new registration screen (input: user information).

[1140] The terminal transmits the input user information to the server (output: user information).

[1141] The server stores the received user information in a database, generates a registration completion message and an authentication token, and returns them to the terminal (data processing: store user information in database, data calculation: generate token) (output: registration completion message, authentication token).

[1142] The terminal displays a registration completion message to the user (action: display message).

[1143] Step 2: User Login

[1144] The user enters their email address and password on the login screen (input: login information).

[1145] The terminal sends the login information to the server (output: login information).

[1146] The server checks the entered login information against the database, and if it matches, it creates an authenticated session; if it doesn't, it generates an error message and sends it to the terminal (data processing: checking login information, data calculation: session generation) (output: authenticated session or error message).

[1147] The terminal maintains the authenticated session and displays a login success message or an error message to the user (action: display message).

[1148] Step 3: Enter your interests and preferences

[1149] The user enters the area of ​​interest, budget, visit date, etc. on the "Create Tour Plan" page and taps the submit button (input: area of ​​interest, budget, visit date).

[1150] The terminal sends the input information to the server (output: areas of interest, budget, date of visit).

[1151] The server stores the received information in a database and generates a prompt statement based on that information (data processing: storing in a database, data calculation: generating a prompt statement) (output: prompt statement).

[1152] Step 4: Generate a tour plan

[1153] The server sends the generated prompt sentence to the generative AI model (input: prompt sentence) (output: generation request).

[1154] The generative AI model generates a tour plan including tourist attractions and the order in which to visit them based on the prompt sentence (Data calculation: Generate tour plan) (Output: Tour plan).

[1155] The server receives the generated tour plan and transmits it to the terminal (output: tour plan).

[1156] Step 5: Present your tour plan

[1157] The terminal displays the received tour plan to the user (operation: display plan).

[1158] The user checks the presented plan and makes fine adjustments as necessary (input: fine adjustment information).

[1159] Step 6: Collect emotion data

[1160] During the tour, the emotion engine collects the user's facial expression and voice data in real time (input: facial expression data, voice data).

[1161] The terminal transmits the collected emotion data to the server in real time (output: emotion data).

[1162] Step 7: Analyze emotional data and adjust your plan

[1163] The server analyzes the emotional data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to adjust a new tour plan (data calculation: analysis of emotional data, regeneration of plan) (output: readjustment request).

[1164] The generative AI model generates a new tour plan with additional rest stops added as needed and sends it to the server (output: adjusted tour plan).

[1165] The server transmits the adjusted tour plan to the terminal (output: adjusted tour plan).

[1166] The terminal presents the adjusted tour plan to the user (action: show plan).

[1167] Step 8: Providing AR Navigation

[1168] When the user starts the tour, he or she uses the AR navigation of the device (action: start AR navigation).

[1169] The device displays route guidance and tourist information superimposed on real-world scenery (operation: composite display of the physical world and digital information).

[1170] Step 9: Feedback and Rating

[1171] After completing the tour, the user enters a rating and comments on the feedback page (input: feedback information).

[1172] The terminal transmits the feedback information to the server (output: feedback information).

[1173] The server stores the feedback and emotion data in a database and takes them into consideration when generating the next plan (data processing: save, data calculation: take into consideration when generating the next plan).

[1174] Through these steps, the system can flexibly adjust the tour plan according to the user's emotional state and provide a personalized sightseeing experience.

[1175] (Application example 2)

[1176] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1177] Conventional sightseeing tour planning systems can generate plans based on the user's interests and preferences, but they cannot adjust the plan to accommodate real-time emotional changes during the tour. Furthermore, no systems exist that can personalize the in-store shopping experience or promote sales based on the user's emotional state. This makes it difficult to provide an optimal and comfortable experience for users.

[1178] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information about interests and desires input by a user, means for generating an optimal sightseeing tour plan based on the received information about interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, means for regenerating the sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour, and means for detecting the user's emotions and adjusting the plan based on the emotions. This makes it possible to adjust the plan in response to real-time changes in the user's emotions, providing a more personalized and comfortable experience.

[1179] The "means for receiving information about interests and desires entered by the user" refers to a device such as a smartphone or tablet that allows the user to enter information about their interests and desires, which is then received by the system.

[1180] "Means for generating optimal sightseeing tour plans based on the received information on interests and desires using generative AI" refers to a means for automatically generating optimal sightseeing tour plans using artificial intelligence technology based on the received information on the user's interests and desires.

[1181] The "means for presenting the generated sightseeing tour plan to the user" is a means for presenting the generated sightseeing tour plan to the user in an easy-to-view manner, allowing the user to confirm and adjust the plan.

[1182] "Means for displaying real-world guidance as augmented reality" refers to the display of virtual information superimposed on real-world scenery, making it easier for users to visually understand the information.

[1183] "Means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour" refers to a means for the generative AI to regenerate a new sightseeing tour plan in response to changes in the user's wishes or situation during the tour.

[1184] The "means for detecting a user's emotions and adjusting the plan based on the emotions" is for detecting a user's real-time emotional state and adjusting the tour plan or shopping experience according to the emotions.

[1185] The present invention is a system that provides users with optimized sightseeing tour plans and in-store shopping plans, and also detects the user's emotions in real time and adjusts the plans based on the emotions.

[1186] Hardware and Software Configuration

[1187] Hardware

[1188] Smart glasses: Devices with built-in camera and display capabilities (e.g., Google Glass, Vuzix Blade).

[1189] Server: A high-performance server device that runs a database and generative AI.

[1190] software

[1191] Emotion engine: A software module that uses libraries such as OpenCV to analyze the user's facial expressions and detect emotions.

[1192] Generative AI: A module that uses generative AI models such as GPT-3 to generate and regenerate optimal plans based on the user's interests, desired information, and emotional data.

[1193] AR module: A software module that displays virtual guidance information overlaid on the real-world scenery.

[1194] System Operation

[1195] User Registration and Login

[1196] The user puts on the smart glasses and accesses the app to register or log in. The server receives the user's registration information or login information and compares it with the user information stored in the database.

[1197] Enter your interests and preferences

[1198] Through the smart glasses, users input information such as their interests, budget, and visit date, which is then sent to the server.

[1199] Plan generation and presentation

[1200] The server then sends the received information about interests and preferences to the generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the information. The generated plans are then sent from the server to the smart glasses and presented to the user.

[1201] Real-time emotion detection and plan adjustment

[1202] As the user tours or shops, the smart glasses' camera captures the user's facial expressions, and the emotion engine detects their emotions in real time. The emotion information is sent to the server, which analyzes the information from the emotion engine. If the user feels stressed or tired, the generative AI regenerates the plan based on the new conditions.

[1203] Augmented reality (AR) guidance

[1204] The smart glasses overlay virtual guide information on real-world scenery, providing users with visually easy-to-understand navigation and tourist information.

[1205] Specific examples

[1206] Scenario 1: If a user feels tired while shopping, the emotion engine detects this. The generative AI recommends a simple rest area and refreshing items (e.g., drinks). The in-store AR display also guides the user to the shortest route.

[1207] Scenario 2: When a user makes an unexpected change during a sightseeing tour, they input their new interests and preferences and submit them to the system. The generative AI then regenerates the optimal tour plan based on the new conditions and displays it on the smart glasses.

[1208] Example prompt sentence:

[1209] The user's interests are "fashion" and "technology," their budget is 10,000 yen, and their current emotion is "tired." Please suggest places to rest and products that will refresh them.

[1210] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1211] Step 1:

[1212] A user puts on smart glasses and accesses the app to register or log in. The input is the user's registration information (name, email address, password) or login information (email address, password). The server receives this information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session and sends it back to the device. The output is the authenticated session information.

[1213] Step 2:

[1214] The user inputs information such as their interests, budget, and visit date through the smart glasses. The input information is sent from the device to the server. The server temporarily stores the received interest and preference information in a database. The input is the interest and preference information entered by the user, and the output is the saved interest and preference information.

[1215] Step 3:

[1216] The server sends the stored interests and preferences to a generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the interests and preferences. The input is the interests and preferences, and the output is the generated plan.

[1217] Step 4:

[1218] The generated plan is sent from the server to the smart glasses. The device presents the generated plan to the user. The input is the generated plan, and the output is the presented plan. The user can review the plan and make fine adjustments if necessary.

[1219] Step 5:

[1220] As the user tours or goes shopping, the camera in the smart glasses continues to capture the user's facial expressions. Based on the captured frames, the device uses an emotion engine to detect the user's emotions in real time. The input is the camera frames, and the output is the detected emotion data.

[1221] Step 6:

[1222] The detected emotional information is sent from the device to the server. The server analyzes the information from the emotion engine, and if the user is feeling stressed or tired, the generative AI regenerates a plan based on the new conditions. The input is emotional data, and the output is the regenerated plan.

[1223] Step 7:

[1224] The smart glasses receive the regenerated plan and overlay virtual guide information on the real-world scenery, providing the user with a visually easy-to-understand navigation or shopping guide. The input is the regenerated plan, and the output is the guide information displayed in AR.

[1225] Step 8:

[1226] After completing the tour or shopping, the user accesses the feedback page and enters their experience rating and comments. The device sends the entered feedback information to the server, which stores the feedback in a database and uses it for analysis. The input is the user's feedback, and the output is the stored feedback information.

[1227] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1228] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1229] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1230] [Third embodiment]

[1231] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1232] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1233] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1234] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1235] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1237] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1238] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1239] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1240] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1241] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1242] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1243] The present invention is a system for providing users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Specific embodiments of the system of the present invention will be described below.

[1244] User Registration and Login

[1245] User

[1246] Users access the "UTour" app through their device (smartphone, tablet, etc.) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[1247] server

[1248] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[1249] Enter your interests and preferences

[1250] User

[1251] Users go to the tour plan creation page of the "UTour" app and enter information such as their areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[1252] Terminal

[1253] The terminal transmits the inputted interest and desired information to the server.

[1254] server

[1255] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[1256] Tour plan generation and provision

[1257] server

[1258] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[1259] Terminal

[1260] The tour plan sent from the server is sent back to the terminal and presented to the user, who can then check the presented plan and make any necessary adjustments.

[1261] AR guidance

[1262] User

[1263] Once users start the tour, they can enable AR navigation on their device to receive on-site guidance.

[1264] Terminal

[1265] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real world. For example, instructions like "The next spot is 500 meters away, turn right" are displayed in real time.

[1266] Changing plans mid-way

[1267] User

[1268] If an unexpected change becomes necessary during the tour, the user re-enters the desired conditions and transmits them to the system.

[1269] server

[1270] The server then re-queues the generative AI based on the new conditions to generate a new sightseeing tour plan. The updated plan is sent to the device and provided to the user in real time.

[1271] Feedback and Ratings

[1272] User

[1273] After completing the tour, users enter feedback and ratings based on their experience into the UTour app.

[1274] Terminal

[1275] The terminal transmits the input feedback information to the server.

[1276] server

[1277] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system.

[1278] Specific examples

[1279] Example 1: One-day tour of historical sites

[1280] User A wishes to visit multiple historical sites in one day, with a budget of 5,000 yen or less. User A logs into "UTour" and enters his / her area of ​​interest as "historical sites," his / her budget as "under 5,000 yen," and the date of visit as "tomorrow." The server then has the generative AI generate a plan. The generated plan includes historical sites A, B, and C, and route guidance is displayed in AR. User A is satisfied with the presented plan and begins the tour.

[1281] Example 2: Changing the plan for an anime pilgrimage tour

[1282] While planning a tour of anime holy sites, User B unexpectedly needs to skip one of the stops. User B enters the new desired information through the app and sends it to the server. The server then requests the generative AI to generate a new plan. The regenerated plan is provided to User B in real time, and the AR navigation is also updated.

[1283] This allows users to always receive the best possible sightseeing experience.

[1284] The system provides a customized sightseeing experience tailored to the user's individual needs and circumstances, and its real-time navigation and information display enable efficient and fulfilling sightseeing.

[1285] The processing flow will be explained below.

[1286] User Registration and Login

[1287] Step 1:

[1288] The user launches the "UTour" app on their device and accesses the new registration page.

[1289] Step 2:

[1290] The user enters the required information such as name, email address, and password.

[1291] Step 3:

[1292] The terminal transmits the input information to the server.

[1293] Step 4:

[1294] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[1295] Step 5:

[1296] The user enters their email address and password on the login page of the "UTour" app.

[1297] Step 6:

[1298] The terminal transmits the login information to the server.

[1299] Step 7:

[1300] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[1301] Enter your interests and preferences

[1302] Step 1:

[1303] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[1304] Step 2:

[1305] The terminal transmits the input desired information to the server.

[1306] Step 3:

[1307] The server temporarily stores the received desired information in a database.

[1308] Tour plan generation and provision

[1309] Step 1:

[1310] The server sends the saved desired information to the generative AI.

[1311] Step 2:

[1312] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[1313] Step 3:

[1314] The generative AI returns the generated tour plan to the server.

[1315] Step 4:

[1316] The server transmits the generated tour plan to the terminal.

[1317] Step 5:

[1318] The terminal presents the received tour plan to the user.

[1319] Step 6:

[1320] The user can review the proposed plan and make fine adjustments as needed.

[1321] AR guidance

[1322] Step 1:

[1323] The user starts the tour and enables AR navigation on the device.

[1324] Step 2:

[1325] The device activates the camera function and prepares for AR display.

[1326] Step 3:

[1327] The server receives the current location information and transmits route information to the next destination.

[1328] Step 4:

[1329] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[1330] Changing plans mid-way

[1331] Step 1:

[1332] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[1333] Step 2:

[1334] The terminal sends the new conditions to the server.

[1335] Step 3:

[1336] The server will then make a new request to the generative AI based on the new conditions.

[1337] Step 4:

[1338] The generative AI generates a revised tour plan based on the new conditions.

[1339] Step 5:

[1340] The generative AI sends the revised tour plan back to the server.

[1341] Step 6:

[1342] The server sends the modified plan to the terminal.

[1343] Step 7:

[1344] The device displays the new tour plan to the user and updates the AR navigation.

[1345] Feedback and Ratings

[1346] Step 1:

[1347] After completing the tour, users can enter their tour rating and comments on the feedback page.

[1348] Step 2:

[1349] The terminal transmits the input feedback to the server.

[1350] Step 3:

[1351] The server stores the feedback in a database and uses it for analysis.

[1352] The above are the specific processing steps of the "UTour" system.

[1353] Example 1

[1354] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1355] Conventional sightseeing tour systems have had difficulty quickly providing optimal plans tailored to individual users' interests and wishes. Furthermore, they lacked the ability to change plans mid-tour and real-time guidance, resulting in an unsatisfactory user experience. The present invention aims to solve these problems and provide customized sightseeing tour plans that satisfy users.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1357] In this invention, the server includes means for receiving information on interests and desires input by a user, means for temporarily storing the received information in a database, means for generating an optimal sightseeing tour plan based on the received information on interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, and means for regenerating the sightseeing tour plan based on the new conditions using the generative AI when the user changes the plan during the tour, thereby enabling users to receive an optimal sightseeing experience tailored to their individual interests and desires in real time.

[1358] "Interests and desired information input by the user" refers to information about the user's individual areas of interest and desired conditions for the sightseeing tour (for example, tourist spots, budget, planned dates of visit, etc.).

[1359] The "receiving means" is a mechanism that has the function of acquiring information entered by the user from the terminal and transmitting it to the server.

[1360] The "means for temporarily storing in a database" is a system for temporarily storing received information about the user's interests and preferences, and can be, for example, a NoSQL database or an SQL database.

[1361] "Generative AI" is an artificial intelligence model that generates specific outputs based on input conditions, such as generating sightseeing tour plans using natural language processing.

[1362] The "means for generating optimal sightseeing tour plans" is an algorithm that uses generative AI to construct the order in which tourist spots should be visited and a schedule based on information input by the user.

[1363] The "means for presenting the generated sightseeing tour plan to the user" is a system for displaying the generated tour plan information on the terminal in a format that is easy for the user to understand.

[1364] "Means for displaying real-world guidance as augmented reality" refers to augmented reality technology that uses camera functions to display route guidance and tourist information overlaid on real-world scenery.

[1365] "Means for regenerating sightseeing tour plans based on new conditions when the plan is changed" refers to a mechanism that, when a user inputs new conditions during a tour, requests the generative AI to create a new sightseeing tour plan again.

[1366] The "means for saving feedback" is a system for saving the feedback entered by the user after the tour has ended in a database.

[1367] "Means for user authentication" refers to the process of verifying the identity of a user when logging into a system, for example by verifying an email address and password.

[1368] "Means for securely storing authenticated user data" refers to technology that protects the personal information and interest / desired information of authenticated users while safely storing them in a database.

[1369] This invention is a system that provides users with optimized sightseeing tour plans, automatically generating plans based on the user's individual interests and wishes, and providing real-time navigation and information. A specific embodiment of this system will be described.

[1370] User Registration and Login

[1371] User

[1372] Users access the "Tourism Planning App" using a device such as a smartphone or tablet and register or log in. When registering, users enter their name, email address, and password, which are then sent to the system. If they already have an account, they enter their email address and password for authentication.

[1373] Terminal

[1374] The terminal sends the input information to the server using the HTTPS protocol. Specifically, it makes a POST request to the API endpoint (e.g., / api / login) on the server.

[1375] server

[1376] The server receives the submitted information and checks it against the user information stored in a database (e.g., MySQL). If the check is successful, it generates an authenticated session ID, stores the session information in an in-memory database such as Redis, and sends it back to the device.

[1377] Enter your interests and preferences

[1378] User

[1379] Users go to the tour plan creation page of the "tourism planning app" and enter information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and planned date of visit.

[1380] Terminal

[1381] The terminal parses the input information into JSON format and sends it to the server. Specifically, it makes a POST request to the / api / planrequest endpoint.

[1382] server

[1383] The server temporarily stores the received information in a NoSQL database (e.g., MongoDB).

[1384] Tour plan generation and provision

[1385] server

[1386] The server generates a prompt based on the stored information and requests it from a generative AI model (e.g., GPT-4). An example prompt is: "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan that matches these."

[1387] Based on these prompts, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[1388] Terminal

[1389] The tour plan sent from the server is received and presented to the user, who can then check the plan and make any necessary adjustments.

[1390] AR guidance

[1391] User

[1392] When users start the tour, they enable AR navigation on their device and receive on-site guidance.

[1393] Terminal

[1394] The device utilizes its camera function to display route guidance and tourist information superimposed on the real landscape as augmented reality (AR). For example, instructions such as "The next spot is 500 meters away, turn right" are displayed in real time.

[1395] Changing plans mid-way

[1396] User

[1397] If an unplanned change occurs during the tour, the user enters new desired conditions and submits them to the system.

[1398] server

[1399] The server then requests the generative AI model again based on the new conditions to generate a new sightseeing tour plan. The updated plan is then sent back to the device and provided to the user in real time.

[1400] Feedback and Ratings

[1401] User

[1402] After completing the tour, users enter feedback and ratings based on their experience into the "tourism planning app."

[1403] Terminal

[1404] The terminal transmits the input feedback information to the server.

[1405] server

[1406] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system's accuracy and user satisfaction.

[1407] This will allow users to enjoy a customized sightseeing experience in real time that is tailored to their individual interests and desires.

[1408] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1409] Step 1: User registers or logs in

[1410] User

[1411] Users access the "Tourism Planning App" from their device and register an account or log in. If registering for the first time, they enter information such as their name, email address, and password, and submit. If they already have an account, they enter their email address and password for authentication.

[1412] Terminal

[1413] The device sends the entered information to the server using HTTPS communication, by sending a POST request to an API endpoint such as / api / login.

[1414] server

[1415] The server compares the received information with a database (e.g., MySQL) and saves it in the database in the case of a new registration. In the case of a login, the authentication information is checked, and if it matches, a session ID is generated, encrypted, and sent back to the terminal. The session information is stored in an in-memory database such as Redis.

[1416] Input and Output

[1417] Input: Registration and login information from the user

[1418] Output: Authenticated session ID

[1419] Step 2: User enters their interests and preferences

[1420] User

[1421] The user proceeds to the tour plan creation page and enters information such as areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[1422] Terminal

[1423] The terminal converts the input information into JSON format and sends it to the server. Specifically, it makes a POST request to / api / planrequest.

[1424] server

[1425] The server temporarily stores the received information in a database, such as MongoDB.

[1426] Input and Output

[1427] Input: User's interests and preferences

[1428] Output: Information ID stored in the database

[1429] Step 3: Request to generate a tour plan

[1430] server

[1431] The server generates prompts based on the stored information and sends them to a generative AI model (e.g., GPT-4). For example, it generates a prompt such as, "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan based on this information."

[1432] Input and Output

[1433] Input: Saved user information

[1434] Output: Prompt for the generative AI

[1435] Step 4: Plan generation using a generative AI model

[1436] Generative AI

[1437] Based on the prompts received, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule.

[1438] Input and Output

[1439] Input: Prompt sentence for generative AI

[1440] Output: Optimal sightseeing tour plan

[1441] Step 5: Present your tour plan

[1442] server

[1443] The generated tour plan is sent to the server, and the server retransmits the result to the terminal.

[1444] Terminal

[1445] The terminal presents the received tour plan to the user, who can then review the plan and make minor adjustments as necessary.

[1446] Input and Output

[1447] Input: Generated tour plan

[1448] Output: The tour plan displayed to the user

[1449] Step 6: Enabling AR Navigation

[1450] User

[1451] The user enables the AR navigation function on their device when starting the tour.

[1452] Terminal

[1453] The device uses its camera to display route guidance to tourist spots in augmented reality (AR) overlaid on the real-world scenery, with real-time guidance such as "Next destination 500m, turn right."

[1454] Input and Output

[1455] Input: Location information based on tour plan

[1456] Output: AR display overlaid on real scenery

[1457] Step 7: Change your plan mid-way

[1458] User

[1459] If an unplanned change is required during the tour, the user enters new requirements and submits them to the system.

[1460] Terminal

[1461] The terminal transmits the new desired conditions to the server.

[1462] server

[1463] The server then resubmits the request to the generative AI model based on the new conditions to generate a new tour plan, which is then sent back to the device and provided to the user in real time.

[1464] Input and Output

[1465] Input: User's new preferences

[1466] Output: New sightseeing tour plan

[1467] Step 8: Enter and save your feedback

[1468] User

[1469] After completing the tour, users provide feedback and a rating based on their experience.

[1470] Terminal

[1471] The terminal transmits the input feedback information to the server.

[1472] server

[1473] The server stores the collected feedback information in a database and uses it as reference information when generating subsequent tour plans.

[1474] Input and Output

[1475] Input: User feedback information

[1476] Output: Feedback information stored in a database

[1477] Through this series of processes, users can enjoy sightseeing tour plans optimized in real time to suit their individual interests and desires.

[1478] (Application example 1)

[1479] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1480] Conventional sightseeing tour planning systems do not take into account the user's consumption preferences or in-store route guidance, which makes it difficult to achieve efficient shopping in stores. Furthermore, guidance using augmented reality (AR) technology is limited, making it difficult for users to quickly and intuitively obtain the information they need in-store. This often results in an unsatisfactory shopping experience for users.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1482] In this invention, the server includes means for receiving information on interests and desires input by a user, means for generating an optimal sightseeing tour plan and in-store route based on the received information on interests and desires using a generative AI, means for presenting the generated plan and route to the user, means for displaying real-world guidance as augmented reality, means for regenerating the plan based on new conditions using the generative AI when the user changes the plan during the tour or shopping, and means for providing in-store product locations as augmented reality, thereby enabling users to enjoy optimal sightseeing and shopping experiences based on their individual interests and consumption preferences.

[1483] "Interest and desired information" is a general term for information that a user inputs about their areas of interest and the experiences they are looking for.

[1484] "Generative AI" is an artificial intelligence technology that automatically generates optimal plans and proposals based on input information.

[1485] A "tourist tour plan" refers to a travel plan that includes the order and schedule of visiting tourist spots.

[1486] Augmented reality (AR) is a technology that displays digital information overlaid on real-world scenery.

[1487] "In-store route" refers to an optimal route plan for a user to move efficiently within the store.

[1488] "Product suggestions" are information about products recommended based on the user's consumption preferences.

[1489] "User authentication" is the process of identifying an individual user and verifying their access rights.

[1490] "Consumption preferences" refers to trends in a user's purchasing behavior and preferences.

[1491] "Feedback" refers to the evaluations and opinions provided by users after using the service.

[1492] "Plan regeneration" is the process of regenerating an existing plan based on new conditions.

[1493] The present invention is a system that provides users with optimized sightseeing tour plans and in-store routes. This system generates plans based on the user's interests and desired information, and provides guide information using augmented reality (AR) technology. Specific embodiments of the present invention are described below.

[1494] User Registration and Login

[1495] User

[1496] Users access a dedicated application through their device (such as a smartphone or tablet) to register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register with the system. When logging in with an existing account, users enter their email address and password for authentication.

[1497] server

[1498] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[1499] Enter your interests and preferences

[1500] User

[1501] The user goes to the planning page of the dedicated application and enters information such as their areas of interest (e.g., history, fashion, food, etc.), budget, and planned dates of visit.

[1502] Terminal

[1503] The terminal transmits the inputted interest and desired information to the server.

[1504] server

[1505] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal plan.

[1506] Generate and deliver plans

[1507] server

[1508] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which tourist spots should be visited, a schedule, in-store routes, and product suggestions, and sends the results back to the server.

[1509] Terminal

[1510] The plan sent from the server is sent back to the terminal and presented to the user, who can then check the plan and make any necessary adjustments.

[1511] AR guidance

[1512] User

[1513] Once the user executes the plan, they can enable AR navigation on their device to receive on-site guidance.

[1514] Terminal

[1515] The device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. For example, instructions such as "The next stop is 500 meters away, turn right" or "Recommended products are on this shelf" are displayed in real time.

[1516] Changing plans mid-way

[1517] User

[1518] If an unexpected change is required during the plan, the user re-enters the desired conditions and sends them to the system.

[1519] server

[1520] The server then re-submits the request to the AI ​​based on the new conditions to generate a new plan. The updated plan is then sent to the device and provided to the user in real time.

[1521] Feedback and Ratings

[1522] User

[1523] After completing the plan, users enter feedback and ratings based on their experience into a dedicated application.

[1524] Terminal

[1525] The terminal transmits the input feedback information to the server.

[1526] server

[1527] The server stores the collected feedback in a database and uses it as reference information when generating future plans, thereby improving the system.

[1528] Specific prompt examples

[1529] "Generate the optimal shopping plan based on the following user preferences and wishes: Preferences: Apparel, Budget: Under 10,000 yen, Season: Spring"

[1530] This system enables users to efficiently enjoy optimal sightseeing and shopping experiences based on their interests and consumption preferences.

[1531] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1532] Step 1:

[1533] A user accesses a dedicated application using a terminal to register or log in. As input, they provide their name, email address, and password. The server receives this information and compares it with the user information stored in a database. As output, it generates an authenticated session and sends it back to the terminal.

[1534] Step 2:

[1535] The user enters information such as their areas of interest, budget, and planned visit date on the plan creation page of the dedicated application. The input information includes interests and wishes. The device sends this information to the server, which then temporarily stores the received information in a database.

[1536] Step 3:

[1537] The server sends the stored information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which to visit tourist spots, a schedule, in-store routes, and product suggestions. The optimal plan is then returned to the server as output.

[1538] Step 4:

[1539] The server sends the plan returned by the generative AI to the device and presents it to the user. The user can then review the plan and make any necessary adjustments. At this stage, the user can also enable AR navigation for when they arrive at the destination.

[1540] Step 5:

[1541] When the user executes the plan, the device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. The input includes the user's current location and the generated plan. The output is the route instructions and product location information displayed on the real world.

[1542] Step 6:

[1543] If an unexpected change is required midway through the plan, the user re-enters their desired conditions and sends them to the system. The new desired conditions are included as input. The server then requests the generative AI again based on the new conditions to generate a new plan. As output, the updated plan is sent to the device and provided to the user in real time.

[1544] Step 7:

[1545] After completing the plan, the user enters feedback and evaluation based on their experience into a dedicated application. The input includes the user's feedback. The device sends this feedback information to the server, which stores it in a database. The output provides reference information for use in improving the system.

[1546] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1547] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[1548] User Registration and Login

[1549] User

[1550] Users access the "UTour" app through their device (smartphone or tablet) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[1551] server

[1552] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[1553] Enter your interests and preferences

[1554] User

[1555] The user goes to the "Tour Plan Creation" page on the terminal and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[1556] Terminal

[1557] The terminal transmits the input information to the server.

[1558] server

[1559] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[1560] Tour plan generation and provision

[1561] server

[1562] The server sends the stored information to a generative AI, which determines tourist spots and the order in which to visit them based on the user's interests and preferences, and generates a tour plan. The generated plan is then sent back to the server, which then sends it to the device.

[1563] Terminal

[1564] The terminal presents the received tour plan to the user, who can then check the plan and make fine adjustments to it as necessary.

[1565] Introducing the Emotion Engine

[1566] User

[1567] During the tour, the emotion engine collects data such as facial expressions and voice in real time, and this emotional information reflects how the user feels throughout the tour experience.

[1568] Terminal

[1569] The terminal transmits the collected emotion information to the server.

[1570] server

[1571] The server analyzes the emotional information sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI to adjust the tour plan. For example, if the user feels tired, it adds rest stops and sends the adjusted plan back to the device.

[1572] AR guidance and plan changes

[1573] User

[1574] Once a user starts a tour, they can activate AR navigation on their device to receive on-site guidance. If an unexpected change is needed during the tour, they can enter their new preferences and submit them to the system.

[1575] Terminal

[1576] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real landscape. When the user inputs new desired conditions, the device sends them to the server.

[1577] server

[1578] The server then requests the generative AI to generate a new sightseeing tour plan based on the new conditions. The updated plan is sent to the device and provided to the user in real time.

[1579] Feedback and Ratings

[1580] User

[1581] After the tour, users can access the feedback page in the UTour app and enter their tour rating and comments.

[1582] Terminal

[1583] The terminal transmits the input feedback information to the server.

[1584] server

[1585] The server stores the feedback in a database and uses it for analysis. Emotion data collected by the emotion engine is also stored as feedback information and taken into account when generating the next plan.

[1586] Specific examples

[1587] Example 1: Tailoring tours to match the emotions of history buffs

[1588] User A plans a one-day tour of historical sites, with his / her area of ​​interest being "historical sites," his / her budget being 5,000 yen, and his / her visit date being the following day. The system uses generative AI to create an optimal tour plan and guides him / her with AR navigation. During the tour, the emotion engine recognizes that User A is tired and sends this to the server. The server then generates a new plan with appropriate rest spots added and sends it to the device.

[1589] Example 2: Anime pilgrimage with changes and emotional feedback

[1590] During a tour of anime holy sites, User B must skip one of the stops due to unplanned circumstances. The emotion engine detects stress and anxiety and reports this to the server. The server regenerates the tour plan based on the new conditions and provides it to the user. After the tour ends, User B inputs feedback and emotion data, and the emotion engine provides new insights for the next tour.

[1591] By incorporating an emotion engine, the present invention makes it possible to provide flexible tour plans that correspond to the user's emotional state, thereby realizing a more advanced and personalized sightseeing experience.

[1592] The processing flow will be explained below.

[1593] User Registration and Login

[1594] Step 1:

[1595] The user launches the "UTour" app on their device and accesses the new registration page.

[1596] Step 2:

[1597] The user enters the required information such as name, email address, and password.

[1598] Step 3:

[1599] The terminal transmits the input information to the server.

[1600] Step 4:

[1601] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[1602] Step 5:

[1603] The user enters their email address and password on the login page of the "UTour" app.

[1604] Step 6:

[1605] The terminal transmits the login information to the server.

[1606] Step 7:

[1607] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[1608] Enter your interests and preferences

[1609] Step 1:

[1610] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[1611] Step 2:

[1612] The terminal transmits the input desired information to the server.

[1613] Step 3:

[1614] The server temporarily stores the received desired information in a database.

[1615] Tour plan generation and provision

[1616] Step 1:

[1617] The server sends the saved desired information to the generative AI.

[1618] Step 2:

[1619] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[1620] Step 3:

[1621] The generative AI returns the generated tour plan to the server.

[1622] Step 4:

[1623] The server transmits the generated tour plan to the terminal.

[1624] Step 5:

[1625] The terminal presents the received tour plan to the user.

[1626] Step 6:

[1627] The user can review the proposed plan and make fine adjustments as needed.

[1628] Introducing the Emotion Engine

[1629] Step 1:

[1630] During the tour, users use the device's camera and microphone to input their facial expressions and voices into the emotion engine.

[1631] Step 2:

[1632] The terminal recognizes the user's emotions in real time using an emotion engine and transmits the data to the server.

[1633] Step 3:

[1634] The server analyzes the emotion information sent from the emotion engine, and if the user's emotion indicates discomfort or stress, it determines that the tour plan needs to be adjusted.

[1635] Step 4:

[1636] The server requests generative AI to adjust the tour plan based on the emotional information.

[1637] Step 5:

[1638] The generative AI generates a revised tour plan based on the new conditions.

[1639] Step 6:

[1640] The generative AI sends the revised tour plan back to the server.

[1641] Step 7:

[1642] The server sends the modified plan to the terminal.

[1643] Step 8:

[1644] The device displays the new tour plan to the user and updates the AR navigation.

[1645] AR guidance and plan changes

[1646] Step 1:

[1647] The user starts the tour and enables AR navigation on the device.

[1648] Step 2:

[1649] The device activates the camera function and prepares for AR display.

[1650] Step 3:

[1651] The server receives the current location information and transmits route information to the next destination.

[1652] Step 4:

[1653] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[1654] Step 5:

[1655] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[1656] Step 6:

[1657] The terminal sends the new conditions to the server.

[1658] Step 7:

[1659] The server will then make a new request to the generative AI based on the new conditions.

[1660] Step 8:

[1661] The generative AI generates a revised tour plan based on the new conditions.

[1662] Step 9:

[1663] The generative AI sends the revised tour plan back to the server.

[1664] Step 10:

[1665] The server sends the modified plan to the terminal.

[1666] Step 11:

[1667] The device displays the new tour plan to the user and updates the AR navigation.

[1668] Feedback and Ratings

[1669] Step 1:

[1670] After completing the tour, users can enter their tour rating and comments on the feedback page.

[1671] Step 2:

[1672] The terminal transmits the input feedback to the server.

[1673] Step 3:

[1674] The server stores the feedback in a database and uses it for analysis. It also stores the emotion data collected by the emotion engine as feedback information.

[1675] The above are the specific processing steps of the "UTour" system, which incorporates an emotion engine.

[1676] Example 2

[1677] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1678] Conventional tourist tour plan generation systems provide tour plans based on the user's interests and preferences, but they are unable to consider information about the user's emotions and psychological state. Furthermore, they lack the means to dynamically adjust tour plans in real time, making it difficult to maximize user satisfaction. Therefore, there is a need for a system that can flexibly adjust tour plans based on the user's emotions and provide a personalized tourist experience.

[1679] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving interest and preference information input by a user; means for generating an optimal sightseeing tour plan based on the received interest and preference information using a generative AI model; means for presenting the generated sightseeing tour plan to the user; means for displaying real-world guidance as augmented reality; means for regenerating the sightseeing tour plan based on new conditions using the generative AI model when the user changes the plan during the tour; means having an emotion engine for collecting the user's facial expressions and voice in real time; and means for analyzing the collected emotion data and adjusting the tour plan according to the emotion. This makes it possible to provide flexible tour plans that correspond to the user's emotional state, resulting in a more advanced and personalized sightseeing experience.

[1680] A "user" is a person who uses the system to create a sightseeing tour plan and enjoy a personalized sightseeing experience.

[1681] "Interest and desired information" is information input by the user such as areas of interest, budget, and dates of visit related to tourism.

[1682] A "generative AI model" is an artificial intelligence model that generates optimal sightseeing tour plans based on the user's interests and desired information.

[1683] A "tour plan" is a tour plan that includes specific tourist spots and the order in which they are to be visited, created by a generative AI model based on the user's interests and preferences.

[1684] "Augmented reality" is a technology that displays digital information overlaid on a real-world environment.

[1685] The "emotion engine" is part of a system that collects and analyzes emotions from the user's facial expressions and voice in real time.

[1686] "Emotion data" refers to data such as facial expressions and voice collected by the user's emotion engine.

[1687] "Feedback" refers to information such as ratings and comments entered by the user after the tour has ended.

[1688] "User authentication" is the process of verifying a user's identity when accessing a system.

[1689] "Generate" means that the generative AI model creates a sightseeing tour plan based on the user's interests and desired information.

[1690] "Adjustment" means making changes or corrections to existing sightseeing tour plans based on the results of analyzing emotional data.

[1691] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[1692] 1. User Registration and Login

[1693] User

[1694] Users access the dedicated application using a device such as a smartphone or tablet. When using for the first time, they create a new account by entering their name, email address, and password on the new registration screen. Existing users can authenticate by entering their email address and password on the login screen.

[1695] Terminal

[1696] The terminal transmits the registration information or login information input by the user to the server.

[1697] server

[1698] The server checks the entered information against the user information in the database. If the check is successful, it creates an authenticated session and returns it to the terminal. If the check is unsuccessful, it creates an error message and returns it to the terminal.

[1699] 2. Enter your interests and preferences

[1700] User

[1701] The user goes to the "Create a Tour Plan" page and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[1702] Terminal

[1703] The terminal transmits the inputted interest and desired information to the server.

[1704] server

[1705] The server stores the received information in a database, and uses a generative AI model to generate prompts based on that information to create optimal sightseeing tour plans.

[1706] 3. Creating and providing tour plans

[1707] server

[1708] The server sends prompts to the generative AI model, which then generates a tour plan based on the user's interests, budget, visit dates, and other information, including the selection of tourist spots and the order in which to visit them.

[1709] Terminal

[1710] The tour plan received from the server is presented to the user, who can then review the plan and make any necessary adjustments.

[1711] 4. Introducing the Emotion Engine

[1712] User

[1713] During the tour, the emotion engine collects the user's facial expressions and voice data in real time, and this emotion information reflects the user's emotional state throughout the tour experience.

[1714] Terminal

[1715] The device transmits the collected emotion data to the server in real time.

[1716] server

[1717] The server analyzes the emotion data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to readjust the tour plan. For example, if the user shows signs of fatigue, it will regenerate the plan by adding rest stops and send it to the device.

[1718] 5. AR Guidance and Plan Changes

[1719] User

[1720] Once the tour begins, users can activate AR navigation on their device to receive on-site guidance. If unplanned changes occur, users can enter new preferences and submit them to the system.

[1721] Terminal

[1722] The device displays route guidance and tourist information superimposed on the real landscape as AR, and also transmits new desired conditions to the server.

[1723] server

[1724] The server then requests the generative AI model to generate a new sightseeing tour plan based on the new conditions, and the plan is provided to the device in real time.

[1725] 6. Feedback and Ratings

[1726] User

[1727] After the tour is over, the user inputs feedback and sends the feedback and emotion data to the server.

[1728] Terminal

[1729] The terminal transmits the feedback information to the server.

[1730] server

[1731] The server stores the feedback and emotion data in a database and takes them into account when generating the next plan.

[1732] Specific examples

[1733] Example 1: Tour plan for history buffs

[1734] User A selects "historical sites" as an area of ​​interest, budgets 5,000 yen, and the date of visit for the following day. The server generates an appropriate tour plan using a generative AI model and sends it to the device. If User A shows signs of fatigue during the tour, the server incorporates rest spots based on information from the emotion engine and resends the adjusted plan.

[1735] Example 2: Anime pilgrimages that require changes mid-trip

[1736] While touring the anime holy sites, User B needs to skip one of the stops. The emotion engine detects the stress and reports it to the server. The server regenerates the tour plan taking the new conditions into account and provides it to User B. After the tour ends, the feedback and emotion data are used to generate the next plan.

[1737] Example prompt sentence:

[1738] "Generate a tour plan for historical sites with a budget of 5000 yen. The visit date is tomorrow."

[1739] The above is an embodiment of the sightseeing tour plan creation system of the present invention. By using this system, it is possible to provide flexible tour plans that correspond to the emotional state of the user, realizing a more advanced and personalized sightseeing experience.

[1740] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1741] Step 1: User Registration

[1742] Users access the application using a smartphone or tablet and enter their name, email address, and password on the new registration screen (input: user information).

[1743] The terminal transmits the input user information to the server (output: user information).

[1744] The server stores the received user information in a database, generates a registration completion message and an authentication token, and returns them to the terminal (data processing: store user information in database, data calculation: generate token) (output: registration completion message, authentication token).

[1745] The terminal displays a registration completion message to the user (action: display message).

[1746] Step 2: User Login

[1747] The user enters their email address and password on the login screen (input: login information).

[1748] The terminal sends the login information to the server (output: login information).

[1749] The server checks the entered login information against the database, and if it matches, it creates an authenticated session; if it doesn't, it generates an error message and sends it to the terminal (data processing: checking login information, data calculation: session generation) (output: authenticated session or error message).

[1750] The terminal maintains the authenticated session and displays a login success message or an error message to the user (action: display message).

[1751] Step 3: Enter your interests and preferences

[1752] The user enters the area of ​​interest, budget, visit date, etc. on the "Create Tour Plan" page and taps the submit button (input: area of ​​interest, budget, visit date).

[1753] The terminal sends the input information to the server (output: areas of interest, budget, date of visit).

[1754] The server stores the received information in a database and generates a prompt statement based on that information (data processing: storing in a database, data calculation: generating a prompt statement) (output: prompt statement).

[1755] Step 4: Generate a tour plan

[1756] The server sends the generated prompt sentence to the generative AI model (input: prompt sentence) (output: generation request).

[1757] The generative AI model generates a tour plan including tourist attractions and the order in which to visit them based on the prompt sentence (Data calculation: Generate tour plan) (Output: Tour plan).

[1758] The server receives the generated tour plan and transmits it to the terminal (output: tour plan).

[1759] Step 5: Present your tour plan

[1760] The terminal displays the received tour plan to the user (operation: display plan).

[1761] The user checks the presented plan and makes fine adjustments as necessary (input: fine adjustment information).

[1762] Step 6: Collect emotion data

[1763] During the tour, the emotion engine collects the user's facial expression and voice data in real time (input: facial expression data, voice data).

[1764] The terminal transmits the collected emotion data to the server in real time (output: emotion data).

[1765] Step 7: Analyze emotional data and adjust your plan

[1766] The server analyzes the emotional data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to adjust a new tour plan (data calculation: analysis of emotional data, regeneration of plan) (output: readjustment request).

[1767] The generative AI model generates a new tour plan with additional rest stops added as needed and sends it to the server (output: adjusted tour plan).

[1768] The server transmits the adjusted tour plan to the terminal (output: adjusted tour plan).

[1769] The terminal presents the adjusted tour plan to the user (action: show plan).

[1770] Step 8: Providing AR Navigation

[1771] When the user starts the tour, he or she uses the AR navigation of the device (action: start AR navigation).

[1772] The device displays route guidance and tourist information superimposed on real-world scenery (operation: composite display of the physical world and digital information).

[1773] Step 9: Feedback and Rating

[1774] After completing the tour, the user enters a rating and comments on the feedback page (input: feedback information).

[1775] The terminal transmits the feedback information to the server (output: feedback information).

[1776] The server stores the feedback and emotion data in a database and takes them into consideration when generating the next plan (data processing: save, data calculation: take into consideration when generating the next plan).

[1777] Through these steps, the system can flexibly adjust the tour plan according to the user's emotional state and provide a personalized sightseeing experience.

[1778] (Application example 2)

[1779] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1780] Conventional sightseeing tour planning systems can generate plans based on the user's interests and preferences, but they cannot adjust the plan to accommodate real-time emotional changes during the tour. Furthermore, no systems exist that can personalize the in-store shopping experience or promote sales based on the user's emotional state. This makes it difficult to provide an optimal and comfortable experience for users.

[1781] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information about interests and desires input by a user, means for generating an optimal sightseeing tour plan based on the received information about interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, means for regenerating the sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour, and means for detecting the user's emotions and adjusting the plan based on the emotions. This makes it possible to adjust the plan in response to real-time changes in the user's emotions, providing a more personalized and comfortable experience.

[1782] The "means for receiving information about interests and desires entered by the user" refers to a device such as a smartphone or tablet that allows the user to enter information about their interests and desires, which is then received by the system.

[1783] "Means for generating optimal sightseeing tour plans based on the received information on interests and desires using generative AI" refers to a means for automatically generating optimal sightseeing tour plans using artificial intelligence technology based on the received information on the user's interests and desires.

[1784] The "means for presenting the generated sightseeing tour plan to the user" is a means for presenting the generated sightseeing tour plan to the user in an easy-to-view manner, allowing the user to confirm and adjust the plan.

[1785] "Means for displaying real-world guidance as augmented reality" refers to the display of virtual information superimposed on real-world scenery, making it easier for users to visually understand the information.

[1786] "Means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour" refers to a means for the generative AI to regenerate a new sightseeing tour plan in response to changes in the user's wishes or situation during the tour.

[1787] The "means for detecting a user's emotions and adjusting the plan based on the emotions" is for detecting a user's real-time emotional state and adjusting the tour plan or shopping experience according to the emotions.

[1788] The present invention is a system that provides users with optimized sightseeing tour plans and in-store shopping plans, and also detects the user's emotions in real time and adjusts the plans based on the emotions.

[1789] Hardware and Software Configuration

[1790] Hardware

[1791] Smart glasses: Devices with built-in camera and display capabilities (e.g., Google Glass, Vuzix Blade).

[1792] Server: A high-performance server device that runs a database and generative AI.

[1793] software

[1794] Emotion engine: A software module that uses libraries such as OpenCV to analyze the user's facial expressions and detect emotions.

[1795] Generative AI: A module that uses generative AI models such as GPT-3 to generate and regenerate optimal plans based on the user's interests, desired information, and emotional data.

[1796] AR module: A software module that displays virtual guidance information overlaid on the real-world scenery.

[1797] System Operation

[1798] User Registration and Login

[1799] The user puts on the smart glasses and accesses the app to register or log in. The server receives the user's registration information or login information and compares it with the user information stored in the database.

[1800] Enter your interests and preferences

[1801] Through the smart glasses, users input information such as their interests, budget, and visit date, which is then sent to the server.

[1802] Plan generation and presentation

[1803] The server then sends the received information about interests and preferences to the generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the information. The generated plans are then sent from the server to the smart glasses and presented to the user.

[1804] Real-time emotion detection and plan adjustment

[1805] As the user tours or shops, the smart glasses' camera captures the user's facial expressions, and the emotion engine detects their emotions in real time. The emotion information is sent to the server, which analyzes the information from the emotion engine. If the user feels stressed or tired, the generative AI regenerates the plan based on the new conditions.

[1806] Augmented reality (AR) guidance

[1807] The smart glasses overlay virtual guide information on real-world scenery, providing users with visually easy-to-understand navigation and tourist information.

[1808] Specific examples

[1809] Scenario 1: If a user feels tired while shopping, the emotion engine detects this. The generative AI recommends a simple rest area and refreshing items (e.g., drinks). The in-store AR display also guides the user to the shortest route.

[1810] Scenario 2: When a user makes an unexpected change during a sightseeing tour, they input their new interests and preferences and submit them to the system. The generative AI then regenerates the optimal tour plan based on the new conditions and displays it on the smart glasses.

[1811] Example prompt sentence:

[1812] The user's interests are "fashion" and "technology," their budget is 10,000 yen, and their current emotion is "tired." Please suggest places to rest and products that will refresh them.

[1813] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1814] Step 1:

[1815] A user puts on smart glasses and accesses the app to register or log in. The input is the user's registration information (name, email address, password) or login information (email address, password). The server receives this information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session and sends it back to the device. The output is the authenticated session information.

[1816] Step 2:

[1817] The user inputs information such as their interests, budget, and visit date through the smart glasses. The input information is sent from the device to the server. The server temporarily stores the received interest and preference information in a database. The input is the interest and preference information entered by the user, and the output is the saved interest and preference information.

[1818] Step 3:

[1819] The server sends the stored interests and preferences to a generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the interests and preferences. The input is the interests and preferences, and the output is the generated plan.

[1820] Step 4:

[1821] The generated plan is sent from the server to the smart glasses. The device presents the generated plan to the user. The input is the generated plan, and the output is the presented plan. The user can review the plan and make fine adjustments if necessary.

[1822] Step 5:

[1823] As the user tours or goes shopping, the camera in the smart glasses continues to capture the user's facial expressions. Based on the captured frames, the device uses an emotion engine to detect the user's emotions in real time. The input is the camera frames, and the output is the detected emotion data.

[1824] Step 6:

[1825] The detected emotional information is sent from the device to the server. The server analyzes the information from the emotion engine, and if the user is feeling stressed or tired, the generative AI regenerates a plan based on the new conditions. The input is emotional data, and the output is the regenerated plan.

[1826] Step 7:

[1827] The smart glasses receive the regenerated plan and overlay virtual guide information on the real-world scenery, providing the user with a visually easy-to-understand navigation or shopping guide. The input is the regenerated plan, and the output is the guide information displayed in AR.

[1828] Step 8:

[1829] After completing the tour or shopping, the user accesses the feedback page and enters their experience rating and comments. The device sends the entered feedback information to the server, which stores the feedback in a database and uses it for analysis. The input is the user's feedback, and the output is the stored feedback information.

[1830] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1831] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1832] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1833] [Fourth embodiment]

[1834] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1835] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1836] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1837] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1838] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1839] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1840] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1841] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1842] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1843] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1844] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1845] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1846] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1847] The present invention is a system for providing users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Specific embodiments of the system of the present invention will be described below.

[1848] User Registration and Login

[1849] User

[1850] Users access the "UTour" app through their device (smartphone, tablet, etc.) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[1851] server

[1852] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[1853] Enter your interests and preferences

[1854] User

[1855] Users go to the tour plan creation page of the "UTour" app and enter information such as their areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[1856] Terminal

[1857] The terminal transmits the inputted interest and desired information to the server.

[1858] server

[1859] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[1860] Tour plan generation and provision

[1861] server

[1862] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[1863] Terminal

[1864] The tour plan sent from the server is sent back to the terminal and presented to the user, who can then check the presented plan and make any necessary adjustments.

[1865] AR guidance

[1866] User

[1867] Once users start the tour, they can enable AR navigation on their device to receive on-site guidance.

[1868] Terminal

[1869] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real world. For example, instructions like "The next spot is 500 meters away, turn right" are displayed in real time.

[1870] Changing plans mid-way

[1871] User

[1872] If an unexpected change becomes necessary during the tour, the user re-enters the desired conditions and transmits them to the system.

[1873] server

[1874] The server then re-queues the generative AI based on the new conditions to generate a new sightseeing tour plan. The updated plan is sent to the device and provided to the user in real time.

[1875] Feedback and Ratings

[1876] User

[1877] After completing the tour, users enter feedback and ratings based on their experience into the UTour app.

[1878] Terminal

[1879] The terminal transmits the input feedback information to the server.

[1880] server

[1881] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system.

[1882] Specific examples

[1883] Example 1: One-day tour of historical sites

[1884] User A wishes to visit multiple historical sites in one day, with a budget of 5,000 yen or less. User A logs into "UTour" and enters his / her area of ​​interest as "historical sites," his / her budget as "under 5,000 yen," and the date of visit as "tomorrow." The server then has the generative AI generate a plan. The generated plan includes historical sites A, B, and C, and route guidance is displayed in AR. User A is satisfied with the presented plan and begins the tour.

[1885] Example 2: Changing the plan for an anime pilgrimage tour

[1886] While planning a tour of anime holy sites, User B unexpectedly needs to skip one of the stops. User B enters the new desired information through the app and sends it to the server. The server then requests the generative AI to generate a new plan. The regenerated plan is provided to User B in real time, and the AR navigation is also updated.

[1887] This allows users to always receive the best possible sightseeing experience.

[1888] The system provides a customized sightseeing experience tailored to the user's individual needs and circumstances, and its real-time navigation and information display enable efficient and fulfilling sightseeing.

[1889] The processing flow will be explained below.

[1890] User Registration and Login

[1891] Step 1:

[1892] The user launches the "UTour" app on their device and accesses the new registration page.

[1893] Step 2:

[1894] The user enters the required information such as name, email address, and password.

[1895] Step 3:

[1896] The terminal transmits the input information to the server.

[1897] Step 4:

[1898] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[1899] Step 5:

[1900] The user enters their email address and password on the login page of the "UTour" app.

[1901] Step 6:

[1902] The terminal transmits the login information to the server.

[1903] Step 7:

[1904] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[1905] Enter your interests and preferences

[1906] Step 1:

[1907] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[1908] Step 2:

[1909] The terminal transmits the input desired information to the server.

[1910] Step 3:

[1911] The server temporarily stores the received desired information in a database.

[1912] Tour plan generation and provision

[1913] Step 1:

[1914] The server sends the saved desired information to the generative AI.

[1915] Step 2:

[1916] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[1917] Step 3:

[1918] The generative AI returns the generated tour plan to the server.

[1919] Step 4:

[1920] The server transmits the generated tour plan to the terminal.

[1921] Step 5:

[1922] The terminal presents the received tour plan to the user.

[1923] Step 6:

[1924] The user can review the proposed plan and make fine adjustments as needed.

[1925] AR guidance

[1926] Step 1:

[1927] The user starts the tour and enables AR navigation on the device.

[1928] Step 2:

[1929] The device activates the camera function and prepares for AR display.

[1930] Step 3:

[1931] The server receives the current location information and transmits route information to the next destination.

[1932] Step 4:

[1933] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[1934] Changing plans mid-way

[1935] Step 1:

[1936] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[1937] Step 2:

[1938] The terminal sends the new conditions to the server.

[1939] Step 3:

[1940] The server will then make a new request to the generative AI based on the new conditions.

[1941] Step 4:

[1942] The generative AI generates a revised tour plan based on the new conditions.

[1943] Step 5:

[1944] The generative AI sends the revised tour plan back to the server.

[1945] Step 6:

[1946] The server sends the modified plan to the terminal.

[1947] Step 7:

[1948] The device displays the new tour plan to the user and updates the AR navigation.

[1949] Feedback and Ratings

[1950] Step 1:

[1951] After completing the tour, users can enter their tour rating and comments on the feedback page.

[1952] Step 2:

[1953] The terminal transmits the input feedback to the server.

[1954] Step 3:

[1955] The server stores the feedback in a database and uses it for analysis.

[1956] The above are the specific processing steps of the "UTour" system.

[1957] Example 1

[1958] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1959] Conventional sightseeing tour systems have had difficulty quickly providing optimal plans tailored to individual users' interests and wishes. Furthermore, they lacked the ability to change plans mid-tour and real-time guidance, resulting in an unsatisfactory user experience. The present invention aims to solve these problems and provide customized sightseeing tour plans that satisfy users.

[1960] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1961] In this invention, the server includes means for receiving information on interests and desires input by a user, means for temporarily storing the received information in a database, means for generating an optimal sightseeing tour plan based on the received information on interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, and means for regenerating the sightseeing tour plan based on the new conditions using the generative AI when the user changes the plan during the tour, thereby enabling users to receive an optimal sightseeing experience tailored to their individual interests and desires in real time.

[1962] "Interests and desired information input by the user" refers to information about the user's individual areas of interest and desired conditions for the sightseeing tour (for example, tourist spots, budget, planned dates of visit, etc.).

[1963] The "receiving means" is a mechanism that has the function of acquiring information entered by the user from the terminal and transmitting it to the server.

[1964] The "means for temporarily storing in a database" is a system for temporarily storing received information about the user's interests and preferences, and can be, for example, a NoSQL database or an SQL database.

[1965] "Generative AI" is an artificial intelligence model that generates specific outputs based on input conditions, such as generating sightseeing tour plans using natural language processing.

[1966] The "means for generating optimal sightseeing tour plans" is an algorithm that uses generative AI to construct the order in which tourist spots should be visited and a schedule based on information input by the user.

[1967] The "means for presenting the generated sightseeing tour plan to the user" is a system for displaying the generated tour plan information on the terminal in a format that is easy for the user to understand.

[1968] "Means for displaying real-world guidance as augmented reality" refers to augmented reality technology that uses camera functions to display route guidance and tourist information overlaid on real-world scenery.

[1969] "Means for regenerating sightseeing tour plans based on new conditions when the plan is changed" refers to a mechanism that, when a user inputs new conditions during a tour, requests the generative AI to create a new sightseeing tour plan again.

[1970] The "means for saving feedback" is a system for saving the feedback entered by the user after the tour has ended in a database.

[1971] "Means for user authentication" refers to the process of verifying the identity of a user when logging into a system, for example by verifying an email address and password.

[1972] "Means for securely storing authenticated user data" refers to technology that protects the personal information and interest / desired information of authenticated users while safely storing them in a database.

[1973] This invention is a system that provides users with optimized sightseeing tour plans, automatically generating plans based on the user's individual interests and wishes, and providing real-time navigation and information. A specific embodiment of this system will be described.

[1974] User Registration and Login

[1975] User

[1976] Users access the "Tourism Planning App" using a device such as a smartphone or tablet and register or log in. When registering, users enter their name, email address, and password, which are then sent to the system. If they already have an account, they enter their email address and password for authentication.

[1977] Terminal

[1978] The terminal sends the input information to the server using the HTTPS protocol. Specifically, it makes a POST request to the API endpoint (e.g., / api / login) on the server.

[1979] server

[1980] The server receives the submitted information and checks it against the user information stored in a database (e.g., MySQL). If the check is successful, it generates an authenticated session ID, stores the session information in an in-memory database such as Redis, and sends it back to the device.

[1981] Enter your interests and preferences

[1982] User

[1983] Users go to the tour plan creation page of the "tourism planning app" and enter information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and planned date of visit.

[1984] Terminal

[1985] The terminal parses the input information into JSON format and sends it to the server. Specifically, it makes a POST request to the / api / planrequest endpoint.

[1986] server

[1987] The server temporarily stores the received information in a NoSQL database (e.g., MongoDB).

[1988] Tour plan generation and provision

[1989] server

[1990] The server generates a prompt based on the stored information and requests it from a generative AI model (e.g., GPT-4). An example prompt is: "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan that matches these."

[1991] Based on these prompts, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule, and sends the results back to the server.

[1992] Terminal

[1993] The tour plan sent from the server is received and presented to the user, who can then check the plan and make any necessary adjustments.

[1994] AR guidance

[1995] User

[1996] When users start the tour, they enable AR navigation on their device and receive on-site guidance.

[1997] Terminal

[1998] The device utilizes its camera function to display route guidance and tourist information superimposed on the real landscape as augmented reality (AR). For example, instructions such as "The next spot is 500 meters away, turn right" are displayed in real time.

[1999] Changing plans mid-way

[2000] User

[2001] If an unplanned change occurs during the tour, the user enters new desired conditions and submits them to the system.

[2002] server

[2003] The server then requests the generative AI model again based on the new conditions to generate a new sightseeing tour plan. The updated plan is then sent back to the device and provided to the user in real time.

[2004] Feedback and Ratings

[2005] User

[2006] After completing the tour, users enter feedback and ratings based on their experience into the "tourism planning app."

[2007] Terminal

[2008] The terminal transmits the input feedback information to the server.

[2009] server

[2010] The server stores the collected feedback in a database and uses it as reference information when generating future tour plans, thereby improving the system's accuracy and user satisfaction.

[2011] This will allow users to enjoy a customized sightseeing experience in real time that is tailored to their individual interests and desires.

[2012] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2013] Step 1: User registers or logs in

[2014] User

[2015] Users access the "Tourism Planning App" from their device and register an account or log in. If registering for the first time, they enter information such as their name, email address, and password, and submit. If they already have an account, they enter their email address and password for authentication.

[2016] Terminal

[2017] The device sends the entered information to the server using HTTPS communication, by sending a POST request to an API endpoint such as / api / login.

[2018] server

[2019] The server compares the received information with a database (e.g., MySQL) and saves it in the database in the case of a new registration. In the case of a login, the authentication information is checked, and if it matches, a session ID is generated, encrypted, and sent back to the terminal. The session information is stored in an in-memory database such as Redis.

[2020] Input and Output

[2021] Input: Registration and login information from the user

[2022] Output: Authenticated session ID

[2023] Step 2: User enters their interests and preferences

[2024] User

[2025] The user proceeds to the tour plan creation page and enters information such as areas of interest (e.g., history, anime, nature, etc.), budget, and planned dates of visit.

[2026] Terminal

[2027] The terminal converts the input information into JSON format and sends it to the server. Specifically, it makes a POST request to / api / planrequest.

[2028] server

[2029] The server temporarily stores the received information in a database, such as MongoDB.

[2030] Input and Output

[2031] Input: User's interests and preferences

[2032] Output: Information ID stored in the database

[2033] Step 3: Request to generate a tour plan

[2034] server

[2035] The server generates prompts based on the stored information and sends them to a generative AI model (e.g., GPT-4). For example, it generates a prompt such as, "The user's interest is history, their budget is within 5,000 yen, and their visit date is tomorrow. Please generate a sightseeing tour plan based on this information."

[2036] Input and Output

[2037] Input: Saved user information

[2038] Output: Prompt for the generative AI

[2039] Step 4: Plan generation using a generative AI model

[2040] Generative AI

[2041] Based on the prompts received, the generative AI model generates an optimal sightseeing tour plan, including the order in which tourist spots are visited and the schedule.

[2042] Input and Output

[2043] Input: Prompt sentence for generative AI

[2044] Output: Optimal sightseeing tour plan

[2045] Step 5: Present your tour plan

[2046] server

[2047] The generated tour plan is sent to the server, and the server retransmits the result to the terminal.

[2048] Terminal

[2049] The terminal presents the received tour plan to the user, who can then review the plan and make minor adjustments as necessary.

[2050] Input and Output

[2051] Input: Generated tour plan

[2052] Output: The tour plan displayed to the user

[2053] Step 6: Enabling AR Navigation

[2054] User

[2055] The user enables the AR navigation function on their device when starting the tour.

[2056] Terminal

[2057] The device uses its camera to display route guidance to tourist spots in augmented reality (AR) overlaid on the real-world scenery, with real-time guidance such as "Next destination 500m, turn right."

[2058] Input and Output

[2059] Input: Location information based on tour plan

[2060] Output: AR display overlaid on real scenery

[2061] Step 7: Change your plan mid-way

[2062] User

[2063] If an unplanned change is required during the tour, the user enters new requirements and submits them to the system.

[2064] Terminal

[2065] The terminal transmits the new desired conditions to the server.

[2066] server

[2067] The server then resubmits the request to the generative AI model based on the new conditions to generate a new tour plan, which is then sent back to the device and provided to the user in real time.

[2068] Input and Output

[2069] Input: User's new preferences

[2070] Output: New sightseeing tour plan

[2071] Step 8: Enter and save your feedback

[2072] User

[2073] After completing the tour, users provide feedback and a rating based on their experience.

[2074] Terminal

[2075] The terminal transmits the input feedback information to the server.

[2076] server

[2077] The server stores the collected feedback information in a database and uses it as reference information when generating subsequent tour plans.

[2078] Input and Output

[2079] Input: User feedback information

[2080] Output: Feedback information stored in a database

[2081] Through this series of processes, users can enjoy sightseeing tour plans optimized in real time to suit their individual interests and desires.

[2082] (Application example 1)

[2083] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2084] Conventional sightseeing tour planning systems do not take into account the user's consumption preferences or in-store route guidance, which makes it difficult to achieve efficient shopping in stores. Furthermore, guidance using augmented reality (AR) technology is limited, making it difficult for users to quickly and intuitively obtain the information they need in-store. This often results in an unsatisfactory shopping experience for users.

[2085] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2086] In this invention, the server includes means for receiving information on interests and desires input by a user, means for generating an optimal sightseeing tour plan and in-store route based on the received information on interests and desires using a generative AI, means for presenting the generated plan and route to the user, means for displaying real-world guidance as augmented reality, means for regenerating the plan based on new conditions using the generative AI when the user changes the plan during the tour or shopping, and means for providing in-store product locations as augmented reality, thereby enabling users to enjoy optimal sightseeing and shopping experiences based on their individual interests and consumption preferences.

[2087] "Interest and desired information" is a general term for information that a user inputs about their areas of interest and the experiences they are looking for.

[2088] "Generative AI" is an artificial intelligence technology that automatically generates optimal plans and proposals based on input information.

[2089] A "tourist tour plan" refers to a travel plan that includes the order and schedule of visiting tourist spots.

[2090] Augmented reality (AR) is a technology that displays digital information overlaid on real-world scenery.

[2091] "In-store route" refers to an optimal route plan for a user to move efficiently within the store.

[2092] "Product suggestions" are information about products recommended based on the user's consumption preferences.

[2093] "User authentication" is the process of identifying an individual user and verifying their access rights.

[2094] "Consumption preferences" refers to trends in a user's purchasing behavior and preferences.

[2095] "Feedback" refers to the evaluations and opinions provided by users after using the service.

[2096] "Plan regeneration" is the process of regenerating an existing plan based on new conditions.

[2097] The present invention is a system that provides users with optimized sightseeing tour plans and in-store routes. This system generates plans based on the user's interests and desired information, and provides guide information using augmented reality (AR) technology. Specific embodiments of the present invention are described below.

[2098] User Registration and Login

[2099] User

[2100] Users access a dedicated application through their device (such as a smartphone or tablet) to register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register with the system. When logging in with an existing account, users enter their email address and password for authentication.

[2101] server

[2102] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[2103] Enter your interests and preferences

[2104] User

[2105] The user goes to the planning page of the dedicated application and enters information such as their areas of interest (e.g., history, fashion, food, etc.), budget, and planned dates of visit.

[2106] Terminal

[2107] The terminal transmits the inputted interest and desired information to the server.

[2108] server

[2109] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal plan.

[2110] Generate and deliver plans

[2111] server

[2112] The server sends the saved information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which tourist spots should be visited, a schedule, in-store routes, and product suggestions, and sends the results back to the server.

[2113] Terminal

[2114] The plan sent from the server is sent back to the terminal and presented to the user, who can then check the plan and make any necessary adjustments.

[2115] AR guidance

[2116] User

[2117] Once the user executes the plan, they can enable AR navigation on their device to receive on-site guidance.

[2118] Terminal

[2119] The device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. For example, instructions such as "The next stop is 500 meters away, turn right" or "Recommended products are on this shelf" are displayed in real time.

[2120] Changing plans mid-way

[2121] User

[2122] If an unexpected change is required during the plan, the user re-enters the desired conditions and sends them to the system.

[2123] server

[2124] The server then re-submits the request to the AI ​​based on the new conditions to generate a new plan. The updated plan is then sent to the device and provided to the user in real time.

[2125] Feedback and Ratings

[2126] User

[2127] After completing the plan, users enter feedback and ratings based on their experience into a dedicated application.

[2128] Terminal

[2129] The terminal transmits the input feedback information to the server.

[2130] server

[2131] The server stores the collected feedback in a database and uses it as reference information when generating future plans, thereby improving the system.

[2132] Specific prompt examples

[2133] "Generate the optimal shopping plan based on the following user preferences and wishes: Preferences: Apparel, Budget: Under 10,000 yen, Season: Spring"

[2134] This system enables users to efficiently enjoy optimal sightseeing and shopping experiences based on their interests and consumption preferences.

[2135] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2136] Step 1:

[2137] A user accesses a dedicated application using a terminal to register or log in. As input, they provide their name, email address, and password. The server receives this information and compares it with the user information stored in a database. As output, it generates an authenticated session and sends it back to the terminal.

[2138] Step 2:

[2139] The user enters information such as their areas of interest, budget, and planned visit date on the plan creation page of the dedicated application. The input information includes interests and wishes. The device sends this information to the server, which then temporarily stores the received information in a database.

[2140] Step 3:

[2141] The server sends the stored information about interests and preferences to the generative AI, which then generates an optimal plan based on this information, including the order in which to visit tourist spots, a schedule, in-store routes, and product suggestions. The optimal plan is then returned to the server as output.

[2142] Step 4:

[2143] The server sends the plan returned by the generative AI to the device and presents it to the user. The user can then review the plan and make any necessary adjustments. At this stage, the user can also enable AR navigation for when they arrive at the destination.

[2144] Step 5:

[2145] When the user executes the plan, the device uses its camera to display route guidance and product information as augmented reality (AR) overlaid on the real world. The input includes the user's current location and the generated plan. The output is the route instructions and product location information displayed on the real world.

[2146] Step 6:

[2147] If an unexpected change is required midway through the plan, the user re-enters their desired conditions and sends them to the system. The new desired conditions are included as input. The server then requests the generative AI again based on the new conditions to generate a new plan. As output, the updated plan is sent to the device and provided to the user in real time.

[2148] Step 7:

[2149] After completing the plan, the user enters feedback and evaluation based on their experience into a dedicated application. The input includes the user's feedback. The device sends this feedback information to the server, which stores it in a database. The output provides reference information for use in improving the system.

[2150] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2151] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[2152] User Registration and Login

[2153] User

[2154] Users access the "UTour" app through their device (smartphone or tablet) and register or log in. When registering for the first time, users enter required information such as their name, email address, and password to register in the system. When logging in with an existing account, users enter their email address and password for authentication.

[2155] server

[2156] The server receives the registration information and login information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session for the user and returns it to the terminal.

[2157] Enter your interests and preferences

[2158] User

[2159] The user goes to the "Tour Plan Creation" page on the terminal and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[2160] Terminal

[2161] The terminal transmits the input information to the server.

[2162] server

[2163] The server temporarily stores the received information on interests and preferences in a database, and then uses that information to request a generative AI to generate the optimal sightseeing tour plan.

[2164] Tour plan generation and provision

[2165] server

[2166] The server sends the stored information to a generative AI, which determines tourist spots and the order in which to visit them based on the user's interests and preferences, and generates a tour plan. The generated plan is then sent back to the server, which then sends it to the device.

[2167] Terminal

[2168] The terminal presents the received tour plan to the user, who can then check the plan and make fine adjustments to it as necessary.

[2169] Introducing the Emotion Engine

[2170] User

[2171] During the tour, the emotion engine collects data such as facial expressions and voice in real time, and this emotional information reflects how the user feels throughout the tour experience.

[2172] Terminal

[2173] The terminal transmits the collected emotion information to the server.

[2174] server

[2175] The server analyzes the emotional information sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI to adjust the tour plan. For example, if the user feels tired, it adds rest stops and sends the adjusted plan back to the device.

[2176] AR guidance and plan changes

[2177] User

[2178] Once a user starts a tour, they can activate AR navigation on their device to receive on-site guidance. If an unexpected change is needed during the tour, they can enter their new preferences and submit them to the system.

[2179] Terminal

[2180] The device uses its camera to display route guidance and tourist information as augmented reality (AR) overlaid on the real landscape. When the user inputs new desired conditions, the device sends them to the server.

[2181] server

[2182] The server then requests the generative AI to generate a new sightseeing tour plan based on the new conditions. The updated plan is sent to the device and provided to the user in real time.

[2183] Feedback and Ratings

[2184] User

[2185] After the tour, users can access the feedback page in the UTour app and enter their tour rating and comments.

[2186] Terminal

[2187] The terminal transmits the input feedback information to the server.

[2188] server

[2189] The server stores the feedback in a database and uses it for analysis. Emotion data collected by the emotion engine is also stored as feedback information and taken into account when generating the next plan.

[2190] Specific examples

[2191] Example 1: Tailoring tours to match the emotions of history buffs

[2192] User A plans a one-day tour of historical sites, with his / her area of ​​interest being "historical sites," his / her budget being 5,000 yen, and his / her visit date being the following day. The system uses generative AI to create an optimal tour plan and guides him / her with AR navigation. During the tour, the emotion engine recognizes that User A is tired and sends this to the server. The server then generates a new plan with appropriate rest spots added and sends it to the device.

[2193] Example 2: Anime pilgrimage with changes and emotional feedback

[2194] During a tour of anime holy sites, User B must skip one of the stops due to unplanned circumstances. The emotion engine detects stress and anxiety and reports this to the server. The server regenerates the tour plan based on the new conditions and provides it to the user. After the tour ends, User B inputs feedback and emotion data, and the emotion engine provides new insights for the next tour.

[2195] By incorporating an emotion engine, the present invention makes it possible to provide flexible tour plans that correspond to the user's emotional state, thereby realizing a more advanced and personalized sightseeing experience.

[2196] The processing flow will be explained below.

[2197] User Registration and Login

[2198] Step 1:

[2199] The user launches the "UTour" app on their device and accesses the new registration page.

[2200] Step 2:

[2201] The user enters the required information such as name, email address, and password.

[2202] Step 3:

[2203] The terminal transmits the input information to the server.

[2204] Step 4:

[2205] The server verifies the received information, stores it in a database, and if the storage is successful, returns a registration completion message to the terminal.

[2206] Step 5:

[2207] The user enters their email address and password on the login page of the "UTour" app.

[2208] Step 6:

[2209] The terminal transmits the login information to the server.

[2210] Step 7:

[2211] The server checks the user's information from the database, and if the authentication is successful, creates a session and returns a success message to the terminal.

[2212] Enter your interests and preferences

[2213] Step 1:

[2214] The user enters desired information such as areas of interest, budget, and dates of visit on the "Tour Plan Creation" page of the terminal.

[2215] Step 2:

[2216] The terminal transmits the input desired information to the server.

[2217] Step 3:

[2218] The server temporarily stores the received desired information in a database.

[2219] Tour plan generation and provision

[2220] Step 1:

[2221] The server sends the saved desired information to the generative AI.

[2222] Step 2:

[2223] The generative AI determines the best tourist spots and the order in which to visit them based on desired information.

[2224] Step 3:

[2225] The generative AI returns the generated tour plan to the server.

[2226] Step 4:

[2227] The server transmits the generated tour plan to the terminal.

[2228] Step 5:

[2229] The terminal presents the received tour plan to the user.

[2230] Step 6:

[2231] The user can review the proposed plan and make fine adjustments as needed.

[2232] Introducing the Emotion Engine

[2233] Step 1:

[2234] During the tour, users use the device's camera and microphone to input their facial expressions and voices into the emotion engine.

[2235] Step 2:

[2236] The terminal recognizes the user's emotions in real time using an emotion engine and transmits the data to the server.

[2237] Step 3:

[2238] The server analyzes the emotion information sent from the emotion engine, and if the user's emotion indicates discomfort or stress, it determines that the tour plan needs to be adjusted.

[2239] Step 4:

[2240] The server requests generative AI to adjust the tour plan based on the emotional information.

[2241] Step 5:

[2242] The generative AI generates a revised tour plan based on the new conditions.

[2243] Step 6:

[2244] The generative AI sends the revised tour plan back to the server.

[2245] Step 7:

[2246] The server sends the modified plan to the terminal.

[2247] Step 8:

[2248] The device displays the new tour plan to the user and updates the AR navigation.

[2249] AR guidance and plan changes

[2250] Step 1:

[2251] The user starts the tour and enables AR navigation on the device.

[2252] Step 2:

[2253] The device activates the camera function and prepares for AR display.

[2254] Step 3:

[2255] The server receives the current location information and transmits route information to the next destination.

[2256] Step 4:

[2257] The device uses AR to display real-time directions and tourist information, such as a message like "The next spot is 500 meters away, turn right."

[2258] Step 5:

[2259] If the user wishes to change the plan during the tour, he or she inputs new desired conditions into the terminal.

[2260] Step 6:

[2261] The terminal sends the new conditions to the server.

[2262] Step 7:

[2263] The server will then make a new request to the generative AI based on the new conditions.

[2264] Step 8:

[2265] The generative AI generates a revised tour plan based on the new conditions.

[2266] Step 9:

[2267] The generative AI sends the revised tour plan back to the server.

[2268] Step 10:

[2269] The server sends the modified plan to the terminal.

[2270] Step 11:

[2271] The device displays the new tour plan to the user and updates the AR navigation.

[2272] Feedback and Ratings

[2273] Step 1:

[2274] After completing the tour, users can enter their tour rating and comments on the feedback page.

[2275] Step 2:

[2276] The terminal transmits the input feedback to the server.

[2277] Step 3:

[2278] The server stores the feedback in a database and uses it for analysis. It also stores the emotion data collected by the emotion engine as feedback information.

[2279] The above are the specific processing steps of the "UTour" system, which incorporates an emotion engine.

[2280] Example 2

[2281] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2282] Conventional tourist tour plan generation systems provide tour plans based on the user's interests and preferences, but they are unable to consider information about the user's emotions and psychological state. Furthermore, they lack the means to dynamically adjust tour plans in real time, making it difficult to maximize user satisfaction. Therefore, there is a need for a system that can flexibly adjust tour plans based on the user's emotions and provide a personalized tourist experience.

[2283] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving interest and preference information input by a user; means for generating an optimal sightseeing tour plan based on the received interest and preference information using a generative AI model; means for presenting the generated sightseeing tour plan to the user; means for displaying real-world guidance as augmented reality; means for regenerating the sightseeing tour plan based on new conditions using the generative AI model when the user changes the plan during the tour; means having an emotion engine for collecting the user's facial expressions and voice in real time; and means for analyzing the collected emotion data and adjusting the tour plan according to the emotion. This makes it possible to provide flexible tour plans that correspond to the user's emotional state, resulting in a more advanced and personalized sightseeing experience.

[2284] A "user" is a person who uses the system to create a sightseeing tour plan and enjoy a personalized sightseeing experience.

[2285] "Interest and desired information" is information input by the user such as areas of interest, budget, and dates of visit related to tourism.

[2286] A "generative AI model" is an artificial intelligence model that generates optimal sightseeing tour plans based on the user's interests and desired information.

[2287] A "tour plan" is a tour plan that includes specific tourist spots and the order in which they are to be visited, created by a generative AI model based on the user's interests and preferences.

[2288] "Augmented reality" is a technology that displays digital information overlaid on a real-world environment.

[2289] The "emotion engine" is part of a system that collects and analyzes emotions from the user's facial expressions and voice in real time.

[2290] "Emotion data" refers to data such as facial expressions and voice collected by the user's emotion engine.

[2291] "Feedback" refers to information such as ratings and comments entered by the user after the tour has ended.

[2292] "User authentication" is the process of verifying a user's identity when accessing a system.

[2293] "Generate" means that the generative AI model creates a sightseeing tour plan based on the user's interests and desired information.

[2294] "Adjustment" means making changes or corrections to existing sightseeing tour plans based on the results of analyzing emotional data.

[2295] The present invention is a system that provides users with an optimized sightseeing tour plan, and various means required for this purpose will be described in detail. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, a more personalized experience can be provided. Specific embodiments of the present invention are described below.

[2296] 1. User Registration and Login

[2297] User

[2298] Users access the dedicated application using a device such as a smartphone or tablet. When using for the first time, they create a new account by entering their name, email address, and password on the new registration screen. Existing users can authenticate by entering their email address and password on the login screen.

[2299] Terminal

[2300] The terminal transmits the registration information or login information input by the user to the server.

[2301] server

[2302] The server checks the entered information against the user information in the database. If the check is successful, it creates an authenticated session and returns it to the terminal. If the check is unsuccessful, it creates an error message and returns it to the terminal.

[2303] 2. Enter your interests and preferences

[2304] User

[2305] The user goes to the "Create a Tour Plan" page and enters information such as their area of ​​interest (e.g., history, anime, nature, etc.), budget, and dates of visit.

[2306] Terminal

[2307] The terminal transmits the inputted interest and desired information to the server.

[2308] server

[2309] The server stores the received information in a database, and uses a generative AI model to generate prompts based on that information to create optimal sightseeing tour plans.

[2310] 3. Creating and providing tour plans

[2311] server

[2312] The server sends prompts to the generative AI model, which then generates a tour plan based on the user's interests, budget, visit dates, and other information, including the selection of tourist spots and the order in which to visit them.

[2313] Terminal

[2314] The tour plan received from the server is presented to the user, who can then review the plan and make any necessary adjustments.

[2315] 4. Introducing the Emotion Engine

[2316] User

[2317] During the tour, the emotion engine collects the user's facial expressions and voice data in real time, and this emotion information reflects the user's emotional state throughout the tour experience.

[2318] Terminal

[2319] The device transmits the collected emotion data to the server in real time.

[2320] server

[2321] The server analyzes the emotion data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to readjust the tour plan. For example, if the user shows signs of fatigue, it will regenerate the plan by adding rest stops and send it to the device.

[2322] 5. AR Guidance and Plan Changes

[2323] User

[2324] Once the tour begins, users can activate AR navigation on their device to receive on-site guidance. If unplanned changes occur, users can enter new preferences and submit them to the system.

[2325] Terminal

[2326] The device displays route guidance and tourist information superimposed on the real landscape as AR, and also transmits new desired conditions to the server.

[2327] server

[2328] The server then requests the generative AI model to generate a new sightseeing tour plan based on the new conditions, and the plan is provided to the device in real time.

[2329] 6. Feedback and Ratings

[2330] User

[2331] After the tour is over, the user inputs feedback and sends the feedback and emotion data to the server.

[2332] Terminal

[2333] The terminal transmits the feedback information to the server.

[2334] server

[2335] The server stores the feedback and emotion data in a database and takes them into account when generating the next plan.

[2336] Specific examples

[2337] Example 1: Tour plan for history buffs

[2338] User A selects "historical sites" as an area of ​​interest, budgets 5,000 yen, and the date of visit for the following day. The server generates an appropriate tour plan using a generative AI model and sends it to the device. If User A shows signs of fatigue during the tour, the server incorporates rest spots based on information from the emotion engine and resends the adjusted plan.

[2339] Example 2: Anime pilgrimages that require changes mid-trip

[2340] While touring the anime holy sites, User B needs to skip one of the stops. The emotion engine detects the stress and reports it to the server. The server regenerates the tour plan taking the new conditions into account and provides it to User B. After the tour ends, the feedback and emotion data are used to generate the next plan.

[2341] Example prompt sentence:

[2342] "Generate a tour plan for historical sites with a budget of 5000 yen. The visit date is tomorrow."

[2343] The above is an embodiment of the sightseeing tour plan creation system of the present invention. By using this system, it is possible to provide flexible tour plans that correspond to the emotional state of the user, realizing a more advanced and personalized sightseeing experience.

[2344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2345] Step 1: User Registration

[2346] Users access the application using a smartphone or tablet and enter their name, email address, and password on the new registration screen (input: user information).

[2347] The terminal transmits the input user information to the server (output: user information).

[2348] The server stores the received user information in a database, generates a registration completion message and an authentication token, and returns them to the terminal (data processing: store user information in database, data calculation: generate token) (output: registration completion message, authentication token).

[2349] The terminal displays a registration completion message to the user (action: display message).

[2350] Step 2: User Login

[2351] The user enters their email address and password on the login screen (input: login information).

[2352] The terminal sends the login information to the server (output: login information).

[2353] The server checks the entered login information against the database, and if it matches, it creates an authenticated session; if it doesn't, it generates an error message and sends it to the terminal (data processing: checking login information, data calculation: session generation) (output: authenticated session or error message).

[2354] The terminal maintains the authenticated session and displays a login success message or an error message to the user (action: display message).

[2355] Step 3: Enter your interests and preferences

[2356] The user enters the area of ​​interest, budget, visit date, etc. on the "Create Tour Plan" page and taps the submit button (input: area of ​​interest, budget, visit date).

[2357] The terminal sends the input information to the server (output: areas of interest, budget, date of visit).

[2358] The server stores the received information in a database and generates a prompt statement based on that information (data processing: storing in a database, data calculation: generating a prompt statement) (output: prompt statement).

[2359] Step 4: Generate a tour plan

[2360] The server sends the generated prompt sentence to the generative AI model (input: prompt sentence) (output: generation request).

[2361] The generative AI model generates a tour plan including tourist attractions and the order in which to visit them based on the prompt sentence (Data calculation: Generate tour plan) (Output: Tour plan).

[2362] The server receives the generated tour plan and transmits it to the terminal (output: tour plan).

[2363] Step 5: Present your tour plan

[2364] The terminal displays the received tour plan to the user (operation: display plan).

[2365] The user checks the presented plan and makes fine adjustments as necessary (input: fine adjustment information).

[2366] Step 6: Collect emotion data

[2367] During the tour, the emotion engine collects the user's facial expression and voice data in real time (input: facial expression data, voice data).

[2368] The terminal transmits the collected emotion data to the server in real time (output: emotion data).

[2369] Step 7: Analyze emotional data and adjust your plan

[2370] The server analyzes the emotional data sent from the emotion engine, and if the user expresses unpleasant emotions, it requests the generative AI model to adjust a new tour plan (data calculation: analysis of emotional data, regeneration of plan) (output: readjustment request).

[2371] The generative AI model generates a new tour plan with additional rest stops added as needed and sends it to the server (output: adjusted tour plan).

[2372] The server transmits the adjusted tour plan to the terminal (output: adjusted tour plan).

[2373] The terminal presents the adjusted tour plan to the user (action: show plan).

[2374] Step 8: Providing AR Navigation

[2375] When the user starts the tour, he or she uses the AR navigation of the device (action: start AR navigation).

[2376] The device displays route guidance and tourist information superimposed on real-world scenery (operation: composite display of the physical world and digital information).

[2377] Step 9: Feedback and Rating

[2378] After completing the tour, the user enters a rating and comments on the feedback page (input: feedback information).

[2379] The terminal transmits the feedback information to the server (output: feedback information).

[2380] The server stores the feedback and emotion data in a database and takes them into consideration when generating the next plan (data processing: save, data calculation: take into consideration when generating the next plan).

[2381] Through these steps, the system can flexibly adjust the tour plan according to the user's emotional state and provide a personalized sightseeing experience.

[2382] (Application example 2)

[2383] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2384] Conventional sightseeing tour planning systems can generate plans based on the user's interests and preferences, but they cannot adjust the plan to accommodate real-time emotional changes during the tour. Furthermore, no systems exist that can personalize the in-store shopping experience or promote sales based on the user's emotional state. This makes it difficult to provide an optimal and comfortable experience for users.

[2385] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information about interests and desires input by a user, means for generating an optimal sightseeing tour plan based on the received information about interests and desires using a generative AI, means for presenting the generated sightseeing tour plan to the user, means for displaying real-world guidance as augmented reality, means for regenerating the sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour, and means for detecting the user's emotions and adjusting the plan based on the emotions. This makes it possible to adjust the plan in response to real-time changes in the user's emotions, providing a more personalized and comfortable experience.

[2386] The "means for receiving information about interests and desires entered by the user" refers to a device such as a smartphone or tablet that allows the user to enter information about their interests and desires, which is then received by the system.

[2387] "Means for generating optimal sightseeing tour plans based on the received information on interests and desires using generative AI" refers to a means for automatically generating optimal sightseeing tour plans using artificial intelligence technology based on the received information on the user's interests and desires.

[2388] The "means for presenting the generated sightseeing tour plan to the user" is a means for presenting the generated sightseeing tour plan to the user in an easy-to-view manner, allowing the user to confirm and adjust the plan.

[2389] "Means for displaying real-world guidance as augmented reality" refers to the display of virtual information superimposed on real-world scenery, making it easier for users to visually understand the information.

[2390] "Means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour" refers to a means for the generative AI to regenerate a new sightseeing tour plan in response to changes in the user's wishes or situation during the tour.

[2391] The "means for detecting a user's emotions and adjusting the plan based on the emotions" is for detecting a user's real-time emotional state and adjusting the tour plan or shopping experience according to the emotions.

[2392] The present invention is a system that provides users with optimized sightseeing tour plans and in-store shopping plans, and also detects the user's emotions in real time and adjusts the plans based on the emotions.

[2393] Hardware and Software Configuration

[2394] Hardware

[2395] Smart glasses: Devices with built-in camera and display capabilities (e.g., Google Glass, Vuzix Blade).

[2396] Server: A high-performance server device that runs a database and generative AI.

[2397] software

[2398] Emotion engine: A software module that uses libraries such as OpenCV to analyze the user's facial expressions and detect emotions.

[2399] Generative AI: A module that uses generative AI models such as GPT-3 to generate and regenerate optimal plans based on the user's interests, desired information, and emotional data.

[2400] AR module: A software module that displays virtual guidance information overlaid on the real-world scenery.

[2401] System Operation

[2402] User Registration and Login

[2403] The user puts on the smart glasses and accesses the app to register or log in. The server receives the user's registration information or login information and compares it with the user information stored in the database.

[2404] Enter your interests and preferences

[2405] Through the smart glasses, users input information such as their interests, budget, and visit date, which is then sent to the server.

[2406] Plan generation and presentation

[2407] The server then sends the received information about interests and preferences to the generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the information. The generated plans are then sent from the server to the smart glasses and presented to the user.

[2408] Real-time emotion detection and plan adjustment

[2409] As the user tours or shops, the smart glasses' camera captures the user's facial expressions, and the emotion engine detects their emotions in real time. The emotion information is sent to the server, which analyzes the information from the emotion engine. If the user feels stressed or tired, the generative AI regenerates the plan based on the new conditions.

[2410] Augmented reality (AR) guidance

[2411] The smart glasses overlay virtual guide information on real-world scenery, providing users with visually easy-to-understand navigation and tourist information.

[2412] Specific examples

[2413] Scenario 1: If a user feels tired while shopping, the emotion engine detects this. The generative AI recommends a simple rest area and refreshing items (e.g., drinks). The in-store AR display also guides the user to the shortest route.

[2414] Scenario 2: When a user makes an unexpected change during a sightseeing tour, they input their new interests and preferences and submit them to the system. The generative AI then regenerates the optimal tour plan based on the new conditions and displays it on the smart glasses.

[2415] Example prompt sentence:

[2416] The user's interests are "fashion" and "technology," their budget is 10,000 yen, and their current emotion is "tired." Please suggest places to rest and products that will refresh them.

[2417] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2418] Step 1:

[2419] A user puts on smart glasses and accesses the app to register or log in. The input is the user's registration information (name, email address, password) or login information (email address, password). The server receives this information and compares it with the user information stored in the database. If the comparison is successful, it creates an authenticated session and sends it back to the device. The output is the authenticated session information.

[2420] Step 2:

[2421] The user inputs information such as their interests, budget, and visit date through the smart glasses. The input information is sent from the device to the server. The server temporarily stores the received interest and preference information in a database. The input is the interest and preference information entered by the user, and the output is the saved interest and preference information.

[2422] Step 3:

[2423] The server sends the stored interests and preferences to a generative AI, which then generates optimal sightseeing tour plans and shopping plans based on the interests and preferences. The input is the interests and preferences, and the output is the generated plan.

[2424] Step 4:

[2425] The generated plan is sent from the server to the smart glasses. The device presents the generated plan to the user. The input is the generated plan, and the output is the presented plan. The user can review the plan and make fine adjustments if necessary.

[2426] Step 5:

[2427] As the user tours or goes shopping, the camera in the smart glasses continues to capture the user's facial expressions. Based on the captured frames, the device uses an emotion engine to detect the user's emotions in real time. The input is the camera frames, and the output is the detected emotion data.

[2428] Step 6:

[2429] The detected emotional information is sent from the device to the server. The server analyzes the information from the emotion engine, and if the user is feeling stressed or tired, the generative AI regenerates a plan based on the new conditions. The input is emotional data, and the output is the regenerated plan.

[2430] Step 7:

[2431] The smart glasses receive the regenerated plan and overlay virtual guide information on the real-world scenery, providing the user with a visually easy-to-understand navigation or shopping guide. The input is the regenerated plan, and the output is the guide information displayed in AR.

[2432] Step 8:

[2433] After completing the tour or shopping, the user accesses the feedback page and enters their experience rating and comments. The device sends the entered feedback information to the server, which stores the feedback in a database and uses it for analysis. The input is the user's feedback, and the output is the stored feedback information.

[2434] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT ...

Claims

1. means for receiving user input of interest and preference information; A means for generating an optimal sightseeing tour plan based on the received interest and desire information using a generative AI; means for presenting the generated sightseeing tour plan to a user; a means for displaying real-world guidance as augmented reality; a means for regenerating a sightseeing tour plan based on new conditions using the generative AI when the user changes the plan during the tour; A system including:

2. 2. The system according to claim 1, further comprising means for allowing a user to input feedback after completing a tour and for storing the feedback.

3. 10. The system of claim 1, further comprising means for performing user authentication and means for securely storing data of authenticated users.

Images