System

Abstract

A system is provided.SOLUTION: A system includes position information acquisition means for acquiring a current position of a user, communication means for transmitting the acquired current position information to a server, display means for displaying sightseeing plan information provided from the server on a terminal, and recommendation means for acquiring recommendation information from a database stored in the server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional tourist information systems require users to obtain information themselves, making it difficult to efficiently gather information. They also lack the functionality to provide optimal sightseeing plans in real time, and their provision of route guidance and detailed guides at tourist spots is limited. Furthermore, when understanding a foreign language is required, the translation function is insufficient, and support for linking with reservation systems is lacking. This results in users having to spend a lot of time and effort during their trip. The objective of the present invention is to solve these problems and improve users' sightseeing experience. [Means for solving the problem]

[0005] The present invention provides a system including a location information acquisition means for acquiring a user's current location, a communication means for transmitting the acquired current location information to a server, a display means for displaying tour plan information provided by the server on a terminal, and a recommendation means for acquiring recommended information from a database stored on the server. The present invention also provides a system including a means for transmitting tour preference information input by the user to the server, a means for receiving the tour plan generated by the server, a visual guide means for displaying location information and detailed information about tourist spots, an audio guide means for providing audio guidance about the displayed information, a means for detecting text information within the user's field of view using a camera, a translation means for translating the detected text information, a display means for displaying the translation results, a reservation means for sending a reservation request to the server and checking the reservation status, and a display and audio notification means for confirming the reservation completion result. This allows users to enjoy optimal tour plans in real time and receive efficient route guidance, detailed tour guides, translation functions, and reservation support.

[0006] The "location information acquisition means" is a technical means for detecting the current location of the user and acquiring the location information.

[0007] "Communication means" refers to the technical means for transmitting acquired location information and other data to the server.

[0008] "Display means" refers to the technical means for visually presenting information and guidance provided by the server to the user.

[0009] "Recommendation means" refers to a technical means for recommending optimal tourist information and services to users based on the database stored on the server.

[0010] "Tourist plan information" includes detailed information such as information about tourist spots and facilities, recommended routes, and experience activities.

[0011] The "visual guide means" is a technical means for providing detailed information such as tourist spots by overlaying the image viewed by the user.

[0012] "Audio guide means" refers to a technical means for providing the user with audio guidance of displayed tourist information, etc.

[0013] A "camera" is a device for capturing the scenery and text that comes into the user's field of vision.

[0014] "Translation means" refers to technical means for translating text information captured by the camera into the user's native language.

[0015] "Reservation means" refers to the technical means for sending a reservation request to the server and checking the reservation status. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[0038] System Overview

[0039] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system is composed of location information acquisition means, communication means, display means, recommendation means, visual guide means, audio guide means, camera, and translation means.

[0040] Program processing overview

[0041] Startup and login

[0042] 1. Startup

[0043] Device: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[0044] 2. Log in

[0045] User: The user enters login information and performs an operation.

[0046] On the device: The entered login information is converted into a security token and sent to the server.

[0047] Server: Checks the login information against a database and returns an authentication token if authentication is successful.

[0048] Obtaining your current location and automatically generating a sightseeing plan

[0049] 3. Get your current location

[0050] Device: The built-in GPS module is activated and acquires the current location. This information is then sent to the server after user confirmation.

[0051] 4. Automatic generation of sightseeing plans

[0052] User: Enters preferences and wishes into the interface.

[0053] Device: The entered information and current location information are sent to the server.

[0054] Server: The generation AI generates the optimal sightseeing plan from the database and sends it back to the device.

[0055] Terminal: Provides the received travel plan to the user visually and audibly.

[0056] Directions to tourist spots

[0057] 5. Destination selection and route guidance

[0058] User: Choose the place you want to go from the presented sightseeing itinerary.

[0059] Terminal: Sends destination information to the server and calculates the optimal route.

[0060] Server: Calculates route information and sends it to the device.

[0061] Terminal: Provides visual and audio route guidance to the user.

[0062] Real-time tourist guide

[0063] 6. Providing detailed information about tourist spots

[0064] Terminal: The camera detects tourist spots and displays them visually. Detailed information is obtained from the server and provided to the user.

[0065] Terminal: Also provides audio guidance to users.

[0066] Example: When a user stands in front of a museum, they are given visual and audio guidance with detailed information about the building's history.

[0067] Providing translation functions

[0068] 7. Translation of text information

[0069] Device: The camera detects text information within the user's field of view.

[0070] Terminal: OCR the text information and send it to the translation API.

[0071] Server: Obtains translation results and sends them to the device.

[0072] Terminal: Provides the user with visual and audio translation results.

[0073] Example: Translating and displaying a restaurant menu.

[0074] Recommendations for nearby information

[0075] 8. Obtaining recommendation information

[0076] User: Submit a request for recommendations.

[0077] Server: Retrieves the best recommendations based on the user's location and preferences.

[0078] Terminal: Provides recommendation information visually and audibly.

[0079] Example: Recommending a popular nearby cafe.

[0080] Experience booking support

[0081] 9. Making a Reservation

[0082] User: Select the experience or restaurant they want.

[0083] Terminal: Sends a reservation request to the server and checks the reservation status.

[0084] Server: Connects to the reservation system, checks, and sends the results to the terminal.

[0085] Terminal: Provides visual and audio confirmation of reservation completion.

[0086] Example: Notifying a customer that a tea time reservation at a cafe has been completed.

[0087] Example

[0088] In a real-world tourism scenario, when a user wears smart glasses and stands at a tourist spot, their current location information is immediately acquired, and the AI ​​automatically generates an optimal sightseeing plan. The user can then follow the guide as they tour the tourist spots, enjoying real-time guidance, translation, and recommendations. All information is provided through the smart glasses' display and audio, allowing users to enjoy sightseeing hands-free.

[0089] This allows users to have an efficient and fulfilling sightseeing experience, freeing them from the language barrier and the hassle of gathering information.

[0090] The processing flow will be explained below.

[0091] Step 1:

[0092] boot

[0093] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[0094] Step 2:

[0095] Log in

[0096] User: The user enters their login information (ID and password) and selects the login button.

[0097] Terminal: Converts the entered login information into a security token and sends it to the server.

[0098] Server: Compares the received login information with the database, and if authentication is successful, issues an authentication token and returns it to the device.

[0099] Step 3:

[0100] Get current location

[0101] Device: Activates the built-in GPS module and obtains the user's current location information (latitude and longitude).

[0102] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[0103] Step 4:

[0104] Sending current location information

[0105] Device: Sends the acquired location information to the server.

[0106] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0107] Step 5:

[0108] Enter your sightseeing plan

[0109] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[0110] Step 6:

[0111] Submitting a plan generation request

[0112] Terminal: Sends a plan generation request including the input sightseeing plan information and current location information to the server.

[0113] Step 7:

[0114] Automatic generation of sightseeing plans

[0115] Server: The AI ​​automatically generates the optimal sightseeing plan based on the user's current location and preferences. The generated sightseeing plan is sent to the device.

[0116] Terminal: Presents the received sightseeing plan to the user with visual and audio guides.

[0117] Step 8:

[0118] Destination selection

[0119] User: Select the destinations they want to visit from the presented itinerary.

[0120] Step 9:

[0121] Sending a route calculation request

[0122] Terminal: Sends a route calculation request including the selected destination information to the server.

[0123] Step 10:

[0124] Route calculation and guidance

[0125] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[0126] Terminal: Provides visual and audio route guidance to the user based on the calculated route information.

[0127] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[0128] Step 11:

[0129] Real-time guide to tourist attractions

[0130] Device: Detects tourist spots within the user's field of view through the smart glasses camera.

[0131] Server: Based on the detected spot information, retrieves detailed information from the database and sends it back to the device.

[0132] Terminal: The acquired detailed information is presented to the user and audio guidance is also provided.

[0133] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[0134] Step 12:

[0135] Translation feature

[0136] Device: Detects text information within the user's field of view through the camera.

[0137] Terminal: OCR the text information and send it to the translation API.

[0138] Server: Translates the source text into the user's native language via the translation API and sends the results to the device.

[0139] Terminal: Presents the translation results to the user visually and audibly.

[0140] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[0141] Step 13:

[0142] Recommendations for nearby information

[0143] User: Sends a request for recommendations.

[0144] Server: Retrieves the most suitable nearby information (tourist attractions, restaurants, etc.) from a database based on the user's current location and preferences.

[0145] Terminal: Provides the acquired recommendation information to the user visually and audibly.

[0146] Example: Information about a popular nearby cafe is displayed, suggesting, "Would you like to take a break here?"

[0147] Step 14:

[0148] Experience booking support

[0149] User: Selects the experience or restaurant they want from the recommendations and submits a reservation request.

[0150] Terminal: Sends a reservation request to the server and asks for availability.

[0151] Server: Connects to the reservation system to check availability and sends the results to the terminal.

[0152] Terminal: Provides visual and audio confirmation of reservation completion.

[0153] Example: A user can enjoy tea time at a cafe at a reserved time.

[0154] Step 15:

[0155] Exit and log out

[0156] User: Select Logout from the menu.

[0157] Terminal: Sends a logout request to the server to end the session.

[0158] Server: Discard the user's session information and notify the user that the logout is complete.

[0159] Terminal: Shut down the system and prepare for next use.

[0160] Example 1

[0161] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] Conventional tourist guide systems require users to search for information and plan their own sightseeing itineraries, and they have problems with information gathering and understanding if they do not understand a foreign language. Furthermore, they are unable to provide real-time guidance, translation, and recommendation functions based on the user's current location, limiting the tourist experience.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0164] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for providing visual and audio guides, translation means for detecting and translating text information using a camera, means for transmitting a reservation request to the server and acquiring reservation information, and means for recommending surrounding information. This allows the user to enjoy optimal sightseeing plans in real time, freeing them from the language barrier and the hassle of information gathering, and enabling a more efficient and fulfilling sightseeing experience.

[0165] The "location information acquisition means" is a device or technology for identifying the user's current location and acquiring that location information.

[0166] The "communication means" is an interface for transmitting acquired location information and user requests to a server and receiving data from the server.

[0167] The "display means" is a device for visually displaying to the user the sightseeing plan information and other data provided by the server.

[0168] A "recommendation means" is an algorithm or technology that extracts information suitable for a user from a database stored on a server and provides it to the user.

[0169] A "visual guide means" is a device or technology that visually guides a user to location information and detailed information about tourist spots.

[0170] "Audio guidance means" refers to a device or technology that provides audio guidance of visually displayed information.

[0171] The "translation means" is a technology for translating text information detected using a camera into another language and providing it to the user.

[0172] The "reservation means" is a means for a user to send a reservation request for a desired experience or restaurant to the server and check the reservation status.

[0173] "Recommendation methods" are algorithms and technologies that select and present optimal information to users based on their current location and preferences.

[0174] The present invention relates to a system that enables users to use smart glasses while sightseeing to receive optimal sightseeing plans, route guidance, and guide services in real time. This system is composed of a location information acquisition means, a communication means, a display means, a recommendation means, a visual guide means, an audio guide means, a camera, and a translation means.

[0175] Startup and login

[0176] First, the user turns on the smart glasses. As a terminal, the smart glasses launch the OS and go through initialization processes, displaying the login screen. The user enters login information using the smart glasses' touchpad or voice input. The terminal converts the entered information into a security token and sends it to the server using a communication method. The server references a database to confirm the user's authentication information, and if authentication is successful, returns an authentication token to the terminal.

[0177] Obtaining your current location and automatically generating a sightseeing plan

[0178] The device's built-in GPS module then obtains its current location and, with the user's permission, transmits this information to a server. The user inputs their preferences and the places they want to visit through the smart glasses' interface. The device then transmits this information to the server, which uses a generative AI model to generate an optimal itinerary. The generated itinerary is then sent back to the device and presented to the user visually and audibly.

[0179] Specific examples

[0180] For example, if a user wishes to visit a historical building, the server will search the database for the most suitable sightseeing plan based on the input information and current location information. The sightseeing plan will include nearby historical buildings, recommended restaurants, etc. This information will be provided to the user through the smart glasses' display and audio output.

[0181] Directions to tourist spots

[0182] When a user selects a destination from the provided sightseeing plan, the device sends this information to the server and requests it to calculate the optimal route. The server calculates the optimal route using GPS information and the selected destination information and sends it to the device. The device then provides the user with visual and audio route guidance.

[0183] Real-time tourist guide

[0184] When users arrive at a tourist spot, the smart glasses' camera detects the tourist spot and sends that information to the server. The server then retrieves detailed information about the tourist spot from a database and sends it to the device. This allows users to obtain detailed information about the tourist spot visually and audibly. For example, when users stand in front of a museum, they will be provided with information about the building's history and exhibits.

[0185] Providing translation functions

[0186] When the smart glasses' camera detects text within the user's field of view, it processes it with OCR and sends it to a translation API. The server then retrieves the translation results and sends them to the device. The user can then confirm the translation visually and audibly. For example, a restaurant menu may be translated and displayed.

[0187] Recommendations for nearby information

[0188] When a user sends a recommendation request, the server retrieves the best recommendations based on the user's current location and preferences. The device then presents this information visually and audibly, recommending nearby popular cafes, tourist attractions, and more.

[0189] Experience booking support

[0190] Once the user selects the desired experience and restaurant, the device sends a reservation request to the server and checks the reservation status. The server connects to the reservation system to confirm whether the reservation can be made and sends the result to the device. The device then notifies the user visually and audibly that the reservation has been completed. For example, it notifies the user that a tea time reservation at a cafe has been completed.

[0191] This system allows users to enjoy optimal sightseeing plans in real time, freeing them from language barriers and the hassle of gathering information, enabling a more efficient and fulfilling sightseeing experience.

[0192] Prompt Sentence Examples

[0193] "Please suggest the best sightseeing plan based on the user's current location."

[0194] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0195] Step 1:

[0196] boot

[0197] Device: When you start up the smart glasses, the OS starts up and the system is initialized. The login screen is displayed.

[0198] Input: User powers on smart glasses.

[0199] Output: The login screen is displayed.

[0200] What happens: The OS checks various modules to make sure they are OK, then renders the login screen in the UI.

[0201] Step 2:

[0202] Log in

[0203] User: The user enters their login information using the smart glasses interface.

[0204] Terminal: Converts the entered login information into a security token and sends it to the server.

[0205] Server: The server checks the user's authentication information by referencing the database, and if authentication is successful, returns an authentication token to the terminal.

[0206] Input: Login information (user ID and password).

[0207] Output: An authentication token.

[0208] Specific operation: Login information is encrypted into a security token and sent to the server, where it is checked against a database. If authentication is successful, an authentication token is generated and sent back to the device as an HTTP response.

[0209] Step 3:

[0210] Get current location

[0211] On the device: The built-in GPS module is activated and acquires the current location, which is then sent to the server after user confirmation.

[0212] Input: Latitude and longitude information from the GPS module.

[0213] Output: Current location information.

[0214] Specific operation: Captures GPS signals and measures latitude and longitude information. Then, parses this information into JSON format and sends it to the server via a communication method.

[0215] Step 4:

[0216] Automatic generation of sightseeing plans

[0217] User: Enters preferences and wishes using the smart glasses interface.

[0218] Device: Sends input information and current location information to the server.

[0219] Server: The generative AI model generates the optimal sightseeing plan from the database and sends it back to the device.

[0220] Input: Your preferences, wishes, and location.

[0221] Output: Tourist plan.

[0222] How it works: After users input their preferences and wishes, they send that information along with their current location to the server, where the generative AI model generates the best options and returns them as a sightseeing plan.

[0223] Step 5:

[0224] Destination selection and route guidance

[0225] User: Choose the place you want to go from the presented sightseeing itinerary.

[0226] Terminal: Sends the selected destination information to the server and requests it to calculate the optimal route.

[0227] Server: Calculates the optimal route and sends it to the device, which then provides visual and audio guidance to the user.

[0228] Input: Destination information.

[0229] Output: Optimal route.

[0230] Specific operation: Using GPS information and the selected destination, the system calls a map API (e.g., Google Maps API) to calculate the optimal route, sends the calculation results back to the device, and provides audio and visual guidance.

[0231] Step 6:

[0232] Providing detailed information about tourist spots

[0233] Device: The camera in the smart glasses detects tourist spots and sends the information to the server.

[0234] Server: Retrieves detailed information about tourist attractions from the database and sends it to the terminal, which then provides the user with detailed information visually and audibly.

[0235] Input: Video of tourist spot.

[0236] Output: Detailed information about the tourist spot.

[0237] Specific operation: Camera images are analyzed using image recognition technology to identify tourist spots. Information about the identified spots is sent to the server, and the corresponding detailed information is retrieved and presented to the user.

[0238] Step 7:

[0239] Text translation

[0240] Device: The camera detects text information within the user's field of view, performs OCR processing, and sends it to the translation API.

[0241] Server: Receives the translation results and sends them to the device, which then provides the translation results to the user visually and audibly.

[0242] Input: Text information.

[0243] Output: The translation result.

[0244] Specific operation: Extracts text from images using OCR technology, sends it to a translation API (e.g., Google Cloud Translation API), receives the translation result, and notifies you by display and voice.

[0245] Step 8:

[0246] Acquiring recommendation information

[0247] User: Send a recommendation request.

[0248] Server: Retrieves optimal recommendations based on the user's current location and preferences. The device provides the recommendations visually and audibly.

[0249] Input: recommendation request, current location, preferences.

[0250] Output: Recommendation information.

[0251] Specific operation: A database search is performed based on the user's request and current location information, and information on nearby tourist attractions and restaurants is processed using a recommendation algorithm to generate optimal recommendations.

[0252] Step 9:

[0253] Making a reservation

[0254] User: Select the experience or restaurant they want.

[0255] Terminal: Sends a reservation request to the server and checks the reservation status.

[0256] Server: Connects to the reservation system, confirms the reservation, and sends the result to the terminal. The terminal then provides the user with visual and audio information about the completion of the reservation.

[0257] Input: Booking request.

[0258] Output: Booking completion information.

[0259] Specific behavior: Sends a reservation request to the reservation system, checks the reservation status, and notifies the user visually and audibly if the reservation is successful.

[0260] (Application example 1)

[0261] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] In modern tourism and shopping experiences, users are required to efficiently obtain and use a large amount of information. However, it is difficult to obtain the necessary information in real time in foreign countries or large stores, and it is often time-consuming to perform complex operations and gather information. Furthermore, it is currently difficult for users to instantly grasp the product information and inventory status they are looking for and receive appropriate guidance. This results in users wasting their valuable time and resulting in an unsatisfactory experience.

[0263] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0264] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, a camera module for the user to acquire product information in a physical store, means for transmitting the acquired product information to the server and receiving detailed information, and means for providing the detailed information visually and audibly. This allows the user to acquire necessary information in real time at tourist spots and physical stores and use it efficiently.

[0265] "User" refers to an individual or organization that uses the system to sightsee or shop.

[0266] "Current Location" refers to the geographic location of the user obtained using GPS or other means.

[0267] "Location information acquisition means" refers to a function or device for identifying the user's current location and acquiring that information.

[0268] "Communication means" refers to the functions and devices for sending acquired information to a server and receiving information from the server.

[0269] "Display means" refers to a device or function for visually conveying information received from the server to the user.

[0270] "Recommendation means" refers to a function for recommending highly relevant information and plans to a user from a database stored on a server.

[0271] "Camera Module" refers to a device or feature that includes a camera that allows a user to scan products or attractions.

[0272] "Detailed information" refers to information provided by the server and displayed to the user, including product descriptions and tourist spot information.

[0273] "Visual guide means" refers to devices or functions for visually displaying the location of products or detailed information about tourist attractions.

[0274] "Audio guide means" refers to a function or device for providing the user with audio guidance of displayed information.

[0275] "Stock status" refers to the availability and quantity of the product selected by the user in the store.

[0276] "Reservation" refers to the procedure or function that allows a user to reserve a selected product in advance for later purchase.

[0277] System Overview

[0278] This invention relates to a system for improving users' sightseeing and shopping experiences. The system uses smart glasses to help users obtain necessary information in real time at tourist spots and brick-and-mortar stores and use it efficiently.

[0279] Hardware and software used

[0280] The hardware and software used in this system are as follows:

[0281] GPSModule: A location information acquisition means for acquiring the user's current location.

[0282] CameraModule: A camera module that allows users to scan products and tourist attractions.

[0283] DisplayModule: A display means for visually conveying information received from the server to the user.

[0284] ServerCommunication: A communication method for sending acquired information to a server and receiving information from the server.

[0285] Program processing overview

[0286] The system's main processing steps are as follows: First, the user puts on the smart glasses and starts the system. The system identifies the user's current location using GPS Module and sends this information to the server via ServerCommunication. The server generates optimal sightseeing plans and shopping guides based on the user's current location, past purchase history, and input preference information, and presents them visually to the user via Display Module.

[0287] Furthermore, when a user explores a product in a physical store, the CameraModule is used to scan the product and retrieve details, reviews, stock status, etc. from the server. This information is also provided to the user through the DisplayModule and audio guide.

[0288] Specific examples

[0289] For example, when a user arrives at a tourist spot, the system immediately acquires their current location information and sends it to the server. The server then uses a generative AI model to automatically generate an optimal sightseeing plan for the user, providing it to the user. This plan includes detailed information about the tourist spots the user should visit and directions. When the user stands in front of a museum, they are given visual and audio guidance about the museum's history and exhibits.

[0290] When users scan an item in-store, they can view detailed product information, reviews, and availability in real time. If the item they select is low in stock, the system will suggest a reservation and guide the user through the process.

[0291] Prompt Sentence Examples

[0292] Examples of prompts to input to a generative AI model include:

[0293] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[0294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0295] Step 1:

[0296] Startup and login

[0297] Terminal: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[0298] Input: The user enters their login information (username and password).

[0299] Output: The entered login information is converted into a security token and sent to the server.

[0300] Server: Checks the login information against a database and, if authentication is successful, returns an authentication token.

[0301] Terminal: Receives the authentication token and notifies the user that login was successful.

[0302] Step 2:

[0303] Obtaining your current location and automatically generating a sightseeing plan

[0304] Device: The built-in GPS module starts up and acquires the current location.

[0305] Input: Current location information obtained.

[0306] Output: Sends current location information to the server.

[0307] Server: Receives the current location information sent by the user and generates the optimal sightseeing plan based on the database.

[0308] Terminal: Provides the generated sightseeing plan to the user visually and audibly.

[0309] Step 3:

[0310] Destination selection and route guidance

[0311] User: Choose the place you want to go from the presented sightseeing plan.

[0312] Input: User selected destination information.

[0313] Output: Sends destination information to the server.

[0314] Server: Calculates the optimal route to the selected destination and sends the route information to the device.

[0315] Terminal: Provides visual and audio route guidance to the user.

[0316] Step 4:

[0317] Scanning products and obtaining information

[0318] User: The customer holds the product in their hand and scans it with the camera module on the smart glasses.

[0319] Input: Captured product image.

[0320] Output: Send product images to the server and get detailed information.

[0321] Server: Analyzes the sent product image, retrieves detailed information from the database, and returns it to the terminal.

[0322] Terminal: Provides detailed product information visually and audibly.

[0323] Step 5:

[0324] Check stock availability and reserve

[0325] User: To check the stock status of the product they wish to purchase, they send a stock check request from the details screen.

[0326] Input: Inventory check request information.

[0327] Output: Sends stock status information to the server and receives confirmation results.

[0328] Server: Connects to the inventory system to check the inventory status and sends the results to the terminal.

[0329] Terminal: Provides visual and audio information on stock availability and guides you through the reservation process if one is available.

[0330] Step 6:

[0331] Sales information and coupon recommendations

[0332] Terminal: As users move around the store, they obtain sale and coupon information based on their current location and purchase history.

[0333] Input: Current location information and purchase history information.

[0334] Output: Receive sale information and coupon information from the server.

[0335] Server: Based on the user's current location information and purchase history, extracts sale and coupon information from the database and sends it to the terminal.

[0336] Terminal: Provides acquired sales and coupon information to the user visually and audibly.

[0337] Prompt Sentence Examples

[0338] Examples of prompts to input to a generative AI model include:

[0339] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[0340] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0341] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[0342] System Overview

[0343] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system consists of the following:

[0344] Location information acquisition means

[0345] communication means

[0346] Display means

[0347] Recommendation method

[0348] Visual Guide Means

[0349] Audio guide means

[0350] Translation tools

[0351] Reservation method

[0352] Emotion Recognition Engine

[0353] Program processing overview

[0354] Startup and login

[0355] 1. Startup

[0356] Device: When the smart glasses are started up, the internal system is initialized and the login screen is displayed.

[0357] 2. Log in

[0358] User: The user enters their login information (ID and password) and selects the login button.

[0359] Terminal: Converts the entered login information into a security token and sends it to the server.

[0360] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[0361] Obtaining your current location and automatically generating a sightseeing plan

[0362] 3. Get your current location

[0363] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[0364] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[0365] 4. Sending current location information

[0366] Device: Sends the acquired location information to the server.

[0367] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0368] Acquisition and application of emotion data

[0369] 5. Acquiring Emotion Data

[0370] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[0371] 6. Transmission and analysis of emotional data

[0372] Device: Sends the acquired emotion data to the server.

[0373] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[0374] Coordination and provision of tourism plans

[0375] 7. Generate a sightseeing plan

[0376] User: The user enters their travel preferences and wishes into the interface.

[0377] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[0378] Terminal: Presents the received travel plan to the user visually and audibly.

[0379] Directions and guides to tourist spots

[0380] 8. Destination selection and route guidance

[0381] User: Choose the places they want to visit from the presented itinerary.

[0382] Terminal: Sends destination information to the server and calculates the optimal route.

[0383] Server: Calculates route information and sends it to the device.

[0384] Terminal: Provides visual and audio route guidance.

[0385] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[0386] Real-time guide and translation

[0387] 9. Providing detailed information about tourist attractions

[0388] Terminal: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and presents it.

[0389] Terminal: Also provides audio guidance to the user.

[0390] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[0391] 10. Text Translation

[0392] Device: Detects text in the user's field of view via the camera.

[0393] Terminal: OCR the text information and send it to the translation API.

[0394] Server: Sends the translation results to the device.

[0395] Terminal: Provides visual and audio translation results.

[0396] Example: Translating a restaurant menu to show "Chicken Curry - 500 yen."

[0397] Local information recommendations and experience reservations

[0398] 11. Obtaining local information and making recommendations

[0399] User: Send a recommendation request.

[0400] Server: Makes recommendations by providing peripheral information while taking into account emotional data.

[0401] Terminal: Provides information visually and audibly.

[0402] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[0403] 12. Booking an Experience

[0404] User: Selects the experience or restaurant they want and submits a reservation request.

[0405] Terminal: Sends a reservation request to the server.

[0406] Server: Connects to the reservation system and checks availability.

[0407] Terminal: Provides visual and audio confirmation of reservation completion.

[0408] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[0409] Example

[0410] For example, when a user puts on smart glasses and starts sightseeing, location information is first acquired. Then, a sightseeing plan tailored to the user's needs is generated based on emotional data. When the user asks for directions or detailed information about tourist spots, guidance tailored to the user's current emotional state is provided. This allows the user to enjoy sightseeing without stress.

[0411] If a translation is needed, the smart glasses' camera detects the text and provides an instant translation. Furthermore, if the emotion recognition engine determines that the user is tired, it will provide a recommendation to take a break. In this way, the system comprehensively supports the user's sightseeing experience.

[0412] The processing flow will be explained below.

[0413] Step 1:

[0414] boot

[0415] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[0416] Step 2:

[0417] Log in

[0418] User: The user enters their login information (ID and password) and selects the login button.

[0419] Terminal: Converts the entered login information into a security token and sends it to the server.

[0420] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[0421] Step 3:

[0422] Get current location

[0423] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[0424] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[0425] Step 4:

[0426] Sending current location information

[0427] Device: Sends the acquired location information to the server.

[0428] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0429] Step 5:

[0430] Acquiring emotion data

[0431] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[0432] Step 6:

[0433] Sending and analyzing emotional data

[0434] Device: Sends the acquired emotion data to the server.

[0435] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[0436] Step 7:

[0437] Generate a sightseeing plan

[0438] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[0439] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[0440] Terminal: Presents the received travel plan to the user visually and audibly.

[0441] Step 8:

[0442] Selecting a destination and calculating a route

[0443] User: Select the places they want to visit from the presented itinerary.

[0444] Terminal: Sends destination information to the server and calculates the optimal route.

[0445] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[0446] Terminal: Provides visual and audio route guidance.

[0447] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[0448] Step 9:

[0449] Real-time Guide

[0450] Device: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and displays it.

[0451] Server: Provides detailed information about tourist spots based on the detected spot information.

[0452] Terminal: Provides both visual and audio guidance to the user.

[0453] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[0454] Step 10:

[0455] Translation feature

[0456] Device: Detects text information within the user's field of view through the camera.

[0457] Terminal: OCR the text information and send it to the translation API.

[0458] Server: Sends the translation results to the device.

[0459] Terminal: Provides visual and audio translation results.

[0460] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[0461] Step 11:

[0462] Recommendations for nearby information

[0463] User: Send a recommendation request.

[0464] Server: Obtains optimal surrounding information based on the user's current location and emotional data, and generates recommendation information.

[0465] Terminal: Provides visual and audio recommendations.

[0466] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[0467] Step 12:

[0468] Book an experience

[0469] User: Select the experience or restaurant they want.

[0470] Terminal: Sends a reservation request to the server and checks availability.

[0471] Server: Connects to the reservation system, checks availability, and sends the results to the device.

[0472] Terminal: Provides visual and audio confirmation of reservation completion.

[0473] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[0474] Step 13:

[0475] Exit and log out

[0476] User: Select Logout from the menu.

[0477] Terminal: Sends a logout request to the server to end the session.

[0478] Server: Discard the user's session information and notify the user that the logout is complete.

[0479] Terminal: Shut down the system and prepare for next use.

[0480] Example 2

[0481] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0482] This invention relates to a system for improving the tourism experience. However, conventional tourism systems often lack features such as optimizing sightseeing plans based on the user's real-time emotional state, translation functions, and reservation functions. As a result, users are often limited in the information and support they receive while sightseeing, which can lead to a decrease in satisfaction. Furthermore, the lack of route guidance to tourist spots and detailed guide information is also problematic. Therefore, there is a need for a system that can comprehensively support users' sightseeing experiences and provide high-quality services.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0484] In this invention, the server

[0485] A means for acquiring location information;

[0486] means for transmitting the acquired location information to a server;

[0487] a means for displaying the sightseeing plan provided by the server on the terminal;

[0488] A means for obtaining recommendation information from a database;

[0489] means for acquiring user emotion data using an emotion recognition engine;

[0490] means for transmitting the acquired emotion data to a server and analyzing it;

[0491] a means of providing real-time visual and audio route guidance;

[0492] A means of recommending surrounding information;

[0493] a means for visually detecting and translating text;

[0494] a means for submitting a reservation request and providing reservation completion information;

[0495] This allows users to receive optimal sightseeing plans, route guidance, and detailed guide information in real time, and also provides recommendation information and translation services based on their emotional state, greatly improving the user's sightseeing experience.

[0496] "Means for acquiring location information" refers to a device or function that acquires the user's current location using a GPS module or other location measurement technology.

[0497] The "means for transmitting the acquired location information to a server" refers to a device or function for transmitting location information data to a server via Wi-Fi, a cellular network, or other communication technology.

[0498] The "means for displaying the sightseeing plan provided by the server on the terminal" refers to a device or function that visually displays the sightseeing plan information received from the server on the user's smart glasses or mobile device.

[0499] The "means for acquiring recommended information from a database" refers to a device or function that searches for and acquires recommended information related to travel and sightseeing from a database stored on a server.

[0500] "Means for acquiring user emotional data using an emotion recognition engine" refers to a device or function that uses sensors such as a camera or microphone to analyze the user's facial expressions and vocal tone and identify the user's emotional state.

[0501] The "means for transmitting acquired emotion data to a server and analyzing it" refers to a device or function that transmits the user's emotion data to a server and analyzes the data in detail on the server side.

[0502] "Means for providing real-time visual and audio route guidance" refers to a device or function that provides the user with visual and audio guidance in real time on the optimal route to a destination specified by the user.

[0503] A "means for recommending surrounding information" is a device or function that recommends nearby tourist spots and facilities based on the user's current location and emotional state.

[0504] A "means for visually detecting and translating text" is a device or function that uses a camera to detect text in its field of view and then OCRs and translates that text.

[0505] The "means for sending a reservation request and providing reservation completion information" refers to a device or function that sends a reservation request for the user's desired experience or facility to the server and notifies the user that the reservation has been completed.

[0506] The present invention relates to a system that uses smart glasses to significantly improve a user's sightseeing experience. This system is designed to enable users to receive optimal sightseeing plans, route guidance, and guide services in real time while sightseeing.

[0507] System configuration

[0508] This system consists of the following main means:

[0509] 1. Location information acquisition means

[0510] 2. Means of communication

[0511] 3. Display means

[0512] 4. Recommendation method

[0513] 5. Emotion recognition means

[0514] 6. Route guidance methods

[0515] 7. Translation Methods

[0516] 8. Reservation Methods

[0517] Hardware and software used

[0518] Smart glasses (terminal): A device worn by the user. This device is equipped with a GPS module, camera, microphone, display, and speaker.

[0519] Server: The back-end processing device, including the database and generative AI model.

[0520] Generative AI model: Runs on the server side and generates sightseeing plans based on the user's location, emotional data, and preferences.

[0521] Processing flow and specific operations

[0522] Below is an overview of the operation of this system and the specific operations for each process.

[0523] Startup and login

[0524] 1. Startup

[0525] Device: When the smart glasses are turned on, the internal system is first initialized and the login screen is displayed.

[0526] 2. Log in

[0527] User: Enter your login information (ID and password) and select the Login button.

[0528] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[0529] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[0530] Obtaining and sending location information

[0531] 3. Get your current location

[0532] Device: The built-in GPS module is activated and obtains the user's current location information in real time.

[0533] User: A confirmation dialog about obtaining location information will be displayed and the user will select "Allow."

[0534] 4. Sending current location information

[0535] Device: Sends the acquired location information to the server.

[0536] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0537] Acquisition and application of emotion data

[0538] 5. Acquiring Emotion Data

[0539] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[0540] 6. Transmission and analysis of emotional data

[0541] Device: Sends the acquired emotion data to the server.

[0542] Server: Analyzes emotion data, identifies the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan.

[0543] Creating and providing tour plans

[0544] 7. Generate a sightseeing plan

[0545] User: Enters travel preferences and wishes into the interface.

[0546] Server: Uses a generative AI model to generate optimal sightseeing plans based on location, preference, and emotion data.

[0547] Server: Sends the generated travel plan to the device.

[0548] Terminal: Presents the received travel plan to the user visually and audibly.

[0549] Directions and guides to tourist spots

[0550] 8. Destination selection and route guidance

[0551] User: Choose the places they want to visit from the presented itinerary.

[0552] Device: Sends the selected destination information to the server and calculates the optimal route.

[0553] Server: Calculates route information and sends it to the device.

[0554] Terminal: Provides visual and audio route guidance.

[0555] Specific examples of facilities

[0556] For example, when a user puts on smart glasses and begins sightseeing, location information is first acquired. Then, the user's desired sightseeing plan is created using a generative AI model and sent to the device. As the user heads to their selected destination, route guidance is provided in real time. Also, when the user arrives at a tourist spot, the smart glasses' camera detects the spot and displays detailed information. If translation is required, the camera detects the text and the translation result is instantly provided visually and audibly.

[0557] Prompt Sentence Examples

[0558] Examples of prompts for operating this system are shown below.

[0559] "Get your current location"

[0560] "Generate the best sightseeing plan based on user preferences and sentiment data."

[0561] "Please provide more information about the attraction the user is looking at."

[0562] "Book your selected experience"

[0563] "Use your camera to translate any text you see"

[0564] Using these prompts, the generative AI model provides a sightseeing experience that is tailored to the user, ensuring the entire system operates smoothly.

[0565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0566] Step 1: Boot up and log in

[0567] 1.1 Startup

[0568] Device: When the smart glasses are turned on, they first initialize their internal systems, which includes checking that the CPU, memory, and sensors are working.

[0569] Device: Once initialization is complete, the login screen will appear.

[0570] Input: The user presses the power button.

[0571] Output: The login screen is displayed.

[0572] 1.2 Login

[0573] User: Enter your ID and password on the login screen and press the login button.

[0574] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[0575] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[0576] Input: The user enters their ID and password and presses the login button.

[0577] Output: If authentication is successful, the main menu is displayed.

[0578] Step 2: Obtaining and sending location information

[0579] 2.1 Obtaining current location

[0580] Device: Activates the built-in GPS module and obtains the user's current location (latitude and longitude) in real time.

[0581] User: A confirmation dialog about obtaining location information will appear, select "Allow".

[0582] Input: The user selects "Allow" in the location information confirmation dialog.

[0583] Output: Current location information (latitude and longitude) is obtained.

[0584] 2.2 Sending current location information

[0585] Terminal: The acquired current location information is sent to the server in packet format.

[0586] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0587] Input: Current location information (latitude and longitude) obtained by the device.

[0588] Output: The location information is sent to the server, and information about nearby tourist spots is searched.

[0589] Step 3: Acquiring and applying emotion data

[0590] 3.1 Acquiring Emotion Data

[0591] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[0592] Input: User's facial expressions and voice captured through camera and microphone.

[0593] Output: The analyzed emotion data is obtained.

[0594] 3.2 Emotional Data Transmission and Analysis

[0595] Device: Sends the acquired emotion data to the server.

[0596] Server: Analyzes the emotion data and identifies the user's state (e.g., joy, excitement, fatigue, etc.).

[0597] Input: The acquired emotion data.

[0598] Output: Emotion data is sent and analyzed on the server.

[0599] Step 4: Generate and provide a tour plan

[0600] 4.1 User Input

[0601] User: Enters travel preferences and wishes into the interface.

[0602] Input: The user enters their travel preferences and wishes into the interface.

[0603] Output: The entered preference information is saved on the device.

[0604] 4.2 Generating a Tourist Plan

[0605] Server: Uses a generative AI model to generate optimal sightseeing plans based on location information, user preferences, and emotional data.

[0606] Input: Location, user preferences, and emotional data.

[0607] Output: The generated itinerary.

[0608] 4.3 Presentation of sightseeing plans

[0609] Server: Sends the generated travel plan to the device.

[0610] Terminal: Presents the received travel plan to the user visually and audibly.

[0611] Input: The generated itinerary.

[0612] Output: The sightseeing plan is presented visually and audibly.

[0613] Step 5: Select a destination and get directions

[0614] 5.1 Selecting a destination

[0615] User: Select the places they want to visit from the presented itinerary.

[0616] Input: The destination selected by the user.

[0617] Output: Selected destination information.

[0618] 5.2 Route calculation

[0619] Terminal: Sends destination information to the server and calculates the optimal route.

[0620] Server: Calculates the optimal route taking into account congestion information and real-time traffic information, and returns the results to the device.

[0621] Input: Selected destination information.

[0622] Output: Calculated route information.

[0623] 5.3 Route guidance

[0624] Device: Provides visual and audio guidance based on the received route.

[0625] Input: Calculated route information.

[0626] Output: Route guidance is provided visually and audibly.

[0627] Step 6: Provide detailed information about the tourist spot

[0628] 6.1 Tourist Spot Detection

[0629] Device: The smart glasses' camera detects and identifies tourist spots.

[0630] Input: Tourist spots detected through the camera.

[0631] Output: Information about detected tourist spots.

[0632] 6.2 Obtain detailed information

[0633] Device: Queries the server for information about the detected tourist spots.

[0634] Server: Retrieves relevant details from the database and sends them to the device.

[0635] Input: Detected tourist spot information.

[0636] Output: The detailed information retrieved.

[0637] 6.3 Detailed information presentation

[0638] Terminal: Provides users with detailed information about tourist attractions through visual and audio means.

[0639] Input: The retrieved details.

[0640] Output: Detailed information about tourist attractions is provided visually and audibly.

[0641] Step 7: Translate the text

[0642] 7.1 Text Detection

[0643] Device: Detects text in the user's field of view via the camera.

[0644] Input: Text information detected through the camera.

[0645] Output: The detected text data.

[0646] 7.2 Translation Process

[0647] Device: OCR the detected text information and send it to the translation API.

[0648] Server: Uses the translation API to return the translated text.

[0649] Input: OCR processed text data.

[0650] Output: The translation result.

[0651] 7.3 Providing translation results

[0652] Terminal: Provides the user with visual and audio translation results.

[0653] Input: Translation result.

[0654] Output: Visual and audio translation results are provided.

[0655] Step 8: Recommend local information and book experiences

[0656] 8.1 Recommendation Request

[0657] User: Request a recommendation.

[0658] Input: The user request.

[0659] Output: The recommendation request is saved to the device.

[0660] 8.2 Providing recommendation information

[0661] Server: Generates nearby recommendation information based on emotion data and sends it to the device.

[0662] Input: Emotion data, peripheral information.

[0663] Output: Recommendation information.

[0664] 8.3 Experience Booking

[0665] User: Selects the experience or restaurant they want and submits a reservation request.

[0666] Terminal: Sends a reservation request to the server.

[0667] Server: Connects to the reservation system and checks availability.

[0668] Terminal: Provides visual and audio confirmation of reservation completion.

[0669] Input: User's booking request.

[0670] Output: Booking completion information is provided.

[0671] (Application example 2)

[0672] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0673] Conventional tourism support systems have limited ways for users to obtain information about tourist destinations, requiring a great deal of effort to obtain information about the destinations they are visiting. Furthermore, if users do not understand the local language, it is difficult to understand information about tourist destinations and prices. Furthermore, there is a lack of product information, price comparisons, and navigation functions within commercial facilities, limiting the user's shopping experience.

[0674] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for acquiring and displaying information about products, means for converting the user's voice input into text using voice recognition, and means for recommending products based on the acquired text information. This not only enables the user to easily acquire information about destinations, but also enables the user to understand tourist destination and product information across language barriers, and enjoy a pleasant shopping experience in commercial facilities.

[0675] The "location information acquisition means" is a device or software for acquiring the user's current location information.

[0676] "Communication means" refers to a device or protocol for transmitting acquired location information and other data to a server.

[0677] The "display means" refers to a display device or software for displaying information provided by the server on the terminal.

[0678] The "recommendation means" is an algorithm or system for obtaining recommendation information from a database stored on a server.

[0679] "Means for acquiring and displaying product information" refers to devices or software that acquire detailed product information and provide it to users visually or audibly.

[0680] "Means for converting a user's voice input into text using voice recognition" refers to technology for recognizing a user's voice and converting it into text data.

[0681] The "means for recommending products based on acquired text information" is a system for recommending related products to a user based on text information acquired by speech recognition.

[0682] The following describes in detail the mode for carrying out the present invention. This system comprehensively supports users' shopping experiences in physical stores. The system is mainly configured using the following hardware and software:

[0683] Smart glasses: Acquire location information and provide visual and audio information to the user.

[0684] Server: Generates and manages travel plans and product recommendation information.

[0685] Speech recognition engine: Converts the user's speech into text.

[0686] Emotion recognition engine: Analyzes the user's emotions.

[0687] Main features of the system

[0688] 1. Location information acquisition: The device (smart glasses) acquires its current location using its built-in GPS module. The location information is sent to the server via communication means. The server uses this location information to search for information on nearby tourist spots and stores and provides it to the user.

[0689] 2. Speech Recognition: A microphone on the smart glasses captures the user's voice input and converts it into text using a speech recognition engine (e.g., Google Speech Recognition API). This text information is used to understand the user's requests or questions.

[0690] 3. Acquisition and display of product information: The device uses a camera or scanner to acquire product information. For example, it scans a barcode or QR code to acquire detailed product information. The information is then visually displayed on the smart glasses display by the display means and is provided to the user via audio guidance.

[0691] 4. Emotion Recognition: The smart glasses are equipped with a camera and an emotion recognition engine that analyzes the user's facial expressions and tone of voice to obtain emotional data. This data is then sent to a server and used to provide recommendations based on the user's emotional state.

[0692] 5. Providing recommendation information: The server uses a generative AI model to recommend optimal products based on the user's location, past purchase history, preferences, and emotional data. For example, if a user says, "I want a new smartphone," the voice recognition engine analyzes the request and sends it to the server. The server then generates a recommendation for the optimal smartphone and sends it to the device.

[0693] 6. Translation function: The smart glasses' camera is used to detect and OCR foreign language text, and then the translation API is used to translate it. The translation results are provided to the user visually and audibly.

[0694] Specific examples

[0695] The user puts on the smart glasses and says, "find sneakers." The voice recognition engine converts the request into text and sends it to the server. The server searches for the location of the sneakers in the store, generates navigation information such as "They're in section B, shelf 4," and sends it to the device. The device then provides visual and audio guidance to the user, telling them to "turn right and go straight for about 10 meters." If the user wants to know more about the sneakers, the smart glasses' camera scans the item to obtain and display more information.

[0696] Example prompt for a generative AI model:

[0697] "Build an assistant app that recommends products taking into account the user's emotional data. Include the ability to get the product's location when the user enters the product name, generate navigation instructions, and guide the user through the app via voice."

[0698] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0699] Step 1:

[0700] The user puts on the smart glasses and says "find sneakers." This voice input is captured by the smart glasses' microphone. The voice recognition engine converts this voice data into text data. In this case, the input is voice data, and the output is the text data "find sneakers."

[0701] Step 2:

[0702] The terminal sends the converted text data "find sneakers" to the server via a communication means. This process transmits the user's request to the server. The input here is the text data, and the output is the delivery of the request to the server.

[0703] Step 3:

[0704] The server analyzes the received text data and searches for information about "sneakers" in the product database. The search results are specific product information and its location information. The input here is the text data and the product database in the server, and the output is product information including location information, such as "It's in Section B, Shelf 4."

[0705] Step 4:

[0706] The server uses the generative AI model to generate in-store navigation information based on the user's current location and product locations. This information is sent to the terminal. The input here is the user's current location and product locations, and the output is navigation instructions.

[0707] Step 5:

[0708] The device then presents the received navigation information to the user visually and audibly. For example, it may provide guidance such as "Turn right, then go straight for about 10 meters." The input here is navigation information, and the output is visual display and audio guidance.

[0709] Step 6:

[0710] When a user wants to know more information about sneakers, they scan the product's barcode with the camera in their smart glasses. This scanned data is sent to the server by the product information acquisition means. The input here is the barcode data, and the output is a request to the server.

[0711] Step 7:

[0712] The server searches for product information based on the acquired barcode data and sends detailed information to the terminal. The input here is the barcode data and product database, and the output is detailed product information.

[0713] Step 8:

[0714] The terminal receives detailed product information from the server and presents it to the user visually and audibly. For example, it may display and guide the user, saying, "This sneaker is the latest model and costs XYZ yen." The input here is detailed product information, and the output is a visual display and audio guidance.

[0715] Step 9:

[0716] When a user wants to translate a foreign text, they use the smart glasses' camera to capture the text. This captured data is sent to the translation tool, which then performs the translation through OCR processing and a translation API. The input is the captured data, and the output is the translated text.

[0717] Step 10:

[0718] The terminal provides the translation result to the user visually and audibly. For example, it displays "Chicken curry - 500 yen." Here, the input is the translated text data, and the output is the visual display and audio guidance.

[0719] The above processing steps allow users to significantly improve their shopping experience in physical stores.

[0720] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0722] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0723] [Second embodiment]

[0724] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0725] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0727] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0731] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0732] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0733] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0734] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0735] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0736] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[0737] System Overview

[0738] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system is composed of location information acquisition means, communication means, display means, recommendation means, visual guide means, audio guide means, camera, and translation means.

[0739] Program processing overview

[0740] Startup and login

[0741] 1. Startup

[0742] Device: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[0743] 2. Log in

[0744] User: The user enters login information and performs an operation.

[0745] On the device: The entered login information is converted into a security token and sent to the server.

[0746] Server: Checks the login information against a database and returns an authentication token if authentication is successful.

[0747] Obtaining your current location and automatically generating a sightseeing plan

[0748] 3. Get your current location

[0749] Device: The built-in GPS module is activated and acquires the current location. This information is then sent to the server after user confirmation.

[0750] 4. Automatic generation of sightseeing plans

[0751] User: Enters preferences and wishes into the interface.

[0752] Device: The entered information and current location information are sent to the server.

[0753] Server: The generation AI generates the optimal sightseeing plan from the database and sends it back to the device.

[0754] Terminal: Provides the received travel plan to the user visually and audibly.

[0755] Directions to tourist spots

[0756] 5. Destination selection and route guidance

[0757] User: Choose the place you want to go from the presented sightseeing itinerary.

[0758] Terminal: Sends destination information to the server and calculates the optimal route.

[0759] Server: Calculates route information and sends it to the device.

[0760] Terminal: Provides visual and audio route guidance to the user.

[0761] Real-time tourist guide

[0762] 6. Providing detailed information about tourist spots

[0763] Terminal: The camera detects tourist spots and displays them visually. Detailed information is obtained from the server and provided to the user.

[0764] Terminal: Also provides audio guidance to users.

[0765] Example: When a user stands in front of a museum, they are given visual and audio guidance with detailed information about the building's history.

[0766] Providing translation functions

[0767] 7. Translation of text information

[0768] Device: The camera detects text information within the user's field of view.

[0769] Terminal: OCR the text information and send it to the translation API.

[0770] Server: Obtains translation results and sends them to the device.

[0771] Terminal: Provides the user with visual and audio translation results.

[0772] Example: Translating and displaying a restaurant menu.

[0773] Recommendations for nearby information

[0774] 8. Obtaining recommendation information

[0775] User: Submit a request for recommendations.

[0776] Server: Retrieves the best recommendations based on the user's location and preferences.

[0777] Terminal: Provides recommendation information visually and audibly.

[0778] Example: Recommending a popular nearby cafe.

[0779] Experience booking support

[0780] 9. Making a Reservation

[0781] User: Select the experience or restaurant they want.

[0782] Terminal: Sends a reservation request to the server and checks the reservation status.

[0783] Server: Connects to the reservation system, checks, and sends the results to the terminal.

[0784] Terminal: Provides visual and audio confirmation of reservation completion.

[0785] Example: Notifying a customer that a tea time reservation at a cafe has been completed.

[0786] Example

[0787] In a real-world tourism scenario, when a user wears smart glasses and stands at a tourist spot, their current location information is immediately acquired, and the AI ​​automatically generates an optimal sightseeing plan. The user can then follow the guide as they tour the tourist spots, enjoying real-time guidance, translation, and recommendations. All information is provided through the smart glasses' display and audio, allowing users to enjoy sightseeing hands-free.

[0788] This allows users to have an efficient and fulfilling sightseeing experience, freeing them from the language barrier and the hassle of gathering information.

[0789] The processing flow will be explained below.

[0790] Step 1:

[0791] boot

[0792] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[0793] Step 2:

[0794] Log in

[0795] User: The user enters their login information (ID and password) and selects the login button.

[0796] Terminal: Converts the entered login information into a security token and sends it to the server.

[0797] Server: Compares the received login information with the database, and if authentication is successful, issues an authentication token and returns it to the device.

[0798] Step 3:

[0799] Get current location

[0800] Device: Activates the built-in GPS module and obtains the user's current location information (latitude and longitude).

[0801] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[0802] Step 4:

[0803] Sending current location information

[0804] Device: Sends the acquired location information to the server.

[0805] Server: Based on the received location information, search the database for information on nearby tourist spots.

[0806] Step 5:

[0807] Enter your sightseeing plan

[0808] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[0809] Step 6:

[0810] Submitting a plan generation request

[0811] Terminal: Sends a plan generation request including the input sightseeing plan information and current location information to the server.

[0812] Step 7:

[0813] Automatic generation of sightseeing plans

[0814] Server: The AI ​​automatically generates the optimal sightseeing plan based on the user's current location and preferences. The generated sightseeing plan is sent to the device.

[0815] Terminal: Presents the received sightseeing plan to the user with visual and audio guides.

[0816] Step 8:

[0817] Destination selection

[0818] User: Select the destinations they want to visit from the presented itinerary.

[0819] Step 9:

[0820] Sending a route calculation request

[0821] Terminal: Sends a route calculation request including the selected destination information to the server.

[0822] Step 10:

[0823] Route calculation and guidance

[0824] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[0825] Terminal: Provides visual and audio route guidance to the user based on the calculated route information.

[0826] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[0827] Step 11:

[0828] Real-time guide to tourist attractions

[0829] Device: Detects tourist spots within the user's field of view through the smart glasses camera.

[0830] Server: Based on the detected spot information, retrieves detailed information from the database and sends it back to the device.

[0831] Terminal: The acquired detailed information is presented to the user and audio guidance is also provided.

[0832] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[0833] Step 12:

[0834] Translation feature

[0835] Device: Detects text information within the user's field of view through the camera.

[0836] Terminal: OCR the text information and send it to the translation API.

[0837] Server: Translates the source text into the user's native language via the translation API and sends the results to the device.

[0838] Terminal: Presents the translation results to the user visually and audibly.

[0839] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[0840] Step 13:

[0841] Recommendations for nearby information

[0842] User: Sends a request for recommendations.

[0843] Server: Retrieves the most suitable nearby information (tourist attractions, restaurants, etc.) from a database based on the user's current location and preferences.

[0844] Terminal: Provides the acquired recommendation information to the user visually and audibly.

[0845] Example: Information about a popular nearby cafe is displayed, suggesting, "Would you like to take a break here?"

[0846] Step 14:

[0847] Experience booking support

[0848] User: Selects the experience or restaurant they want from the recommendations and submits a reservation request.

[0849] Terminal: Sends a reservation request to the server and asks for availability.

[0850] Server: Connects to the reservation system to check availability and sends the results to the terminal.

[0851] Terminal: Provides visual and audio confirmation of reservation completion.

[0852] Example: A user can enjoy tea time at a cafe at a reserved time.

[0853] Step 15:

[0854] Exit and log out

[0855] User: Select Logout from the menu.

[0856] Terminal: Sends a logout request to the server to end the session.

[0857] Server: Discard the user's session information and notify the user that the logout is complete.

[0858] Terminal: Shut down the system and prepare for next use.

[0859] Example 1

[0860] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0861] Conventional tourist guide systems require users to search for information and plan their own sightseeing itineraries, and they have problems with information gathering and understanding if they do not understand a foreign language. Furthermore, they are unable to provide real-time guidance, translation, and recommendation functions based on the user's current location, limiting the tourist experience.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0863] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for providing visual and audio guides, translation means for detecting and translating text information using a camera, means for transmitting a reservation request to the server and acquiring reservation information, and means for recommending surrounding information. This allows the user to enjoy optimal sightseeing plans in real time, freeing them from the language barrier and the hassle of information gathering, and enabling a more efficient and fulfilling sightseeing experience.

[0864] The "location information acquisition means" is a device or technology for identifying the user's current location and acquiring that location information.

[0865] The "communication means" is an interface for transmitting acquired location information and user requests to a server and receiving data from the server.

[0866] The "display means" is a device for visually displaying to the user the sightseeing plan information and other data provided by the server.

[0867] A "recommendation means" is an algorithm or technology that extracts information suitable for a user from a database stored on a server and provides it to the user.

[0868] A "visual guide means" is a device or technology that visually guides a user to location information and detailed information about tourist spots.

[0869] "Audio guidance means" refers to a device or technology that provides audio guidance of visually displayed information.

[0870] The "translation means" is a technology for translating text information detected using a camera into another language and providing it to the user.

[0871] The "reservation means" is a means for a user to send a reservation request for a desired experience or restaurant to the server and check the reservation status.

[0872] "Recommendation methods" are algorithms and technologies that select and present optimal information to users based on their current location and preferences.

[0873] The present invention relates to a system that enables users to use smart glasses while sightseeing to receive optimal sightseeing plans, route guidance, and guide services in real time. This system is composed of a location information acquisition means, a communication means, a display means, a recommendation means, a visual guide means, an audio guide means, a camera, and a translation means.

[0874] Startup and login

[0875] First, the user turns on the smart glasses. As a terminal, the smart glasses launch the OS and go through initialization processes, displaying the login screen. The user enters login information using the smart glasses' touchpad or voice input. The terminal converts the entered information into a security token and sends it to the server using a communication method. The server references a database to confirm the user's authentication information, and if authentication is successful, returns an authentication token to the terminal.

[0876] Obtaining your current location and automatically generating a sightseeing plan

[0877] The device's built-in GPS module then obtains its current location and, with the user's permission, transmits this information to a server. The user inputs their preferences and the places they want to visit through the smart glasses' interface. The device then transmits this information to the server, which uses a generative AI model to generate an optimal itinerary. The generated itinerary is then sent back to the device and presented to the user visually and audibly.

[0878] Specific examples

[0879] For example, if a user wishes to visit a historical building, the server will search the database for the most suitable sightseeing plan based on the input information and current location information. The sightseeing plan will include nearby historical buildings, recommended restaurants, etc. This information will be provided to the user through the smart glasses' display and audio output.

[0880] Directions to tourist spots

[0881] When a user selects a destination from the provided sightseeing plan, the device sends this information to the server and requests it to calculate the optimal route. The server calculates the optimal route using GPS information and the selected destination information and sends it to the device. The device then provides the user with visual and audio route guidance.

[0882] Real-time tourist guide

[0883] When users arrive at a tourist spot, the smart glasses' camera detects the tourist spot and sends that information to the server. The server then retrieves detailed information about the tourist spot from a database and sends it to the device. This allows users to obtain detailed information about the tourist spot visually and audibly. For example, when users stand in front of a museum, they will be provided with information about the building's history and exhibits.

[0884] Providing translation functions

[0885] When the smart glasses' camera detects text within the user's field of view, it processes it with OCR and sends it to a translation API. The server then retrieves the translation results and sends them to the device. The user can then confirm the translation visually and audibly. For example, a restaurant menu may be translated and displayed.

[0886] Recommendations for nearby information

[0887] When a user sends a recommendation request, the server retrieves the best recommendations based on the user's current location and preferences. The device then presents this information visually and audibly, recommending nearby popular cafes, tourist attractions, and more.

[0888] Experience booking support

[0889] Once the user selects the desired experience and restaurant, the device sends a reservation request to the server and checks the reservation status. The server connects to the reservation system to confirm whether the reservation can be made and sends the result to the device. The device then notifies the user visually and audibly that the reservation has been completed. For example, it notifies the user that a tea time reservation at a cafe has been completed.

[0890] This system allows users to enjoy optimal sightseeing plans in real time, freeing them from language barriers and the hassle of gathering information, enabling a more efficient and fulfilling sightseeing experience.

[0891] Prompt Sentence Examples

[0892] "Please suggest the best sightseeing plan based on the user's current location."

[0893] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0894] Step 1:

[0895] boot

[0896] Device: When you start up the smart glasses, the OS starts up and the system is initialized. The login screen is displayed.

[0897] Input: User powers on smart glasses.

[0898] Output: The login screen is displayed.

[0899] What happens: The OS checks various modules to make sure they are OK, then renders the login screen in the UI.

[0900] Step 2:

[0901] Log in

[0902] User: The user enters their login information using the smart glasses interface.

[0903] Terminal: Converts the entered login information into a security token and sends it to the server.

[0904] Server: The server checks the user's authentication information by referencing the database, and if authentication is successful, returns an authentication token to the terminal.

[0905] Input: Login information (user ID and password).

[0906] Output: An authentication token.

[0907] Specific operation: Login information is encrypted into a security token and sent to the server, where it is checked against a database. If authentication is successful, an authentication token is generated and sent back to the device as an HTTP response.

[0908] Step 3:

[0909] Get current location

[0910] On the device: The built-in GPS module is activated and acquires the current location, which is then sent to the server after user confirmation.

[0911] Input: Latitude and longitude information from the GPS module.

[0912] Output: Current location information.

[0913] Specific operation: Captures GPS signals and measures latitude and longitude information. Then, parses this information into JSON format and sends it to the server via a communication method.

[0914] Step 4:

[0915] Automatic generation of sightseeing plans

[0916] User: Enters preferences and wishes using the smart glasses interface.

[0917] Device: Sends input information and current location information to the server.

[0918] Server: The generative AI model generates the optimal sightseeing plan from the database and sends it back to the device.

[0919] Input: Your preferences, wishes, and location.

[0920] Output: Tourist plan.

[0921] How it works: After users input their preferences and wishes, they send that information along with their current location to the server, where the generative AI model generates the best options and returns them as a sightseeing plan.

[0922] Step 5:

[0923] Destination selection and route guidance

[0924] User: Choose the place you want to go from the presented sightseeing itinerary.

[0925] Terminal: Sends the selected destination information to the server and requests it to calculate the optimal route.

[0926] Server: Calculates the optimal route and sends it to the device, which then provides visual and audio guidance to the user.

[0927] Input: Destination information.

[0928] Output: Optimal route.

[0929] Specific operation: Using GPS information and the selected destination, the system calls a map API (e.g., Google Maps API) to calculate the optimal route, sends the calculation results back to the device, and provides audio and visual guidance.

[0930] Step 6:

[0931] Providing detailed information about tourist spots

[0932] Device: The camera in the smart glasses detects tourist spots and sends the information to the server.

[0933] Server: Retrieves detailed information about tourist attractions from the database and sends it to the terminal, which then provides the user with detailed information visually and audibly.

[0934] Input: Video of tourist spot.

[0935] Output: Detailed information about the tourist spot.

[0936] Specific operation: Camera images are analyzed using image recognition technology to identify tourist spots. Information about the identified spots is sent to the server, and the corresponding detailed information is retrieved and presented to the user.

[0937] Step 7:

[0938] Text translation

[0939] Device: The camera detects text information within the user's field of view, performs OCR processing, and sends it to the translation API.

[0940] Server: Receives the translation results and sends them to the device, which then provides the translation results to the user visually and audibly.

[0941] Input: Text information.

[0942] Output: The translation result.

[0943] Specific operation: Extracts text from images using OCR technology, sends it to a translation API (e.g., Google Cloud Translation API), receives the translation result, and notifies you by display and voice.

[0944] Step 8:

[0945] Acquiring recommendation information

[0946] User: Send a recommendation request.

[0947] Server: Retrieves optimal recommendations based on the user's current location and preferences. The device provides the recommendations visually and audibly.

[0948] Input: recommendation request, current location, preferences.

[0949] Output: Recommendation information.

[0950] Specific operation: A database search is performed based on the user's request and current location information, and information on nearby tourist attractions and restaurants is processed using a recommendation algorithm to generate optimal recommendations.

[0951] Step 9:

[0952] Making a reservation

[0953] User: Select the experience or restaurant they want.

[0954] Terminal: Sends a reservation request to the server and checks the reservation status.

[0955] Server: Connects to the reservation system, confirms the reservation, and sends the result to the terminal. The terminal then provides the user with visual and audio information about the completion of the reservation.

[0956] Input: Booking request.

[0957] Output: Booking completion information.

[0958] Specific behavior: Sends a reservation request to the reservation system, checks the reservation status, and notifies the user visually and audibly if the reservation is successful.

[0959] (Application example 1)

[0960] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0961] In modern tourism and shopping experiences, users are required to efficiently obtain and use a large amount of information. However, it is difficult to obtain the necessary information in real time in foreign countries or large stores, and it is often time-consuming to perform complex operations and gather information. Furthermore, it is currently difficult for users to instantly grasp the product information and inventory status they are looking for and receive appropriate guidance. This results in users wasting their valuable time and resulting in an unsatisfactory experience.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0963] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, a camera module for the user to acquire product information in a physical store, means for transmitting the acquired product information to the server and receiving detailed information, and means for providing the detailed information visually and audibly. This allows the user to acquire necessary information in real time at tourist spots and physical stores and use it efficiently.

[0964] "User" refers to an individual or organization that uses the system to sightsee or shop.

[0965] "Current Location" refers to the geographic location of the user obtained using GPS or other means.

[0966] "Location information acquisition means" refers to a function or device for identifying the user's current location and acquiring that information.

[0967] "Communication means" refers to the functions and devices for sending acquired information to a server and receiving information from the server.

[0968] "Display means" refers to a device or function for visually conveying information received from the server to the user.

[0969] "Recommendation means" refers to a function for recommending highly relevant information and plans to a user from a database stored on a server.

[0970] "Camera Module" refers to a device or feature that includes a camera that allows a user to scan products or attractions.

[0971] "Detailed information" refers to information provided by the server and displayed to the user, including product descriptions and tourist spot information.

[0972] "Visual guide means" refers to devices or functions for visually displaying the location of products or detailed information about tourist attractions.

[0973] "Audio guide means" refers to a function or device for providing the user with audio guidance of displayed information.

[0974] "Stock status" refers to the availability and quantity of the product selected by the user in the store.

[0975] "Reservation" refers to the procedure or function that allows a user to reserve a selected product in advance for later purchase.

[0976] System Overview

[0977] This invention relates to a system for improving users' sightseeing and shopping experiences. The system uses smart glasses to help users obtain necessary information in real time at tourist spots and brick-and-mortar stores and use it efficiently.

[0978] Hardware and software used

[0979] The hardware and software used in this system are as follows:

[0980] GPSModule: A location information acquisition means for acquiring the user's current location.

[0981] CameraModule: A camera module that allows users to scan products and tourist attractions.

[0982] DisplayModule: A display means for visually conveying information received from the server to the user.

[0983] ServerCommunication: A communication method for sending acquired information to a server and receiving information from the server.

[0984] Program processing overview

[0985] The system's main processing steps are as follows: First, the user puts on the smart glasses and starts the system. The system identifies the user's current location using GPS Module and sends this information to the server via ServerCommunication. The server generates optimal sightseeing plans and shopping guides based on the user's current location, past purchase history, and input preference information, and presents them visually to the user via Display Module.

[0986] Furthermore, when a user explores a product in a physical store, the CameraModule is used to scan the product and retrieve details, reviews, stock status, etc. from the server. This information is also provided to the user through the DisplayModule and audio guide.

[0987] Specific examples

[0988] For example, when a user arrives at a tourist spot, the system immediately acquires their current location information and sends it to the server. The server then uses a generative AI model to automatically generate an optimal sightseeing plan for the user, providing it to the user. This plan includes detailed information about the tourist spots the user should visit and directions. When the user stands in front of a museum, they are given visual and audio guidance about the museum's history and exhibits.

[0989] When users scan an item in-store, they can view detailed product information, reviews, and availability in real time. If the item they select is low in stock, the system will suggest a reservation and guide the user through the process.

[0990] Prompt Sentence Examples

[0991] Examples of prompts to input to a generative AI model include:

[0992] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[0993] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0994] Step 1:

[0995] Startup and login

[0996] Terminal: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[0997] Input: The user enters their login information (username and password).

[0998] Output: The entered login information is converted into a security token and sent to the server.

[0999] Server: Checks the login information against a database and, if authentication is successful, returns an authentication token.

[1000] Terminal: Receives the authentication token and notifies the user that login was successful.

[1001] Step 2:

[1002] Obtaining your current location and automatically generating a sightseeing plan

[1003] Device: The built-in GPS module starts up and acquires the current location.

[1004] Input: Current location information obtained.

[1005] Output: Sends current location information to the server.

[1006] Server: Receives the current location information sent by the user and generates the optimal sightseeing plan based on the database.

[1007] Terminal: Provides the generated sightseeing plan to the user visually and audibly.

[1008] Step 3:

[1009] Destination selection and route guidance

[1010] User: Choose the place you want to go from the presented sightseeing plan.

[1011] Input: User selected destination information.

[1012] Output: Sends destination information to the server.

[1013] Server: Calculates the optimal route to the selected destination and sends the route information to the device.

[1014] Terminal: Provides visual and audio route guidance to the user.

[1015] Step 4:

[1016] Scanning products and obtaining information

[1017] User: The customer holds the product in their hand and scans it with the camera module on the smart glasses.

[1018] Input: Captured product image.

[1019] Output: Send product images to the server and get detailed information.

[1020] Server: Analyzes the sent product image, retrieves detailed information from the database, and returns it to the terminal.

[1021] Terminal: Provides detailed product information visually and audibly.

[1022] Step 5:

[1023] Check stock availability and reserve

[1024] User: To check the stock status of the product they wish to purchase, they send a stock check request from the details screen.

[1025] Input: Inventory check request information.

[1026] Output: Sends stock status information to the server and receives confirmation results.

[1027] Server: Connects to the inventory system to check the inventory status and sends the results to the terminal.

[1028] Terminal: Provides visual and audio information on stock availability and guides you through the reservation process if one is available.

[1029] Step 6:

[1030] Sales information and coupon recommendations

[1031] Terminal: As users move around the store, they obtain sale and coupon information based on their current location and purchase history.

[1032] Input: Current location information and purchase history information.

[1033] Output: Receive sale information and coupon information from the server.

[1034] Server: Based on the user's current location information and purchase history, extracts sale and coupon information from the database and sends it to the terminal.

[1035] Terminal: Provides acquired sales and coupon information to the user visually and audibly.

[1036] Prompt Sentence Examples

[1037] Examples of prompts to input to a generative AI model include:

[1038] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[1039] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1040] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[1041] System Overview

[1042] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system consists of the following:

[1043] Location information acquisition means

[1044] communication means

[1045] Display means

[1046] Recommendation method

[1047] Visual Guide Means

[1048] Audio guide means

[1049] Translation tools

[1050] Reservation method

[1051] Emotion Recognition Engine

[1052] Program processing overview

[1053] Startup and login

[1054] 1. Startup

[1055] Device: When the smart glasses are started up, the internal system is initialized and the login screen is displayed.

[1056] 2. Log in

[1057] User: The user enters their login information (ID and password) and selects the login button.

[1058] Terminal: Converts the entered login information into a security token and sends it to the server.

[1059] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[1060] Obtaining your current location and automatically generating a sightseeing plan

[1061] 3. Get your current location

[1062] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[1063] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[1064] 4. Sending current location information

[1065] Device: Sends the acquired location information to the server.

[1066] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1067] Acquisition and application of emotion data

[1068] 5. Acquiring Emotion Data

[1069] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[1070] 6. Transmission and analysis of emotional data

[1071] Device: Sends the acquired emotion data to the server.

[1072] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[1073] Coordination and provision of tourism plans

[1074] 7. Generate a sightseeing plan

[1075] User: The user enters their travel preferences and wishes into the interface.

[1076] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[1077] Terminal: Presents the received travel plan to the user visually and audibly.

[1078] Directions and guides to tourist spots

[1079] 8. Destination selection and route guidance

[1080] User: Choose the places they want to visit from the presented itinerary.

[1081] Terminal: Sends destination information to the server and calculates the optimal route.

[1082] Server: Calculates route information and sends it to the device.

[1083] Terminal: Provides visual and audio route guidance.

[1084] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[1085] Real-time guide and translation

[1086] 9. Providing detailed information about tourist attractions

[1087] Terminal: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and presents it.

[1088] Terminal: Also provides audio guidance to the user.

[1089] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[1090] 10. Text Translation

[1091] Device: Detects text in the user's field of view via the camera.

[1092] Terminal: OCR the text information and send it to the translation API.

[1093] Server: Sends the translation results to the device.

[1094] Terminal: Provides visual and audio translation results.

[1095] Example: Translating a restaurant menu to show "Chicken Curry - 500 yen."

[1096] Local information recommendations and experience reservations

[1097] 11. Obtaining local information and making recommendations

[1098] User: Send a recommendation request.

[1099] Server: Makes recommendations by providing peripheral information while taking into account emotional data.

[1100] Terminal: Provides information visually and audibly.

[1101] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[1102] 12. Booking an Experience

[1103] User: Selects the experience or restaurant they want and submits a reservation request.

[1104] Terminal: Sends a reservation request to the server.

[1105] Server: Connects to the reservation system and checks availability.

[1106] Terminal: Provides visual and audio confirmation of reservation completion.

[1107] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[1108] Example

[1109] For example, when a user puts on smart glasses and starts sightseeing, location information is first acquired. Then, a sightseeing plan tailored to the user's needs is generated based on emotional data. When the user asks for directions or detailed information about tourist spots, guidance tailored to the user's current emotional state is provided. This allows the user to enjoy sightseeing without stress.

[1110] If a translation is needed, the smart glasses' camera detects the text and provides an instant translation. Furthermore, if the emotion recognition engine determines that the user is tired, it will provide a recommendation to take a break. In this way, the system comprehensively supports the user's sightseeing experience.

[1111] The processing flow will be explained below.

[1112] Step 1:

[1113] boot

[1114] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[1115] Step 2:

[1116] Log in

[1117] User: The user enters their login information (ID and password) and selects the login button.

[1118] Terminal: Converts the entered login information into a security token and sends it to the server.

[1119] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[1120] Step 3:

[1121] Get current location

[1122] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[1123] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[1124] Step 4:

[1125] Sending current location information

[1126] Device: Sends the acquired location information to the server.

[1127] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1128] Step 5:

[1129] Acquiring emotion data

[1130] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[1131] Step 6:

[1132] Sending and analyzing emotional data

[1133] Device: Sends the acquired emotion data to the server.

[1134] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[1135] Step 7:

[1136] Generate a sightseeing plan

[1137] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[1138] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[1139] Terminal: Presents the received travel plan to the user visually and audibly.

[1140] Step 8:

[1141] Selecting a destination and calculating a route

[1142] User: Select the places they want to visit from the presented itinerary.

[1143] Terminal: Sends destination information to the server and calculates the optimal route.

[1144] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[1145] Terminal: Provides visual and audio route guidance.

[1146] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[1147] Step 9:

[1148] Real-time Guide

[1149] Device: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and displays it.

[1150] Server: Provides detailed information about tourist spots based on the detected spot information.

[1151] Terminal: Provides both visual and audio guidance to the user.

[1152] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[1153] Step 10:

[1154] Translation feature

[1155] Device: Detects text information within the user's field of view through the camera.

[1156] Terminal: OCR the text information and send it to the translation API.

[1157] Server: Sends the translation results to the device.

[1158] Terminal: Provides visual and audio translation results.

[1159] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[1160] Step 11:

[1161] Recommendations for nearby information

[1162] User: Send a recommendation request.

[1163] Server: Obtains optimal surrounding information based on the user's current location and emotional data, and generates recommendation information.

[1164] Terminal: Provides visual and audio recommendations.

[1165] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[1166] Step 12:

[1167] Book an experience

[1168] User: Select the experience or restaurant they want.

[1169] Terminal: Sends a reservation request to the server and checks availability.

[1170] Server: Connects to the reservation system, checks availability, and sends the results to the device.

[1171] Terminal: Provides visual and audio confirmation of reservation completion.

[1172] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[1173] Step 13:

[1174] Exit and log out

[1175] User: Select Logout from the menu.

[1176] Terminal: Sends a logout request to the server to end the session.

[1177] Server: Discard the user's session information and notify the user that the logout is complete.

[1178] Terminal: Shut down the system and prepare for next use.

[1179] Example 2

[1180] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1181] This invention relates to a system for improving the tourism experience. However, conventional tourism systems often lack features such as optimizing sightseeing plans based on the user's real-time emotional state, translation functions, and reservation functions. As a result, users are often limited in the information and support they receive while sightseeing, which can lead to a decrease in satisfaction. Furthermore, the lack of route guidance to tourist spots and detailed guide information is also problematic. Therefore, there is a need for a system that can comprehensively support users' sightseeing experiences and provide high-quality services.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1183] In this invention, the server

[1184] A means for acquiring location information;

[1185] means for transmitting the acquired location information to a server;

[1186] a means for displaying the sightseeing plan provided by the server on the terminal;

[1187] A means for obtaining recommendation information from a database;

[1188] means for acquiring user emotion data using an emotion recognition engine;

[1189] means for transmitting the acquired emotion data to a server and analyzing it;

[1190] a means of providing real-time visual and audio route guidance;

[1191] A means of recommending surrounding information;

[1192] a means for visually detecting and translating text;

[1193] a means for submitting a reservation request and providing reservation completion information;

[1194] This allows users to receive optimal sightseeing plans, route guidance, and detailed guide information in real time, and also provides recommendation information and translation services based on their emotional state, greatly improving the user's sightseeing experience.

[1195] "Means for acquiring location information" refers to a device or function that acquires the user's current location using a GPS module or other location measurement technology.

[1196] The "means for transmitting the acquired location information to a server" refers to a device or function for transmitting location information data to a server via Wi-Fi, a cellular network, or other communication technology.

[1197] The "means for displaying the sightseeing plan provided by the server on the terminal" refers to a device or function that visually displays the sightseeing plan information received from the server on the user's smart glasses or mobile device.

[1198] The "means for acquiring recommended information from a database" refers to a device or function that searches for and acquires recommended information related to travel and sightseeing from a database stored on a server.

[1199] "Means for acquiring user emotional data using an emotion recognition engine" refers to a device or function that uses sensors such as a camera or microphone to analyze the user's facial expressions and vocal tone and identify the user's emotional state.

[1200] The "means for transmitting acquired emotion data to a server and analyzing it" refers to a device or function that transmits the user's emotion data to a server and analyzes the data in detail on the server side.

[1201] "Means for providing real-time visual and audio route guidance" refers to a device or function that provides the user with visual and audio guidance in real time on the optimal route to a destination specified by the user.

[1202] A "means for recommending surrounding information" is a device or function that recommends nearby tourist spots and facilities based on the user's current location and emotional state.

[1203] A "means for visually detecting and translating text" is a device or function that uses a camera to detect text in its field of view and then OCRs and translates that text.

[1204] The "means for sending a reservation request and providing reservation completion information" refers to a device or function that sends a reservation request for the user's desired experience or facility to the server and notifies the user that the reservation has been completed.

[1205] The present invention relates to a system that uses smart glasses to significantly improve a user's sightseeing experience. This system is designed to enable users to receive optimal sightseeing plans, route guidance, and guide services in real time while sightseeing.

[1206] System configuration

[1207] This system consists of the following main means:

[1208] 1. Location information acquisition means

[1209] 2. Means of communication

[1210] 3. Display means

[1211] 4. Recommendation method

[1212] 5. Emotion recognition means

[1213] 6. Route guidance methods

[1214] 7. Translation Methods

[1215] 8. Reservation Methods

[1216] Hardware and software used

[1217] Smart glasses (terminal): A device worn by the user. This device is equipped with a GPS module, camera, microphone, display, and speaker.

[1218] Server: The back-end processing device, including the database and generative AI model.

[1219] Generative AI model: Runs on the server side and generates sightseeing plans based on the user's location, emotional data, and preferences.

[1220] Processing flow and specific operations

[1221] Below is an overview of the operation of this system and the specific operations for each process.

[1222] Startup and login

[1223] 1. Startup

[1224] Device: When the smart glasses are turned on, the internal system is first initialized and the login screen is displayed.

[1225] 2. Log in

[1226] User: Enter your login information (ID and password) and select the Login button.

[1227] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[1228] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[1229] Obtaining and sending location information

[1230] 3. Get your current location

[1231] Device: The built-in GPS module is activated and obtains the user's current location information in real time.

[1232] User: A confirmation dialog about obtaining location information will be displayed and the user will select "Allow."

[1233] 4. Sending current location information

[1234] Device: Sends the acquired location information to the server.

[1235] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1236] Acquisition and application of emotion data

[1237] 5. Acquiring Emotion Data

[1238] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[1239] 6. Transmission and analysis of emotional data

[1240] Device: Sends the acquired emotion data to the server.

[1241] Server: Analyzes emotion data, identifies the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan.

[1242] Creating and providing tour plans

[1243] 7. Generate a sightseeing plan

[1244] User: Enters travel preferences and wishes into the interface.

[1245] Server: Uses a generative AI model to generate optimal sightseeing plans based on location, preference, and emotion data.

[1246] Server: Sends the generated travel plan to the device.

[1247] Terminal: Presents the received travel plan to the user visually and audibly.

[1248] Directions and guides to tourist spots

[1249] 8. Destination selection and route guidance

[1250] User: Choose the places they want to visit from the presented itinerary.

[1251] Device: Sends the selected destination information to the server and calculates the optimal route.

[1252] Server: Calculates route information and sends it to the device.

[1253] Terminal: Provides visual and audio route guidance.

[1254] Specific examples of facilities

[1255] For example, when a user puts on smart glasses and begins sightseeing, location information is first acquired. Then, the user's desired sightseeing plan is created using a generative AI model and sent to the device. As the user heads to their selected destination, route guidance is provided in real time. Also, when the user arrives at a tourist spot, the smart glasses' camera detects the spot and displays detailed information. If translation is required, the camera detects the text and the translation result is instantly provided visually and audibly.

[1256] Prompt Sentence Examples

[1257] Examples of prompts for operating this system are shown below.

[1258] "Get your current location"

[1259] "Generate the best sightseeing plan based on user preferences and sentiment data."

[1260] "Please provide more information about the attraction the user is looking at."

[1261] "Book your selected experience"

[1262] "Use your camera to translate any text you see"

[1263] Using these prompts, the generative AI model provides a sightseeing experience that is tailored to the user, ensuring the entire system operates smoothly.

[1264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1265] Step 1: Boot up and log in

[1266] 1.1 Startup

[1267] Device: When the smart glasses are turned on, they first initialize their internal systems, which includes checking that the CPU, memory, and sensors are working.

[1268] Device: Once initialization is complete, the login screen will appear.

[1269] Input: The user presses the power button.

[1270] Output: The login screen is displayed.

[1271] 1.2 Login

[1272] User: Enter your ID and password on the login screen and press the login button.

[1273] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[1274] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[1275] Input: The user enters their ID and password and presses the login button.

[1276] Output: If authentication is successful, the main menu is displayed.

[1277] Step 2: Obtaining and sending location information

[1278] 2.1 Obtaining current location

[1279] Device: Activates the built-in GPS module and obtains the user's current location (latitude and longitude) in real time.

[1280] User: A confirmation dialog about obtaining location information will appear, select "Allow".

[1281] Input: The user selects "Allow" in the location information confirmation dialog.

[1282] Output: Current location information (latitude and longitude) is obtained.

[1283] 2.2 Sending current location information

[1284] Terminal: The acquired current location information is sent to the server in packet format.

[1285] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1286] Input: Current location information (latitude and longitude) obtained by the device.

[1287] Output: The location information is sent to the server, and information about nearby tourist spots is searched.

[1288] Step 3: Acquiring and applying emotion data

[1289] 3.1 Acquiring Emotion Data

[1290] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[1291] Input: User's facial expressions and voice captured through camera and microphone.

[1292] Output: The analyzed emotion data is obtained.

[1293] 3.2 Emotional Data Transmission and Analysis

[1294] Device: Sends the acquired emotion data to the server.

[1295] Server: Analyzes the emotion data and identifies the user's state (e.g., joy, excitement, fatigue, etc.).

[1296] Input: The acquired emotion data.

[1297] Output: Emotion data is sent and analyzed on the server.

[1298] Step 4: Generate and provide a tour plan

[1299] 4.1 User Input

[1300] User: Enters travel preferences and wishes into the interface.

[1301] Input: The user enters their travel preferences and wishes into the interface.

[1302] Output: The entered preference information is saved on the device.

[1303] 4.2 Generating a Tourist Plan

[1304] Server: Uses a generative AI model to generate optimal sightseeing plans based on location information, user preferences, and emotional data.

[1305] Input: Location, user preferences, and emotional data.

[1306] Output: The generated itinerary.

[1307] 4.3 Presentation of sightseeing plans

[1308] Server: Sends the generated travel plan to the device.

[1309] Terminal: Presents the received travel plan to the user visually and audibly.

[1310] Input: The generated itinerary.

[1311] Output: The sightseeing plan is presented visually and audibly.

[1312] Step 5: Select a destination and get directions

[1313] 5.1 Selecting a destination

[1314] User: Select the places they want to visit from the presented itinerary.

[1315] Input: The destination selected by the user.

[1316] Output: Selected destination information.

[1317] 5.2 Route calculation

[1318] Terminal: Sends destination information to the server and calculates the optimal route.

[1319] Server: Calculates the optimal route taking into account congestion information and real-time traffic information, and returns the results to the device.

[1320] Input: Selected destination information.

[1321] Output: Calculated route information.

[1322] 5.3 Route guidance

[1323] Device: Provides visual and audio guidance based on the received route.

[1324] Input: Calculated route information.

[1325] Output: Route guidance is provided visually and audibly.

[1326] Step 6: Provide detailed information about the tourist spot

[1327] 6.1 Tourist Spot Detection

[1328] Device: The smart glasses' camera detects and identifies tourist spots.

[1329] Input: Tourist spots detected through the camera.

[1330] Output: Information about detected tourist spots.

[1331] 6.2 Obtain detailed information

[1332] Device: Queries the server for information about the detected tourist spots.

[1333] Server: Retrieves relevant details from the database and sends them to the device.

[1334] Input: Detected tourist spot information.

[1335] Output: The detailed information retrieved.

[1336] 6.3 Detailed information presentation

[1337] Terminal: Provides users with detailed information about tourist attractions through visual and audio means.

[1338] Input: The retrieved details.

[1339] Output: Detailed information about tourist attractions is provided visually and audibly.

[1340] Step 7: Translate the text

[1341] 7.1 Text Detection

[1342] Device: Detects text in the user's field of view via the camera.

[1343] Input: Text information detected through the camera.

[1344] Output: The detected text data.

[1345] 7.2 Translation Process

[1346] Device: OCR the detected text information and send it to the translation API.

[1347] Server: Uses the translation API to return the translated text.

[1348] Input: OCR processed text data.

[1349] Output: The translation result.

[1350] 7.3 Providing translation results

[1351] Terminal: Provides the user with visual and audio translation results.

[1352] Input: Translation result.

[1353] Output: Visual and audio translation results are provided.

[1354] Step 8: Recommend local information and book experiences

[1355] 8.1 Recommendation Request

[1356] User: Request a recommendation.

[1357] Input: The user request.

[1358] Output: The recommendation request is saved to the device.

[1359] 8.2 Providing recommendation information

[1360] Server: Generates nearby recommendation information based on emotion data and sends it to the device.

[1361] Input: Emotion data, peripheral information.

[1362] Output: Recommendation information.

[1363] 8.3 Experience Booking

[1364] User: Selects the experience or restaurant they want and submits a reservation request.

[1365] Terminal: Sends a reservation request to the server.

[1366] Server: Connects to the reservation system and checks availability.

[1367] Terminal: Provides visual and audio confirmation of reservation completion.

[1368] Input: User's booking request.

[1369] Output: Booking completion information is provided.

[1370] (Application example 2)

[1371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1372] Conventional tourism support systems have limited ways for users to obtain information about tourist destinations, requiring a great deal of effort to obtain information about the destinations they are visiting. Furthermore, if users do not understand the local language, it is difficult to understand information about tourist destinations and prices. Furthermore, there is a lack of product information, price comparisons, and navigation functions within commercial facilities, limiting the user's shopping experience.

[1373] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for acquiring and displaying information about products, means for converting the user's voice input into text using voice recognition, and means for recommending products based on the acquired text information. This not only enables the user to easily acquire information about destinations, but also enables the user to understand tourist destination and product information across language barriers, and enjoy a pleasant shopping experience in commercial facilities.

[1374] The "location information acquisition means" is a device or software for acquiring the user's current location information.

[1375] "Communication means" refers to a device or protocol for transmitting acquired location information and other data to a server.

[1376] The "display means" refers to a display device or software for displaying information provided by the server on the terminal.

[1377] The "recommendation means" is an algorithm or system for obtaining recommendation information from a database stored on a server.

[1378] "Means for acquiring and displaying product information" refers to devices or software that acquire detailed product information and provide it to users visually or audibly.

[1379] "Means for converting a user's voice input into text using voice recognition" refers to technology for recognizing a user's voice and converting it into text data.

[1380] The "means for recommending products based on acquired text information" is a system for recommending related products to a user based on text information acquired by speech recognition.

[1381] The following describes in detail the mode for carrying out the present invention. This system comprehensively supports users' shopping experiences in physical stores. The system is mainly configured using the following hardware and software:

[1382] Smart glasses: Acquire location information and provide visual and audio information to the user.

[1383] Server: Generates and manages travel plans and product recommendation information.

[1384] Speech recognition engine: Converts the user's speech into text.

[1385] Emotion recognition engine: Analyzes the user's emotions.

[1386] Main features of the system

[1387] 1. Location information acquisition: The device (smart glasses) acquires its current location using its built-in GPS module. The location information is sent to the server via communication means. The server uses this location information to search for information on nearby tourist spots and stores and provides it to the user.

[1388] 2. Speech Recognition: A microphone on the smart glasses captures the user's voice input and converts it into text using a speech recognition engine (e.g., Google Speech Recognition API). This text information is used to understand the user's requests or questions.

[1389] 3. Acquisition and display of product information: The device uses a camera or scanner to acquire product information. For example, it scans a barcode or QR code to acquire detailed product information. The information is then visually displayed on the smart glasses display by the display means and is provided to the user via audio guidance.

[1390] 4. Emotion Recognition: The smart glasses are equipped with a camera and an emotion recognition engine that analyzes the user's facial expressions and tone of voice to obtain emotional data. This data is then sent to a server and used to provide recommendations based on the user's emotional state.

[1391] 5. Providing recommendation information: The server uses a generative AI model to recommend optimal products based on the user's location, past purchase history, preferences, and emotional data. For example, if a user says, "I want a new smartphone," the voice recognition engine analyzes the request and sends it to the server. The server then generates a recommendation for the optimal smartphone and sends it to the device.

[1392] 6. Translation function: The smart glasses' camera is used to detect and OCR foreign language text, and then the translation API is used to translate it. The translation results are provided to the user visually and audibly.

[1393] Specific examples

[1394] The user puts on the smart glasses and says, "find sneakers." The voice recognition engine converts the request into text and sends it to the server. The server searches for the location of the sneakers in the store, generates navigation information such as "They're in section B, shelf 4," and sends it to the device. The device then provides visual and audio guidance to the user, telling them to "turn right and go straight for about 10 meters." If the user wants to know more about the sneakers, the smart glasses' camera scans the item to obtain and display more information.

[1395] Example prompt for a generative AI model:

[1396] "Build an assistant app that recommends products taking into account the user's emotional data. Include the ability to get the product's location when the user enters the product name, generate navigation instructions, and guide the user through the app via voice."

[1397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1398] Step 1:

[1399] The user puts on the smart glasses and says "find sneakers." This voice input is captured by the smart glasses' microphone. The voice recognition engine converts this voice data into text data. In this case, the input is voice data, and the output is the text data "find sneakers."

[1400] Step 2:

[1401] The terminal sends the converted text data "find sneakers" to the server via a communication means. This process transmits the user's request to the server. The input here is the text data, and the output is the delivery of the request to the server.

[1402] Step 3:

[1403] The server analyzes the received text data and searches for information about "sneakers" in the product database. The search results are specific product information and its location information. The input here is the text data and the product database in the server, and the output is product information including location information, such as "It's in Section B, Shelf 4."

[1404] Step 4:

[1405] The server uses the generative AI model to generate in-store navigation information based on the user's current location and product locations. This information is sent to the terminal. The input here is the user's current location and product locations, and the output is navigation instructions.

[1406] Step 5:

[1407] The device then presents the received navigation information to the user visually and audibly. For example, it may provide guidance such as "Turn right, then go straight for about 10 meters." The input here is navigation information, and the output is visual display and audio guidance.

[1408] Step 6:

[1409] When a user wants to know more information about sneakers, they scan the product's barcode with the camera in their smart glasses. This scanned data is sent to the server by the product information acquisition means. The input here is the barcode data, and the output is a request to the server.

[1410] Step 7:

[1411] The server searches for product information based on the acquired barcode data and sends detailed information to the terminal. The input here is the barcode data and product database, and the output is detailed product information.

[1412] Step 8:

[1413] The terminal receives detailed product information from the server and presents it to the user visually and audibly. For example, it may display and guide the user, saying, "This sneaker is the latest model and costs XYZ yen." The input here is detailed product information, and the output is a visual display and audio guidance.

[1414] Step 9:

[1415] When a user wants to translate a foreign text, they use the smart glasses' camera to capture the text. This captured data is sent to the translation tool, which then performs the translation through OCR processing and a translation API. The input is the captured data, and the output is the translated text.

[1416] Step 10:

[1417] The terminal provides the translation result to the user visually and audibly. For example, it displays "Chicken curry - 500 yen." Here, the input is the translated text data, and the output is the visual display and audio guidance.

[1418] The above processing steps allow users to significantly improve their shopping experience in physical stores.

[1419] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1421] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1422] [Third embodiment]

[1423] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1424] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1426] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1430] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1433] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1435] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[1436] System Overview

[1437] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system is composed of location information acquisition means, communication means, display means, recommendation means, visual guide means, audio guide means, camera, and translation means.

[1438] Program processing overview

[1439] Startup and login

[1440] 1. Startup

[1441] Device: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[1442] 2. Log in

[1443] User: The user enters login information and performs an operation.

[1444] On the device: The entered login information is converted into a security token and sent to the server.

[1445] Server: Checks the login information against a database and returns an authentication token if authentication is successful.

[1446] Obtaining your current location and automatically generating a sightseeing plan

[1447] 3. Get your current location

[1448] Device: The built-in GPS module is activated and acquires the current location. This information is then sent to the server after user confirmation.

[1449] 4. Automatic generation of sightseeing plans

[1450] User: Enters preferences and wishes into the interface.

[1451] Device: The entered information and current location information are sent to the server.

[1452] Server: The generation AI generates the optimal sightseeing plan from the database and sends it back to the device.

[1453] Terminal: Provides the received travel plan to the user visually and audibly.

[1454] Directions to tourist spots

[1455] 5. Destination selection and route guidance

[1456] User: Choose the place you want to go from the presented sightseeing itinerary.

[1457] Terminal: Sends destination information to the server and calculates the optimal route.

[1458] Server: Calculates route information and sends it to the device.

[1459] Terminal: Provides visual and audio route guidance to the user.

[1460] Real-time tourist guide

[1461] 6. Providing detailed information about tourist spots

[1462] Terminal: The camera detects tourist spots and displays them visually. Detailed information is obtained from the server and provided to the user.

[1463] Terminal: Also provides audio guidance to users.

[1464] Example: When a user stands in front of a museum, they are given visual and audio guidance with detailed information about the building's history.

[1465] Providing translation functions

[1466] 7. Translation of text information

[1467] Device: The camera detects text information within the user's field of view.

[1468] Terminal: OCR the text information and send it to the translation API.

[1469] Server: Obtains translation results and sends them to the device.

[1470] Terminal: Provides the user with visual and audio translation results.

[1471] Example: Translating and displaying a restaurant menu.

[1472] Recommendations for nearby information

[1473] 8. Obtaining recommendation information

[1474] User: Submit a request for recommendations.

[1475] Server: Retrieves the best recommendations based on the user's location and preferences.

[1476] Terminal: Provides recommendation information visually and audibly.

[1477] Example: Recommending a popular nearby cafe.

[1478] Experience booking support

[1479] 9. Making a Reservation

[1480] User: Select the experience or restaurant they want.

[1481] Terminal: Sends a reservation request to the server and checks the reservation status.

[1482] Server: Connects to the reservation system, checks, and sends the results to the terminal.

[1483] Terminal: Provides visual and audio confirmation of reservation completion.

[1484] Example: Notifying a customer that a tea time reservation at a cafe has been completed.

[1485] Example

[1486] In a real-world tourism scenario, when a user wears smart glasses and stands at a tourist spot, their current location information is immediately acquired, and the AI ​​automatically generates an optimal sightseeing plan. The user can then follow the guide as they tour the tourist spots, enjoying real-time guidance, translation, and recommendations. All information is provided through the smart glasses' display and audio, allowing users to enjoy sightseeing hands-free.

[1487] This allows users to have an efficient and fulfilling sightseeing experience, freeing them from the language barrier and the hassle of gathering information.

[1488] The processing flow will be explained below.

[1489] Step 1:

[1490] boot

[1491] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[1492] Step 2:

[1493] Log in

[1494] User: The user enters their login information (ID and password) and selects the login button.

[1495] Terminal: Converts the entered login information into a security token and sends it to the server.

[1496] Server: Compares the received login information with the database, and if authentication is successful, issues an authentication token and returns it to the device.

[1497] Step 3:

[1498] Get current location

[1499] Device: Activates the built-in GPS module and obtains the user's current location information (latitude and longitude).

[1500] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[1501] Step 4:

[1502] Sending current location information

[1503] Device: Sends the acquired location information to the server.

[1504] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1505] Step 5:

[1506] Enter your sightseeing plan

[1507] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[1508] Step 6:

[1509] Submitting a plan generation request

[1510] Terminal: Sends a plan generation request including the input sightseeing plan information and current location information to the server.

[1511] Step 7:

[1512] Automatic generation of sightseeing plans

[1513] Server: The AI ​​automatically generates the optimal sightseeing plan based on the user's current location and preferences. The generated sightseeing plan is sent to the device.

[1514] Terminal: Presents the received sightseeing plan to the user with visual and audio guides.

[1515] Step 8:

[1516] Destination selection

[1517] User: Select the destinations they want to visit from the presented itinerary.

[1518] Step 9:

[1519] Sending a route calculation request

[1520] Terminal: Sends a route calculation request including the selected destination information to the server.

[1521] Step 10:

[1522] Route calculation and guidance

[1523] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[1524] Terminal: Provides visual and audio route guidance to the user based on the calculated route information.

[1525] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[1526] Step 11:

[1527] Real-time guide to tourist attractions

[1528] Device: Detects tourist spots within the user's field of view through the smart glasses camera.

[1529] Server: Based on the detected spot information, retrieves detailed information from the database and sends it back to the device.

[1530] Terminal: The acquired detailed information is presented to the user and audio guidance is also provided.

[1531] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[1532] Step 12:

[1533] Translation feature

[1534] Device: Detects text information within the user's field of view through the camera.

[1535] Terminal: OCR the text information and send it to the translation API.

[1536] Server: Translates the source text into the user's native language via the translation API and sends the results to the device.

[1537] Terminal: Presents the translation results to the user visually and audibly.

[1538] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[1539] Step 13:

[1540] Recommendations for nearby information

[1541] User: Sends a request for recommendations.

[1542] Server: Retrieves the most suitable nearby information (tourist attractions, restaurants, etc.) from a database based on the user's current location and preferences.

[1543] Terminal: Provides the acquired recommendation information to the user visually and audibly.

[1544] Example: Information about a popular nearby cafe is displayed, suggesting, "Would you like to take a break here?"

[1545] Step 14:

[1546] Experience booking support

[1547] User: Selects the experience or restaurant they want from the recommendations and submits a reservation request.

[1548] Terminal: Sends a reservation request to the server and asks for availability.

[1549] Server: Connects to the reservation system to check availability and sends the results to the terminal.

[1550] Terminal: Provides visual and audio confirmation of reservation completion.

[1551] Example: A user can enjoy tea time at a cafe at a reserved time.

[1552] Step 15:

[1553] Exit and log out

[1554] User: Select Logout from the menu.

[1555] Terminal: Sends a logout request to the server to end the session.

[1556] Server: Discard the user's session information and notify the user that the logout is complete.

[1557] Terminal: Shut down the system and prepare for next use.

[1558] Example 1

[1559] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1560] Conventional tourist guide systems require users to search for information and plan their own sightseeing itineraries, and they have problems with information gathering and understanding if they do not understand a foreign language. Furthermore, they are unable to provide real-time guidance, translation, and recommendation functions based on the user's current location, limiting the tourist experience.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1562] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for providing visual and audio guides, translation means for detecting and translating text information using a camera, means for transmitting a reservation request to the server and acquiring reservation information, and means for recommending surrounding information. This allows the user to enjoy optimal sightseeing plans in real time, freeing them from the language barrier and the hassle of information gathering, and enabling a more efficient and fulfilling sightseeing experience.

[1563] The "location information acquisition means" is a device or technology for identifying the user's current location and acquiring that location information.

[1564] The "communication means" is an interface for transmitting acquired location information and user requests to a server and receiving data from the server.

[1565] The "display means" is a device for visually displaying to the user the sightseeing plan information and other data provided by the server.

[1566] A "recommendation means" is an algorithm or technology that extracts information suitable for a user from a database stored on a server and provides it to the user.

[1567] A "visual guide means" is a device or technology that visually guides a user to location information and detailed information about tourist spots.

[1568] "Audio guidance means" refers to a device or technology that provides audio guidance of visually displayed information.

[1569] The "translation means" is a technology for translating text information detected using a camera into another language and providing it to the user.

[1570] The "reservation means" is a means for a user to send a reservation request for a desired experience or restaurant to the server and check the reservation status.

[1571] "Recommendation methods" are algorithms and technologies that select and present optimal information to users based on their current location and preferences.

[1572] The present invention relates to a system that enables users to use smart glasses while sightseeing to receive optimal sightseeing plans, route guidance, and guide services in real time. This system is composed of a location information acquisition means, a communication means, a display means, a recommendation means, a visual guide means, an audio guide means, a camera, and a translation means.

[1573] Startup and login

[1574] First, the user turns on the smart glasses. As a terminal, the smart glasses launch the OS and go through initialization processes, displaying the login screen. The user enters login information using the smart glasses' touchpad or voice input. The terminal converts the entered information into a security token and sends it to the server using a communication method. The server references a database to confirm the user's authentication information, and if authentication is successful, returns an authentication token to the terminal.

[1575] Obtaining your current location and automatically generating a sightseeing plan

[1576] The device's built-in GPS module then obtains its current location and, with the user's permission, transmits this information to a server. The user inputs their preferences and the places they want to visit through the smart glasses' interface. The device then transmits this information to the server, which uses a generative AI model to generate an optimal itinerary. The generated itinerary is then sent back to the device and presented to the user visually and audibly.

[1577] Specific examples

[1578] For example, if a user wishes to visit a historical building, the server will search the database for the most suitable sightseeing plan based on the input information and current location information. The sightseeing plan will include nearby historical buildings, recommended restaurants, etc. This information will be provided to the user through the smart glasses' display and audio output.

[1579] Directions to tourist spots

[1580] When a user selects a destination from the provided sightseeing plan, the device sends this information to the server and requests it to calculate the optimal route. The server calculates the optimal route using GPS information and the selected destination information and sends it to the device. The device then provides the user with visual and audio route guidance.

[1581] Real-time tourist guide

[1582] When users arrive at a tourist spot, the smart glasses' camera detects the tourist spot and sends that information to the server. The server then retrieves detailed information about the tourist spot from a database and sends it to the device. This allows users to obtain detailed information about the tourist spot visually and audibly. For example, when users stand in front of a museum, they will be provided with information about the building's history and exhibits.

[1583] Providing translation functions

[1584] When the smart glasses' camera detects text within the user's field of view, it processes it with OCR and sends it to a translation API. The server then retrieves the translation results and sends them to the device. The user can then confirm the translation visually and audibly. For example, a restaurant menu may be translated and displayed.

[1585] Recommendations for nearby information

[1586] When a user sends a recommendation request, the server retrieves the best recommendations based on the user's current location and preferences. The device then presents this information visually and audibly, recommending nearby popular cafes, tourist attractions, and more.

[1587] Experience booking support

[1588] Once the user selects the desired experience and restaurant, the device sends a reservation request to the server and checks the reservation status. The server connects to the reservation system to confirm whether the reservation can be made and sends the result to the device. The device then notifies the user visually and audibly that the reservation has been completed. For example, it notifies the user that a tea time reservation at a cafe has been completed.

[1589] This system allows users to enjoy optimal sightseeing plans in real time, freeing them from language barriers and the hassle of gathering information, enabling a more efficient and fulfilling sightseeing experience.

[1590] Prompt Sentence Examples

[1591] "Please suggest the best sightseeing plan based on the user's current location."

[1592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1593] Step 1:

[1594] boot

[1595] Device: When you start up the smart glasses, the OS starts up and the system is initialized. The login screen is displayed.

[1596] Input: User powers on smart glasses.

[1597] Output: The login screen is displayed.

[1598] What happens: The OS checks various modules to make sure they are OK, then renders the login screen in the UI.

[1599] Step 2:

[1600] Log in

[1601] User: The user enters their login information using the smart glasses interface.

[1602] Terminal: Converts the entered login information into a security token and sends it to the server.

[1603] Server: The server checks the user's authentication information by referencing the database, and if authentication is successful, returns an authentication token to the terminal.

[1604] Input: Login information (user ID and password).

[1605] Output: An authentication token.

[1606] Specific operation: Login information is encrypted into a security token and sent to the server, where it is checked against a database. If authentication is successful, an authentication token is generated and sent back to the device as an HTTP response.

[1607] Step 3:

[1608] Get current location

[1609] On the device: The built-in GPS module is activated and acquires the current location, which is then sent to the server after user confirmation.

[1610] Input: Latitude and longitude information from the GPS module.

[1611] Output: Current location information.

[1612] Specific operation: Captures GPS signals and measures latitude and longitude information. Then, parses this information into JSON format and sends it to the server via a communication method.

[1613] Step 4:

[1614] Automatic generation of sightseeing plans

[1615] User: Enters preferences and wishes using the smart glasses interface.

[1616] Device: Sends input information and current location information to the server.

[1617] Server: The generative AI model generates the optimal sightseeing plan from the database and sends it back to the device.

[1618] Input: Your preferences, wishes, and location.

[1619] Output: Tourist plan.

[1620] How it works: After users input their preferences and wishes, they send that information along with their current location to the server, where the generative AI model generates the best options and returns them as a sightseeing plan.

[1621] Step 5:

[1622] Destination selection and route guidance

[1623] User: Choose the place you want to go from the presented sightseeing itinerary.

[1624] Terminal: Sends the selected destination information to the server and requests it to calculate the optimal route.

[1625] Server: Calculates the optimal route and sends it to the device, which then provides visual and audio guidance to the user.

[1626] Input: Destination information.

[1627] Output: Optimal route.

[1628] Specific operation: Using GPS information and the selected destination, the system calls a map API (e.g., Google Maps API) to calculate the optimal route, sends the calculation results back to the device, and provides audio and visual guidance.

[1629] Step 6:

[1630] Providing detailed information about tourist spots

[1631] Device: The camera in the smart glasses detects tourist spots and sends the information to the server.

[1632] Server: Retrieves detailed information about tourist attractions from the database and sends it to the terminal, which then provides the user with detailed information visually and audibly.

[1633] Input: Video of tourist spot.

[1634] Output: Detailed information about the tourist spot.

[1635] Specific operation: Camera images are analyzed using image recognition technology to identify tourist spots. Information about the identified spots is sent to the server, and the corresponding detailed information is retrieved and presented to the user.

[1636] Step 7:

[1637] Text translation

[1638] Device: The camera detects text information within the user's field of view, performs OCR processing, and sends it to the translation API.

[1639] Server: Receives the translation results and sends them to the device, which then provides the translation results to the user visually and audibly.

[1640] Input: Text information.

[1641] Output: The translation result.

[1642] Specific operation: Extracts text from images using OCR technology, sends it to a translation API (e.g., Google Cloud Translation API), receives the translation result, and notifies you by display and voice.

[1643] Step 8:

[1644] Acquiring recommendation information

[1645] User: Send a recommendation request.

[1646] Server: Retrieves optimal recommendations based on the user's current location and preferences. The device provides the recommendations visually and audibly.

[1647] Input: recommendation request, current location, preferences.

[1648] Output: Recommendation information.

[1649] Specific operation: A database search is performed based on the user's request and current location information, and information on nearby tourist attractions and restaurants is processed using a recommendation algorithm to generate optimal recommendations.

[1650] Step 9:

[1651] Making a reservation

[1652] User: Select the experience or restaurant they want.

[1653] Terminal: Sends a reservation request to the server and checks the reservation status.

[1654] Server: Connects to the reservation system, confirms the reservation, and sends the result to the terminal. The terminal then provides the user with visual and audio information about the completion of the reservation.

[1655] Input: Booking request.

[1656] Output: Booking completion information.

[1657] Specific behavior: Sends a reservation request to the reservation system, checks the reservation status, and notifies the user visually and audibly if the reservation is successful.

[1658] (Application example 1)

[1659] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1660] In modern tourism and shopping experiences, users are required to efficiently obtain and use a large amount of information. However, it is difficult to obtain the necessary information in real time in foreign countries or large stores, and it is often time-consuming to perform complex operations and gather information. Furthermore, it is currently difficult for users to instantly grasp the product information and inventory status they are looking for and receive appropriate guidance. This results in users wasting their valuable time and resulting in an unsatisfactory experience.

[1661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1662] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, a camera module for the user to acquire product information in a physical store, means for transmitting the acquired product information to the server and receiving detailed information, and means for providing the detailed information visually and audibly. This allows the user to acquire necessary information in real time at tourist spots and physical stores and use it efficiently.

[1663] "User" refers to an individual or organization that uses the system to sightsee or shop.

[1664] "Current Location" refers to the geographic location of the user obtained using GPS or other means.

[1665] "Location information acquisition means" refers to a function or device for identifying the user's current location and acquiring that information.

[1666] "Communication means" refers to the functions and devices for sending acquired information to a server and receiving information from the server.

[1667] "Display means" refers to a device or function for visually conveying information received from the server to the user.

[1668] "Recommendation means" refers to a function for recommending highly relevant information and plans to a user from a database stored on a server.

[1669] "Camera Module" refers to a device or feature that includes a camera that allows a user to scan products or attractions.

[1670] "Detailed information" refers to information provided by the server and displayed to the user, including product descriptions and tourist spot information.

[1671] "Visual guide means" refers to devices or functions for visually displaying the location of products or detailed information about tourist attractions.

[1672] "Audio guide means" refers to a function or device for providing the user with audio guidance of displayed information.

[1673] "Stock status" refers to the availability and quantity of the product selected by the user in the store.

[1674] "Reservation" refers to the procedure or function that allows a user to reserve a selected product in advance for later purchase.

[1675] System Overview

[1676] This invention relates to a system for improving users' sightseeing and shopping experiences. The system uses smart glasses to help users obtain necessary information in real time at tourist spots and brick-and-mortar stores and use it efficiently.

[1677] Hardware and software used

[1678] The hardware and software used in this system are as follows:

[1679] GPSModule: A location information acquisition means for acquiring the user's current location.

[1680] CameraModule: A camera module that allows users to scan products and tourist attractions.

[1681] DisplayModule: A display means for visually conveying information received from the server to the user.

[1682] ServerCommunication: A communication method for sending acquired information to a server and receiving information from the server.

[1683] Program processing overview

[1684] The system's main processing steps are as follows: First, the user puts on the smart glasses and starts the system. The system identifies the user's current location using GPS Module and sends this information to the server via ServerCommunication. The server generates optimal sightseeing plans and shopping guides based on the user's current location, past purchase history, and input preference information, and presents them visually to the user via Display Module.

[1685] Furthermore, when a user explores a product in a physical store, the CameraModule is used to scan the product and retrieve details, reviews, stock status, etc. from the server. This information is also provided to the user through the DisplayModule and audio guide.

[1686] Specific examples

[1687] For example, when a user arrives at a tourist spot, the system immediately acquires their current location information and sends it to the server. The server then uses a generative AI model to automatically generate an optimal sightseeing plan for the user, providing it to the user. This plan includes detailed information about the tourist spots the user should visit and directions. When the user stands in front of a museum, they are given visual and audio guidance about the museum's history and exhibits.

[1688] When users scan an item in-store, they can view detailed product information, reviews, and availability in real time. If the item they select is low in stock, the system will suggest a reservation and guide the user through the process.

[1689] Prompt Sentence Examples

[1690] Examples of prompts to input to a generative AI model include:

[1691] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[1692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1693] Step 1:

[1694] Startup and login

[1695] Terminal: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[1696] Input: The user enters their login information (username and password).

[1697] Output: The entered login information is converted into a security token and sent to the server.

[1698] Server: Checks the login information against a database and, if authentication is successful, returns an authentication token.

[1699] Terminal: Receives the authentication token and notifies the user that login was successful.

[1700] Step 2:

[1701] Obtaining your current location and automatically generating a sightseeing plan

[1702] Device: The built-in GPS module starts up and acquires the current location.

[1703] Input: Current location information obtained.

[1704] Output: Sends current location information to the server.

[1705] Server: Receives the current location information sent by the user and generates the optimal sightseeing plan based on the database.

[1706] Terminal: Provides the generated sightseeing plan to the user visually and audibly.

[1707] Step 3:

[1708] Destination selection and route guidance

[1709] User: Choose the place you want to go from the presented sightseeing plan.

[1710] Input: User selected destination information.

[1711] Output: Sends destination information to the server.

[1712] Server: Calculates the optimal route to the selected destination and sends the route information to the device.

[1713] Terminal: Provides visual and audio route guidance to the user.

[1714] Step 4:

[1715] Scanning products and obtaining information

[1716] User: The customer holds the product in their hand and scans it with the camera module on the smart glasses.

[1717] Input: Captured product image.

[1718] Output: Send product images to the server and get detailed information.

[1719] Server: Analyzes the sent product image, retrieves detailed information from the database, and returns it to the terminal.

[1720] Terminal: Provides detailed product information visually and audibly.

[1721] Step 5:

[1722] Check stock availability and reserve

[1723] User: To check the stock status of the product they wish to purchase, they send a stock check request from the details screen.

[1724] Input: Inventory check request information.

[1725] Output: Sends stock status information to the server and receives confirmation results.

[1726] Server: Connects to the inventory system to check the inventory status and sends the results to the terminal.

[1727] Terminal: Provides visual and audio information on stock availability and guides you through the reservation process if one is available.

[1728] Step 6:

[1729] Sales information and coupon recommendations

[1730] Terminal: As users move around the store, they obtain sale and coupon information based on their current location and purchase history.

[1731] Input: Current location information and purchase history information.

[1732] Output: Receive sale information and coupon information from the server.

[1733] Server: Based on the user's current location information and purchase history, extracts sale and coupon information from the database and sends it to the terminal.

[1734] Terminal: Provides acquired sales and coupon information to the user visually and audibly.

[1735] Prompt Sentence Examples

[1736] Examples of prompts to input to a generative AI model include:

[1737] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[1738] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1739] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[1740] System Overview

[1741] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system consists of the following:

[1742] Location information acquisition means

[1743] communication means

[1744] Display means

[1745] Recommendation method

[1746] Visual Guide Means

[1747] Audio guide means

[1748] Translation tools

[1749] Reservation method

[1750] Emotion Recognition Engine

[1751] Program processing overview

[1752] Startup and login

[1753] 1. Startup

[1754] Device: When the smart glasses are started up, the internal system is initialized and the login screen is displayed.

[1755] 2. Log in

[1756] User: The user enters their login information (ID and password) and selects the login button.

[1757] Terminal: Converts the entered login information into a security token and sends it to the server.

[1758] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[1759] Obtaining your current location and automatically generating a sightseeing plan

[1760] 3. Get your current location

[1761] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[1762] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[1763] 4. Sending current location information

[1764] Device: Sends the acquired location information to the server.

[1765] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1766] Acquisition and application of emotion data

[1767] 5. Acquiring Emotion Data

[1768] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[1769] 6. Transmission and analysis of emotional data

[1770] Device: Sends the acquired emotion data to the server.

[1771] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[1772] Coordination and provision of tourism plans

[1773] 7. Generate a sightseeing plan

[1774] User: The user enters their travel preferences and wishes into the interface.

[1775] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[1776] Terminal: Presents the received travel plan to the user visually and audibly.

[1777] Directions and guides to tourist spots

[1778] 8. Destination selection and route guidance

[1779] User: Choose the places they want to visit from the presented itinerary.

[1780] Terminal: Sends destination information to the server and calculates the optimal route.

[1781] Server: Calculates route information and sends it to the device.

[1782] Terminal: Provides visual and audio route guidance.

[1783] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[1784] Real-time guide and translation

[1785] 9. Providing detailed information about tourist attractions

[1786] Terminal: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and presents it.

[1787] Terminal: Also provides audio guidance to the user.

[1788] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[1789] 10. Text Translation

[1790] Device: Detects text in the user's field of view via the camera.

[1791] Terminal: OCR the text information and send it to the translation API.

[1792] Server: Sends the translation results to the device.

[1793] Terminal: Provides visual and audio translation results.

[1794] Example: Translating a restaurant menu to show "Chicken Curry - 500 yen."

[1795] Local information recommendations and experience reservations

[1796] 11. Obtaining local information and making recommendations

[1797] User: Send a recommendation request.

[1798] Server: Makes recommendations by providing peripheral information while taking into account emotional data.

[1799] Terminal: Provides information visually and audibly.

[1800] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[1801] 12. Booking an Experience

[1802] User: Selects the experience or restaurant they want and submits a reservation request.

[1803] Terminal: Sends a reservation request to the server.

[1804] Server: Connects to the reservation system and checks availability.

[1805] Terminal: Provides visual and audio confirmation of reservation completion.

[1806] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[1807] Example

[1808] For example, when a user puts on smart glasses and starts sightseeing, location information is first acquired. Then, a sightseeing plan tailored to the user's needs is generated based on emotional data. When the user asks for directions or detailed information about tourist spots, guidance tailored to the user's current emotional state is provided. This allows the user to enjoy sightseeing without stress.

[1809] If a translation is needed, the smart glasses' camera detects the text and provides an instant translation. Furthermore, if the emotion recognition engine determines that the user is tired, it will provide a recommendation to take a break. In this way, the system comprehensively supports the user's sightseeing experience.

[1810] The processing flow will be explained below.

[1811] Step 1:

[1812] boot

[1813] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[1814] Step 2:

[1815] Log in

[1816] User: The user enters their login information (ID and password) and selects the login button.

[1817] Terminal: Converts the entered login information into a security token and sends it to the server.

[1818] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[1819] Step 3:

[1820] Get current location

[1821] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[1822] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[1823] Step 4:

[1824] Sending current location information

[1825] Device: Sends the acquired location information to the server.

[1826] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1827] Step 5:

[1828] Acquiring emotion data

[1829] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[1830] Step 6:

[1831] Sending and analyzing emotional data

[1832] Device: Sends the acquired emotion data to the server.

[1833] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[1834] Step 7:

[1835] Generate a sightseeing plan

[1836] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[1837] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[1838] Terminal: Presents the received travel plan to the user visually and audibly.

[1839] Step 8:

[1840] Selecting a destination and calculating a route

[1841] User: Select the places they want to visit from the presented itinerary.

[1842] Terminal: Sends destination information to the server and calculates the optimal route.

[1843] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[1844] Terminal: Provides visual and audio route guidance.

[1845] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[1846] Step 9:

[1847] Real-time Guide

[1848] Device: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and displays it.

[1849] Server: Provides detailed information about tourist spots based on the detected spot information.

[1850] Terminal: Provides both visual and audio guidance to the user.

[1851] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[1852] Step 10:

[1853] Translation feature

[1854] Device: Detects text information within the user's field of view through the camera.

[1855] Terminal: OCR the text information and send it to the translation API.

[1856] Server: Sends the translation results to the device.

[1857] Terminal: Provides visual and audio translation results.

[1858] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[1859] Step 11:

[1860] Recommendations for nearby information

[1861] User: Send a recommendation request.

[1862] Server: Obtains optimal surrounding information based on the user's current location and emotional data, and generates recommendation information.

[1863] Terminal: Provides visual and audio recommendations.

[1864] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[1865] Step 12:

[1866] Book an experience

[1867] User: Select the experience or restaurant they want.

[1868] Terminal: Sends a reservation request to the server and checks availability.

[1869] Server: Connects to the reservation system, checks availability, and sends the results to the device.

[1870] Terminal: Provides visual and audio confirmation of reservation completion.

[1871] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[1872] Step 13:

[1873] Exit and log out

[1874] User: Select Logout from the menu.

[1875] Terminal: Sends a logout request to the server to end the session.

[1876] Server: Discard the user's session information and notify the user that the logout is complete.

[1877] Terminal: Shut down the system and prepare for next use.

[1878] Example 2

[1879] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1880] This invention relates to a system for improving the tourism experience. However, conventional tourism systems often lack features such as optimizing sightseeing plans based on the user's real-time emotional state, translation functions, and reservation functions. As a result, users are often limited in the information and support they receive while sightseeing, which can lead to a decrease in satisfaction. Furthermore, the lack of route guidance to tourist spots and detailed guide information is also problematic. Therefore, there is a need for a system that can comprehensively support users' sightseeing experiences and provide high-quality services.

[1881] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1882] In this invention, the server

[1883] A means for acquiring location information;

[1884] means for transmitting the acquired location information to a server;

[1885] a means for displaying the sightseeing plan provided by the server on the terminal;

[1886] A means for obtaining recommendation information from a database;

[1887] means for acquiring user emotion data using an emotion recognition engine;

[1888] means for transmitting the acquired emotion data to a server and analyzing it;

[1889] a means of providing real-time visual and audio route guidance;

[1890] A means of recommending surrounding information;

[1891] a means for visually detecting and translating text;

[1892] a means for submitting a reservation request and providing reservation completion information;

[1893] This allows users to receive optimal sightseeing plans, route guidance, and detailed guide information in real time, and also provides recommendation information and translation services based on their emotional state, greatly improving the user's sightseeing experience.

[1894] "Means for acquiring location information" refers to a device or function that acquires the user's current location using a GPS module or other location measurement technology.

[1895] The "means for transmitting the acquired location information to a server" refers to a device or function for transmitting location information data to a server via Wi-Fi, a cellular network, or other communication technology.

[1896] The "means for displaying the sightseeing plan provided by the server on the terminal" refers to a device or function that visually displays the sightseeing plan information received from the server on the user's smart glasses or mobile device.

[1897] The "means for acquiring recommended information from a database" refers to a device or function that searches for and acquires recommended information related to travel and sightseeing from a database stored on a server.

[1898] "Means for acquiring user emotional data using an emotion recognition engine" refers to a device or function that uses sensors such as a camera or microphone to analyze the user's facial expressions and vocal tone and identify the user's emotional state.

[1899] The "means for transmitting acquired emotion data to a server and analyzing it" refers to a device or function that transmits the user's emotion data to a server and analyzes the data in detail on the server side.

[1900] "Means for providing real-time visual and audio route guidance" refers to a device or function that provides the user with visual and audio guidance in real time on the optimal route to a destination specified by the user.

[1901] A "means for recommending surrounding information" is a device or function that recommends nearby tourist spots and facilities based on the user's current location and emotional state.

[1902] A "means for visually detecting and translating text" is a device or function that uses a camera to detect text in its field of view and then OCRs and translates that text.

[1903] The "means for sending a reservation request and providing reservation completion information" refers to a device or function that sends a reservation request for the user's desired experience or facility to the server and notifies the user that the reservation has been completed.

[1904] The present invention relates to a system that uses smart glasses to significantly improve a user's sightseeing experience. This system is designed to enable users to receive optimal sightseeing plans, route guidance, and guide services in real time while sightseeing.

[1905] System configuration

[1906] This system consists of the following main means:

[1907] 1. Location information acquisition means

[1908] 2. Means of communication

[1909] 3. Display means

[1910] 4. Recommendation method

[1911] 5. Emotion recognition means

[1912] 6. Route guidance methods

[1913] 7. Translation Methods

[1914] 8. Reservation Methods

[1915] Hardware and software used

[1916] Smart glasses (terminal): A device worn by the user. This device is equipped with a GPS module, camera, microphone, display, and speaker.

[1917] Server: The back-end processing device, including the database and generative AI model.

[1918] Generative AI model: Runs on the server side and generates sightseeing plans based on the user's location, emotional data, and preferences.

[1919] Processing flow and specific operations

[1920] Below is an overview of the operation of this system and the specific operations for each process.

[1921] Startup and login

[1922] 1. Startup

[1923] Device: When the smart glasses are turned on, the internal system is first initialized and the login screen is displayed.

[1924] 2. Log in

[1925] User: Enter your login information (ID and password) and select the Login button.

[1926] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[1927] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[1928] Obtaining and sending location information

[1929] 3. Get your current location

[1930] Device: The built-in GPS module is activated and obtains the user's current location information in real time.

[1931] User: A confirmation dialog about obtaining location information will be displayed and the user will select "Allow."

[1932] 4. Sending current location information

[1933] Device: Sends the acquired location information to the server.

[1934] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1935] Acquisition and application of emotion data

[1936] 5. Acquiring Emotion Data

[1937] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[1938] 6. Transmission and analysis of emotional data

[1939] Device: Sends the acquired emotion data to the server.

[1940] Server: Analyzes emotion data, identifies the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan.

[1941] Creating and providing tour plans

[1942] 7. Generate a sightseeing plan

[1943] User: Enters travel preferences and wishes into the interface.

[1944] Server: Uses a generative AI model to generate optimal sightseeing plans based on location, preference, and emotion data.

[1945] Server: Sends the generated travel plan to the device.

[1946] Terminal: Presents the received travel plan to the user visually and audibly.

[1947] Directions and guides to tourist spots

[1948] 8. Destination selection and route guidance

[1949] User: Choose the places they want to visit from the presented itinerary.

[1950] Device: Sends the selected destination information to the server and calculates the optimal route.

[1951] Server: Calculates route information and sends it to the device.

[1952] Terminal: Provides visual and audio route guidance.

[1953] Specific examples of facilities

[1954] For example, when a user puts on smart glasses and begins sightseeing, location information is first acquired. Then, the user's desired sightseeing plan is created using a generative AI model and sent to the device. As the user heads to their selected destination, route guidance is provided in real time. Also, when the user arrives at a tourist spot, the smart glasses' camera detects the spot and displays detailed information. If translation is required, the camera detects the text and the translation result is instantly provided visually and audibly.

[1955] Prompt Sentence Examples

[1956] Examples of prompts for operating this system are shown below.

[1957] "Get your current location"

[1958] "Generate the best sightseeing plan based on user preferences and sentiment data."

[1959] "Please provide more information about the attraction the user is looking at."

[1960] "Book your selected experience"

[1961] "Use your camera to translate any text you see"

[1962] Using these prompts, the generative AI model provides a sightseeing experience that is tailored to the user, ensuring the entire system operates smoothly.

[1963] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1964] Step 1: Boot up and log in

[1965] 1.1 Startup

[1966] Device: When the smart glasses are turned on, they first initialize their internal systems, which includes checking that the CPU, memory, and sensors are working.

[1967] Device: Once initialization is complete, the login screen will appear.

[1968] Input: The user presses the power button.

[1969] Output: The login screen is displayed.

[1970] 1.2 Login

[1971] User: Enter your ID and password on the login screen and press the login button.

[1972] Terminal: The entered login information is encrypted, converted into a security token, and sent to the server.

[1973] Server: Checks the login information against the database, and if authentication is successful, generates an authentication token and sends it back to the device.

[1974] Input: The user enters their ID and password and presses the login button.

[1975] Output: If authentication is successful, the main menu is displayed.

[1976] Step 2: Obtaining and sending location information

[1977] 2.1 Obtaining current location

[1978] Device: Activates the built-in GPS module and obtains the user's current location (latitude and longitude) in real time.

[1979] User: A confirmation dialog about obtaining location information will appear, select "Allow".

[1980] Input: The user selects "Allow" in the location information confirmation dialog.

[1981] Output: Current location information (latitude and longitude) is obtained.

[1982] 2.2 Sending current location information

[1983] Terminal: The acquired current location information is sent to the server in packet format.

[1984] Server: Based on the received location information, search the database for information on nearby tourist spots.

[1985] Input: Current location information (latitude and longitude) obtained by the device.

[1986] Output: The location information is sent to the server, and information about nearby tourist spots is searched.

[1987] Step 3: Acquiring and applying emotion data

[1988] 3.1 Acquiring Emotion Data

[1989] Device: Using the camera and microphone installed in the smart glasses, emotional data is obtained by analyzing the user's facial expressions and tone of voice.

[1990] Input: User's facial expressions and voice captured through camera and microphone.

[1991] Output: The analyzed emotion data is obtained.

[1992] 3.2 Emotional Data Transmission and Analysis

[1993] Device: Sends the acquired emotion data to the server.

[1994] Server: Analyzes the emotion data and identifies the user's state (e.g., joy, excitement, fatigue, etc.).

[1995] Input: The acquired emotion data.

[1996] Output: Emotion data is sent and analyzed on the server.

[1997] Step 4: Generate and provide a tour plan

[1998] 4.1 User Input

[1999] User: Enters travel preferences and wishes into the interface.

[2000] Input: The user enters their travel preferences and wishes into the interface.

[2001] Output: The entered preference information is saved on the device.

[2002] 4.2 Generating a Tourist Plan

[2003] Server: Uses a generative AI model to generate optimal sightseeing plans based on location information, user preferences, and emotional data.

[2004] Input: Location, user preferences, and emotional data.

[2005] Output: The generated itinerary.

[2006] 4.3 Presentation of sightseeing plans

[2007] Server: Sends the generated travel plan to the device.

[2008] Terminal: Presents the received travel plan to the user visually and audibly.

[2009] Input: The generated itinerary.

[2010] Output: The sightseeing plan is presented visually and audibly.

[2011] Step 5: Select a destination and get directions

[2012] 5.1 Selecting a destination

[2013] User: Select the places they want to visit from the presented itinerary.

[2014] Input: The destination selected by the user.

[2015] Output: Selected destination information.

[2016] 5.2 Route calculation

[2017] Terminal: Sends destination information to the server and calculates the optimal route.

[2018] Server: Calculates the optimal route taking into account congestion information and real-time traffic information, and returns the results to the device.

[2019] Input: Selected destination information.

[2020] Output: Calculated route information.

[2021] 5.3 Route guidance

[2022] Device: Provides visual and audio guidance based on the received route.

[2023] Input: Calculated route information.

[2024] Output: Route guidance is provided visually and audibly.

[2025] Step 6: Provide detailed information about the tourist spot

[2026] 6.1 Tourist Spot Detection

[2027] Device: The smart glasses' camera detects and identifies tourist spots.

[2028] Input: Tourist spots detected through the camera.

[2029] Output: Information about detected tourist spots.

[2030] 6.2 Obtain detailed information

[2031] Device: Queries the server for information about the detected tourist spots.

[2032] Server: Retrieves relevant details from the database and sends them to the device.

[2033] Input: Detected tourist spot information.

[2034] Output: The detailed information retrieved.

[2035] 6.3 Detailed information presentation

[2036] Terminal: Provides users with detailed information about tourist attractions through visual and audio means.

[2037] Input: The retrieved details.

[2038] Output: Detailed information about tourist attractions is provided visually and audibly.

[2039] Step 7: Translate the text

[2040] 7.1 Text Detection

[2041] Device: Detects text in the user's field of view via the camera.

[2042] Input: Text information detected through the camera.

[2043] Output: The detected text data.

[2044] 7.2 Translation Process

[2045] Device: OCR the detected text information and send it to the translation API.

[2046] Server: Uses the translation API to return the translated text.

[2047] Input: OCR processed text data.

[2048] Output: The translation result.

[2049] 7.3 Providing translation results

[2050] Terminal: Provides the user with visual and audio translation results.

[2051] Input: Translation result.

[2052] Output: Visual and audio translation results are provided.

[2053] Step 8: Recommend local information and book experiences

[2054] 8.1 Recommendation Request

[2055] User: Request a recommendation.

[2056] Input: The user request.

[2057] Output: The recommendation request is saved to the device.

[2058] 8.2 Providing recommendation information

[2059] Server: Generates nearby recommendation information based on emotion data and sends it to the device.

[2060] Input: Emotion data, peripheral information.

[2061] Output: Recommendation information.

[2062] 8.3 Experience Booking

[2063] User: Selects the experience or restaurant they want and submits a reservation request.

[2064] Terminal: Sends a reservation request to the server.

[2065] Server: Connects to the reservation system and checks availability.

[2066] Terminal: Provides visual and audio confirmation of reservation completion.

[2067] Input: User's booking request.

[2068] Output: Booking completion information is provided.

[2069] (Application example 2)

[2070] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2071] Conventional tourism support systems have limited ways for users to obtain information about tourist destinations, requiring a great deal of effort to obtain information about the destinations they are visiting. Furthermore, if users do not understand the local language, it is difficult to understand information about tourist destinations and prices. Furthermore, there is a lack of product information, price comparisons, and navigation functions within commercial facilities, limiting the user's shopping experience.

[2072] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for acquiring and displaying information about products, means for converting the user's voice input into text using voice recognition, and means for recommending products based on the acquired text information. This not only enables the user to easily acquire information about destinations, but also enables the user to understand tourist destination and product information across language barriers, and enjoy a pleasant shopping experience in commercial facilities.

[2073] The "location information acquisition means" is a device or software for acquiring the user's current location information.

[2074] "Communication means" refers to a device or protocol for transmitting acquired location information and other data to a server.

[2075] The "display means" refers to a display device or software for displaying information provided by the server on the terminal.

[2076] The "recommendation means" is an algorithm or system for obtaining recommendation information from a database stored on a server.

[2077] "Means for acquiring and displaying product information" refers to devices or software that acquire detailed product information and provide it to users visually or audibly.

[2078] "Means for converting a user's voice input into text using voice recognition" refers to technology for recognizing a user's voice and converting it into text data.

[2079] The "means for recommending products based on acquired text information" is a system for recommending related products to a user based on text information acquired by speech recognition.

[2080] The following describes in detail the mode for carrying out the present invention. This system comprehensively supports users' shopping experiences in physical stores. The system is mainly configured using the following hardware and software:

[2081] Smart glasses: Acquire location information and provide visual and audio information to the user.

[2082] Server: Generates and manages travel plans and product recommendation information.

[2083] Speech recognition engine: Converts the user's speech into text.

[2084] Emotion recognition engine: Analyzes the user's emotions.

[2085] Main features of the system

[2086] 1. Location information acquisition: The device (smart glasses) acquires its current location using its built-in GPS module. The location information is sent to the server via communication means. The server uses this location information to search for information on nearby tourist spots and stores and provides it to the user.

[2087] 2. Speech Recognition: A microphone on the smart glasses captures the user's voice input and converts it into text using a speech recognition engine (e.g., Google Speech Recognition API). This text information is used to understand the user's requests or questions.

[2088] 3. Acquisition and display of product information: The device uses a camera or scanner to acquire product information. For example, it scans a barcode or QR code to acquire detailed product information. The information is then visually displayed on the smart glasses display by the display means and is provided to the user via audio guidance.

[2089] 4. Emotion Recognition: The smart glasses are equipped with a camera and an emotion recognition engine that analyzes the user's facial expressions and tone of voice to obtain emotional data. This data is then sent to a server and used to provide recommendations based on the user's emotional state.

[2090] 5. Providing recommendation information: The server uses a generative AI model to recommend optimal products based on the user's location, past purchase history, preferences, and emotional data. For example, if a user says, "I want a new smartphone," the voice recognition engine analyzes the request and sends it to the server. The server then generates a recommendation for the optimal smartphone and sends it to the device.

[2091] 6. Translation function: The smart glasses' camera is used to detect and OCR foreign language text, and then the translation API is used to translate it. The translation results are provided to the user visually and audibly.

[2092] Specific examples

[2093] The user puts on the smart glasses and says, "find sneakers." The voice recognition engine converts the request into text and sends it to the server. The server searches for the location of the sneakers in the store, generates navigation information such as "They're in section B, shelf 4," and sends it to the device. The device then provides visual and audio guidance to the user, telling them to "turn right and go straight for about 10 meters." If the user wants to know more about the sneakers, the smart glasses' camera scans the item to obtain and display more information.

[2094] Example prompt for a generative AI model:

[2095] "Build an assistant app that recommends products taking into account the user's emotional data. Include the ability to get the product's location when the user enters the product name, generate navigation instructions, and guide the user through the app via voice."

[2096] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2097] Step 1:

[2098] The user puts on the smart glasses and says "find sneakers." This voice input is captured by the smart glasses' microphone. The voice recognition engine converts this voice data into text data. In this case, the input is voice data, and the output is the text data "find sneakers."

[2099] Step 2:

[2100] The terminal sends the converted text data "find sneakers" to the server via a communication means. This process transmits the user's request to the server. The input here is the text data, and the output is the delivery of the request to the server.

[2101] Step 3:

[2102] The server analyzes the received text data and searches for information about "sneakers" in the product database. The search results are specific product information and its location information. The input here is the text data and the product database in the server, and the output is product information including location information, such as "It's in Section B, Shelf 4."

[2103] Step 4:

[2104] The server uses the generative AI model to generate in-store navigation information based on the user's current location and product locations. This information is sent to the terminal. The input here is the user's current location and product locations, and the output is navigation instructions.

[2105] Step 5:

[2106] The device then presents the received navigation information to the user visually and audibly. For example, it may provide guidance such as "Turn right, then go straight for about 10 meters." The input here is navigation information, and the output is visual display and audio guidance.

[2107] Step 6:

[2108] When a user wants to know more information about sneakers, they scan the product's barcode with the camera in their smart glasses. This scanned data is sent to the server by the product information acquisition means. The input here is the barcode data, and the output is a request to the server.

[2109] Step 7:

[2110] The server searches for product information based on the acquired barcode data and sends detailed information to the terminal. The input here is the barcode data and product database, and the output is detailed product information.

[2111] Step 8:

[2112] The terminal receives detailed product information from the server and presents it to the user visually and audibly. For example, it may display and guide the user, saying, "This sneaker is the latest model and costs XYZ yen." The input here is detailed product information, and the output is a visual display and audio guidance.

[2113] Step 9:

[2114] When a user wants to translate a foreign text, they use the smart glasses' camera to capture the text. This captured data is sent to the translation tool, which then performs the translation through OCR processing and a translation API. The input is the captured data, and the output is the translated text.

[2115] Step 10:

[2116] The terminal provides the translation result to the user visually and audibly. For example, it displays "Chicken curry - 500 yen." Here, the input is the translated text data, and the output is the visual display and audio guidance.

[2117] The above processing steps allow users to significantly improve their shopping experience in physical stores.

[2118] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2119] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2120] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2121] [Fourth embodiment]

[2122] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2123] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2124] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2125] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2126] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2127] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2128] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2129] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2130] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2131] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2133] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2134] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2135] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[2136] System Overview

[2137] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system is composed of location information acquisition means, communication means, display means, recommendation means, visual guide means, audio guide means, camera, and translation means.

[2138] Program processing overview

[2139] Startup and login

[2140] 1. Startup

[2141] Device: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[2142] 2. Log in

[2143] User: The user enters login information and performs an operation.

[2144] On the device: The entered login information is converted into a security token and sent to the server.

[2145] Server: Checks the login information against a database and returns an authentication token if authentication is successful.

[2146] Obtaining your current location and automatically generating a sightseeing plan

[2147] 3. Get your current location

[2148] Device: The built-in GPS module is activated and acquires the current location. This information is then sent to the server after user confirmation.

[2149] 4. Automatic generation of sightseeing plans

[2150] User: Enters preferences and wishes into the interface.

[2151] Device: The entered information and current location information are sent to the server.

[2152] Server: The generation AI generates the optimal sightseeing plan from the database and sends it back to the device.

[2153] Terminal: Provides the received travel plan to the user visually and audibly.

[2154] Directions to tourist spots

[2155] 5. Destination selection and route guidance

[2156] User: Choose the place you want to go from the presented sightseeing itinerary.

[2157] Terminal: Sends destination information to the server and calculates the optimal route.

[2158] Server: Calculates route information and sends it to the device.

[2159] Terminal: Provides visual and audio route guidance to the user.

[2160] Real-time tourist guide

[2161] 6. Providing detailed information about tourist spots

[2162] Terminal: The camera detects tourist spots and displays them visually. Detailed information is obtained from the server and provided to the user.

[2163] Terminal: Also provides audio guidance to users.

[2164] Example: When a user stands in front of a museum, they are given visual and audio guidance with detailed information about the building's history.

[2165] Providing translation functions

[2166] 7. Translation of text information

[2167] Device: The camera detects text information within the user's field of view.

[2168] Terminal: OCR the text information and send it to the translation API.

[2169] Server: Obtains translation results and sends them to the device.

[2170] Terminal: Provides the user with visual and audio translation results.

[2171] Example: Translating and displaying a restaurant menu.

[2172] Recommendations for nearby information

[2173] 8. Obtaining recommendation information

[2174] User: Submit a request for recommendations.

[2175] Server: Retrieves the best recommendations based on the user's location and preferences.

[2176] Terminal: Provides recommendation information visually and audibly.

[2177] Example: Recommending a popular nearby cafe.

[2178] Experience booking support

[2179] 9. Making a Reservation

[2180] User: Select the experience or restaurant they want.

[2181] Terminal: Sends a reservation request to the server and checks the reservation status.

[2182] Server: Connects to the reservation system, checks, and sends the results to the terminal.

[2183] Terminal: Provides visual and audio confirmation of reservation completion.

[2184] Example: Notifying a customer that a tea time reservation at a cafe has been completed.

[2185] Example

[2186] In a real-world tourism scenario, when a user wears smart glasses and stands at a tourist spot, their current location information is immediately acquired, and the AI ​​automatically generates an optimal sightseeing plan. The user can then follow the guide as they tour the tourist spots, enjoying real-time guidance, translation, and recommendations. All information is provided through the smart glasses' display and audio, allowing users to enjoy sightseeing hands-free.

[2187] This allows users to have an efficient and fulfilling sightseeing experience, freeing them from the language barrier and the hassle of gathering information.

[2188] The processing flow will be explained below.

[2189] Step 1:

[2190] boot

[2191] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[2192] Step 2:

[2193] Log in

[2194] User: The user enters their login information (ID and password) and selects the login button.

[2195] Terminal: Converts the entered login information into a security token and sends it to the server.

[2196] Server: Compares the received login information with the database, and if authentication is successful, issues an authentication token and returns it to the device.

[2197] Step 3:

[2198] Get current location

[2199] Device: Activates the built-in GPS module and obtains the user's current location information (latitude and longitude).

[2200] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[2201] Step 4:

[2202] Sending current location information

[2203] Device: Sends the acquired location information to the server.

[2204] Server: Based on the received location information, search the database for information on nearby tourist spots.

[2205] Step 5:

[2206] Enter your sightseeing plan

[2207] User: Enters information such as travel preferences and hobbies (e.g., historical sites, museums, nature) into the smartglasses interface.

[2208] Step 6:

[2209] Submitting a plan generation request

[2210] Terminal: Sends a plan generation request including the input sightseeing plan information and current location information to the server.

[2211] Step 7:

[2212] Automatic generation of sightseeing plans

[2213] Server: The AI ​​automatically generates the optimal sightseeing plan based on the user's current location and preferences. The generated sightseeing plan is sent to the device.

[2214] Terminal: Presents the received sightseeing plan to the user with visual and audio guides.

[2215] Step 8:

[2216] Destination selection

[2217] User: Select the destinations they want to visit from the presented itinerary.

[2218] Step 9:

[2219] Sending a route calculation request

[2220] Terminal: Sends a route calculation request including the selected destination information to the server.

[2221] Step 10:

[2222] Route calculation and guidance

[2223] Server: Calculates the optimal route from the current location to the destination and sends the route information to the device.

[2224] Terminal: Provides visual and audio route guidance to the user based on the calculated route information.

[2225] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[2226] Step 11:

[2227] Real-time guide to tourist attractions

[2228] Device: Detects tourist spots within the user's field of view through the smart glasses camera.

[2229] Server: Based on the detected spot information, retrieves detailed information from the database and sends it back to the device.

[2230] Terminal: The acquired detailed information is presented to the user and audio guidance is also provided.

[2231] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[2232] Step 12:

[2233] Translation feature

[2234] Device: Detects text information within the user's field of view through the camera.

[2235] Terminal: OCR the text information and send it to the translation API.

[2236] Server: Translates the source text into the user's native language via the translation API and sends the results to the device.

[2237] Terminal: Presents the translation results to the user visually and audibly.

[2238] Example: Translating a local restaurant menu to show "Chicken Curry - 500 yen."

[2239] Step 13:

[2240] Recommendations for nearby information

[2241] User: Sends a request for recommendations.

[2242] Server: Retrieves the most suitable nearby information (tourist attractions, restaurants, etc.) from a database based on the user's current location and preferences.

[2243] Terminal: Provides the acquired recommendation information to the user visually and audibly.

[2244] Example: Information about a popular nearby cafe is displayed, suggesting, "Would you like to take a break here?"

[2245] Step 14:

[2246] Experience booking support

[2247] User: Selects the experience or restaurant they want from the recommendations and submits a reservation request.

[2248] Terminal: Sends a reservation request to the server and asks for availability.

[2249] Server: Connects to the reservation system to check availability and sends the results to the terminal.

[2250] Terminal: Provides visual and audio confirmation of reservation completion.

[2251] Example: A user can enjoy tea time at a cafe at a reserved time.

[2252] Step 15:

[2253] Exit and log out

[2254] User: Select Logout from the menu.

[2255] Terminal: Sends a logout request to the server to end the session.

[2256] Server: Discard the user's session information and notify the user that the logout is complete.

[2257] Terminal: Shut down the system and prepare for next use.

[2258] Example 1

[2259] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2260] Conventional tourist guide systems require users to search for information and plan their own sightseeing itineraries, and they have problems with information gathering and understanding if they do not understand a foreign language. Furthermore, they are unable to provide real-time guidance, translation, and recommendation functions based on the user's current location, limiting the tourist experience.

[2261] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2262] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, means for providing visual and audio guides, translation means for detecting and translating text information using a camera, means for transmitting a reservation request to the server and acquiring reservation information, and means for recommending surrounding information. This allows the user to enjoy optimal sightseeing plans in real time, freeing them from the language barrier and the hassle of information gathering, and enabling a more efficient and fulfilling sightseeing experience.

[2263] The "location information acquisition means" is a device or technology for identifying the user's current location and acquiring that location information.

[2264] The "communication means" is an interface for transmitting acquired location information and user requests to a server and receiving data from the server.

[2265] The "display means" is a device for visually displaying to the user the sightseeing plan information and other data provided by the server.

[2266] A "recommendation means" is an algorithm or technology that extracts information suitable for a user from a database stored on a server and provides it to the user.

[2267] A "visual guide means" is a device or technology that visually guides a user to location information and detailed information about tourist spots.

[2268] "Audio guidance means" refers to a device or technology that provides audio guidance of visually displayed information.

[2269] The "translation means" is a technology for translating text information detected using a camera into another language and providing it to the user.

[2270] The "reservation means" is a means for a user to send a reservation request for a desired experience or restaurant to the server and check the reservation status.

[2271] "Recommendation methods" are algorithms and technologies that select and present optimal information to users based on their current location and preferences.

[2272] The present invention relates to a system that enables users to use smart glasses while sightseeing to receive optimal sightseeing plans, route guidance, and guide services in real time. This system is composed of a location information acquisition means, a communication means, a display means, a recommendation means, a visual guide means, an audio guide means, a camera, and a translation means.

[2273] Startup and login

[2274] First, the user turns on the smart glasses. As a terminal, the smart glasses launch the OS and go through initialization processes, displaying the login screen. The user enters login information using the smart glasses' touchpad or voice input. The terminal converts the entered information into a security token and sends it to the server using a communication method. The server references a database to confirm the user's authentication information, and if authentication is successful, returns an authentication token to the terminal.

[2275] Obtaining your current location and automatically generating a sightseeing plan

[2276] The device's built-in GPS module then obtains its current location and, with the user's permission, transmits this information to a server. The user inputs their preferences and the places they want to visit through the smart glasses' interface. The device then transmits this information to the server, which uses a generative AI model to generate an optimal itinerary. The generated itinerary is then sent back to the device and presented to the user visually and audibly.

[2277] Specific examples

[2278] For example, if a user wishes to visit a historical building, the server will search the database for the most suitable sightseeing plan based on the input information and current location information. The sightseeing plan will include nearby historical buildings, recommended restaurants, etc. This information will be provided to the user through the smart glasses' display and audio output.

[2279] Directions to tourist spots

[2280] When a user selects a destination from the provided sightseeing plan, the device sends this information to the server and requests it to calculate the optimal route. The server calculates the optimal route using GPS information and the selected destination information and sends it to the device. The device then provides the user with visual and audio route guidance.

[2281] Real-time tourist guide

[2282] When users arrive at a tourist spot, the smart glasses' camera detects the tourist spot and sends that information to the server. The server then retrieves detailed information about the tourist spot from a database and sends it to the device. This allows users to obtain detailed information about the tourist spot visually and audibly. For example, when users stand in front of a museum, they will be provided with information about the building's history and exhibits.

[2283] Providing translation functions

[2284] When the smart glasses' camera detects text within the user's field of view, it processes it with OCR and sends it to a translation API. The server then retrieves the translation results and sends them to the device. The user can then confirm the translation visually and audibly. For example, a restaurant menu may be translated and displayed.

[2285] Recommendations for nearby information

[2286] When a user sends a recommendation request, the server retrieves the best recommendations based on the user's current location and preferences. The device then presents this information visually and audibly, recommending nearby popular cafes, tourist attractions, and more.

[2287] Experience booking support

[2288] Once the user selects the desired experience and restaurant, the device sends a reservation request to the server and checks the reservation status. The server connects to the reservation system to confirm whether the reservation can be made and sends the result to the device. The device then notifies the user visually and audibly that the reservation has been completed. For example, it notifies the user that a tea time reservation at a cafe has been completed.

[2289] This system allows users to enjoy optimal sightseeing plans in real time, freeing them from language barriers and the hassle of gathering information, enabling a more efficient and fulfilling sightseeing experience.

[2290] Prompt Sentence Examples

[2291] "Please suggest the best sightseeing plan based on the user's current location."

[2292] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2293] Step 1:

[2294] boot

[2295] Device: When you start up the smart glasses, the OS starts up and the system is initialized. The login screen is displayed.

[2296] Input: User powers on smart glasses.

[2297] Output: The login screen is displayed.

[2298] What happens: The OS checks various modules to make sure they are OK, then renders the login screen in the UI.

[2299] Step 2:

[2300] Log in

[2301] User: The user enters their login information using the smart glasses interface.

[2302] Terminal: Converts the entered login information into a security token and sends it to the server.

[2303] Server: The server checks the user's authentication information by referencing the database, and if authentication is successful, returns an authentication token to the terminal.

[2304] Input: Login information (user ID and password).

[2305] Output: An authentication token.

[2306] Specific operation: Login information is encrypted into a security token and sent to the server, where it is checked against a database. If authentication is successful, an authentication token is generated and sent back to the device as an HTTP response.

[2307] Step 3:

[2308] Get current location

[2309] On the device: The built-in GPS module is activated and acquires the current location, which is then sent to the server after user confirmation.

[2310] Input: Latitude and longitude information from the GPS module.

[2311] Output: Current location information.

[2312] Specific operation: Captures GPS signals and measures latitude and longitude information. Then, parses this information into JSON format and sends it to the server via a communication method.

[2313] Step 4:

[2314] Automatic generation of sightseeing plans

[2315] User: Enters preferences and wishes using the smart glasses interface.

[2316] Device: Sends input information and current location information to the server.

[2317] Server: The generative AI model generates the optimal sightseeing plan from the database and sends it back to the device.

[2318] Input: Your preferences, wishes, and location.

[2319] Output: Tourist plan.

[2320] How it works: After users input their preferences and wishes, they send that information along with their current location to the server, where the generative AI model generates the best options and returns them as a sightseeing plan.

[2321] Step 5:

[2322] Destination selection and route guidance

[2323] User: Choose the place you want to go from the presented sightseeing itinerary.

[2324] Terminal: Sends the selected destination information to the server and requests it to calculate the optimal route.

[2325] Server: Calculates the optimal route and sends it to the device, which then provides visual and audio guidance to the user.

[2326] Input: Destination information.

[2327] Output: Optimal route.

[2328] Specific operation: Using GPS information and the selected destination, the system calls a map API (e.g., Google Maps API) to calculate the optimal route, sends the calculation results back to the device, and provides audio and visual guidance.

[2329] Step 6:

[2330] Providing detailed information about tourist spots

[2331] Device: The camera in the smart glasses detects tourist spots and sends the information to the server.

[2332] Server: Retrieves detailed information about tourist attractions from the database and sends it to the terminal, which then provides the user with detailed information visually and audibly.

[2333] Input: Video of tourist spot.

[2334] Output: Detailed information about the tourist spot.

[2335] Specific operation: Camera images are analyzed using image recognition technology to identify tourist spots. Information about the identified spots is sent to the server, and the corresponding detailed information is retrieved and presented to the user.

[2336] Step 7:

[2337] Text translation

[2338] Device: The camera detects text information within the user's field of view, performs OCR processing, and sends it to the translation API.

[2339] Server: Receives the translation results and sends them to the device, which then provides the translation results to the user visually and audibly.

[2340] Input: Text information.

[2341] Output: The translation result.

[2342] Specific operation: Extracts text from images using OCR technology, sends it to a translation API (e.g., Google Cloud Translation API), receives the translation result, and notifies you by display and voice.

[2343] Step 8:

[2344] Acquiring recommendation information

[2345] User: Send a recommendation request.

[2346] Server: Retrieves optimal recommendations based on the user's current location and preferences. The device provides the recommendations visually and audibly.

[2347] Input: recommendation request, current location, preferences.

[2348] Output: Recommendation information.

[2349] Specific operation: A database search is performed based on the user's request and current location information, and information on nearby tourist attractions and restaurants is processed using a recommendation algorithm to generate optimal recommendations.

[2350] Step 9:

[2351] Making a reservation

[2352] User: Select the experience or restaurant they want.

[2353] Terminal: Sends a reservation request to the server and checks the reservation status.

[2354] Server: Connects to the reservation system, confirms the reservation, and sends the result to the terminal. The terminal then provides the user with visual and audio information about the completion of the reservation.

[2355] Input: Booking request.

[2356] Output: Booking completion information.

[2357] Specific behavior: Sends a reservation request to the reservation system, checks the reservation status, and notifies the user visually and audibly if the reservation is successful.

[2358] (Application example 1)

[2359] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2360] In modern tourism and shopping experiences, users are required to efficiently obtain and use a large amount of information. However, it is difficult to obtain the necessary information in real time in foreign countries or large stores, and it is often time-consuming to perform complex operations and gather information. Furthermore, it is currently difficult for users to instantly grasp the product information and inventory status they are looking for and receive appropriate guidance. This results in users wasting their valuable time and resulting in an unsatisfactory experience.

[2361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2362] In this invention, the server includes location information acquisition means for acquiring the user's current location, communication means for transmitting the acquired current location information to the server, display means for displaying on the terminal sightseeing plan information provided by the server, recommendation means for acquiring recommended information from a database stored in the server, a camera module for the user to acquire product information in a physical store, means for transmitting the acquired product information to the server and receiving detailed information, and means for providing the detailed information visually and audibly. This allows the user to acquire necessary information in real time at tourist spots and physical stores and use it efficiently.

[2363] "User" refers to an individual or organization that uses the system to sightsee or shop.

[2364] "Current Location" refers to the geographic location of the user obtained using GPS or other means.

[2365] "Location information acquisition means" refers to a function or device for identifying the user's current location and acquiring that information.

[2366] "Communication means" refers to the functions and devices for sending acquired information to a server and receiving information from the server.

[2367] "Display means" refers to a device or function for visually conveying information received from the server to the user.

[2368] "Recommendation means" refers to a function for recommending highly relevant information and plans to a user from a database stored on a server.

[2369] "Camera Module" refers to a device or feature that includes a camera that allows a user to scan products or attractions.

[2370] "Detailed information" refers to information provided by the server and displayed to the user, including product descriptions and tourist spot information.

[2371] "Visual guide means" refers to devices or functions for visually displaying the location of products or detailed information about tourist attractions.

[2372] "Audio guide means" refers to a function or device for providing the user with audio guidance of displayed information.

[2373] "Stock status" refers to the availability and quantity of the product selected by the user in the store.

[2374] "Reservation" refers to the procedure or function that allows a user to reserve a selected product in advance for later purchase.

[2375] System Overview

[2376] This invention relates to a system for improving users' sightseeing and shopping experiences. The system uses smart glasses to help users obtain necessary information in real time at tourist spots and brick-and-mortar stores and use it efficiently.

[2377] Hardware and software used

[2378] The hardware and software used in this system are as follows:

[2379] GPSModule: A location information acquisition means for acquiring the user's current location.

[2380] CameraModule: A camera module that allows users to scan products and tourist attractions.

[2381] DisplayModule: A display means for visually conveying information received from the server to the user.

[2382] ServerCommunication: A communication method for sending acquired information to a server and receiving information from the server.

[2383] Program processing overview

[2384] The system's main processing steps are as follows: First, the user puts on the smart glasses and starts the system. The system identifies the user's current location using GPS Module and sends this information to the server via ServerCommunication. The server generates optimal sightseeing plans and shopping guides based on the user's current location, past purchase history, and input preference information, and presents them visually to the user via Display Module.

[2385] Furthermore, when a user explores a product in a physical store, the CameraModule is used to scan the product and retrieve details, reviews, stock status, etc. from the server. This information is also provided to the user through the DisplayModule and audio guide.

[2386] Specific examples

[2387] For example, when a user arrives at a tourist spot, the system immediately acquires their current location information and sends it to the server. The server then uses a generative AI model to automatically generate an optimal sightseeing plan for the user, providing it to the user. This plan includes detailed information about the tourist spots the user should visit and directions. When the user stands in front of a museum, they are given visual and audio guidance about the museum's history and exhibits.

[2388] When users scan an item in-store, they can view detailed product information, reviews, and availability in real time. If the item they select is low in stock, the system will suggest a reservation and guide the user through the process.

[2389] Prompt Sentence Examples

[2390] Examples of prompts to input to a generative AI model include:

[2391] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[2392] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2393] Step 1:

[2394] Startup and login

[2395] Terminal: When the smart glasses start up, the internal system is initialized and the login screen is displayed.

[2396] Input: The user enters their login information (username and password).

[2397] Output: The entered login information is converted into a security token and sent to the server.

[2398] Server: Checks the login information against a database and, if authentication is successful, returns an authentication token.

[2399] Terminal: Receives the authentication token and notifies the user that login was successful.

[2400] Step 2:

[2401] Obtaining your current location and automatically generating a sightseeing plan

[2402] Device: The built-in GPS module starts up and acquires the current location.

[2403] Input: Current location information obtained.

[2404] Output: Sends current location information to the server.

[2405] Server: Receives the current location information sent by the user and generates the optimal sightseeing plan based on the database.

[2406] Terminal: Provides the generated sightseeing plan to the user visually and audibly.

[2407] Step 3:

[2408] Destination selection and route guidance

[2409] User: Choose the place you want to go from the presented sightseeing plan.

[2410] Input: User selected destination information.

[2411] Output: Sends destination information to the server.

[2412] Server: Calculates the optimal route to the selected destination and sends the route information to the device.

[2413] Terminal: Provides visual and audio route guidance to the user.

[2414] Step 4:

[2415] Scanning products and obtaining information

[2416] User: The customer holds the product in their hand and scans it with the camera module on the smart glasses.

[2417] Input: Captured product image.

[2418] Output: Send product images to the server and get detailed information.

[2419] Server: Analyzes the sent product image, retrieves detailed information from the database, and returns it to the terminal.

[2420] Terminal: Provides detailed product information visually and audibly.

[2421] Step 5:

[2422] Check stock availability and reserve

[2423] User: To check the stock status of the product they wish to purchase, they send a stock check request from the details screen.

[2424] Input: Inventory check request information.

[2425] Output: Sends stock status information to the server and receives confirmation results.

[2426] Server: Connects to the inventory system to check the inventory status and sends the results to the terminal.

[2427] Terminal: Provides visual and audio information on stock availability and guides you through the reservation process if one is available.

[2428] Step 6:

[2429] Sales information and coupon recommendations

[2430] Terminal: As users move around the store, they obtain sale and coupon information based on their current location and purchase history.

[2431] Input: Current location information and purchase history information.

[2432] Output: Receive sale information and coupon information from the server.

[2433] Server: Based on the user's current location information and purchase history, extracts sale and coupon information from the database and sends it to the terminal.

[2434] Terminal: Provides acquired sales and coupon information to the user visually and audibly.

[2435] Prompt Sentence Examples

[2436] Examples of prompts to input to a generative AI model include:

[2437] "Design your application so that users can scan products using smart glasses and view detailed information and reviews. For example, use the smart glasses' camera to read the QR code on a product and retrieve the information from the server and display it on the display."

[2438] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2439] This section describes specific embodiments of the present invention. The present invention relates to a system using smart glasses that significantly improves the user's sightseeing experience. The main processing flow of the system and the specific operations performed in each process are described below.

[2440] System Overview

[2441] This system is designed to enable users to use smart glasses while sightseeing to receive real-time optimal sightseeing plans, route guidance, and guide services. The system consists of the following:

[2442] Location information acquisition means

[2443] communication means

[2444] Display means

[2445] Recommendation method

[2446] Visual Guide Means

[2447] Audio guide means

[2448] Translation tools

[2449] Reservation method

[2450] Emotion Recognition Engine

[2451] Program processing overview

[2452] Startup and login

[2453] 1. Startup

[2454] Device: When the smart glasses are started up, the internal system is initialized and the login screen is displayed.

[2455] 2. Log in

[2456] User: The user enters their login information (ID and password) and selects the login button.

[2457] Terminal: Converts the entered login information into a security token and sends it to the server.

[2458] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[2459] Obtaining your current location and automatically generating a sightseeing plan

[2460] 3. Get your current location

[2461] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longitude).

[2462] User: A confirmation dialog about obtaining location information will be displayed and the user will select to allow it.

[2463] 4. Sending current location information

[2464] Device: Sends the acquired location information to the server.

[2465] Server: Based on the received location information, search the database for information on nearby tourist spots.

[2466] Acquisition and application of emotion data

[2467] 5. Acquiring Emotion Data

[2468] Device: The emotion recognition engine installed in the smart glasses analyzes the user's facial expressions and tone of voice to obtain emotional data.

[2469] 6. Transmission and analysis of emotional data

[2470] Device: Sends the acquired emotion data to the server.

[2471] Server: Analyzes emotion data to identify the user's state (happiness, excitement, fatigue, etc.), and dynamically adjusts the sightseeing plan accordingly.

[2472] Coordination and provision of tourism plans

[2473] 7. Generate a sightseeing plan

[2474] User: The user enters their travel preferences and wishes into the interface.

[2475] Server: The AI ​​generates the optimal sightseeing plan based on location, preference, and emotional data. The plan is then sent to the device.

[2476] Terminal: Presents the received travel plan to the user visually and audibly.

[2477] Directions and guides to tourist spots

[2478] 8. Destination selection and route guidance

[2479] User: Choose the places they want to visit from the presented itinerary.

[2480] Terminal: Sends destination information to the server and calculates the optimal route.

[2481] Server: Calculates route information and sends it to the device.

[2482] Terminal: Provides visual and audio route guidance.

[2483] Example: Display "Go straight for 200 meters and turn right at the next intersection."

[2484] Real-time guide and translation

[2485] 9. Providing detailed information about tourist attractions

[2486] Terminal: Detects tourist spots through the smart glasses camera and displays them visually. Retrieves detailed information from the server and presents it.

[2487] Terminal: Also provides audio guidance to the user.

[2488] Example: When a user arrives in front of a historic building, they are given visual and audio information about the building.

[2489] 10. Text Translation

[2490] Device: Detects text in the user's field of view via the camera.

[2491] Terminal: OCR the text information and send it to the translation API.

[2492] Server: Sends the translation results to the device.

[2493] Terminal: Provides visual and audio translation results.

[2494] Example: Translating a restaurant menu to show "Chicken Curry - 500 yen."

[2495] Local information recommendations and experience reservations

[2496] 11. Obtaining local information and making recommendations

[2497] User: Send a recommendation request.

[2498] Server: Makes recommendations by providing peripheral information while taking into account emotional data.

[2499] Terminal: Provides information visually and audibly.

[2500] Example: If the user is suspected to be tired, suggest nearby cafes or rest areas.

[2501] 12. Booking an Experience

[2502] User: Selects the experience or restaurant they want and submits a reservation request.

[2503] Terminal: Sends a reservation request to the server.

[2504] Server: Connects to the reservation system and checks availability.

[2505] Terminal: Provides visual and audio confirmation of reservation completion.

[2506] Example: Book a tea time at a cafe and receive a notification when the reservation is complete.

[2507] Example

[2508] For example, when a user puts on smart glasses and starts sightseeing, location information is first acquired. Then, a sightseeing plan tailored to the user's needs is generated based on emotional data. When the user asks for directions or detailed information about tourist spots, guidance tailored to the user's current emotional state is provided. This allows the user to enjoy sightseeing without stress.

[2509] If a translation is needed, the smart glasses' camera detects the text and provides an instant translation. Furthermore, if the emotion recognition engine determines that the user is tired, it will provide a recommendation to take a break. In this way, the system comprehensively supports the user's sightseeing experience.

[2510] The processing flow will be explained below.

[2511] Step 1:

[2512] boot

[2513] Device: The smart glasses start up, the internal system initializes, and the login screen appears.

[2514] Step 2:

[2515] Log in

[2516] User: The user enters their login information (ID and password) and selects the login button.

[2517] Terminal: Converts the entered login information into a security token and sends it to the server.

[2518] Server: Checks the login information against the database, and if authentication is successful, returns an authentication token to the device.

[2519] Step 3:

[2520] Get current location

[2521] Device: The built-in GPS module is activated and acquires the user's current location information (latitude and longi...

Claims

1. location information acquisition means for acquiring a current location of a user; A communication means for transmitting the acquired current location information to a server; a display means for displaying the sightseeing plan information provided by the server on the terminal; a recommendation means for acquiring recommendation information from a database stored in a server; A system including:

2. means for transmitting the tourist preference information input by the user to a server; means for receiving the tour plan generated by the server; The system of claim 1 further comprising:

3. a visual guide means for displaying location information and detailed information of tourist attractions; a voice guide means for providing voice guidance on the displayed information; 3. The system according to claim 1 or 2, comprising:

4. means for detecting text information within the user's field of view using a camera; a translation means for translating the detected text information; a display means for displaying the translation result; 4. The system according to claim 1, further comprising:

5. A reservation means for sending a reservation request to the server and checking the reservation status; means for displaying and announcing the completion of the reservation; 5. The system of claim 1, further comprising:

Images