System

Abstract

A system is provided.SOLUTION: A system comprising: a traveler terminal; a visual device connected to the terminal; camera means for acquiring information within a field of view on the visual device; translation means for translating the information acquired by the camera means into multiple languages; display means for displaying the translated information on the visual device; and voice output means for outputting the translated information as voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional tourism systems, language barriers make it difficult for travelers to fully understand information about tourist destinations. Furthermore, the provision of information in different languages ​​is insufficient, preventing travelers from fully experiencing the attractions of tourist destinations. Furthermore, functions such as recording the history of tourist destination visits and recommending the next tourist destination are not adequately provided. The present invention aims to solve these problems and provide a system that allows travelers to comfortably tour tourist destinations in different language areas. [Means for solving the problem]

[0005] The present invention provides a system including a traveler's terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view on the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, and an audio output means for outputting the translated information as audio. Furthermore, the system also includes a recommendation means for recording the traveler's visit history and presenting potential next tourist destinations based on the visit history, and a feedback means for collecting traveler ratings and feedback and improving the system based on the collected data, thereby making it easier for travelers to understand information about tourist destinations across language barriers and enabling them to enjoy sightseeing more enjoyably.

[0006] A "traveler's terminal" is an electronic device carried by a traveler, used to view tourist information and communicate.

[0007] A "visual device" is a glasses-type device worn by a traveler that has a built-in camera and display and displays information within the field of view.

[0008] "Camera means" refers to a camera installed in the visual device, which is a device for capturing text information and images within the field of view.

[0009] The "translation means" is software or a system for converting text information and audio information acquired by the camera means into the user's native language.

[0010] The "display means" is a function for displaying the information translated by the translation means on the display of the visual device.

[0011] The "audio output means" refers to a speaker or earphones that converts the information translated by the translation means into audio and provides it to the user.

[0012] A "recommendation method" is an algorithm or system that presents potential next tourist destinations based on a traveler's visiting history.

[0013] "Feedback means" is a function for collecting evaluations and feedback from users and using the information to improve the system. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] MODE FOR CARRYING OUT THE INVENTION

[0036] System configuration

[0037] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, and a feedback means, thereby enabling travelers to obtain visual and audio information that can be understood in different language areas.

[0038] Program processing

[0039] Registering and configuring user devices

[0040] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0041] Visual device attachment and linkage

[0042] After arriving at the airport, the user picks up the visual device at the rental counter.

[0043] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[0044] Information translation during sightseeing

[0045] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[0046] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[0047] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[0048] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[0049] Audio guide

[0050] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[0051] Users can listen to audio guidance in their native language through the visual device's earphones.

[0052] Recommendation feature

[0053] The terminal records the user's visit history and transmits it to the server.

[0054] The server analyzes the visit history and generates the next tourist spot candidates.

[0055] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[0056] Feedback collection

[0057] After completing their tour, users provide ratings and feedback through a dedicated app.

[0058] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[0059] Specific examples

[0060] If you are visiting the Great Buddha of Nara

[0061] 1. The user arrives at the Great Buddha of Nara.

[0062] 2. The user wears the visual device and scans the information board with the camera means.

[0063] 3. The visual device sends the text data to the terminal.

[0064] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[0065] 5. The terminal returns the translation result to the visual device, which displays it.

[0066] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[0067] 7. After the tour, the user submits feedback via the app.

[0068] 8. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[0069] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and plan their next sightseeing destination.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user downloads and installs the dedicated app.

[0073] Step 2:

[0074] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[0075] Step 3:

[0076] The terminal transmits the input information to the server.

[0077] Step 4:

[0078] The server stores the received information in a database and creates a profile for the user.

[0079] Step 5:

[0080] A user arrives at the airport and turns on location services on their smartphone.

[0081] Step 6:

[0082] The device sends its current location to the server.

[0083] Step 7:

[0084] The server uses the location information to identify the nearest rental counter and check the stock of audio glasses.

[0085] Step 8:

[0086] The server transmits the location of the rental counter and reservation information for the audio glasses to the terminal.

[0087] Step 9:

[0088] The user receives the audio glasses at the rental counter.

[0089] Step 10:

[0090] The device establishes a connection with the audio glasses via Bluetooth or Wi-Fi and transfers user settings to the glasses.

[0091] Step 11:

[0092] The user arrives at a tourist spot and puts on the audio glasses.

[0093] Step 12:

[0094] The audio glasses use a camera means to acquire text information within the field of view.

[0095] Step 13:

[0096] The information acquired by the audio glasses is sent to the terminal as text data.

[0097] Step 14:

[0098] The device sends text data to a translation AI and receives the text translated into the user's native language.

[0099] Step 15:

[0100] The device sends the translated text to the audio glasses.

[0101] Step 16:

[0102] Audio glasses display the translated text in the user's field of view.

[0103] Step 17:

[0104] The device converts the translated information into audio data and sends it to the earphones of the audio glasses.

[0105] Step 18:

[0106] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[0107] Step 19:

[0108] The terminal sends the visit history to the server.

[0109] Step 20:

[0110] The server analyzes the visit history and generates the next tourist spot candidates.

[0111] Step 21:

[0112] The device notifies the audio glasses of information about the next potential tourist spot.

[0113] Step 22:

[0114] The user checks the notification and selects the next destination.

[0115] Step 23:

[0116] After sightseeing, users send feedback through a dedicated app.

[0117] Step 24:

[0118] The terminal transmits the feedback data to the server.

[0119] Step 25:

[0120] The server analyzes the feedback and makes improvements to the system.

[0121] Step 26:

[0122] The server generates a discount code for the survey respondent and sends it to the terminal.

[0123] Step 27:

[0124] The terminal displays the discount code to the user.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] One challenge is that travelers in different language regions have difficulty accurately understanding local guide information and signs. Travelers also have a need to receive information visually and audibly in their native language. Furthermore, there is a lack of recommendations for next tourist spots during a trip, and a means to collect user feedback to improve the system.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes a visual device connected to a traveler's terminal, an imaging means for acquiring information within the field of view on the visual device, a translation means for translating the information acquired by the imaging means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a text-to-speech synthesis means for the voice output means to convert the translated information into voice data, a registration means for the traveler to register personal information and native language using a dedicated app and transmit the information to the server, and a setting means for the traveler to receive the visual device at the rental counter, connect the visual device to the traveler's terminal, and transfer setting information. This allows travelers from different language-speaking regions to understand information in real time, receive recommendations for next tourist spots, and improve the system based on feedback.

[0130] A "traveler's device" is an electronic device that a traveler can carry around, such as a smartphone or tablet, which can be operated by installing a dedicated application.

[0131] "Visual devices" refer to devices such as head-mounted displays and smart glasses that can be worn by travelers and can display information within their field of vision.

[0132] "Imaging means" refers to cameras and sensors built into the visual device, including hardware and software functionality for capturing information within the field of view.

[0133] "Translation tools" means software or services for converting acquired information into another language, including machine translation APIs and translation software.

[0134] "Display means" refers to the technology and devices for displaying translated information on the display of a visual device, specifically including the screen portion of a head-mounted display or smart glasses.

[0135] "Audio output means" refers to devices or technologies for providing translated information to travelers as audio, and specifically includes earphones and speakers.

[0136] "Text-to-speech synthesis means" refers to a technology for converting text-format information into voice data, and specifically refers to a text-to-speech synthesis engine (TTS engine).

[0137] "Registration means" includes a function that allows travelers to enter personal information and their native language using a dedicated app and send that information to a server.

[0138] The "setting means" includes a function for a traveler to receive a visual device at a rental counter, connect the visual device to a terminal, and transfer setting information to the visual device.

[0139] "Recommendation methods" refer to algorithms and functions for recommending the next tourist destination based on a traveler's visiting history, and specifically include analytical techniques using machine learning algorithms.

[0140] "Feedback means" includes functions for collecting traveler ratings and feedback and improving the system based on that data.

[0141] The system of the present invention includes a traveler's terminal, a visual device, an imaging means, a translation means, a display means, a voice output means, a text-to-speech synthesis means, a registration means, a setting means, a recommendation means, and a feedback means, and allows travelers to visually and audibly obtain information that can be understood in different language areas.

[0142] The terminal is a portable communication device such as a smartphone or tablet, and is operated by installing a dedicated application. Travelers download this dedicated application and register their personal information and native language. The terminal sends the registered information to a server, where the traveler's information is stored in a database.

[0143] The visual device is a smart glass or head-mounted display worn by the traveler. The traveler picks up the visual device at the airport rental counter and connects the device to the terminal via Bluetooth. The terminal transfers the user's setting information to the visual device, allowing the visual device to be operated in the user's native language.

[0144] The imaging means is a camera built into the visual device, and the traveler scans the guide signs and explanatory text at the tourist spot. This camera has high resolution and can accurately capture the tourist spot information. The text data acquired by the imaging means is transmitted to the terminal in real time.

[0145] The terminal translates the text data sent from the imaging means using the Google Translate API. The translation means translates the text data into the traveler's native language. The translated text data is sent from the terminal to the visual device and displayed on the visual device's built-in display. The user can visually check the information on the signboard in their native language.

[0146] Furthermore, the terminal uses a text-to-speech synthesis means (TTS engine) to convert the translated text data into voice data, which is provided to the user through the earphone of the visual device, allowing the user to hear the voice guidance in their native language.

[0147] As a recommendation method, the device records the traveler's visit history and periodically sends this data to the server, which then uses a machine learning algorithm to generate next tourist destination candidates from the visit history and propose the best sightseeing routes and spots for the traveler.

[0148] As a feedback method, travelers provide feedback using a dedicated app after completing their tour. The device sends this feedback data to a server, which analyzes the data and uses it to improve the system. Specific analytical methods include data mining and statistical analysis.

[0149] Specific examples

[0150] If you are visiting the Great Buddha of Nara

[0151] 1. The user arrives at the Great Buddha of Nara.

[0152] 2. The user wears the visual device and scans the information board with the built-in camera.

[0153] 3. The text data acquired by the visual device is sent to the terminal.

[0154] 4. The device uses the Google Translate API to translate the text data into the user's native language.

[0155] 5. The translated text data is displayed on the visual device's display.

[0156] 6. At the same time, the device uses a text-to-speech engine (TTS engine) to convert the data into voice data and provide voice guidance in the user's native language through the visual device's earphones.

[0157] 7. After the tour is completed, the user submits feedback via a dedicated app.

[0158] 8. The device sends the feedback data to the server, which analyzes the data and generates the next tourist destination candidates.

[0159] Prompt Sentence Examples

[0160] "Please explain in detail about the Great Buddha of Nara. Please include important historical background and tourist attractions to interest visitors."

[0161] This will make travellers feel more comfortable and understandable when travelling in different language areas.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Step 1:

[0164] The user downloads the dedicated app and launches it.

[0165] Input: Smartphone or tablet, internet connection.

[0166] Specific operation: The user downloads and installs the dedicated app from the app store. After installation, the user launches the app.

[0167] Output: The dedicated app starts and the start screen is displayed.

[0168] Step 2:

[0169] The user registers personal information and native language, and the terminal transmits this information to the server.

[0170] Input: Personal information such as name, nationality, native language, etc.

[0171] Specific operation: The user enters personal information into the app's registration screen and presses the submit button. The device then sends the entered information to the server using HTTPS.

[0172] Output: Personal information is stored on the server.

[0173] Step 3:

[0174] A user rents a visual device at an airport and connects the device to a terminal.

[0175] Input: reservation number, passport, visual device.

[0176] Specific operation: The user receives the visual device at the airport rental counter, connects the device to the terminal via Bluetooth, and the terminal transfers the configuration information to the visual device.

[0177] Output: The visual device can be operated in the user's native language.

[0178] Step 4:

[0179] A user wears a visual device at a tourist spot and scans guide signs and explanatory texts with an imaging means.

[0180] Input: Text information for signs and explanations.

[0181] Specific operation: A user wears a visual device at a tourist spot and takes a picture of text information with a camera. The image capturing means acquires the text data.

[0182] Output: The captured text data is sent to the terminal.

[0183] Step 5:

[0184] The device translates the acquired text data using the Google Translate API.

[0185] Input: The captured text data.

[0186] Specific operation: The device sends the acquired text data to the Google Translate API and receives the translation result.

[0187] Output: The translated text data is saved on the device.

[0188] Step 6:

[0189] The terminal transmits the translated text data to the visual device, which displays it.

[0190] Input: Translated text data.

[0191] Specific operation: The terminal sends the translated text data to the visual device, which then displays it on the display of the visual device.

[0192] Output: The translated information is displayed in the user's field of view in their native language.

[0193] Step 7:

[0194] The terminal converts the translated text data into voice data and provides it to the user through the voice output means.

[0195] Input: Translated text data.

[0196] Specific operation: The device uses a text-to-speech engine (TTS engine) to convert the translated text data into audio data, which is then provided to the user through the earphones of the visual device.

[0197] Output: The user can hear the translated audio guidance in their native language.

[0198] Step 8:

[0199] The terminal records the user's visit history and sends it to the server.

[0200] Input: Information about the tourist attractions visited and the time.

[0201] Specific operation: The device records the user's visit history using GPS and timestamps. The recorded information is periodically sent to the server.

[0202] Output: The user's visit history is saved on the server.

[0203] Step 9:

[0204] The server uses a machine learning algorithm to generate the next tourist destination candidates.

[0205] Input: User's visit history data.

[0206] Specific operation: The server analyzes the user's visit history and generates the next tourist spot candidate using a machine learning algorithm. The generated candidate spot information is sent to the device.

[0207] Output: A list of potential next tourist spots will be displayed on the terminal.

[0208] Step 10:

[0209] After completing their tour, users provide feedback through a dedicated app.

[0210] Input: Traveler ratings and feedback.

[0211] Specific operation: The user enters ratings and comments about the sightseeing experience on the feedback screen within the dedicated app.

[0212] Output: The feedback data is sent to the server.

[0213] Step 11:

[0214] The server analyzes the feedback data and makes improvements to the system.

[0215] Input: Feedback data.

[0216] What happens: The server analyzes the collected feedback data and identifies areas for improvement. The analysis results are sent to the development team to help improve the system.

[0217] Output: The improved system is reflected in the next use.

[0218] (Application example 1)

[0219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0220] In today's globalized society, language differences are a major barrier for travelers and customers in brick-and-mortar stores. This has created a demand for systems that can easily understand information across different language regions and provide personalized information to individual users. Furthermore, conventional systems have difficulty integrating visual and auditory information, and have not provided sufficient convenience to improve the user experience.

[0221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0222] In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a recommendation means for analyzing the acquired information and recommending information based on the user's past behavior history, a user interface means for presenting the information to the user on the visual device, and a data integration means for linking the visual display of the information and the voice guidance, thereby enabling users from different language backgrounds to understand information in real time and receive individually personalized information.

[0223] A "traveler's terminal" is an electronic device that can be carried by a traveler and is a device for acquiring and displaying information by connecting to a visual device, translation means, display means, etc.

[0224] A "visual device" is a device worn by a user that acquires information within the field of view and displays that information, and includes, in particular, camera means and display means.

[0225] The "camera means" is a device that is mounted on the visual device and has the function of acquiring information within the field of view as an image.

[0226] The "translation means" is a means capable of translating acquired information into a language designated by the user and capable of handling multiple languages.

[0227] "Display means" means a device capable of displaying translated information and other important information on the display of a visual device.

[0228] The "audio output means" is a device that has the function of providing the translated information to the user as audio.

[0229] The "recommendation means" is a device that has the function of selecting and presenting the next recommended information or product based on the user's past behavioral history and purchase history.

[0230] "User interface means" refers to a means by which a user accesses information and performs operations on a visual device, and is an interface that mediates operations between the user and the system.

[0231] The "data integration means" is a means for centrally managing the data for visual displays and audio guidance, linking them together, and providing them to the user.

[0232] System configuration

[0233] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, a recommendation means, a user interface means, and a data integration means, thereby enabling users to understand information in real time and receive personalized information in different languages.

[0234] Program processing

[0235] Registering and configuring user devices

[0236] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0237] Visual device attachment and linkage

[0238] When a user arrives at a physical store, they put on the smart glasses. The device connects with the visual device, and the user's settings are transferred to the visual device, allowing the device to be operated in the user's native language.

[0239] Product information translation and display

[0240] A user uses a visual device in a physical store and scans a product label or price tag using a camera means. The camera means of the visual device acquires product information and transmits the text data to a terminal. The terminal uses a translation means to translate the text into the user's native language. The translated text data is transmitted to the visual device, which displays it in the user's field of view.

[0241] Audio guide

[0242] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means, so that the user can hear the product description in their own language through the earphones of the visual device.

[0243] Recommendation feature

[0244] The device records the user's purchase history and sends it to the server. The server analyzes the purchase history and generates the next recommended product. The device notifies the user of the next recommended product information, helping the user select a new product.

[0245] Hardware and Software Used

[0246] The following hardware and software are used to realize the system.

[0247] Visual device: smart glasses

[0248] Camera means: Built-in camera in smart glasses

[0249] Translation method: Google Translate API

[0250] Audio output method: Pyttsx3 audio engine

[0251] Display: Smart glasses display

[0252] User interface: A dedicated application that works with smart glasses

[0253] Data integration tool: Software that integrates acquired information and links visual displays with audio guidance

[0254] Specific examples

[0255] When a user wears smart glasses in a physical store and scans a product label with the camera, the product name and details are translated into the user's native language and displayed on the smart glasses' display. The information is also presented audibly, allowing the user to obtain both visual and auditory information. Meanwhile, the server analyzes the user's past purchase history and recommends new related products. These recommendations are also provided to the user via the smart glasses' display and audio.

[0256] Prompt Sentence Examples

[0257] 1. When a user wears smart glasses in a physical store and scans a product label, the product information is translated and displayed.

[0258] 2. The text data acquired by the camera is translated into English using Google Translate.

[0259] 3. The translation results are displayed on the smart glasses screen and also explained aloud.

[0260] 4. Recommended products are displayed based on the user's purchase history and guidance is provided via voice.

[0261] This allows users to overcome language barriers and enjoy fashionable shopping.

[0262] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0263] Step 1:

[0264] A user puts on the smart glasses and launches the dedicated application. The input is the user's personal information and native language setting, which are recorded on the device and sent to the server. The output is that the smart glasses are ready for use and the user information is saved on the server.

[0265] Step 2:

[0266] The terminal is connected to the smart glasses, and the user's setting information is transferred to the smart glasses. The input is the user setting information from the server, and the output is that the smart glasses can be operated in the user's native language.

[0267] Step 3:

[0268] A user uses a camera in a physical store to scan product labels or price tags. The input is image data of the product information acquired by the camera, and the output is the product information extracted as text data. This text data is sent to a terminal.

[0269] Step 4:

[0270] The server receives the text data and translates it into the user's native language using a translation means. The input is the text data of the product information and the user's native language information, and the output is the translated text data of the product information.

[0271] Step 5:

[0272] The translated text data is sent to the terminal, which then displays it on the display of the smart glasses. The input is the translated text data of the product information, and the output is the product information displayed on the display of the smart glasses.

[0273] Step 6:

[0274] The terminal converts the translated text data into audio data and provides it to the user through the audio output means. The input is the translated text data of the product information, and the output is audio data. The user can listen to the product explanation in their own language through the earphones of the visual device.

[0275] Step 7:

[0276] The terminal records the user's purchase history and sends the data to the server. The input is the user's purchase history, and the output is the purchase history data stored on the server.

[0277] Step 8:

[0278] The server analyzes the purchase history and generates the next recommended product. The input is the user's purchase history, and the output is the information about the recommended product.

[0279] Step 9:

[0280] Recommended product information is sent to the terminal, which then displays the information on the smart glasses' display and provides audio guidance. The input is the recommended product information, and the output is the information displayed on the smart glasses' display and an audio product description.

[0281] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0282] MODE FOR CARRYING OUT THE INVENTION

[0283] System configuration

[0284] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, a feedback means, and an emotion engine, allowing travelers to obtain visual and audio information that can be understood in different language areas, and providing a personalized experience based on emotions.

[0285] Program processing

[0286] Registering and configuring user devices

[0287] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0288] Visual device attachment and linkage

[0289] After arriving at the airport, the user picks up the visual device at the rental counter.

[0290] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[0291] Information translation during sightseeing

[0292] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[0293] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[0294] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[0295] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[0296] Audio guide

[0297] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[0298] Users can listen to audio guidance in their native language through the visual device's earphones.

[0299] Recommendation feature

[0300] The terminal records the user's visit history and transmits it to the server.

[0301] The server analyzes the visit history and generates the next tourist spot candidates.

[0302] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[0303] Feedback collection

[0304] After completing their tour, users provide ratings and feedback through a dedicated app.

[0305] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[0306] emotion recognition

[0307] The visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[0308] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[0309] Emotion-based information provision

[0310] The device can automatically adjust the content of the information provided based on the recognized emotion. For example, if it determines that the user is tired, it can switch to information recommending tourist spots and rest areas where they can relax.

[0311] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[0312] Specific examples

[0313] If you are visiting the Great Buddha of Nara

[0314] 1. The user arrives at the Great Buddha of Nara.

[0315] 2. The user wears the visual device and scans the information board with the camera means.

[0316] 3. The visual device sends the text data to the terminal.

[0317] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[0318] 5. The terminal returns the translation result to the visual device, which displays it.

[0319] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[0320] 7. The terminal and camera means analyze the user's facial expressions and recognize the user's emotions in real time.

[0321] 8. The device adjusts tourist information and next recommended locations based on the user's current emotional state.

[0322] 9. After the tour, the user submits feedback via the app.

[0323] 10. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[0324] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and to have the optimal sightseeing experience according to their own emotional state.

[0325] The processing flow will be explained below.

[0326] Step 1:

[0327] The user downloads and installs the dedicated app.

[0328] Step 2:

[0329] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[0330] Step 3:

[0331] The terminal transmits the input information to the server.

[0332] Step 4:

[0333] The server stores the received information in a database and creates a profile for the user.

[0334] Step 5:

[0335] A user arrives at the airport and turns on location services on their smartphone.

[0336] Step 6:

[0337] The device sends its current location to the server.

[0338] Step 7:

[0339] The server uses the location information to identify the nearest rental counter and check the inventory of the visual device.

[0340] Step 8:

[0341] The server transmits the location of the rental counter and reservation information for the visual device to the terminal.

[0342] Step 9:

[0343] The user picks up the visual device at the rental counter.

[0344] Step 10:

[0345] The device establishes a connection with the visual device via Bluetooth or Wi-Fi and transfers user settings to the visual device.

[0346] Step 11:

[0347] The user arrives at the tourist spot and puts on the visual device.

[0348] Step 12:

[0349] The vision device utilizes a camera means to capture textual information within the field of view.

[0350] Step 13:

[0351] The information acquired by the visual device is sent to the terminal as text data.

[0352] Step 14:

[0353] The device sends text data to a translation AI and receives the text translated into the user's native language.

[0354] Step 15:

[0355] The terminal sends the translated text to the visual device.

[0356] Step 16:

[0357] A visual device displays the translated text in the user's field of view.

[0358] Step 17:

[0359] The device converts the translated information into audio data and transmits it to the visual device's earphones.

[0360] Step 18:

[0361] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[0362] Step 19:

[0363] The terminal sends the visit history to the server.

[0364] Step 20:

[0365] The server analyzes the visit history and generates the next tourist spot candidates.

[0366] Step 21:

[0367] The terminal notifies the visual device of information about the next tourist spot candidate.

[0368] Step 22:

[0369] The user checks the notification and selects the next destination.

[0370] Step 23:

[0371] While the user is sightseeing, the visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[0372] Step 24:

[0373] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[0374] Step 25:

[0375] The device automatically adjusts the content of the information provided based on the recognized emotion.

[0376] Step 26:

[0377] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[0378] Step 27:

[0379] After completing their tour, users submit their ratings and feedback through a dedicated app.

[0380] Step 28:

[0381] The terminal transmits the feedback data to the server.

[0382] Step 29:

[0383] The server analyzes the feedback and makes improvements to the system.

[0384] Step 30:

[0385] The server generates a discount code for the survey respondent and sends it to the terminal.

[0386] Step 31:

[0387] The terminal displays the discount code to the user.

[0388] Example 2

[0389] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] There is a need to solve the problems of travelers being unable to effectively obtain information in different language environments, and of tourism experiences being uniform and not personalized based on the feelings and preferences of individual travelers. Another issue is that recommendations for next destinations are not made using visit history and feedback, and the optimal travel plan for travelers is not proposed.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0392] In this invention, the server includes a user terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, an emotion recognition means for analyzing the user's facial expressions and tone of voice to evaluate their emotional state, and an information provision adjustment means for adjusting the information provided based on the emotion acquired by the emotion recognition means. This allows travelers to easily acquire information in different language environments and provides a personalized sightseeing experience tailored to their individual emotional state and preferences. It also enables appropriate recommendations of next destinations based on visit history and feedback.

[0393] A "user terminal" is a portable information processing device such as a smartphone or tablet that is carried by a traveler.

[0394] "Visual devices" is a general term for head-mounted displays and AR glasses that have the function of expanding visual information when worn by the user.

[0395] "Camera means" refers to a camera built into the visual device, which is a device that captures specific information within its field of view.

[0396] The "translation means" is a function including software and hardware that converts acquired text data into a different language.

[0397] "Display means" refers to a display device for displaying the translated information in the user's field of view.

[0398] The "audio output means" refers to an earphone or speaker device that provides the translated information to the user as audio.

[0399] "Emotion recognition means" refers to technology and software for analyzing a user's facial expressions and tone of voice to assess their emotional state.

[0400] The "information provision adjustment means" is a function that adjusts the content and format of the information to be provided based on the emotion data obtained by the emotion recognition means.

[0401] The "recommendation method" is a function that records and analyzes the user's visit history and suggests the next tourist spot to visit.

[0402] "Feedback means" is a function for collecting evaluations and opinions from users and using them to improve the system.

[0403] The present invention provides a system that allows travelers to enjoy sightseeing without stress, even in different language environments. This system includes a user terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, an information provision adjustment means, a recommendation means, and a feedback means. The specific configuration and processing flow are described below.

[0404] Registering and configuring user devices

[0405] First, the user downloads a dedicated app and registers their personal information and native language. This information is sent from the device to the server and stored on the server. The user's personal information is used by the system to provide the user with the most appropriate information.

[0406] Visual device attachment and linkage

[0407] When a user picks up a visual device at an airport or rental counter, the terminal and the visual device are connected via Bluetooth or other wireless communication means. The terminal obtains the user's setting information from the server and transfers it to the visual device. Based on this setting information, the visual device is configured to allow the user to operate it in their native language.

[0408] Information translation during sightseeing

[0409] When a user uses a visual device at a tourist spot, the camera built into the visual device captures information signs and explanatory text within its field of view. The visual device extracts text data using OCR technology and sends this data to the device. The device then translates the text data into the user's native language using a translation API (e.g., Google Translate API). The translated result is sent from the device to the visual device, which then displays this information in the user's field of view.

[0410] Audio guide

[0411] The terminal converts the translated text data into audio data using TTS (Text-to-Speech) technology through the earphones of the visual device, and the visual device provides audio guidance to the user in their native language through the audio output means, allowing the user to obtain not only visual information but also audio information.

[0412] Recommendation feature

[0413] The device records the user's visit history and sends it to the server. The server analyzes this visit history and generates the next tourist destination candidates. The generated tourist destination candidates are notified to the user from the device, and a new sightseeing plan based on the user's preferences is provided.

[0414] Feedback collection

[0415] After completing a tour, users provide ratings and feedback through a dedicated app. The devices send this feedback data to a server, which analyzes it and uses it to improve the system. This ensures that the system can always respond to the latest user needs.

[0416] Emotion recognition and information provision coordination

[0417] The visual device and camera means capture the user's facial expressions and tone of voice and transmit them to the terminal. The terminal then analyzes this data using an emotion engine to evaluate the user's emotional state. Based on the evaluation results, the terminal automatically adjusts the content and format of the information it provides. For example, if it recognizes that the user is tired, it will prioritize providing information about places to relax and rest facilities. The adjusted information is then provided to the user through the visual device and audio output means.

[0418] Specific examples

[0419] When visiting the Great Buddha in Nara, the user wears a visual device and scans the information board with the camera. The device extracts the text data using OCR technology and translates it using a translation API (e.g., Google Translate API). The translated results are displayed on the visual device and simultaneously played through earphones as an audio guide. In addition, the system analyzes the user's facial expressions, and if fatigue is detected, information about nearby cafes and relaxation spots is provided. After the user finishes sightseeing and submits feedback, the system generates the next tourist destination candidate and notifies the user.

[0420] This allows users to enjoy a personalized sightseeing experience without having to worry about language barriers.

[0421] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0422] Step 1:

[0423] The user downloads and installs the dedicated app. Specifically, they search for the dedicated app in the app store and click the download button. They then follow the installation process to install the app on their device. The input data is the user's device information, and the output is the installed dedicated app.

[0424] Step 2:

[0425] The device launches the app and displays a screen for the user to enter personal information and their native language. The user enters the required information, such as their name, email address, and native language. The input data is the user's personal information and native language, and the output is the registration information stored on the device.

[0426] Step 3:

[0427] The device sends the registration information to the server. This process involves sending the user's personal information and native language to the server using an HTTP request. The input data is the registration information stored on the device, and the output is the information stored on the server.

[0428] Step 4:

[0429] The user receives and wears the visual device at an airport or rental counter. The user is instructed on how to wear the visual device and wears it correctly. The input data is the visual device received by the user, and the output is the visual device worn.

[0430] Step 5:

[0431] The device is connected to the visual device. The device pairs with the visual device using Bluetooth or other wireless communication methods. Input data is from the user's device and the visual device, and output is from the connected visual device.

[0432] Step 6:

[0433] The terminal obtains the user's setting information from the server and transfers it to the visual device. An HTTP request is made to obtain the registered information from the server. The obtained information is sent to the visual device, which then applies the native language setting. The input data is the user's setting information, and the output is the visual device with the native language setting applied.

[0434] Step 7:

[0435] The user arrives at a tourist spot and uses a visual device to capture signs and explanations within the field of view using a camera. The input data is the tourist spot's field of view information, and the output is the captured text data.

[0436] Step 8:

[0437] The visual device uses OCR technology to extract text data and send it to the terminal. The image data captured by the camera is analyzed by the OCR engine and converted into text data. The input data is the captured image, and the output is the text data sent to the terminal.

[0438] Step 9:

[0439] The device uses the translation API to translate the text data into the user's native language. The device sends the text data to the translation API and receives the translation result. The input data is the acquired text data, and the output is the text data translated into the user's native language.

[0440] Step 10:

[0441] The terminal sends the translation result to the visual device, which then displays the translation result on the AR display of the visual device. The input data is text data translated into the user's native language, and the output is the translation result displayed on the visual device.

[0442] Step 11:

[0443] The device converts the translated text data into voice data using a TTS engine. The text data is input into the TTS engine, which generates voice data. The input data is the translated text data, and the output is the generated voice data.

[0444] Step 12:

[0445] The terminal transmits audio data to the visual device, and the audio guide is reproduced by the audio output means. The user listens to the audio guide through earphones. The input data is the generated audio data, and the output is the reproduced audio guide.

[0446] Step 13:

[0447] Vision and camera means capture the user's facial expressions and tone of voice, with the input data being the user's facial expressions and tone of voice, and the output being the captured emotion data.

[0448] Step 14:

[0449] The device analyzes the captured emotion data with an emotion engine to evaluate the emotional state. The emotion engine analyzes the data and determines the user's emotional state. The input data is the captured emotion data, and the output is the analyzed emotional state.

[0450] Step 15:

[0451] The device adjusts the information it provides based on the user's emotional state. For example, if it determines that the user is tired, it prioritizes providing information about relaxing tourist spots and rest areas. The input data is the analyzed emotional state, and the output is the adjusted information.

[0452] Step 16:

[0453] The terminal transmits the adjusted information to the visual device and provides it to the user through the visual device and the audio output means, where the input data is the adjusted information and the output is the information provided through the visual device and the audio output means.

[0454] Step 17:

[0455] The device records the user's visit history and sends it to the server. Information about visited locations and GPS data are sent to the server. The input data is the recorded visit history, and the output is the visit history stored on the server.

[0456] Step 18:

[0457] The server analyzes the visit history and generates the next tourist destination candidates. Based on the visit history, an algorithm is used to recommend the next tourist destination. The input data is the saved visit history, and the output is the generated tourist destination candidates.

[0458] Step 19:

[0459] The device notifies the user of the next tourist destination candidate. The device suggests the next tourist destination to the user via a pop-up notification or message. The input data are the generated tourist destination candidates, and the output is the information notified to the user.

[0460] Step 20:

[0461] After the user has finished sightseeing, they use a dedicated app to write their feedback, which is then sent from the device to the server. The input data is the user's feedback, and the output is the feedback data sent to the server.

[0462] Step 21:

[0463] The server analyzes the feedback data and uses it to improve the system. The analysis results are used to update and improve the system. The input data is the feedback data stored on the server, and the output is an improved system.

[0464] (Application example 2)

[0465] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] When tourists travel in different language areas, it is necessary to make it easier for them to understand information visually and aurally, and to personalize their travel experience based on their emotions, recommending places to visit, translating visual information while traveling, and providing audio guides easily and effectively.It is also important to improve the quality of the travel experience by presenting potential next tourist destinations based on the traveler's emotional state and visiting history.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, camera means for acquiring information within the field of view of the visual device, translation means for translating the information acquired by the camera means into multiple languages, display means for displaying the translated information on the visual device, audio output means for outputting the translated information as audio, emotion recognition means for analyzing facial expressions to recognize emotions, and information provision means for adjusting the provision of information based on emotion data recognized by the emotion recognition means. This makes it easier for travelers from different language backgrounds to understand information visually and audibly, and enables them to have a personalized experience based on emotions.

[0468] A "traveler's terminal" is a portable information terminal carried by a tourist while sightseeing, and is a device for registering and setting information, connecting to a visual device, and sending and receiving information.

[0469] A "visual device" is a device worn by a traveler that acquires information within the field of view and displays translated information. For example, smart glasses are examples of such devices.

[0470] The "camera means" is a camera device that is mounted on the visual device and has the function of acquiring information within the field of view.

[0471] The "translation means" refers to software and hardware for translating information acquired by the camera means into multiple languages.

[0472] The "display means" refers to a technology for displaying the information translated by the translation means on a visual device, and includes a display and the like.

[0473] "Audio output means" refers to technology that generates translated information as audio data and allows travelers to listen to it. This includes speakers and earphones.

[0474] The "emotion recognition means" is a technology that analyzes travelers' facial expressions and recognizes their emotions in real time. It includes a camera and emotion analysis software.

[0475] The "information provision means" is a technology that adjusts the information to be provided based on the emotion data recognized by the emotion recognition means. This includes displaying information and generating audio data.

[0476] "Recommendation means" is a technology that records a traveler's visit history and presents potential next tourist destinations based on this.

[0477] "Feedback methods" are technologies for collecting traveler ratings and feedback and improving the system based on the collected data.

[0478]

[0479] A system for realizing the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, and an information providing means.

[0480] Registering and configuring user devices

[0481] 1. User Registration:

[0482] Users download a dedicated app onto their mobile devices and register their personal information and native language. This information is sent from the device to the server and stored on the server.

[0483] Introduction of visual devices and information acquisition by camera means

[0484] 2. Visual equipment connection:

[0485] Visual devices (such as smart glasses) are connected to a terminal and user settings are transferred, allowing the visual device to be operated in the user's native language.

[0486] 3. Information acquisition:

[0487] At a tourist spot, a user wears a visual device and uses a camera means to acquire information within the field of view. The camera means scans guide signs and explanatory text at the tourist spot and acquires the text data.

[0488] Multilingual support through translation and display means

[0489] 4. Translation process:

[0490] The camera means transmits the captured text data to the terminal, which then translates the text into the user's native language using a translation means (such as Google Translate API). The translated text data is then transmitted to the visual device again and displayed on the visual device.

[0491] Audio guide provided

[0492] 5. Audio guide:

[0493] The device converts the text data translated by the translation means into voice data, and provides the voice data to the user through the voice output means using a voice synthesis engine (such as IBM Watson Text to Speech API).The user can then listen to the voice guide in their native language through the earphones of the visual device.

[0494] Emotion recognition and information provision

[0495] 6. Emotion recognition:

[0496] The visual device and camera means analyze the user's facial expressions and recognize emotions in real time. The device analyzes the emotion data using an emotion analysis engine (such as Microsoft Azure Emotion API).

[0497] 7. Emotional information:

[0498] The device automatically adjusts the content of the information provided based on the user's emotions. For example, if the device determines that the user is tired, it will switch to recommending tourist spots and rest areas where they can relax.

[0499] Recommendations and Feedback

[0500] 8. Recommendation function:

[0501] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidates, helping the user select a new tourist destination.

[0502] 9. Feedback Collection:

[0503] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to a server, which analyzes it and uses it to improve the system.

[0504] Specific examples

[0505] For example, when a user arrives at the Great Buddha of Nara, he or she wears a visual device and scans an information board with a camera means. The visual device transmits text data to a terminal, and the terminal translates the text into the user's native language with a translation means. The translation result is displayed on the visual device, and an audio guide is also provided through a speech synthesis engine. The user's facial expressions are analyzed in real time, and appropriate information is automatically provided.

[0506] Example prompt sentence:

[0507] "Please translate the Japanese description of the Great Buddha of Nara and provide the following information by voice. Also, please analyze the user's facial expressions and recommend tourist spots where they can relax."

[0508] In this way, the system enables travelers to overcome language barriers and enjoy a personalized, emotion-based tourism experience.

[0509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0510] Step 1:

[0511] Users download a dedicated app and register their personal information and native language. This information is sent from the device to the server and saved on the server. The user inputs personal information such as their name and native language, and the initial setup is completed when the server saves this information.

[0512] Step 2:

[0513] After arriving at a tourist destination, the user puts on the visual device (smart glasses) and connects it to the terminal. The terminal transfers the user's setting information to the visual device, which then receives it. This allows the visual device to be operated in the user's native language. The input is the setting information from the terminal, and the output is the language setting of the visual device.

[0514] Step 3:

[0515] The user uses the camera means via the visual device to acquire information within the field of view. The camera means scans guide signs and explanatory text at tourist spots and acquires the text data. The input is the text information within the field of view, and the output is the text data.

[0516] Step 4:

[0517] The camera means transmits the captured text data to the terminal. The terminal uses the translation means (Google Translate API) to translate the text into the user's native language. The translated text data is transmitted again to the visual device and displayed on the visual device. The input is text data, and the output is translated text data.

[0518] Step 5:

[0519] The device converts the translated text data into audio data. It uses a speech synthesis engine (IBM Watson Text to Speech API) to provide audio guidance through the audio output means. The user can listen to the audio guidance in their native language through the earphones of the visual device. The input is the translated text data, and the output is audio data.

[0520] Step 6:

[0521] The visual device and camera analyze the user's facial expressions in real time and recognize emotions. The device acquires emotion data using an emotion analysis engine (Microsoft Azure Emotion API). The input is the user's facial expression data, and the output is emotion data.

[0522] Step 7:

[0523] The device automatically adjusts the content of the information it provides based on the recognized emotions. For example, if it determines that the user is tired, it will switch to recommending tourist spots and rest areas where users can relax. The input is emotion data, and the output is adjusted recommendation information.

[0524] Step 8:

[0525] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidate information and helps the user select a new tourist destination. The input is visit history data, and the output is information on the next tourist destination candidate.

[0526] Step 9:

[0527] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to the server, which analyzes it and uses it to improve the system. The input is feedback data, and the output is improved system information.

[0528] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0529] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0530] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0531] [Second embodiment]

[0532] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0533] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0534] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0535] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0536] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0537] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0538] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0539] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0540] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0541] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0542] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0543] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0544] MODE FOR CARRYING OUT THE INVENTION

[0545] System configuration

[0546] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, and a feedback means, thereby enabling travelers to obtain visual and audio information that can be understood in different language areas.

[0547] Program processing

[0548] Registering and configuring user devices

[0549] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0550] Visual device attachment and linkage

[0551] After arriving at the airport, the user picks up the visual device at the rental counter.

[0552] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[0553] Information translation during sightseeing

[0554] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[0555] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[0556] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[0557] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[0558] Audio guide

[0559] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[0560] Users can listen to audio guidance in their native language through the visual device's earphones.

[0561] Recommendation feature

[0562] The terminal records the user's visit history and transmits it to the server.

[0563] The server analyzes the visit history and generates the next tourist spot candidates.

[0564] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[0565] Feedback collection

[0566] After completing their tour, users provide ratings and feedback through a dedicated app.

[0567] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[0568] Specific examples

[0569] If you are visiting the Great Buddha of Nara

[0570] 1. The user arrives at the Great Buddha of Nara.

[0571] 2. The user wears the visual device and scans the information board with the camera means.

[0572] 3. The visual device sends the text data to the terminal.

[0573] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[0574] 5. The terminal returns the translation result to the visual device, which displays it.

[0575] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[0576] 7. After the tour, the user submits feedback via the app.

[0577] 8. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[0578] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and plan their next sightseeing destination.

[0579] The processing flow will be explained below.

[0580] Step 1:

[0581] The user downloads and installs the dedicated app.

[0582] Step 2:

[0583] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[0584] Step 3:

[0585] The terminal transmits the input information to the server.

[0586] Step 4:

[0587] The server stores the received information in a database and creates a profile for the user.

[0588] Step 5:

[0589] A user arrives at the airport and turns on location services on their smartphone.

[0590] Step 6:

[0591] The device sends its current location to the server.

[0592] Step 7:

[0593] The server uses the location information to identify the nearest rental counter and check the stock of audio glasses.

[0594] Step 8:

[0595] The server transmits the location of the rental counter and reservation information for the audio glasses to the terminal.

[0596] Step 9:

[0597] The user receives the audio glasses at the rental counter.

[0598] Step 10:

[0599] The device establishes a connection with the audio glasses via Bluetooth or Wi-Fi and transfers user settings to the glasses.

[0600] Step 11:

[0601] The user arrives at a tourist spot and puts on the audio glasses.

[0602] Step 12:

[0603] The audio glasses use a camera means to acquire text information within the field of view.

[0604] Step 13:

[0605] The information acquired by the audio glasses is sent to the terminal as text data.

[0606] Step 14:

[0607] The device sends text data to a translation AI and receives the text translated into the user's native language.

[0608] Step 15:

[0609] The device sends the translated text to the audio glasses.

[0610] Step 16:

[0611] Audio glasses display the translated text in the user's field of view.

[0612] Step 17:

[0613] The device converts the translated information into audio data and sends it to the earphones of the audio glasses.

[0614] Step 18:

[0615] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[0616] Step 19:

[0617] The terminal sends the visit history to the server.

[0618] Step 20:

[0619] The server analyzes the visit history and generates the next tourist spot candidates.

[0620] Step 21:

[0621] The device notifies the audio glasses of information about the next potential tourist spot.

[0622] Step 22:

[0623] The user checks the notification and selects the next destination.

[0624] Step 23:

[0625] After sightseeing, users send feedback through a dedicated app.

[0626] Step 24:

[0627] The terminal transmits the feedback data to the server.

[0628] Step 25:

[0629] The server analyzes the feedback and makes improvements to the system.

[0630] Step 26:

[0631] The server generates a discount code for the survey respondent and sends it to the terminal.

[0632] Step 27:

[0633] The terminal displays the discount code to the user.

[0634] Example 1

[0635] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0636] One challenge is that travelers in different language regions have difficulty accurately understanding local guide information and signs. Travelers also have a need to receive information visually and audibly in their native language. Furthermore, there is a lack of recommendations for next tourist spots during a trip, and a means to collect user feedback to improve the system.

[0637] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0638] In this invention, the server includes a visual device connected to a traveler's terminal, an imaging means for acquiring information within the field of view on the visual device, a translation means for translating the information acquired by the imaging means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a text-to-speech synthesis means for the voice output means to convert the translated information into voice data, a registration means for the traveler to register personal information and native language using a dedicated app and transmit the information to the server, and a setting means for the traveler to receive the visual device at the rental counter, connect the visual device to the traveler's terminal, and transfer setting information. This allows travelers from different language-speaking regions to understand information in real time, receive recommendations for next tourist spots, and improve the system based on feedback.

[0639] A "traveler's device" is an electronic device that a traveler can carry around, such as a smartphone or tablet, which can be operated by installing a dedicated application.

[0640] "Visual devices" refer to devices such as head-mounted displays and smart glasses that can be worn by travelers and can display information within their field of vision.

[0641] "Imaging means" refers to cameras and sensors built into the visual device, including hardware and software functionality for capturing information within the field of view.

[0642] "Translation tools" means software or services for converting acquired information into another language, including machine translation APIs and translation software.

[0643] "Display means" refers to the technology and devices for displaying translated information on the display of a visual device, specifically including the screen portion of a head-mounted display or smart glasses.

[0644] "Audio output means" refers to devices or technologies for providing translated information to travelers as audio, and specifically includes earphones and speakers.

[0645] "Text-to-speech synthesis means" refers to a technology for converting text-format information into voice data, and specifically refers to a text-to-speech synthesis engine (TTS engine).

[0646] "Registration means" includes a function that allows travelers to enter personal information and their native language using a dedicated app and send that information to a server.

[0647] The "setting means" includes a function for a traveler to receive a visual device at a rental counter, connect the visual device to a terminal, and transfer setting information to the visual device.

[0648] "Recommendation methods" refer to algorithms and functions for recommending the next tourist destination based on a traveler's visiting history, and specifically include analytical techniques using machine learning algorithms.

[0649] "Feedback means" includes functions for collecting traveler ratings and feedback and improving the system based on that data.

[0650] The system of the present invention includes a traveler's terminal, a visual device, an imaging means, a translation means, a display means, a voice output means, a text-to-speech synthesis means, a registration means, a setting means, a recommendation means, and a feedback means, and allows travelers to visually and audibly obtain information that can be understood in different language areas.

[0651] The terminal is a portable communication device such as a smartphone or tablet, and is operated by installing a dedicated application. Travelers download this dedicated application and register their personal information and native language. The terminal sends the registered information to a server, where the traveler's information is stored in a database.

[0652] The visual device is a smart glass or head-mounted display worn by the traveler. The traveler picks up the visual device at the airport rental counter and connects the device to the terminal via Bluetooth. The terminal transfers the user's setting information to the visual device, allowing the visual device to be operated in the user's native language.

[0653] The imaging means is a camera built into the visual device, and the traveler scans the guide signs and explanatory text at the tourist spot. This camera has high resolution and can accurately capture the tourist spot information. The text data acquired by the imaging means is transmitted to the terminal in real time.

[0654] The terminal translates the text data sent from the imaging means using the Google Translate API. The translation means translates the text data into the traveler's native language. The translated text data is sent from the terminal to the visual device and displayed on the visual device's built-in display. The user can visually check the information on the signboard in their native language.

[0655] Furthermore, the terminal uses a text-to-speech synthesis means (TTS engine) to convert the translated text data into voice data, which is provided to the user through the earphone of the visual device, allowing the user to hear the voice guidance in their native language.

[0656] As a recommendation method, the device records the traveler's visit history and periodically sends this data to the server, which then uses a machine learning algorithm to generate next tourist destination candidates from the visit history and propose the best sightseeing routes and spots for the traveler.

[0657] As a feedback method, travelers provide feedback using a dedicated app after completing their tour. The device sends this feedback data to a server, which analyzes the data and uses it to improve the system. Specific analytical methods include data mining and statistical analysis.

[0658] Specific examples

[0659] If you are visiting the Great Buddha of Nara

[0660] 1. The user arrives at the Great Buddha of Nara.

[0661] 2. The user wears the visual device and scans the information board with the built-in camera.

[0662] 3. The text data acquired by the visual device is sent to the terminal.

[0663] 4. The device uses the Google Translate API to translate the text data into the user's native language.

[0664] 5. The translated text data is displayed on the visual device's display.

[0665] 6. At the same time, the device uses a text-to-speech engine (TTS engine) to convert the data into voice data and provide voice guidance in the user's native language through the visual device's earphones.

[0666] 7. After the tour is completed, the user submits feedback via a dedicated app.

[0667] 8. The device sends the feedback data to the server, which analyzes the data and generates the next tourist destination candidates.

[0668] Prompt Sentence Examples

[0669] "Please explain in detail about the Great Buddha of Nara. Please include important historical background and tourist attractions to interest visitors."

[0670] This will make travellers feel more comfortable and understandable when travelling in different language areas.

[0671] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0672] Step 1:

[0673] The user downloads the dedicated app and launches it.

[0674] Input: Smartphone or tablet, internet connection.

[0675] Specific operation: The user downloads and installs the dedicated app from the app store. After installation, the user launches the app.

[0676] Output: The dedicated app starts and the start screen is displayed.

[0677] Step 2:

[0678] The user registers personal information and native language, and the terminal transmits this information to the server.

[0679] Input: Personal information such as name, nationality, native language, etc.

[0680] Specific operation: The user enters personal information into the app's registration screen and presses the submit button. The device then sends the entered information to the server using HTTPS.

[0681] Output: Personal information is stored on the server.

[0682] Step 3:

[0683] A user rents a visual device at an airport and connects the device to a terminal.

[0684] Input: reservation number, passport, visual device.

[0685] Specific operation: The user receives the visual device at the airport rental counter, connects the device to the terminal via Bluetooth, and the terminal transfers the configuration information to the visual device.

[0686] Output: The visual device can be operated in the user's native language.

[0687] Step 4:

[0688] A user wears a visual device at a tourist spot and scans guide signs and explanatory texts with an imaging means.

[0689] Input: Text information for signs and explanations.

[0690] Specific operation: A user wears a visual device at a tourist spot and takes a picture of text information with a camera. The image capturing means acquires the text data.

[0691] Output: The captured text data is sent to the terminal.

[0692] Step 5:

[0693] The device translates the acquired text data using the Google Translate API.

[0694] Input: The captured text data.

[0695] Specific operation: The device sends the acquired text data to the Google Translate API and receives the translation result.

[0696] Output: The translated text data is saved on the device.

[0697] Step 6:

[0698] The terminal transmits the translated text data to the visual device, which displays it.

[0699] Input: Translated text data.

[0700] Specific operation: The terminal sends the translated text data to the visual device, which then displays it on the display of the visual device.

[0701] Output: The translated information is displayed in the user's field of view in their native language.

[0702] Step 7:

[0703] The terminal converts the translated text data into voice data and provides it to the user through the voice output means.

[0704] Input: Translated text data.

[0705] Specific operation: The device uses a text-to-speech engine (TTS engine) to convert the translated text data into audio data, which is then provided to the user through the earphones of the visual device.

[0706] Output: The user can hear the translated audio guidance in their native language.

[0707] Step 8:

[0708] The terminal records the user's visit history and sends it to the server.

[0709] Input: Information about the tourist attractions visited and the time.

[0710] Specific operation: The device records the user's visit history using GPS and timestamps. The recorded information is periodically sent to the server.

[0711] Output: The user's visit history is saved on the server.

[0712] Step 9:

[0713] The server uses a machine learning algorithm to generate the next tourist destination candidates.

[0714] Input: User's visit history data.

[0715] Specific operation: The server analyzes the user's visit history and generates the next tourist spot candidate using a machine learning algorithm. The generated candidate spot information is sent to the device.

[0716] Output: A list of potential next tourist spots will be displayed on the terminal.

[0717] Step 10:

[0718] After completing their tour, users provide feedback through a dedicated app.

[0719] Input: Traveler ratings and feedback.

[0720] Specific operation: The user enters ratings and comments about the sightseeing experience on the feedback screen within the dedicated app.

[0721] Output: The feedback data is sent to the server.

[0722] Step 11:

[0723] The server analyzes the feedback data and makes improvements to the system.

[0724] Input: Feedback data.

[0725] What happens: The server analyzes the collected feedback data and identifies areas for improvement. The analysis results are sent to the development team to help improve the system.

[0726] Output: The improved system is reflected in the next use.

[0727] (Application example 1)

[0728] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] In today's globalized society, language differences are a major barrier for travelers and customers in brick-and-mortar stores. This has created a demand for systems that can easily understand information across different language regions and provide personalized information to individual users. Furthermore, conventional systems have difficulty integrating visual and auditory information, and have not provided sufficient convenience to improve the user experience.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0731] In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a recommendation means for analyzing the acquired information and recommending information based on the user's past behavior history, a user interface means for presenting the information to the user on the visual device, and a data integration means for linking the visual display of the information and the voice guidance, thereby enabling users from different language backgrounds to understand information in real time and receive individually personalized information.

[0732] A "traveler's terminal" is an electronic device that can be carried by a traveler and is a device for acquiring and displaying information by connecting to a visual device, translation means, display means, etc.

[0733] A "visual device" is a device worn by a user that acquires information within the field of view and displays that information, and includes, in particular, camera means and display means.

[0734] The "camera means" is a device that is mounted on the visual device and has the function of acquiring information within the field of view as an image.

[0735] The "translation means" is a means capable of translating acquired information into a language designated by the user and capable of handling multiple languages.

[0736] "Display means" means a device capable of displaying translated information and other important information on the display of a visual device.

[0737] The "audio output means" is a device that has the function of providing the translated information to the user as audio.

[0738] The "recommendation means" is a device that has the function of selecting and presenting the next recommended information or product based on the user's past behavioral history and purchase history.

[0739] "User interface means" refers to a means by which a user accesses information and performs operations on a visual device, and is an interface that mediates operations between the user and the system.

[0740] The "data integration means" is a means for centrally managing the data for visual displays and audio guidance, linking them together, and providing them to the user.

[0741] System configuration

[0742] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, a recommendation means, a user interface means, and a data integration means, thereby enabling users to understand information in real time and receive personalized information in different languages.

[0743] Program processing

[0744] Registering and configuring user devices

[0745] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0746] Visual device attachment and linkage

[0747] When a user arrives at a physical store, they put on the smart glasses. The device connects with the visual device, and the user's settings are transferred to the visual device, allowing the device to be operated in the user's native language.

[0748] Product information translation and display

[0749] A user uses a visual device in a physical store and scans a product label or price tag using a camera means. The camera means of the visual device acquires product information and transmits the text data to a terminal. The terminal uses a translation means to translate the text into the user's native language. The translated text data is transmitted to the visual device, which displays it in the user's field of view.

[0750] Audio guide

[0751] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means, so that the user can hear the product description in their own language through the earphones of the visual device.

[0752] Recommendation feature

[0753] The device records the user's purchase history and sends it to the server. The server analyzes the purchase history and generates the next recommended product. The device notifies the user of the next recommended product information, helping the user select a new product.

[0754] Hardware and Software Used

[0755] The following hardware and software are used to realize the system.

[0756] Visual device: smart glasses

[0757] Camera means: Built-in camera in smart glasses

[0758] Translation method: Google Translate API

[0759] Audio output method: Pyttsx3 audio engine

[0760] Display: Smart glasses display

[0761] User interface: A dedicated application that works with smart glasses

[0762] Data integration tool: Software that integrates acquired information and links visual displays with audio guidance

[0763] Specific examples

[0764] When a user wears smart glasses in a physical store and scans a product label with the camera, the product name and details are translated into the user's native language and displayed on the smart glasses' display. The information is also presented audibly, allowing the user to obtain both visual and auditory information. Meanwhile, the server analyzes the user's past purchase history and recommends new related products. These recommendations are also provided to the user via the smart glasses' display and audio.

[0765] Prompt Sentence Examples

[0766] 1. When a user wears smart glasses in a physical store and scans a product label, the product information is translated and displayed.

[0767] 2. The text data acquired by the camera is translated into English using Google Translate.

[0768] 3. The translation results are displayed on the smart glasses screen and also explained aloud.

[0769] 4. Recommended products are displayed based on the user's purchase history and guidance is provided via voice.

[0770] This allows users to overcome language barriers and enjoy fashionable shopping.

[0771] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0772] Step 1:

[0773] A user puts on the smart glasses and launches the dedicated application. The input is the user's personal information and native language setting, which are recorded on the device and sent to the server. The output is that the smart glasses are ready for use and the user information is saved on the server.

[0774] Step 2:

[0775] The terminal is connected to the smart glasses, and the user's setting information is transferred to the smart glasses. The input is the user setting information from the server, and the output is that the smart glasses can be operated in the user's native language.

[0776] Step 3:

[0777] A user uses a camera in a physical store to scan product labels or price tags. The input is image data of the product information acquired by the camera, and the output is the product information extracted as text data. This text data is sent to a terminal.

[0778] Step 4:

[0779] The server receives the text data and translates it into the user's native language using a translation means. The input is the text data of the product information and the user's native language information, and the output is the translated text data of the product information.

[0780] Step 5:

[0781] The translated text data is sent to the terminal, which then displays it on the display of the smart glasses. The input is the translated text data of the product information, and the output is the product information displayed on the display of the smart glasses.

[0782] Step 6:

[0783] The terminal converts the translated text data into audio data and provides it to the user through the audio output means. The input is the translated text data of the product information, and the output is audio data. The user can listen to the product explanation in their own language through the earphones of the visual device.

[0784] Step 7:

[0785] The terminal records the user's purchase history and sends the data to the server. The input is the user's purchase history, and the output is the purchase history data stored on the server.

[0786] Step 8:

[0787] The server analyzes the purchase history and generates the next recommended product. The input is the user's purchase history, and the output is the information about the recommended product.

[0788] Step 9:

[0789] Recommended product information is sent to the terminal, which then displays the information on the smart glasses' display and provides audio guidance. The input is the recommended product information, and the output is the information displayed on the smart glasses' display and an audio product description.

[0790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0791] MODE FOR CARRYING OUT THE INVENTION

[0792] System configuration

[0793] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, a feedback means, and an emotion engine, allowing travelers to obtain visual and audio information that can be understood in different language areas, and providing a personalized experience based on emotions.

[0794] Program processing

[0795] Registering and configuring user devices

[0796] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[0797] Visual device attachment and linkage

[0798] After arriving at the airport, the user picks up the visual device at the rental counter.

[0799] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[0800] Information translation during sightseeing

[0801] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[0802] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[0803] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[0804] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[0805] Audio guide

[0806] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[0807] Users can listen to audio guidance in their native language through the visual device's earphones.

[0808] Recommendation feature

[0809] The terminal records the user's visit history and transmits it to the server.

[0810] The server analyzes the visit history and generates the next tourist spot candidates.

[0811] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[0812] Feedback collection

[0813] After completing their tour, users provide ratings and feedback through a dedicated app.

[0814] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[0815] emotion recognition

[0816] The visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[0817] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[0818] Emotion-based information provision

[0819] The device can automatically adjust the content of the information provided based on the recognized emotion. For example, if it determines that the user is tired, it can switch to information recommending tourist spots and rest areas where they can relax.

[0820] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[0821] Specific examples

[0822] If you are visiting the Great Buddha of Nara

[0823] 1. The user arrives at the Great Buddha of Nara.

[0824] 2. The user wears the visual device and scans the information board with the camera means.

[0825] 3. The visual device sends the text data to the terminal.

[0826] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[0827] 5. The terminal returns the translation result to the visual device, which displays it.

[0828] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[0829] 7. The terminal and camera means analyze the user's facial expressions and recognize the user's emotions in real time.

[0830] 8. The device adjusts tourist information and next recommended locations based on the user's current emotional state.

[0831] 9. After the tour, the user submits feedback via the app.

[0832] 10. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[0833] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and to have the optimal sightseeing experience according to their own emotional state.

[0834] The processing flow will be explained below.

[0835] Step 1:

[0836] The user downloads and installs the dedicated app.

[0837] Step 2:

[0838] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[0839] Step 3:

[0840] The terminal transmits the input information to the server.

[0841] Step 4:

[0842] The server stores the received information in a database and creates a profile for the user.

[0843] Step 5:

[0844] A user arrives at the airport and turns on location services on their smartphone.

[0845] Step 6:

[0846] The device sends its current location to the server.

[0847] Step 7:

[0848] The server uses the location information to identify the nearest rental counter and check the inventory of the visual device.

[0849] Step 8:

[0850] The server transmits the location of the rental counter and reservation information for the visual device to the terminal.

[0851] Step 9:

[0852] The user picks up the visual device at the rental counter.

[0853] Step 10:

[0854] The device establishes a connection with the visual device via Bluetooth or Wi-Fi and transfers user settings to the visual device.

[0855] Step 11:

[0856] The user arrives at the tourist spot and puts on the visual device.

[0857] Step 12:

[0858] The vision device utilizes a camera means to capture textual information within the field of view.

[0859] Step 13:

[0860] The information acquired by the visual device is sent to the terminal as text data.

[0861] Step 14:

[0862] The device sends text data to a translation AI and receives the text translated into the user's native language.

[0863] Step 15:

[0864] The terminal sends the translated text to the visual device.

[0865] Step 16:

[0866] A visual device displays the translated text in the user's field of view.

[0867] Step 17:

[0868] The device converts the translated information into audio data and transmits it to the visual device's earphones.

[0869] Step 18:

[0870] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[0871] Step 19:

[0872] The terminal sends the visit history to the server.

[0873] Step 20:

[0874] The server analyzes the visit history and generates the next tourist spot candidates.

[0875] Step 21:

[0876] The terminal notifies the visual device of information about the next tourist spot candidate.

[0877] Step 22:

[0878] The user checks the notification and selects the next destination.

[0879] Step 23:

[0880] While the user is sightseeing, the visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[0881] Step 24:

[0882] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[0883] Step 25:

[0884] The device automatically adjusts the content of the information provided based on the recognized emotion.

[0885] Step 26:

[0886] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[0887] Step 27:

[0888] After completing their tour, users submit their ratings and feedback through a dedicated app.

[0889] Step 28:

[0890] The terminal transmits the feedback data to the server.

[0891] Step 29:

[0892] The server analyzes the feedback and makes improvements to the system.

[0893] Step 30:

[0894] The server generates a discount code for the survey respondent and sends it to the terminal.

[0895] Step 31:

[0896] The terminal displays the discount code to the user.

[0897] Example 2

[0898] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0899] There is a need to solve the problems of travelers being unable to effectively obtain information in different language environments, and of tourism experiences being uniform and not personalized based on the feelings and preferences of individual travelers. Another issue is that recommendations for next destinations are not made using visit history and feedback, and the optimal travel plan for travelers is not proposed.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0901] In this invention, the server includes a user terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, an emotion recognition means for analyzing the user's facial expressions and tone of voice to evaluate their emotional state, and an information provision adjustment means for adjusting the information provided based on the emotion acquired by the emotion recognition means. This allows travelers to easily acquire information in different language environments and provides a personalized sightseeing experience tailored to their individual emotional state and preferences. It also enables appropriate recommendations of next destinations based on visit history and feedback.

[0902] A "user terminal" is a portable information processing device such as a smartphone or tablet that is carried by a traveler.

[0903] "Visual devices" is a general term for head-mounted displays and AR glasses that have the function of expanding visual information when worn by the user.

[0904] "Camera means" refers to a camera built into the visual device, which is a device that captures specific information within its field of view.

[0905] The "translation means" is a function including software and hardware that converts acquired text data into a different language.

[0906] "Display means" refers to a display device for displaying the translated information in the user's field of view.

[0907] The "audio output means" refers to an earphone or speaker device that provides the translated information to the user as audio.

[0908] "Emotion recognition means" refers to technology and software for analyzing a user's facial expressions and tone of voice to assess their emotional state.

[0909] The "information provision adjustment means" is a function that adjusts the content and format of the information to be provided based on the emotion data obtained by the emotion recognition means.

[0910] The "recommendation method" is a function that records and analyzes the user's visit history and suggests the next tourist spot to visit.

[0911] "Feedback means" is a function for collecting evaluations and opinions from users and using them to improve the system.

[0912] The present invention provides a system that allows travelers to enjoy sightseeing without stress, even in different language environments. This system includes a user terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, an information provision adjustment means, a recommendation means, and a feedback means. The specific configuration and processing flow are described below.

[0913] Registering and configuring user devices

[0914] First, the user downloads a dedicated app and registers their personal information and native language. This information is sent from the device to the server and stored on the server. The user's personal information is used by the system to provide the user with the most appropriate information.

[0915] Visual device attachment and linkage

[0916] When a user picks up a visual device at an airport or rental counter, the terminal and the visual device are connected via Bluetooth or other wireless communication means. The terminal obtains the user's setting information from the server and transfers it to the visual device. Based on this setting information, the visual device is configured to allow the user to operate it in their native language.

[0917] Information translation during sightseeing

[0918] When a user uses a visual device at a tourist spot, the camera built into the visual device captures information signs and explanatory text within its field of view. The visual device extracts text data using OCR technology and sends this data to the device. The device then translates the text data into the user's native language using a translation API (e.g., Google Translate API). The translated result is sent from the device to the visual device, which then displays this information in the user's field of view.

[0919] Audio guide

[0920] The terminal converts the translated text data into audio data using TTS (Text-to-Speech) technology through the earphones of the visual device, and the visual device provides audio guidance to the user in their native language through the audio output means, allowing the user to obtain not only visual information but also audio information.

[0921] Recommendation feature

[0922] The device records the user's visit history and sends it to the server. The server analyzes this visit history and generates the next tourist destination candidates. The generated tourist destination candidates are notified to the user from the device, and a new sightseeing plan based on the user's preferences is provided.

[0923] Feedback collection

[0924] After completing a tour, users provide ratings and feedback through a dedicated app. The devices send this feedback data to a server, which analyzes it and uses it to improve the system. This ensures that the system can always respond to the latest user needs.

[0925] Emotion recognition and information provision coordination

[0926] The visual device and camera means capture the user's facial expressions and tone of voice and transmit them to the terminal. The terminal then analyzes this data using an emotion engine to evaluate the user's emotional state. Based on the evaluation results, the terminal automatically adjusts the content and format of the information it provides. For example, if it recognizes that the user is tired, it will prioritize providing information about places to relax and rest facilities. The adjusted information is then provided to the user through the visual device and audio output means.

[0927] Specific examples

[0928] When visiting the Great Buddha in Nara, the user wears a visual device and scans the information board with the camera. The device extracts the text data using OCR technology and translates it using a translation API (e.g., Google Translate API). The translated results are displayed on the visual device and simultaneously played through earphones as an audio guide. In addition, the system analyzes the user's facial expressions, and if fatigue is detected, information about nearby cafes and relaxation spots is provided. After the user finishes sightseeing and submits feedback, the system generates the next tourist destination candidate and notifies the user.

[0929] This allows users to enjoy a personalized sightseeing experience without having to worry about language barriers.

[0930] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0931] Step 1:

[0932] The user downloads and installs the dedicated app. Specifically, they search for the dedicated app in the app store and click the download button. They then follow the installation process to install the app on their device. The input data is the user's device information, and the output is the installed dedicated app.

[0933] Step 2:

[0934] The device launches the app and displays a screen for the user to enter personal information and their native language. The user enters the required information, such as their name, email address, and native language. The input data is the user's personal information and native language, and the output is the registration information stored on the device.

[0935] Step 3:

[0936] The device sends the registration information to the server. This process involves sending the user's personal information and native language to the server using an HTTP request. The input data is the registration information stored on the device, and the output is the information stored on the server.

[0937] Step 4:

[0938] The user receives and wears the visual device at an airport or rental counter. The user is instructed on how to wear the visual device and wears it correctly. The input data is the visual device received by the user, and the output is the visual device worn.

[0939] Step 5:

[0940] The device is connected to the visual device. The device pairs with the visual device using Bluetooth or other wireless communication methods. Input data is from the user's device and the visual device, and output is from the connected visual device.

[0941] Step 6:

[0942] The terminal obtains the user's setting information from the server and transfers it to the visual device. An HTTP request is made to obtain the registered information from the server. The obtained information is sent to the visual device, which then applies the native language setting. The input data is the user's setting information, and the output is the visual device with the native language setting applied.

[0943] Step 7:

[0944] The user arrives at a tourist spot and uses a visual device to capture signs and explanations within the field of view using a camera. The input data is the tourist spot's field of view information, and the output is the captured text data.

[0945] Step 8:

[0946] The visual device uses OCR technology to extract text data and send it to the terminal. The image data captured by the camera is analyzed by the OCR engine and converted into text data. The input data is the captured image, and the output is the text data sent to the terminal.

[0947] Step 9:

[0948] The device uses the translation API to translate the text data into the user's native language. The device sends the text data to the translation API and receives the translation result. The input data is the acquired text data, and the output is the text data translated into the user's native language.

[0949] Step 10:

[0950] The terminal sends the translation result to the visual device, which then displays the translation result on the AR display of the visual device. The input data is text data translated into the user's native language, and the output is the translation result displayed on the visual device.

[0951] Step 11:

[0952] The device converts the translated text data into voice data using a TTS engine. The text data is input into the TTS engine, which generates voice data. The input data is the translated text data, and the output is the generated voice data.

[0953] Step 12:

[0954] The terminal transmits audio data to the visual device, and the audio guide is reproduced by the audio output means. The user listens to the audio guide through earphones. The input data is the generated audio data, and the output is the reproduced audio guide.

[0955] Step 13:

[0956] Vision and camera means capture the user's facial expressions and tone of voice, with the input data being the user's facial expressions and tone of voice, and the output being the captured emotion data.

[0957] Step 14:

[0958] The device analyzes the captured emotion data with an emotion engine to evaluate the emotional state. The emotion engine analyzes the data and determines the user's emotional state. The input data is the captured emotion data, and the output is the analyzed emotional state.

[0959] Step 15:

[0960] The device adjusts the information it provides based on the user's emotional state. For example, if it determines that the user is tired, it prioritizes providing information about relaxing tourist spots and rest areas. The input data is the analyzed emotional state, and the output is the adjusted information.

[0961] Step 16:

[0962] The terminal transmits the adjusted information to the visual device and provides it to the user through the visual device and the audio output means, where the input data is the adjusted information and the output is the information provided through the visual device and the audio output means.

[0963] Step 17:

[0964] The device records the user's visit history and sends it to the server. Information about visited locations and GPS data are sent to the server. The input data is the recorded visit history, and the output is the visit history stored on the server.

[0965] Step 18:

[0966] The server analyzes the visit history and generates the next tourist destination candidates. Based on the visit history, an algorithm is used to recommend the next tourist destination. The input data is the saved visit history, and the output is the generated tourist destination candidates.

[0967] Step 19:

[0968] The device notifies the user of the next tourist destination candidate. The device suggests the next tourist destination to the user via a pop-up notification or message. The input data are the generated tourist destination candidates, and the output is the information notified to the user.

[0969] Step 20:

[0970] After the user has finished sightseeing, they use a dedicated app to write their feedback, which is then sent from the device to the server. The input data is the user's feedback, and the output is the feedback data sent to the server.

[0971] Step 21:

[0972] The server analyzes the feedback data and uses it to improve the system. The analysis results are used to update and improve the system. The input data is the feedback data stored on the server, and the output is an improved system.

[0973] (Application example 2)

[0974] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0975] When tourists travel in different language areas, it is necessary to make it easier for them to understand information visually and aurally, and to personalize their travel experience based on their emotions, recommending places to visit, translating visual information while traveling, and providing audio guides easily and effectively.It is also important to improve the quality of the travel experience by presenting potential next tourist destinations based on the traveler's emotional state and visiting history.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, camera means for acquiring information within the field of view of the visual device, translation means for translating the information acquired by the camera means into multiple languages, display means for displaying the translated information on the visual device, audio output means for outputting the translated information as audio, emotion recognition means for analyzing facial expressions to recognize emotions, and information provision means for adjusting the provision of information based on emotion data recognized by the emotion recognition means. This makes it easier for travelers from different language backgrounds to understand information visually and audibly, and enables them to have a personalized experience based on emotions.

[0977] A "traveler's terminal" is a portable information terminal carried by a tourist while sightseeing, and is a device for registering and setting information, connecting to a visual device, and sending and receiving information.

[0978] A "visual device" is a device worn by a traveler that acquires information within the field of view and displays translated information. For example, smart glasses are examples of such devices.

[0979] The "camera means" is a camera device that is mounted on the visual device and has the function of acquiring information within the field of view.

[0980] The "translation means" refers to software and hardware for translating information acquired by the camera means into multiple languages.

[0981] The "display means" refers to a technology for displaying the information translated by the translation means on a visual device, and includes a display and the like.

[0982] "Audio output means" refers to technology that generates translated information as audio data and allows travelers to listen to it. This includes speakers and earphones.

[0983] The "emotion recognition means" is a technology that analyzes travelers' facial expressions and recognizes their emotions in real time. It includes a camera and emotion analysis software.

[0984] The "information provision means" is a technology that adjusts the information to be provided based on the emotion data recognized by the emotion recognition means. This includes displaying information and generating audio data.

[0985] "Recommendation means" is a technology that records a traveler's visit history and presents potential next tourist destinations based on this.

[0986] "Feedback methods" are technologies for collecting traveler ratings and feedback and improving the system based on the collected data.

[0987]

[0988] A system for realizing the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, and an information providing means.

[0989] Registering and configuring user devices

[0990] 1. User Registration:

[0991] Users download a dedicated app onto their mobile devices and register their personal information and native language. This information is sent from the device to the server and stored on the server.

[0992] Introduction of visual devices and information acquisition by camera means

[0993] 2. Visual equipment connection:

[0994] Visual devices (such as smart glasses) are connected to a terminal and user settings are transferred, allowing the visual device to be operated in the user's native language.

[0995] 3. Information acquisition:

[0996] At a tourist spot, a user wears a visual device and uses a camera means to acquire information within the field of view. The camera means scans guide signs and explanatory text at the tourist spot and acquires the text data.

[0997] Multilingual support through translation and display means

[0998] 4. Translation process:

[0999] The camera means transmits the captured text data to the terminal, which then translates the text into the user's native language using a translation means (such as Google Translate API). The translated text data is then transmitted to the visual device again and displayed on the visual device.

[1000] Audio guide provided

[1001] 5. Audio guide:

[1002] The device converts the text data translated by the translation means into voice data, and provides the voice data to the user through the voice output means using a voice synthesis engine (such as IBM Watson Text to Speech API).The user can then listen to the voice guide in their native language through the earphones of the visual device.

[1003] Emotion recognition and information provision

[1004] 6. Emotion recognition:

[1005] The visual device and camera means analyze the user's facial expressions and recognize emotions in real time. The device analyzes the emotion data using an emotion analysis engine (such as Microsoft Azure Emotion API).

[1006] 7. Emotional information:

[1007] The device automatically adjusts the content of the information provided based on the user's emotions. For example, if the device determines that the user is tired, it will switch to recommending tourist spots and rest areas where they can relax.

[1008] Recommendations and Feedback

[1009] 8. Recommendation function:

[1010] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidates, helping the user select a new tourist destination.

[1011] 9. Feedback Collection:

[1012] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to a server, which analyzes it and uses it to improve the system.

[1013] Specific examples

[1014] For example, when a user arrives at the Great Buddha of Nara, he or she wears a visual device and scans an information board with a camera means. The visual device transmits text data to a terminal, and the terminal translates the text into the user's native language with a translation means. The translation result is displayed on the visual device, and an audio guide is also provided through a speech synthesis engine. The user's facial expressions are analyzed in real time, and appropriate information is automatically provided.

[1015] Example prompt sentence:

[1016] "Please translate the Japanese description of the Great Buddha of Nara and provide the following information by voice. Also, please analyze the user's facial expressions and recommend tourist spots where they can relax."

[1017] In this way, the system enables travelers to overcome language barriers and enjoy a personalized, emotion-based tourism experience.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] Users download a dedicated app and register their personal information and native language. This information is sent from the device to the server and saved on the server. The user inputs personal information such as their name and native language, and the initial setup is completed when the server saves this information.

[1021] Step 2:

[1022] After arriving at a tourist destination, the user puts on the visual device (smart glasses) and connects it to the terminal. The terminal transfers the user's setting information to the visual device, which then receives it. This allows the visual device to be operated in the user's native language. The input is the setting information from the terminal, and the output is the language setting of the visual device.

[1023] Step 3:

[1024] The user uses the camera means via the visual device to acquire information within the field of view. The camera means scans guide signs and explanatory text at tourist spots and acquires the text data. The input is the text information within the field of view, and the output is the text data.

[1025] Step 4:

[1026] The camera means transmits the captured text data to the terminal. The terminal uses the translation means (Google Translate API) to translate the text into the user's native language. The translated text data is transmitted again to the visual device and displayed on the visual device. The input is text data, and the output is translated text data.

[1027] Step 5:

[1028] The device converts the translated text data into audio data. It uses a speech synthesis engine (IBM Watson Text to Speech API) to provide audio guidance through the audio output means. The user can listen to the audio guidance in their native language through the earphones of the visual device. The input is the translated text data, and the output is audio data.

[1029] Step 6:

[1030] The visual device and camera analyze the user's facial expressions in real time and recognize emotions. The device acquires emotion data using an emotion analysis engine (Microsoft Azure Emotion API). The input is the user's facial expression data, and the output is emotion data.

[1031] Step 7:

[1032] The device automatically adjusts the content of the information it provides based on the recognized emotions. For example, if it determines that the user is tired, it will switch to recommending tourist spots and rest areas where users can relax. The input is emotion data, and the output is adjusted recommendation information.

[1033] Step 8:

[1034] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidate information and helps the user select a new tourist destination. The input is visit history data, and the output is information on the next tourist destination candidate.

[1035] Step 9:

[1036] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to the server, which analyzes it and uses it to improve the system. The input is feedback data, and the output is improved system information.

[1037] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1038] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1039] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1040] [Third embodiment]

[1041] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1042] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1043] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1044] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1045] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1046] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1047] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1048] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1049] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1051] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1052] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1053] MODE FOR CARRYING OUT THE INVENTION

[1054] System configuration

[1055] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, and a feedback means, thereby enabling travelers to obtain visual and audio information that can be understood in different language areas.

[1056] Program processing

[1057] Registering and configuring user devices

[1058] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1059] Visual device attachment and linkage

[1060] After arriving at the airport, the user picks up the visual device at the rental counter.

[1061] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[1062] Information translation during sightseeing

[1063] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[1064] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[1065] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[1066] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[1067] Audio guide

[1068] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[1069] Users can listen to audio guidance in their native language through the visual device's earphones.

[1070] Recommendation feature

[1071] The terminal records the user's visit history and transmits it to the server.

[1072] The server analyzes the visit history and generates the next tourist spot candidates.

[1073] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[1074] Feedback collection

[1075] After completing their tour, users provide ratings and feedback through a dedicated app.

[1076] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[1077] Specific examples

[1078] If you are visiting the Great Buddha of Nara

[1079] 1. The user arrives at the Great Buddha of Nara.

[1080] 2. The user wears the visual device and scans the information board with the camera means.

[1081] 3. The visual device sends the text data to the terminal.

[1082] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[1083] 5. The terminal returns the translation result to the visual device, which displays it.

[1084] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[1085] 7. After the tour, the user submits feedback via the app.

[1086] 8. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[1087] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and plan their next sightseeing destination.

[1088] The processing flow will be explained below.

[1089] Step 1:

[1090] The user downloads and installs the dedicated app.

[1091] Step 2:

[1092] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[1093] Step 3:

[1094] The terminal transmits the input information to the server.

[1095] Step 4:

[1096] The server stores the received information in a database and creates a profile for the user.

[1097] Step 5:

[1098] A user arrives at the airport and turns on location services on their smartphone.

[1099] Step 6:

[1100] The device sends its current location to the server.

[1101] Step 7:

[1102] The server uses the location information to identify the nearest rental counter and check the stock of audio glasses.

[1103] Step 8:

[1104] The server transmits the location of the rental counter and reservation information for the audio glasses to the terminal.

[1105] Step 9:

[1106] The user receives the audio glasses at the rental counter.

[1107] Step 10:

[1108] The device establishes a connection with the audio glasses via Bluetooth or Wi-Fi and transfers user settings to the glasses.

[1109] Step 11:

[1110] The user arrives at a tourist spot and puts on the audio glasses.

[1111] Step 12:

[1112] The audio glasses use a camera means to acquire text information within the field of view.

[1113] Step 13:

[1114] The information acquired by the audio glasses is sent to the terminal as text data.

[1115] Step 14:

[1116] The device sends text data to a translation AI and receives the text translated into the user's native language.

[1117] Step 15:

[1118] The device sends the translated text to the audio glasses.

[1119] Step 16:

[1120] Audio glasses display the translated text in the user's field of view.

[1121] Step 17:

[1122] The device converts the translated information into audio data and sends it to the earphones of the audio glasses.

[1123] Step 18:

[1124] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[1125] Step 19:

[1126] The terminal sends the visit history to the server.

[1127] Step 20:

[1128] The server analyzes the visit history and generates the next tourist spot candidates.

[1129] Step 21:

[1130] The device notifies the audio glasses of information about the next potential tourist spot.

[1131] Step 22:

[1132] The user checks the notification and selects the next destination.

[1133] Step 23:

[1134] After sightseeing, users send feedback through a dedicated app.

[1135] Step 24:

[1136] The terminal transmits the feedback data to the server.

[1137] Step 25:

[1138] The server analyzes the feedback and makes improvements to the system.

[1139] Step 26:

[1140] The server generates a discount code for the survey respondent and sends it to the terminal.

[1141] Step 27:

[1142] The terminal displays the discount code to the user.

[1143] Example 1

[1144] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] One challenge is that travelers in different language regions have difficulty accurately understanding local guide information and signs. Travelers also have a need to receive information visually and audibly in their native language. Furthermore, there is a lack of recommendations for next tourist spots during a trip, and a means to collect user feedback to improve the system.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1147] In this invention, the server includes a visual device connected to a traveler's terminal, an imaging means for acquiring information within the field of view on the visual device, a translation means for translating the information acquired by the imaging means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a text-to-speech synthesis means for the voice output means to convert the translated information into voice data, a registration means for the traveler to register personal information and native language using a dedicated app and transmit the information to the server, and a setting means for the traveler to receive the visual device at the rental counter, connect the visual device to the traveler's terminal, and transfer setting information. This allows travelers from different language-speaking regions to understand information in real time, receive recommendations for next tourist spots, and improve the system based on feedback.

[1148] A "traveler's device" is an electronic device that a traveler can carry around, such as a smartphone or tablet, which can be operated by installing a dedicated application.

[1149] "Visual devices" refer to devices such as head-mounted displays and smart glasses that can be worn by travelers and can display information within their field of vision.

[1150] "Imaging means" refers to cameras and sensors built into the visual device, including hardware and software functionality for capturing information within the field of view.

[1151] "Translation tools" means software or services for converting acquired information into another language, including machine translation APIs and translation software.

[1152] "Display means" refers to the technology and devices for displaying translated information on the display of a visual device, specifically including the screen portion of a head-mounted display or smart glasses.

[1153] "Audio output means" refers to devices or technologies for providing translated information to travelers as audio, and specifically includes earphones and speakers.

[1154] "Text-to-speech synthesis means" refers to a technology for converting text-format information into voice data, and specifically refers to a text-to-speech synthesis engine (TTS engine).

[1155] "Registration means" includes a function that allows travelers to enter personal information and their native language using a dedicated app and send that information to a server.

[1156] The "setting means" includes a function for a traveler to receive a visual device at a rental counter, connect the visual device to a terminal, and transfer setting information to the visual device.

[1157] "Recommendation methods" refer to algorithms and functions for recommending the next tourist destination based on a traveler's visiting history, and specifically include analytical techniques using machine learning algorithms.

[1158] "Feedback means" includes functions for collecting traveler ratings and feedback and improving the system based on that data.

[1159] The system of the present invention includes a traveler's terminal, a visual device, an imaging means, a translation means, a display means, a voice output means, a text-to-speech synthesis means, a registration means, a setting means, a recommendation means, and a feedback means, and allows travelers to visually and audibly obtain information that can be understood in different language areas.

[1160] The terminal is a portable communication device such as a smartphone or tablet, and is operated by installing a dedicated application. Travelers download this dedicated application and register their personal information and native language. The terminal sends the registered information to a server, where the traveler's information is stored in a database.

[1161] The visual device is a smart glass or head-mounted display worn by the traveler. The traveler picks up the visual device at the airport rental counter and connects the device to the terminal via Bluetooth. The terminal transfers the user's setting information to the visual device, allowing the visual device to be operated in the user's native language.

[1162] The imaging means is a camera built into the visual device, and the traveler scans the guide signs and explanatory text at the tourist spot. This camera has high resolution and can accurately capture the tourist spot information. The text data acquired by the imaging means is transmitted to the terminal in real time.

[1163] The terminal translates the text data sent from the imaging means using the Google Translate API. The translation means translates the text data into the traveler's native language. The translated text data is sent from the terminal to the visual device and displayed on the visual device's built-in display. The user can visually check the information on the signboard in their native language.

[1164] Furthermore, the terminal uses a text-to-speech synthesis means (TTS engine) to convert the translated text data into voice data, which is provided to the user through the earphone of the visual device, allowing the user to hear the voice guidance in their native language.

[1165] As a recommendation method, the device records the traveler's visit history and periodically sends this data to the server, which then uses a machine learning algorithm to generate next tourist destination candidates from the visit history and propose the best sightseeing routes and spots for the traveler.

[1166] As a feedback method, travelers provide feedback using a dedicated app after completing their tour. The device sends this feedback data to a server, which analyzes the data and uses it to improve the system. Specific analytical methods include data mining and statistical analysis.

[1167] Specific examples

[1168] If you are visiting the Great Buddha of Nara

[1169] 1. The user arrives at the Great Buddha of Nara.

[1170] 2. The user wears the visual device and scans the information board with the built-in camera.

[1171] 3. The text data acquired by the visual device is sent to the terminal.

[1172] 4. The device uses the Google Translate API to translate the text data into the user's native language.

[1173] 5. The translated text data is displayed on the visual device's display.

[1174] 6. At the same time, the device uses a text-to-speech engine (TTS engine) to convert the data into voice data and provide voice guidance in the user's native language through the visual device's earphones.

[1175] 7. After the tour is completed, the user submits feedback via a dedicated app.

[1176] 8. The device sends the feedback data to the server, which analyzes the data and generates the next tourist destination candidates.

[1177] Prompt Sentence Examples

[1178] "Please explain in detail about the Great Buddha of Nara. Please include important historical background and tourist attractions to interest visitors."

[1179] This will make travellers feel more comfortable and understandable when travelling in different language areas.

[1180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1181] Step 1:

[1182] The user downloads the dedicated app and launches it.

[1183] Input: Smartphone or tablet, internet connection.

[1184] Specific operation: The user downloads and installs the dedicated app from the app store. After installation, the user launches the app.

[1185] Output: The dedicated app starts and the start screen is displayed.

[1186] Step 2:

[1187] The user registers personal information and native language, and the terminal transmits this information to the server.

[1188] Input: Personal information such as name, nationality, native language, etc.

[1189] Specific operation: The user enters personal information into the app's registration screen and presses the submit button. The device then sends the entered information to the server using HTTPS.

[1190] Output: Personal information is stored on the server.

[1191] Step 3:

[1192] A user rents a visual device at an airport and connects the device to a terminal.

[1193] Input: reservation number, passport, visual device.

[1194] Specific operation: The user receives the visual device at the airport rental counter, connects the device to the terminal via Bluetooth, and the terminal transfers the configuration information to the visual device.

[1195] Output: The visual device can be operated in the user's native language.

[1196] Step 4:

[1197] A user wears a visual device at a tourist spot and scans guide signs and explanatory texts with an imaging means.

[1198] Input: Text information for signs and explanations.

[1199] Specific operation: A user wears a visual device at a tourist spot and takes a picture of text information with a camera. The image capturing means acquires the text data.

[1200] Output: The captured text data is sent to the terminal.

[1201] Step 5:

[1202] The device translates the acquired text data using the Google Translate API.

[1203] Input: The captured text data.

[1204] Specific operation: The device sends the acquired text data to the Google Translate API and receives the translation result.

[1205] Output: The translated text data is saved on the device.

[1206] Step 6:

[1207] The terminal transmits the translated text data to the visual device, which displays it.

[1208] Input: Translated text data.

[1209] Specific operation: The terminal sends the translated text data to the visual device, which then displays it on the display of the visual device.

[1210] Output: The translated information is displayed in the user's field of view in their native language.

[1211] Step 7:

[1212] The terminal converts the translated text data into voice data and provides it to the user through the voice output means.

[1213] Input: Translated text data.

[1214] Specific operation: The device uses a text-to-speech engine (TTS engine) to convert the translated text data into audio data, which is then provided to the user through the earphones of the visual device.

[1215] Output: The user can hear the translated audio guidance in their native language.

[1216] Step 8:

[1217] The terminal records the user's visit history and sends it to the server.

[1218] Input: Information about the tourist attractions visited and the time.

[1219] Specific operation: The device records the user's visit history using GPS and timestamps. The recorded information is periodically sent to the server.

[1220] Output: The user's visit history is saved on the server.

[1221] Step 9:

[1222] The server uses a machine learning algorithm to generate the next tourist destination candidates.

[1223] Input: User's visit history data.

[1224] Specific operation: The server analyzes the user's visit history and generates the next tourist spot candidate using a machine learning algorithm. The generated candidate spot information is sent to the device.

[1225] Output: A list of potential next tourist spots will be displayed on the terminal.

[1226] Step 10:

[1227] After completing their tour, users provide feedback through a dedicated app.

[1228] Input: Traveler ratings and feedback.

[1229] Specific operation: The user enters ratings and comments about the sightseeing experience on the feedback screen within the dedicated app.

[1230] Output: The feedback data is sent to the server.

[1231] Step 11:

[1232] The server analyzes the feedback data and makes improvements to the system.

[1233] Input: Feedback data.

[1234] What happens: The server analyzes the collected feedback data and identifies areas for improvement. The analysis results are sent to the development team to help improve the system.

[1235] Output: The improved system is reflected in the next use.

[1236] (Application example 1)

[1237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1238] In today's globalized society, language differences are a major barrier for travelers and customers in brick-and-mortar stores. This has created a demand for systems that can easily understand information across different language regions and provide personalized information to individual users. Furthermore, conventional systems have difficulty integrating visual and auditory information, and have not provided sufficient convenience to improve the user experience.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1240] In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a recommendation means for analyzing the acquired information and recommending information based on the user's past behavior history, a user interface means for presenting the information to the user on the visual device, and a data integration means for linking the visual display of the information and the voice guidance, thereby enabling users from different language backgrounds to understand information in real time and receive individually personalized information.

[1241] A "traveler's terminal" is an electronic device that can be carried by a traveler and is a device for acquiring and displaying information by connecting to a visual device, translation means, display means, etc.

[1242] A "visual device" is a device worn by a user that acquires information within the field of view and displays that information, and includes, in particular, camera means and display means.

[1243] The "camera means" is a device that is mounted on the visual device and has the function of acquiring information within the field of view as an image.

[1244] The "translation means" is a means capable of translating acquired information into a language designated by the user and capable of handling multiple languages.

[1245] "Display means" means a device capable of displaying translated information and other important information on the display of a visual device.

[1246] The "audio output means" is a device that has the function of providing the translated information to the user as audio.

[1247] The "recommendation means" is a device that has the function of selecting and presenting the next recommended information or product based on the user's past behavioral history and purchase history.

[1248] "User interface means" refers to a means by which a user accesses information and performs operations on a visual device, and is an interface that mediates operations between the user and the system.

[1249] The "data integration means" is a means for centrally managing the data for visual displays and audio guidance, linking them together, and providing them to the user.

[1250] System configuration

[1251] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, a recommendation means, a user interface means, and a data integration means, thereby enabling users to understand information in real time and receive personalized information in different languages.

[1252] Program processing

[1253] Registering and configuring user devices

[1254] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1255] Visual device attachment and linkage

[1256] When a user arrives at a physical store, they put on the smart glasses. The device connects with the visual device, and the user's settings are transferred to the visual device, allowing the device to be operated in the user's native language.

[1257] Product information translation and display

[1258] A user uses a visual device in a physical store and scans a product label or price tag using a camera means. The camera means of the visual device acquires product information and transmits the text data to a terminal. The terminal uses a translation means to translate the text into the user's native language. The translated text data is transmitted to the visual device, which displays it in the user's field of view.

[1259] Audio guide

[1260] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means, so that the user can hear the product description in their own language through the earphones of the visual device.

[1261] Recommendation feature

[1262] The device records the user's purchase history and sends it to the server. The server analyzes the purchase history and generates the next recommended product. The device notifies the user of the next recommended product information, helping the user select a new product.

[1263] Hardware and Software Used

[1264] The following hardware and software are used to realize the system.

[1265] Visual device: smart glasses

[1266] Camera means: Built-in camera in smart glasses

[1267] Translation method: Google Translate API

[1268] Audio output method: Pyttsx3 audio engine

[1269] Display: Smart glasses display

[1270] User interface: A dedicated application that works with smart glasses

[1271] Data integration tool: Software that integrates acquired information and links visual displays with audio guidance

[1272] Specific examples

[1273] When a user wears smart glasses in a physical store and scans a product label with the camera, the product name and details are translated into the user's native language and displayed on the smart glasses' display. The information is also presented audibly, allowing the user to obtain both visual and auditory information. Meanwhile, the server analyzes the user's past purchase history and recommends new related products. These recommendations are also provided to the user via the smart glasses' display and audio.

[1274] Prompt Sentence Examples

[1275] 1. When a user wears smart glasses in a physical store and scans a product label, the product information is translated and displayed.

[1276] 2. The text data acquired by the camera is translated into English using Google Translate.

[1277] 3. The translation results are displayed on the smart glasses screen and also explained aloud.

[1278] 4. Recommended products are displayed based on the user's purchase history and guidance is provided via voice.

[1279] This allows users to overcome language barriers and enjoy fashionable shopping.

[1280] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1281] Step 1:

[1282] A user puts on the smart glasses and launches the dedicated application. The input is the user's personal information and native language setting, which are recorded on the device and sent to the server. The output is that the smart glasses are ready for use and the user information is saved on the server.

[1283] Step 2:

[1284] The terminal is connected to the smart glasses, and the user's setting information is transferred to the smart glasses. The input is the user setting information from the server, and the output is that the smart glasses can be operated in the user's native language.

[1285] Step 3:

[1286] A user uses a camera in a physical store to scan product labels or price tags. The input is image data of the product information acquired by the camera, and the output is the product information extracted as text data. This text data is sent to a terminal.

[1287] Step 4:

[1288] The server receives the text data and translates it into the user's native language using a translation means. The input is the text data of the product information and the user's native language information, and the output is the translated text data of the product information.

[1289] Step 5:

[1290] The translated text data is sent to the terminal, which then displays it on the display of the smart glasses. The input is the translated text data of the product information, and the output is the product information displayed on the display of the smart glasses.

[1291] Step 6:

[1292] The terminal converts the translated text data into audio data and provides it to the user through the audio output means. The input is the translated text data of the product information, and the output is audio data. The user can listen to the product explanation in their own language through the earphones of the visual device.

[1293] Step 7:

[1294] The terminal records the user's purchase history and sends the data to the server. The input is the user's purchase history, and the output is the purchase history data stored on the server.

[1295] Step 8:

[1296] The server analyzes the purchase history and generates the next recommended product. The input is the user's purchase history, and the output is the information about the recommended product.

[1297] Step 9:

[1298] Recommended product information is sent to the terminal, which then displays the information on the smart glasses' display and provides audio guidance. The input is the recommended product information, and the output is the information displayed on the smart glasses' display and an audio product description.

[1299] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1300] MODE FOR CARRYING OUT THE INVENTION

[1301] System configuration

[1302] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, a feedback means, and an emotion engine, allowing travelers to obtain visual and audio information that can be understood in different language areas, and providing a personalized experience based on emotions.

[1303] Program processing

[1304] Registering and configuring user devices

[1305] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1306] Visual device attachment and linkage

[1307] After arriving at the airport, the user picks up the visual device at the rental counter.

[1308] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[1309] Information translation during sightseeing

[1310] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[1311] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[1312] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[1313] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[1314] Audio guide

[1315] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[1316] Users can listen to audio guidance in their native language through the visual device's earphones.

[1317] Recommendation feature

[1318] The terminal records the user's visit history and transmits it to the server.

[1319] The server analyzes the visit history and generates the next tourist spot candidates.

[1320] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[1321] Feedback collection

[1322] After completing their tour, users provide ratings and feedback through a dedicated app.

[1323] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[1324] emotion recognition

[1325] The visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[1326] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[1327] Emotion-based information provision

[1328] The device can automatically adjust the content of the information provided based on the recognized emotion. For example, if it determines that the user is tired, it can switch to information recommending tourist spots and rest areas where they can relax.

[1329] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[1330] Specific examples

[1331] If you are visiting the Great Buddha of Nara

[1332] 1. The user arrives at the Great Buddha of Nara.

[1333] 2. The user wears the visual device and scans the information board with the camera means.

[1334] 3. The visual device sends the text data to the terminal.

[1335] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[1336] 5. The terminal returns the translation result to the visual device, which displays it.

[1337] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[1338] 7. The terminal and camera means analyze the user's facial expressions and recognize the user's emotions in real time.

[1339] 8. The device adjusts tourist information and next recommended locations based on the user's current emotional state.

[1340] 9. After the tour, the user submits feedback via the app.

[1341] 10. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[1342] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and to have the optimal sightseeing experience according to their own emotional state.

[1343] The processing flow will be explained below.

[1344] Step 1:

[1345] The user downloads and installs the dedicated app.

[1346] Step 2:

[1347] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[1348] Step 3:

[1349] The terminal transmits the input information to the server.

[1350] Step 4:

[1351] The server stores the received information in a database and creates a profile for the user.

[1352] Step 5:

[1353] A user arrives at the airport and turns on location services on their smartphone.

[1354] Step 6:

[1355] The device sends its current location to the server.

[1356] Step 7:

[1357] The server uses the location information to identify the nearest rental counter and check the inventory of the visual device.

[1358] Step 8:

[1359] The server transmits the location of the rental counter and reservation information for the visual device to the terminal.

[1360] Step 9:

[1361] The user picks up the visual device at the rental counter.

[1362] Step 10:

[1363] The device establishes a connection with the visual device via Bluetooth or Wi-Fi and transfers user settings to the visual device.

[1364] Step 11:

[1365] The user arrives at the tourist spot and puts on the visual device.

[1366] Step 12:

[1367] The vision device utilizes a camera means to capture textual information within the field of view.

[1368] Step 13:

[1369] The information acquired by the visual device is sent to the terminal as text data.

[1370] Step 14:

[1371] The device sends text data to a translation AI and receives the text translated into the user's native language.

[1372] Step 15:

[1373] The terminal sends the translated text to the visual device.

[1374] Step 16:

[1375] A visual device displays the translated text in the user's field of view.

[1376] Step 17:

[1377] The device converts the translated information into audio data and transmits it to the visual device's earphones.

[1378] Step 18:

[1379] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[1380] Step 19:

[1381] The terminal sends the visit history to the server.

[1382] Step 20:

[1383] The server analyzes the visit history and generates the next tourist spot candidates.

[1384] Step 21:

[1385] The terminal notifies the visual device of information about the next tourist spot candidate.

[1386] Step 22:

[1387] The user checks the notification and selects the next destination.

[1388] Step 23:

[1389] While the user is sightseeing, the visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[1390] Step 24:

[1391] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[1392] Step 25:

[1393] The device automatically adjusts the content of the information provided based on the recognized emotion.

[1394] Step 26:

[1395] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[1396] Step 27:

[1397] After completing their tour, users submit their ratings and feedback through a dedicated app.

[1398] Step 28:

[1399] The terminal transmits the feedback data to the server.

[1400] Step 29:

[1401] The server analyzes the feedback and makes improvements to the system.

[1402] Step 30:

[1403] The server generates a discount code for the survey respondent and sends it to the terminal.

[1404] Step 31:

[1405] The terminal displays the discount code to the user.

[1406] Example 2

[1407] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1408] There is a need to solve the problems of travelers being unable to effectively obtain information in different language environments, and of tourism experiences being uniform and not personalized based on the feelings and preferences of individual travelers. Another issue is that recommendations for next destinations are not made using visit history and feedback, and the optimal travel plan for travelers is not proposed.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1410] In this invention, the server includes a user terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, an emotion recognition means for analyzing the user's facial expressions and tone of voice to evaluate their emotional state, and an information provision adjustment means for adjusting the information provided based on the emotion acquired by the emotion recognition means. This allows travelers to easily acquire information in different language environments and provides a personalized sightseeing experience tailored to their individual emotional state and preferences. It also enables appropriate recommendations of next destinations based on visit history and feedback.

[1411] A "user terminal" is a portable information processing device such as a smartphone or tablet that is carried by a traveler.

[1412] "Visual devices" is a general term for head-mounted displays and AR glasses that have the function of expanding visual information when worn by the user.

[1413] "Camera means" refers to a camera built into the visual device, which is a device that captures specific information within its field of view.

[1414] The "translation means" is a function including software and hardware that converts acquired text data into a different language.

[1415] "Display means" refers to a display device for displaying the translated information in the user's field of view.

[1416] The "audio output means" refers to an earphone or speaker device that provides the translated information to the user as audio.

[1417] "Emotion recognition means" refers to technology and software for analyzing a user's facial expressions and tone of voice to assess their emotional state.

[1418] The "information provision adjustment means" is a function that adjusts the content and format of the information to be provided based on the emotion data obtained by the emotion recognition means.

[1419] The "recommendation method" is a function that records and analyzes the user's visit history and suggests the next tourist spot to visit.

[1420] "Feedback means" is a function for collecting evaluations and opinions from users and using them to improve the system.

[1421] The present invention provides a system that allows travelers to enjoy sightseeing without stress, even in different language environments. This system includes a user terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, an information provision adjustment means, a recommendation means, and a feedback means. The specific configuration and processing flow are described below.

[1422] Registering and configuring user devices

[1423] First, the user downloads a dedicated app and registers their personal information and native language. This information is sent from the device to the server and stored on the server. The user's personal information is used by the system to provide the user with the most appropriate information.

[1424] Visual device attachment and linkage

[1425] When a user picks up a visual device at an airport or rental counter, the terminal and the visual device are connected via Bluetooth or other wireless communication means. The terminal obtains the user's setting information from the server and transfers it to the visual device. Based on this setting information, the visual device is configured to allow the user to operate it in their native language.

[1426] Information translation during sightseeing

[1427] When a user uses a visual device at a tourist spot, the camera built into the visual device captures information signs and explanatory text within its field of view. The visual device extracts text data using OCR technology and sends this data to the device. The device then translates the text data into the user's native language using a translation API (e.g., Google Translate API). The translated result is sent from the device to the visual device, which then displays this information in the user's field of view.

[1428] Audio guide

[1429] The terminal converts the translated text data into audio data using TTS (Text-to-Speech) technology through the earphones of the visual device, and the visual device provides audio guidance to the user in their native language through the audio output means, allowing the user to obtain not only visual information but also audio information.

[1430] Recommendation feature

[1431] The device records the user's visit history and sends it to the server. The server analyzes this visit history and generates the next tourist destination candidates. The generated tourist destination candidates are notified to the user from the device, and a new sightseeing plan based on the user's preferences is provided.

[1432] Feedback collection

[1433] After completing a tour, users provide ratings and feedback through a dedicated app. The devices send this feedback data to a server, which analyzes it and uses it to improve the system. This ensures that the system can always respond to the latest user needs.

[1434] Emotion recognition and information provision coordination

[1435] The visual device and camera means capture the user's facial expressions and tone of voice and transmit them to the terminal. The terminal then analyzes this data using an emotion engine to evaluate the user's emotional state. Based on the evaluation results, the terminal automatically adjusts the content and format of the information it provides. For example, if it recognizes that the user is tired, it will prioritize providing information about places to relax and rest facilities. The adjusted information is then provided to the user through the visual device and audio output means.

[1436] Specific examples

[1437] When visiting the Great Buddha in Nara, the user wears a visual device and scans the information board with the camera. The device extracts the text data using OCR technology and translates it using a translation API (e.g., Google Translate API). The translated results are displayed on the visual device and simultaneously played through earphones as an audio guide. In addition, the system analyzes the user's facial expressions, and if fatigue is detected, information about nearby cafes and relaxation spots is provided. After the user finishes sightseeing and submits feedback, the system generates the next tourist destination candidate and notifies the user.

[1438] This allows users to enjoy a personalized sightseeing experience without having to worry about language barriers.

[1439] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] The user downloads and installs the dedicated app. Specifically, they search for the dedicated app in the app store and click the download button. They then follow the installation process to install the app on their device. The input data is the user's device information, and the output is the installed dedicated app.

[1442] Step 2:

[1443] The device launches the app and displays a screen for the user to enter personal information and their native language. The user enters the required information, such as their name, email address, and native language. The input data is the user's personal information and native language, and the output is the registration information stored on the device.

[1444] Step 3:

[1445] The device sends the registration information to the server. This process involves sending the user's personal information and native language to the server using an HTTP request. The input data is the registration information stored on the device, and the output is the information stored on the server.

[1446] Step 4:

[1447] The user receives and wears the visual device at an airport or rental counter. The user is instructed on how to wear the visual device and wears it correctly. The input data is the visual device received by the user, and the output is the visual device worn.

[1448] Step 5:

[1449] The device is connected to the visual device. The device pairs with the visual device using Bluetooth or other wireless communication methods. Input data is from the user's device and the visual device, and output is from the connected visual device.

[1450] Step 6:

[1451] The terminal obtains the user's setting information from the server and transfers it to the visual device. An HTTP request is made to obtain the registered information from the server. The obtained information is sent to the visual device, which then applies the native language setting. The input data is the user's setting information, and the output is the visual device with the native language setting applied.

[1452] Step 7:

[1453] The user arrives at a tourist spot and uses a visual device to capture signs and explanations within the field of view using a camera. The input data is the tourist spot's field of view information, and the output is the captured text data.

[1454] Step 8:

[1455] The visual device uses OCR technology to extract text data and send it to the terminal. The image data captured by the camera is analyzed by the OCR engine and converted into text data. The input data is the captured image, and the output is the text data sent to the terminal.

[1456] Step 9:

[1457] The device uses the translation API to translate the text data into the user's native language. The device sends the text data to the translation API and receives the translation result. The input data is the acquired text data, and the output is the text data translated into the user's native language.

[1458] Step 10:

[1459] The terminal sends the translation result to the visual device, which then displays the translation result on the AR display of the visual device. The input data is text data translated into the user's native language, and the output is the translation result displayed on the visual device.

[1460] Step 11:

[1461] The device converts the translated text data into voice data using a TTS engine. The text data is input into the TTS engine, which generates voice data. The input data is the translated text data, and the output is the generated voice data.

[1462] Step 12:

[1463] The terminal transmits audio data to the visual device, and the audio guide is reproduced by the audio output means. The user listens to the audio guide through earphones. The input data is the generated audio data, and the output is the reproduced audio guide.

[1464] Step 13:

[1465] Vision and camera means capture the user's facial expressions and tone of voice, with the input data being the user's facial expressions and tone of voice, and the output being the captured emotion data.

[1466] Step 14:

[1467] The device analyzes the captured emotion data with an emotion engine to evaluate the emotional state. The emotion engine analyzes the data and determines the user's emotional state. The input data is the captured emotion data, and the output is the analyzed emotional state.

[1468] Step 15:

[1469] The device adjusts the information it provides based on the user's emotional state. For example, if it determines that the user is tired, it prioritizes providing information about relaxing tourist spots and rest areas. The input data is the analyzed emotional state, and the output is the adjusted information.

[1470] Step 16:

[1471] The terminal transmits the adjusted information to the visual device and provides it to the user through the visual device and the audio output means, where the input data is the adjusted information and the output is the information provided through the visual device and the audio output means.

[1472] Step 17:

[1473] The device records the user's visit history and sends it to the server. Information about visited locations and GPS data are sent to the server. The input data is the recorded visit history, and the output is the visit history stored on the server.

[1474] Step 18:

[1475] The server analyzes the visit history and generates the next tourist destination candidates. Based on the visit history, an algorithm is used to recommend the next tourist destination. The input data is the saved visit history, and the output is the generated tourist destination candidates.

[1476] Step 19:

[1477] The device notifies the user of the next tourist destination candidate. The device suggests the next tourist destination to the user via a pop-up notification or message. The input data are the generated tourist destination candidates, and the output is the information notified to the user.

[1478] Step 20:

[1479] After the user has finished sightseeing, they use a dedicated app to write their feedback, which is then sent from the device to the server. The input data is the user's feedback, and the output is the feedback data sent to the server.

[1480] Step 21:

[1481] The server analyzes the feedback data and uses it to improve the system. The analysis results are used to update and improve the system. The input data is the feedback data stored on the server, and the output is an improved system.

[1482] (Application example 2)

[1483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1484] When tourists travel in different language areas, it is necessary to make it easier for them to understand information visually and aurally, and to personalize their travel experience based on their emotions, recommending places to visit, translating visual information while traveling, and providing audio guides easily and effectively.It is also important to improve the quality of the travel experience by presenting potential next tourist destinations based on the traveler's emotional state and visiting history.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, camera means for acquiring information within the field of view of the visual device, translation means for translating the information acquired by the camera means into multiple languages, display means for displaying the translated information on the visual device, audio output means for outputting the translated information as audio, emotion recognition means for analyzing facial expressions to recognize emotions, and information provision means for adjusting the provision of information based on emotion data recognized by the emotion recognition means. This makes it easier for travelers from different language backgrounds to understand information visually and audibly, and enables them to have a personalized experience based on emotions.

[1486] A "traveler's terminal" is a portable information terminal carried by a tourist while sightseeing, and is a device for registering and setting information, connecting to a visual device, and sending and receiving information.

[1487] A "visual device" is a device worn by a traveler that acquires information within the field of view and displays translated information. For example, smart glasses are examples of such devices.

[1488] The "camera means" is a camera device that is mounted on the visual device and has the function of acquiring information within the field of view.

[1489] The "translation means" refers to software and hardware for translating information acquired by the camera means into multiple languages.

[1490] The "display means" refers to a technology for displaying the information translated by the translation means on a visual device, and includes a display and the like.

[1491] "Audio output means" refers to technology that generates translated information as audio data and allows travelers to listen to it. This includes speakers and earphones.

[1492] The "emotion recognition means" is a technology that analyzes travelers' facial expressions and recognizes their emotions in real time. It includes a camera and emotion analysis software.

[1493] The "information provision means" is a technology that adjusts the information to be provided based on the emotion data recognized by the emotion recognition means. This includes displaying information and generating audio data.

[1494] "Recommendation means" is a technology that records a traveler's visit history and presents potential next tourist destinations based on this.

[1495] "Feedback methods" are technologies for collecting traveler ratings and feedback and improving the system based on the collected data.

[1496]

[1497] A system for realizing the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, and an information providing means.

[1498] Registering and configuring user devices

[1499] 1. User Registration:

[1500] Users download a dedicated app onto their mobile devices and register their personal information and native language. This information is sent from the device to the server and stored on the server.

[1501] Introduction of visual devices and information acquisition by camera means

[1502] 2. Visual equipment connection:

[1503] Visual devices (such as smart glasses) are connected to a terminal and user settings are transferred, allowing the visual device to be operated in the user's native language.

[1504] 3. Information acquisition:

[1505] At a tourist spot, a user wears a visual device and uses a camera means to acquire information within the field of view. The camera means scans guide signs and explanatory text at the tourist spot and acquires the text data.

[1506] Multilingual support through translation and display means

[1507] 4. Translation process:

[1508] The camera means transmits the captured text data to the terminal, which then translates the text into the user's native language using a translation means (such as Google Translate API). The translated text data is then transmitted to the visual device again and displayed on the visual device.

[1509] Audio guide provided

[1510] 5. Audio guide:

[1511] The device converts the text data translated by the translation means into voice data, and provides the voice data to the user through the voice output means using a voice synthesis engine (such as IBM Watson Text to Speech API).The user can then listen to the voice guide in their native language through the earphones of the visual device.

[1512] Emotion recognition and information provision

[1513] 6. Emotion recognition:

[1514] The visual device and camera means analyze the user's facial expressions and recognize emotions in real time. The device analyzes the emotion data using an emotion analysis engine (such as Microsoft Azure Emotion API).

[1515] 7. Emotional information:

[1516] The device automatically adjusts the content of the information provided based on the user's emotions. For example, if the device determines that the user is tired, it will switch to recommending tourist spots and rest areas where they can relax.

[1517] Recommendations and Feedback

[1518] 8. Recommendation function:

[1519] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidates, helping the user select a new tourist destination.

[1520] 9. Feedback Collection:

[1521] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to a server, which analyzes it and uses it to improve the system.

[1522] Specific examples

[1523] For example, when a user arrives at the Great Buddha of Nara, he or she wears a visual device and scans an information board with a camera means. The visual device transmits text data to a terminal, and the terminal translates the text into the user's native language with a translation means. The translation result is displayed on the visual device, and an audio guide is also provided through a speech synthesis engine. The user's facial expressions are analyzed in real time, and appropriate information is automatically provided.

[1524] Example prompt sentence:

[1525] "Please translate the Japanese description of the Great Buddha of Nara and provide the following information by voice. Also, please analyze the user's facial expressions and recommend tourist spots where they can relax."

[1526] In this way, the system enables travelers to overcome language barriers and enjoy a personalized, emotion-based tourism experience.

[1527] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1528] Step 1:

[1529] Users download a dedicated app and register their personal information and native language. This information is sent from the device to the server and saved on the server. The user inputs personal information such as their name and native language, and the initial setup is completed when the server saves this information.

[1530] Step 2:

[1531] After arriving at a tourist destination, the user puts on the visual device (smart glasses) and connects it to the terminal. The terminal transfers the user's setting information to the visual device, which then receives it. This allows the visual device to be operated in the user's native language. The input is the setting information from the terminal, and the output is the language setting of the visual device.

[1532] Step 3:

[1533] The user uses the camera means via the visual device to acquire information within the field of view. The camera means scans guide signs and explanatory text at tourist spots and acquires the text data. The input is the text information within the field of view, and the output is the text data.

[1534] Step 4:

[1535] The camera means transmits the captured text data to the terminal. The terminal uses the translation means (Google Translate API) to translate the text into the user's native language. The translated text data is transmitted again to the visual device and displayed on the visual device. The input is text data, and the output is translated text data.

[1536] Step 5:

[1537] The device converts the translated text data into audio data. It uses a speech synthesis engine (IBM Watson Text to Speech API) to provide audio guidance through the audio output means. The user can listen to the audio guidance in their native language through the earphones of the visual device. The input is the translated text data, and the output is audio data.

[1538] Step 6:

[1539] The visual device and camera analyze the user's facial expressions in real time and recognize emotions. The device acquires emotion data using an emotion analysis engine (Microsoft Azure Emotion API). The input is the user's facial expression data, and the output is emotion data.

[1540] Step 7:

[1541] The device automatically adjusts the content of the information it provides based on the recognized emotions. For example, if it determines that the user is tired, it will switch to recommending tourist spots and rest areas where users can relax. The input is emotion data, and the output is adjusted recommendation information.

[1542] Step 8:

[1543] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidate information and helps the user select a new tourist destination. The input is visit history data, and the output is information on the next tourist destination candidate.

[1544] Step 9:

[1545] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to the server, which analyzes it and uses it to improve the system. The input is feedback data, and the output is improved system information.

[1546] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1548] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1549] [Fourth embodiment]

[1550] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1551] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1552] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1553] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1554] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1555] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1556] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1557] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1558] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1559] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1560] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1561] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1562] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1563] MODE FOR CARRYING OUT THE INVENTION

[1564] System configuration

[1565] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, and a feedback means, thereby enabling travelers to obtain visual and audio information that can be understood in different language areas.

[1566] Program processing

[1567] Registering and configuring user devices

[1568] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1569] Visual device attachment and linkage

[1570] After arriving at the airport, the user picks up the visual device at the rental counter.

[1571] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[1572] Information translation during sightseeing

[1573] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[1574] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[1575] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[1576] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[1577] Audio guide

[1578] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[1579] Users can listen to audio guidance in their native language through the visual device's earphones.

[1580] Recommendation feature

[1581] The terminal records the user's visit history and transmits it to the server.

[1582] The server analyzes the visit history and generates the next tourist spot candidates.

[1583] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[1584] Feedback collection

[1585] After completing their tour, users provide ratings and feedback through a dedicated app.

[1586] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[1587] Specific examples

[1588] If you are visiting the Great Buddha of Nara

[1589] 1. The user arrives at the Great Buddha of Nara.

[1590] 2. The user wears the visual device and scans the information board with the camera means.

[1591] 3. The visual device sends the text data to the terminal.

[1592] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[1593] 5. The terminal returns the translation result to the visual device, which displays it.

[1594] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[1595] 7. After the tour, the user submits feedback via the app.

[1596] 8. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[1597] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and plan their next sightseeing destination.

[1598] The processing flow will be explained below.

[1599] Step 1:

[1600] The user downloads and installs the dedicated app.

[1601] Step 2:

[1602] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[1603] Step 3:

[1604] The terminal transmits the input information to the server.

[1605] Step 4:

[1606] The server stores the received information in a database and creates a profile for the user.

[1607] Step 5:

[1608] A user arrives at the airport and turns on location services on their smartphone.

[1609] Step 6:

[1610] The device sends its current location to the server.

[1611] Step 7:

[1612] The server uses the location information to identify the nearest rental counter and check the stock of audio glasses.

[1613] Step 8:

[1614] The server transmits the location of the rental counter and reservation information for the audio glasses to the terminal.

[1615] Step 9:

[1616] The user receives the audio glasses at the rental counter.

[1617] Step 10:

[1618] The device establishes a connection with the audio glasses via Bluetooth or Wi-Fi and transfers user settings to the glasses.

[1619] Step 11:

[1620] The user arrives at a tourist spot and puts on the audio glasses.

[1621] Step 12:

[1622] The audio glasses use a camera means to acquire text information within the field of view.

[1623] Step 13:

[1624] The information acquired by the audio glasses is sent to the terminal as text data.

[1625] Step 14:

[1626] The device sends text data to a translation AI and receives the text translated into the user's native language.

[1627] Step 15:

[1628] The device sends the translated text to the audio glasses.

[1629] Step 16:

[1630] Audio glasses display the translated text in the user's field of view.

[1631] Step 17:

[1632] The device converts the translated information into audio data and sends it to the earphones of the audio glasses.

[1633] Step 18:

[1634] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[1635] Step 19:

[1636] The terminal sends the visit history to the server.

[1637] Step 20:

[1638] The server analyzes the visit history and generates the next tourist spot candidates.

[1639] Step 21:

[1640] The device notifies the audio glasses of information about the next potential tourist spot.

[1641] Step 22:

[1642] The user checks the notification and selects the next destination.

[1643] Step 23:

[1644] After sightseeing, users send feedback through a dedicated app.

[1645] Step 24:

[1646] The terminal transmits the feedback data to the server.

[1647] Step 25:

[1648] The server analyzes the feedback and makes improvements to the system.

[1649] Step 26:

[1650] The server generates a discount code for the survey respondent and sends it to the terminal.

[1651] Step 27:

[1652] The terminal displays the discount code to the user.

[1653] Example 1

[1654] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] One challenge is that travelers in different language regions have difficulty accurately understanding local guide information and signs. Travelers also have a need to receive information visually and audibly in their native language. Furthermore, there is a lack of recommendations for next tourist spots during a trip, and a means to collect user feedback to improve the system.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1657] In this invention, the server includes a visual device connected to a traveler's terminal, an imaging means for acquiring information within the field of view on the visual device, a translation means for translating the information acquired by the imaging means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a text-to-speech synthesis means for the voice output means to convert the translated information into voice data, a registration means for the traveler to register personal information and native language using a dedicated app and transmit the information to the server, and a setting means for the traveler to receive the visual device at the rental counter, connect the visual device to the traveler's terminal, and transfer setting information. This allows travelers from different language-speaking regions to understand information in real time, receive recommendations for next tourist spots, and improve the system based on feedback.

[1658] A "traveler's device" is an electronic device that a traveler can carry around, such as a smartphone or tablet, which can be operated by installing a dedicated application.

[1659] "Visual devices" refer to devices such as head-mounted displays and smart glasses that can be worn by travelers and can display information within their field of vision.

[1660] "Imaging means" refers to cameras and sensors built into the visual device, including hardware and software functionality for capturing information within the field of view.

[1661] "Translation tools" means software or services for converting acquired information into another language, including machine translation APIs and translation software.

[1662] "Display means" refers to the technology and devices for displaying translated information on the display of a visual device, specifically including the screen portion of a head-mounted display or smart glasses.

[1663] "Audio output means" refers to devices or technologies for providing translated information to travelers as audio, and specifically includes earphones and speakers.

[1664] "Text-to-speech synthesis means" refers to a technology for converting text-format information into voice data, and specifically refers to a text-to-speech synthesis engine (TTS engine).

[1665] "Registration means" includes a function that allows travelers to enter personal information and their native language using a dedicated app and send that information to a server.

[1666] The "setting means" includes a function for a traveler to receive a visual device at a rental counter, connect the visual device to a terminal, and transfer setting information to the visual device.

[1667] "Recommendation methods" refer to algorithms and functions for recommending the next tourist destination based on a traveler's visiting history, and specifically include analytical techniques using machine learning algorithms.

[1668] "Feedback means" includes functions for collecting traveler ratings and feedback and improving the system based on that data.

[1669] The system of the present invention includes a traveler's terminal, a visual device, an imaging means, a translation means, a display means, a voice output means, a text-to-speech synthesis means, a registration means, a setting means, a recommendation means, and a feedback means, and allows travelers to visually and audibly obtain information that can be understood in different language areas.

[1670] The terminal is a portable communication device such as a smartphone or tablet, and is operated by installing a dedicated application. Travelers download this dedicated application and register their personal information and native language. The terminal sends the registered information to a server, where the traveler's information is stored in a database.

[1671] The visual device is a smart glass or head-mounted display worn by the traveler. The traveler picks up the visual device at the airport rental counter and connects the device to the terminal via Bluetooth. The terminal transfers the user's setting information to the visual device, allowing the visual device to be operated in the user's native language.

[1672] The imaging means is a camera built into the visual device, and the traveler scans the guide signs and explanatory text at the tourist spot. This camera has high resolution and can accurately capture the tourist spot information. The text data acquired by the imaging means is transmitted to the terminal in real time.

[1673] The terminal translates the text data sent from the imaging means using the Google Translate API. The translation means translates the text data into the traveler's native language. The translated text data is sent from the terminal to the visual device and displayed on the visual device's built-in display. The user can visually check the information on the signboard in their native language.

[1674] Furthermore, the terminal uses a text-to-speech synthesis means (TTS engine) to convert the translated text data into voice data, which is provided to the user through the earphone of the visual device, allowing the user to hear the voice guidance in their native language.

[1675] As a recommendation method, the device records the traveler's visit history and periodically sends this data to the server, which then uses a machine learning algorithm to generate next tourist destination candidates from the visit history and propose the best sightseeing routes and spots for the traveler.

[1676] As a feedback method, travelers provide feedback using a dedicated app after completing their tour. The device sends this feedback data to a server, which analyzes the data and uses it to improve the system. Specific analytical methods include data mining and statistical analysis.

[1677] Specific examples

[1678] If you are visiting the Great Buddha of Nara

[1679] 1. The user arrives at the Great Buddha of Nara.

[1680] 2. The user wears the visual device and scans the information board with the built-in camera.

[1681] 3. The text data acquired by the visual device is sent to the terminal.

[1682] 4. The device uses the Google Translate API to translate the text data into the user's native language.

[1683] 5. The translated text data is displayed on the visual device's display.

[1684] 6. At the same time, the device uses a text-to-speech engine (TTS engine) to convert the data into voice data and provide voice guidance in the user's native language through the visual device's earphones.

[1685] 7. After the tour is completed, the user submits feedback via a dedicated app.

[1686] 8. The device sends the feedback data to the server, which analyzes the data and generates the next tourist destination candidates.

[1687] Prompt Sentence Examples

[1688] "Please explain in detail about the Great Buddha of Nara. Please include important historical background and tourist attractions to interest visitors."

[1689] This will make travellers feel more comfortable and understandable when travelling in different language areas.

[1690] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1691] Step 1:

[1692] The user downloads the dedicated app and launches it.

[1693] Input: Smartphone or tablet, internet connection.

[1694] Specific operation: The user downloads and installs the dedicated app from the app store. After installation, the user launches the app.

[1695] Output: The dedicated app starts and the start screen is displayed.

[1696] Step 2:

[1697] The user registers personal information and native language, and the terminal transmits this information to the server.

[1698] Input: Personal information such as name, nationality, native language, etc.

[1699] Specific operation: The user enters personal information into the app's registration screen and presses the submit button. The device then sends the entered information to the server using HTTPS.

[1700] Output: Personal information is stored on the server.

[1701] Step 3:

[1702] A user rents a visual device at an airport and connects the device to a terminal.

[1703] Input: reservation number, passport, visual device.

[1704] Specific operation: The user receives the visual device at the airport rental counter, connects the device to the terminal via Bluetooth, and the terminal transfers the configuration information to the visual device.

[1705] Output: The visual device can be operated in the user's native language.

[1706] Step 4:

[1707] A user wears a visual device at a tourist spot and scans guide signs and explanatory texts with an imaging means.

[1708] Input: Text information for signs and explanations.

[1709] Specific operation: A user wears a visual device at a tourist spot and takes a picture of text information with a camera. The image capturing means acquires the text data.

[1710] Output: The captured text data is sent to the terminal.

[1711] Step 5:

[1712] The device translates the acquired text data using the Google Translate API.

[1713] Input: The captured text data.

[1714] Specific operation: The device sends the acquired text data to the Google Translate API and receives the translation result.

[1715] Output: The translated text data is saved on the device.

[1716] Step 6:

[1717] The terminal transmits the translated text data to the visual device, which displays it.

[1718] Input: Translated text data.

[1719] Specific operation: The terminal sends the translated text data to the visual device, which then displays it on the display of the visual device.

[1720] Output: The translated information is displayed in the user's field of view in their native language.

[1721] Step 7:

[1722] The terminal converts the translated text data into voice data and provides it to the user through the voice output means.

[1723] Input: Translated text data.

[1724] Specific operation: The device uses a text-to-speech engine (TTS engine) to convert the translated text data into audio data, which is then provided to the user through the earphones of the visual device.

[1725] Output: The user can hear the translated audio guidance in their native language.

[1726] Step 8:

[1727] The terminal records the user's visit history and sends it to the server.

[1728] Input: Information about the tourist attractions visited and the time.

[1729] Specific operation: The device records the user's visit history using GPS and timestamps. The recorded information is periodically sent to the server.

[1730] Output: The user's visit history is saved on the server.

[1731] Step 9:

[1732] The server uses a machine learning algorithm to generate the next tourist destination candidates.

[1733] Input: User's visit history data.

[1734] Specific operation: The server analyzes the user's visit history and generates the next tourist spot candidate using a machine learning algorithm. The generated candidate spot information is sent to the device.

[1735] Output: A list of potential next tourist spots will be displayed on the terminal.

[1736] Step 10:

[1737] After completing their tour, users provide feedback through a dedicated app.

[1738] Input: Traveler ratings and feedback.

[1739] Specific operation: The user enters ratings and comments about the sightseeing experience on the feedback screen within the dedicated app.

[1740] Output: The feedback data is sent to the server.

[1741] Step 11:

[1742] The server analyzes the feedback data and makes improvements to the system.

[1743] Input: Feedback data.

[1744] What happens: The server analyzes the collected feedback data and identifies areas for improvement. The analysis results are sent to the development team to help improve the system.

[1745] Output: The improved system is reflected in the next use.

[1746] (Application example 1)

[1747] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1748] In today's globalized society, language differences are a major barrier for travelers and customers in brick-and-mortar stores. This has created a demand for systems that can easily understand information across different language regions and provide personalized information to individual users. Furthermore, conventional systems have difficulty integrating visual and auditory information, and have not provided sufficient convenience to improve the user experience.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1750] In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, a recommendation means for analyzing the acquired information and recommending information based on the user's past behavior history, a user interface means for presenting the information to the user on the visual device, and a data integration means for linking the visual display of the information and the voice guidance, thereby enabling users from different language backgrounds to understand information in real time and receive individually personalized information.

[1751] A "traveler's terminal" is an electronic device that can be carried by a traveler and is a device for acquiring and displaying information by connecting to a visual device, translation means, display means, etc.

[1752] A "visual device" is a device worn by a user that acquires information within the field of view and displays that information, and includes, in particular, camera means and display means.

[1753] The "camera means" is a device that is mounted on the visual device and has the function of acquiring information within the field of view as an image.

[1754] The "translation means" is a means capable of translating acquired information into a language designated by the user and capable of handling multiple languages.

[1755] "Display means" means a device capable of displaying translated information and other important information on the display of a visual device.

[1756] The "audio output means" is a device that has the function of providing the translated information to the user as audio.

[1757] The "recommendation means" is a device that has the function of selecting and presenting the next recommended information or product based on the user's past behavioral history and purchase history.

[1758] "User interface means" refers to a means by which a user accesses information and performs operations on a visual device, and is an interface that mediates operations between the user and the system.

[1759] The "data integration means" is a means for centrally managing the data for visual displays and audio guidance, linking them together, and providing them to the user.

[1760] System configuration

[1761] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, a recommendation means, a user interface means, and a data integration means, thereby enabling users to understand information in real time and receive personalized information in different languages.

[1762] Program processing

[1763] Registering and configuring user devices

[1764] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1765] Visual device attachment and linkage

[1766] When a user arrives at a physical store, they put on the smart glasses. The device connects with the visual device, and the user's settings are transferred to the visual device, allowing the device to be operated in the user's native language.

[1767] Product information translation and display

[1768] A user uses a visual device in a physical store and scans a product label or price tag using a camera means. The camera means of the visual device acquires product information and transmits the text data to a terminal. The terminal uses a translation means to translate the text into the user's native language. The translated text data is transmitted to the visual device, which displays it in the user's field of view.

[1769] Audio guide

[1770] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means, so that the user can hear the product description in their own language through the earphones of the visual device.

[1771] Recommendation feature

[1772] The device records the user's purchase history and sends it to the server. The server analyzes the purchase history and generates the next recommended product. The device notifies the user of the next recommended product information, helping the user select a new product.

[1773] Hardware and Software Used

[1774] The following hardware and software are used to realize the system.

[1775] Visual device: smart glasses

[1776] Camera means: Built-in camera in smart glasses

[1777] Translation method: Google Translate API

[1778] Audio output method: Pyttsx3 audio engine

[1779] Display: Smart glasses display

[1780] User interface: A dedicated application that works with smart glasses

[1781] Data integration tool: Software that integrates acquired information and links visual displays with audio guidance

[1782] Specific examples

[1783] When a user wears smart glasses in a physical store and scans a product label with the camera, the product name and details are translated into the user's native language and displayed on the smart glasses' display. The information is also presented audibly, allowing the user to obtain both visual and auditory information. Meanwhile, the server analyzes the user's past purchase history and recommends new related products. These recommendations are also provided to the user via the smart glasses' display and audio.

[1784] Prompt Sentence Examples

[1785] 1. When a user wears smart glasses in a physical store and scans a product label, the product information is translated and displayed.

[1786] 2. The text data acquired by the camera is translated into English using Google Translate.

[1787] 3. The translation results are displayed on the smart glasses screen and also explained aloud.

[1788] 4. Recommended products are displayed based on the user's purchase history and guidance is provided via voice.

[1789] This allows users to overcome language barriers and enjoy fashionable shopping.

[1790] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1791] Step 1:

[1792] A user puts on the smart glasses and launches the dedicated application. The input is the user's personal information and native language setting, which are recorded on the device and sent to the server. The output is that the smart glasses are ready for use and the user information is saved on the server.

[1793] Step 2:

[1794] The terminal is connected to the smart glasses, and the user's setting information is transferred to the smart glasses. The input is the user setting information from the server, and the output is that the smart glasses can be operated in the user's native language.

[1795] Step 3:

[1796] A user uses a camera in a physical store to scan product labels or price tags. The input is image data of the product information acquired by the camera, and the output is the product information extracted as text data. This text data is sent to a terminal.

[1797] Step 4:

[1798] The server receives the text data and translates it into the user's native language using a translation means. The input is the text data of the product information and the user's native language information, and the output is the translated text data of the product information.

[1799] Step 5:

[1800] The translated text data is sent to the terminal, which then displays it on the display of the smart glasses. The input is the translated text data of the product information, and the output is the product information displayed on the display of the smart glasses.

[1801] Step 6:

[1802] The terminal converts the translated text data into audio data and provides it to the user through the audio output means. The input is the translated text data of the product information, and the output is audio data. The user can listen to the product explanation in their own language through the earphones of the visual device.

[1803] Step 7:

[1804] The terminal records the user's purchase history and sends the data to the server. The input is the user's purchase history, and the output is the purchase history data stored on the server.

[1805] Step 8:

[1806] The server analyzes the purchase history and generates the next recommended product. The input is the user's purchase history, and the output is the information about the recommended product.

[1807] Step 9:

[1808] Recommended product information is sent to the terminal, which then displays the information on the smart glasses' display and provides audio guidance. The input is the recommended product information, and the output is the information displayed on the smart glasses' display and an audio product description.

[1809] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1810] MODE FOR CARRYING OUT THE INVENTION

[1811] System configuration

[1812] The system of the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, an audio output means, a recommendation means, a feedback means, and an emotion engine, allowing travelers to obtain visual and audio information that can be understood in different language areas, and providing a personalized experience based on emotions.

[1813] Program processing

[1814] Registering and configuring user devices

[1815] Users first download the app and register their personal information and native language, which is then sent from the device to the server and stored on the server.

[1816] Visual device attachment and linkage

[1817] After arriving at the airport, the user picks up the visual device at the rental counter.

[1818] The terminal is connected to the visual device, and the user's setting information is transferred to the visual device, allowing the visual device to be operated in the user's native language.

[1819] Information translation during sightseeing

[1820] A user uses a visual device at a tourist spot and acquires information within the field of view using a camera means.

[1821] The camera means of the visual device scans guide signs and explanatory texts at tourist spots and acquires the text data.

[1822] The visual device transmits the acquired text data to the terminal, which uses a translation means to translate the text into the user's native language.

[1823] The terminal transmits the translated text data to the visual device, which displays it in the user's field of view.

[1824] Audio guide

[1825] The terminal converts the text data translated by the translation means into voice data and provides it to the user through the voice output means.

[1826] Users can listen to audio guidance in their native language through the visual device's earphones.

[1827] Recommendation feature

[1828] The terminal records the user's visit history and transmits it to the server.

[1829] The server analyzes the visit history and generates the next tourist spot candidates.

[1830] The terminal notifies the user of information about the next tourist spot candidates, helping the user to select a new tourist spot.

[1831] Feedback collection

[1832] After completing their tour, users provide ratings and feedback through a dedicated app.

[1833] The device sends feedback data to the server, which analyzes it and uses it to improve the system.

[1834] emotion recognition

[1835] The visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[1836] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[1837] Emotion-based information provision

[1838] The device can automatically adjust the content of the information provided based on the recognized emotion. For example, if it determines that the user is tired, it can switch to information recommending tourist spots and rest areas where they can relax.

[1839] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[1840] Specific examples

[1841] If you are visiting the Great Buddha of Nara

[1842] 1. The user arrives at the Great Buddha of Nara.

[1843] 2. The user wears the visual device and scans the information board with the camera means.

[1844] 3. The visual device sends the text data to the terminal.

[1845] 4. The terminal passes the data to the translation means, which translates it into the user's native language.

[1846] 5. The terminal returns the translation result to the visual device, which displays it.

[1847] 6. The terminal simultaneously converts the translated text into voice data and provides it to the user through the voice output means.

[1848] 7. The terminal and camera means analyze the user's facial expressions and recognize the user's emotions in real time.

[1849] 8. The device adjusts tourist information and next recommended locations based on the user's current emotional state.

[1850] 9. After the tour, the user submits feedback via the app.

[1851] 10. The device sends the visit history and feedback to the server, and the server generates new tourist destination candidates.

[1852] This allows users to gain a deeper understanding of the Great Buddha of Nara without language barriers and to have the optimal sightseeing experience according to their own emotional state.

[1853] The processing flow will be explained below.

[1854] Step 1:

[1855] The user downloads and installs the dedicated app.

[1856] Step 2:

[1857] A user launches the app and enters their personal information (name, native language, contact details, etc.).

[1858] Step 3:

[1859] The terminal transmits the input information to the server.

[1860] Step 4:

[1861] The server stores the received information in a database and creates a profile for the user.

[1862] Step 5:

[1863] A user arrives at the airport and turns on location services on their smartphone.

[1864] Step 6:

[1865] The device sends its current location to the server.

[1866] Step 7:

[1867] The server uses the location information to identify the nearest rental counter and check the inventory of the visual device.

[1868] Step 8:

[1869] The server transmits the location of the rental counter and reservation information for the visual device to the terminal.

[1870] Step 9:

[1871] The user picks up the visual device at the rental counter.

[1872] Step 10:

[1873] The device establishes a connection with the visual device via Bluetooth or Wi-Fi and transfers user settings to the visual device.

[1874] Step 11:

[1875] The user arrives at the tourist spot and puts on the visual device.

[1876] Step 12:

[1877] The vision device utilizes a camera means to capture textual information within the field of view.

[1878] Step 13:

[1879] The information acquired by the visual device is sent to the terminal as text data.

[1880] Step 14:

[1881] The device sends text data to a translation AI and receives the text translated into the user's native language.

[1882] Step 15:

[1883] The terminal sends the translated text to the visual device.

[1884] Step 16:

[1885] A visual device displays the translated text in the user's field of view.

[1886] Step 17:

[1887] The device converts the translated information into audio data and transmits it to the visual device's earphones.

[1888] Step 18:

[1889] As the user continues sightseeing, information about the places visited is recorded on the terminal.

[1890] Step 19:

[1891] The terminal sends the visit history to the server.

[1892] Step 20:

[1893] The server analyzes the visit history and generates the next tourist spot candidates.

[1894] Step 21:

[1895] The terminal notifies the visual device of information about the next tourist spot candidate.

[1896] Step 22:

[1897] The user checks the notification and selects the next destination.

[1898] Step 23:

[1899] While the user is sightseeing, the visual device and camera means analyze the user's facial expressions and voice to recognize emotions in real time.

[1900] Step 24:

[1901] The terminal uses an emotion engine to analyze the recognized emotion data and determine the user's current emotional state.

[1902] Step 25:

[1903] The device automatically adjusts the content of the information provided based on the recognized emotion.

[1904] Step 26:

[1905] Information that matches the user's emotions is provided by display and voice through the display means and voice output means.

[1906] Step 27:

[1907] After completing their tour, users submit their ratings and feedback through a dedicated app.

[1908] Step 28:

[1909] The terminal transmits the feedback data to the server.

[1910] Step 29:

[1911] The server analyzes the feedback and makes improvements to the system.

[1912] Step 30:

[1913] The server generates a discount code for the survey respondent and sends it to the terminal.

[1914] Step 31:

[1915] The terminal displays the discount code to the user.

[1916] Example 2

[1917] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1918] There is a need to solve the problems of travelers being unable to effectively obtain information in different language environments, and of tourism experiences being uniform and not personalized based on the feelings and preferences of individual travelers. Another issue is that recommendations for next destinations are not made using visit history and feedback, and the optimal travel plan for travelers is not proposed.

[1919] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1920] In this invention, the server includes a user terminal, a visual device connected to the terminal, a camera means for acquiring information within the field of view of the visual device, a translation means for translating the information acquired by the camera means into multiple languages, a display means for displaying the translated information on the visual device, a voice output means for outputting the translated information as voice, an emotion recognition means for analyzing the user's facial expressions and tone of voice to evaluate their emotional state, and an information provision adjustment means for adjusting the information provided based on the emotion acquired by the emotion recognition means. This allows travelers to easily acquire information in different language environments and provides a personalized sightseeing experience tailored to their individual emotional state and preferences. It also enables appropriate recommendations of next destinations based on visit history and feedback.

[1921] A "user terminal" is a portable information processing device such as a smartphone or tablet that is carried by a traveler.

[1922] "Visual devices" is a general term for head-mounted displays and AR glasses that have the function of expanding visual information when worn by the user.

[1923] "Camera means" refers to a camera built into the visual device, which is a device that captures specific information within its field of view.

[1924] The "translation means" is a function including software and hardware that converts acquired text data into a different language.

[1925] "Display means" refers to a display device for displaying the translated information in the user's field of view.

[1926] The "audio output means" refers to an earphone or speaker device that provides the translated information to the user as audio.

[1927] "Emotion recognition means" refers to technology and software for analyzing a user's facial expressions and tone of voice to assess their emotional state.

[1928] The "information provision adjustment means" is a function that adjusts the content and format of the information to be provided based on the emotion data obtained by the emotion recognition means.

[1929] The "recommendation method" is a function that records and analyzes the user's visit history and suggests the next tourist spot to visit.

[1930] "Feedback means" is a function for collecting evaluations and opinions from users and using them to improve the system.

[1931] The present invention provides a system that allows travelers to enjoy sightseeing without stress, even in different language environments. This system includes a user terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, an information provision adjustment means, a recommendation means, and a feedback means. The specific configuration and processing flow are described below.

[1932] Registering and configuring user devices

[1933] First, the user downloads a dedicated app and registers their personal information and native language. This information is sent from the device to the server and stored on the server. The user's personal information is used by the system to provide the user with the most appropriate information.

[1934] Visual device attachment and linkage

[1935] When a user picks up a visual device at an airport or rental counter, the terminal and the visual device are connected via Bluetooth or other wireless communication means. The terminal obtains the user's setting information from the server and transfers it to the visual device. Based on this setting information, the visual device is configured to allow the user to operate it in their native language.

[1936] Information translation during sightseeing

[1937] When a user uses a visual device at a tourist spot, the camera built into the visual device captures information signs and explanatory text within its field of view. The visual device extracts text data using OCR technology and sends this data to the device. The device then translates the text data into the user's native language using a translation API (e.g., Google Translate API). The translated result is sent from the device to the visual device, which then displays this information in the user's field of view.

[1938] Audio guide

[1939] The terminal converts the translated text data into audio data using TTS (Text-to-Speech) technology through the earphones of the visual device, and the visual device provides audio guidance to the user in their native language through the audio output means, allowing the user to obtain not only visual information but also audio information.

[1940] Recommendation feature

[1941] The device records the user's visit history and sends it to the server. The server analyzes this visit history and generates the next tourist destination candidates. The generated tourist destination candidates are notified to the user from the device, and a new sightseeing plan based on the user's preferences is provided.

[1942] Feedback collection

[1943] After completing a tour, users provide ratings and feedback through a dedicated app. The devices send this feedback data to a server, which analyzes it and uses it to improve the system. This ensures that the system can always respond to the latest user needs.

[1944] Emotion recognition and information provision coordination

[1945] The visual device and camera means capture the user's facial expressions and tone of voice and transmit them to the terminal. The terminal then analyzes this data using an emotion engine to evaluate the user's emotional state. Based on the evaluation results, the terminal automatically adjusts the content and format of the information it provides. For example, if it recognizes that the user is tired, it will prioritize providing information about places to relax and rest facilities. The adjusted information is then provided to the user through the visual device and audio output means.

[1946] Specific examples

[1947] When visiting the Great Buddha in Nara, the user wears a visual device and scans the information board with the camera. The device extracts the text data using OCR technology and translates it using a translation API (e.g., Google Translate API). The translated results are displayed on the visual device and simultaneously played through earphones as an audio guide. In addition, the system analyzes the user's facial expressions, and if fatigue is detected, information about nearby cafes and relaxation spots is provided. After the user finishes sightseeing and submits feedback, the system generates the next tourist destination candidate and notifies the user.

[1948] This allows users to enjoy a personalized sightseeing experience without having to worry about language barriers.

[1949] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1950] Step 1:

[1951] The user downloads and installs the dedicated app. Specifically, they search for the dedicated app in the app store and click the download button. They then follow the installation process to install the app on their device. The input data is the user's device information, and the output is the installed dedicated app.

[1952] Step 2:

[1953] The device launches the app and displays a screen for the user to enter personal information and their native language. The user enters the required information, such as their name, email address, and native language. The input data is the user's personal information and native language, and the output is the registration information stored on the device.

[1954] Step 3:

[1955] The device sends the registration information to the server. This process involves sending the user's personal information and native language to the server using an HTTP request. The input data is the registration information stored on the device, and the output is the information stored on the server.

[1956] Step 4:

[1957] The user receives and wears the visual device at an airport or rental counter. The user is instructed on how to wear the visual device and wears it correctly. The input data is the visual device received by the user, and the output is the visual device worn.

[1958] Step 5:

[1959] The device is connected to the visual device. The device pairs with the visual device using Bluetooth or other wireless communication methods. Input data is from the user's device and the visual device, and output is from the connected visual device.

[1960] Step 6:

[1961] The terminal obtains the user's setting information from the server and transfers it to the visual device. An HTTP request is made to obtain the registered information from the server. The obtained information is sent to the visual device, which then applies the native language setting. The input data is the user's setting information, and the output is the visual device with the native language setting applied.

[1962] Step 7:

[1963] The user arrives at a tourist spot and uses a visual device to capture signs and explanations within the field of view using a camera. The input data is the tourist spot's field of view information, and the output is the captured text data.

[1964] Step 8:

[1965] The visual device uses OCR technology to extract text data and send it to the terminal. The image data captured by the camera is analyzed by the OCR engine and converted into text data. The input data is the captured image, and the output is the text data sent to the terminal.

[1966] Step 9:

[1967] The device uses the translation API to translate the text data into the user's native language. The device sends the text data to the translation API and receives the translation result. The input data is the acquired text data, and the output is the text data translated into the user's native language.

[1968] Step 10:

[1969] The terminal sends the translation result to the visual device, which then displays the translation result on the AR display of the visual device. The input data is text data translated into the user's native language, and the output is the translation result displayed on the visual device.

[1970] Step 11:

[1971] The device converts the translated text data into voice data using a TTS engine. The text data is input into the TTS engine, which generates voice data. The input data is the translated text data, and the output is the generated voice data.

[1972] Step 12:

[1973] The terminal transmits audio data to the visual device, and the audio guide is reproduced by the audio output means. The user listens to the audio guide through earphones. The input data is the generated audio data, and the output is the reproduced audio guide.

[1974] Step 13:

[1975] Vision and camera means capture the user's facial expressions and tone of voice, with the input data being the user's facial expressions and tone of voice, and the output being the captured emotion data.

[1976] Step 14:

[1977] The device analyzes the captured emotion data with an emotion engine to evaluate the emotional state. The emotion engine analyzes the data and determines the user's emotional state. The input data is the captured emotion data, and the output is the analyzed emotional state.

[1978] Step 15:

[1979] The device adjusts the information it provides based on the user's emotional state. For example, if it determines that the user is tired, it prioritizes providing information about relaxing tourist spots and rest areas. The input data is the analyzed emotional state, and the output is the adjusted information.

[1980] Step 16:

[1981] The terminal transmits the adjusted information to the visual device and provides it to the user through the visual device and the audio output means, where the input data is the adjusted information and the output is the information provided through the visual device and the audio output means.

[1982] Step 17:

[1983] The device records the user's visit history and sends it to the server. Information about visited locations and GPS data are sent to the server. The input data is the recorded visit history, and the output is the visit history stored on the server.

[1984] Step 18:

[1985] The server analyzes the visit history and generates the next tourist destination candidates. Based on the visit history, an algorithm is used to recommend the next tourist destination. The input data is the saved visit history, and the output is the generated tourist destination candidates.

[1986] Step 19:

[1987] The device notifies the user of the next tourist destination candidate. The device suggests the next tourist destination to the user via a pop-up notification or message. The input data are the generated tourist destination candidates, and the output is the information notified to the user.

[1988] Step 20:

[1989] After the user has finished sightseeing, they use a dedicated app to write their feedback, which is then sent from the device to the server. The input data is the user's feedback, and the output is the feedback data sent to the server.

[1990] Step 21:

[1991] The server analyzes the feedback data and uses it to improve the system. The analysis results are used to update and improve the system. The input data is the feedback data stored on the server, and the output is an improved system.

[1992] (Application example 2)

[1993] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1994] When tourists travel in different language areas, it is necessary to make it easier for them to understand information visually and aurally, and to personalize their travel experience based on their emotions, recommending places to visit, translating visual information while traveling, and providing audio guides easily and effectively.It is also important to improve the quality of the travel experience by presenting potential next tourist destinations based on the traveler's emotional state and visiting history.

[1995] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a traveler's terminal, a visual device connected to the terminal, camera means for acquiring information within the field of view of the visual device, translation means for translating the information acquired by the camera means into multiple languages, display means for displaying the translated information on the visual device, audio output means for outputting the translated information as audio, emotion recognition means for analyzing facial expressions to recognize emotions, and information provision means for adjusting the provision of information based on emotion data recognized by the emotion recognition means. This makes it easier for travelers from different language backgrounds to understand information visually and audibly, and enables them to have a personalized experience based on emotions.

[1996] A "traveler's terminal" is a portable information terminal carried by a tourist while sightseeing, and is a device for registering and setting information, connecting to a visual device, and sending and receiving information.

[1997] A "visual device" is a device worn by a traveler that acquires information within the field of view and displays translated information. For example, smart glasses are examples of such devices.

[1998] The "camera means" is a camera device that is mounted on the visual device and has the function of acquiring information within the field of view.

[1999] The "translation means" refers to software and hardware for translating information acquired by the camera means into multiple languages.

[2000] The "display means" refers to a technology for displaying the information translated by the translation means on a visual device, and includes a display and the like.

[2001] "Audio output means" refers to technology that generates translated information as audio data and allows travelers to listen to it. This includes speakers and earphones.

[2002] The "emotion recognition means" is a technology that analyzes travelers' facial expressions and recognizes their emotions in real time. It includes a camera and emotion analysis software.

[2003] The "information provision means" is a technology that adjusts the information to be provided based on the emotion data recognized by the emotion recognition means. This includes displaying information and generating audio data.

[2004] "Recommendation means" is a technology that records a traveler's visit history and presents potential next tourist destinations based on this.

[2005] "Feedback methods" are technologies for collecting traveler ratings and feedback and improving the system based on the collected data.

[2006]

[2007] A system for realizing the present invention includes a traveler's terminal, a visual device, a camera means, a translation means, a display means, a voice output means, an emotion recognition means, and an information providing means.

[2008] Registering and configuring user devices

[2009] 1. User Registration:

[2010] Users download a dedicated app onto their mobile devices and register their personal information and native language. This information is sent from the device to the server and stored on the server.

[2011] Introduction of visual devices and information acquisition by camera means

[2012] 2. Visual equipment connection:

[2013] Visual devices (such as smart glasses) are connected to a terminal and user settings are transferred, allowing the visual device to be operated in the user's native language.

[2014] 3. Information acquisition:

[2015] At a tourist spot, a user wears a visual device and uses a camera means to acquire information within the field of view. The camera means scans guide signs and explanatory text at the tourist spot and acquires the text data.

[2016] Multilingual support through translation and display means

[2017] 4. Translation process:

[2018] The camera means transmits the captured text data to the terminal, which then translates the text into the user's native language using a translation means (such as Google Translate API). The translated text data is then transmitted to the visual device again and displayed on the visual device.

[2019] Audio guide provided

[2020] 5. Audio guide:

[2021] The device converts the text data translated by the translation means into voice data, and provides the voice data to the user through the voice output means using a voice synthesis engine (such as IBM Watson Text to Speech API).The user can then listen to the voice guide in their native language through the earphones of the visual device.

[2022] Emotion recognition and information provision

[2023] 6. Emotion recognition:

[2024] The visual device and camera means analyze the user's facial expressions and recognize emotions in real time. The device analyzes the emotion data using an emotion analysis engine (such as Microsoft Azure Emotion API).

[2025] 7. Emotional information:

[2026] The device automatically adjusts the content of the information provided based on the user's emotions. For example, if the device determines that the user is tired, it will switch to recommending tourist spots and rest areas where they can relax.

[2027] Recommendations and Feedback

[2028] 8. Recommendation function:

[2029] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidates, helping the user select a new tourist destination.

[2030] 9. Feedback Collection:

[2031] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to a server, which analyzes it and uses it to improve the system.

[2032] Specific examples

[2033] For example, when a user arrives at the Great Buddha of Nara, he or she wears a visual device and scans an information board with a camera means. The visual device transmits text data to a terminal, and the terminal translates the text into the user's native language with a translation means. The translation result is displayed on the visual device, and an audio guide is also provided through a speech synthesis engine. The user's facial expressions are analyzed in real time, and appropriate information is automatically provided.

[2034] Example prompt sentence:

[2035] "Please translate the Japanese description of the Great Buddha of Nara and provide the following information by voice. Also, please analyze the user's facial expressions and recommend tourist spots where they can relax."

[2036] In this way, the system enables travelers to overcome language barriers and enjoy a personalized, emotion-based tourism experience.

[2037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2038] Step 1:

[2039] Users download a dedicated app and register their personal information and native language. This information is sent from the device to the server and saved on the server. The user inputs personal information such as their name and native language, and the initial setup is completed when the server saves this information.

[2040] Step 2:

[2041] After arriving at a tourist destination, the user puts on the visual device (smart glasses) and connects it to the terminal. The terminal transfers the user's setting information to the visual device, which then receives it. This allows the visual device to be operated in the user's native language. The input is the setting information from the terminal, and the output is the language setting of the visual device.

[2042] Step 3:

[2043] The user uses the camera means via the visual device to acquire information within the field of view. The camera means scans guide signs and explanatory text at tourist spots and acquires the text data. The input is the text information within the field of view, and the output is the text data.

[2044] Step 4:

[2045] The camera means transmits the captured text data to the terminal. The terminal uses the translation means (Google Translate API) to translate the text into the user's native language. The translated text data is transmitted again to the visual device and displayed on the visual device. The input is text data, and the output is translated text data.

[2046] Step 5:

[2047] The device converts the translated text data into audio data. It uses a speech synthesis engine (IBM Watson Text to Speech API) to provide audio guidance through the audio output means. The user can listen to the audio guidance in their native language through the earphones of the visual device. The input is the translated text data, and the output is audio data.

[2048] Step 6:

[2049] The visual device and camera analyze the user's facial expressions in real time and recognize emotions. The device acquires emotion data using an emotion analysis engine (Microsoft Azure Emotion API). The input is the user's facial expression data, and the output is emotion data.

[2050] Step 7:

[2051] The device automatically adjusts the content of the information it provides based on the recognized emotions. For example, if it determines that the user is tired, it will switch to recommending tourist spots and rest areas where users can relax. The input is emotion data, and the output is adjusted recommendation information.

[2052] Step 8:

[2053] The device records the user's visit history and sends it to the server. The server analyzes the visit history and generates the next tourist destination candidates. The device notifies the user of the next tourist destination candidate information and helps the user select a new tourist destination. The input is visit history data, and the output is information on the next tourist destination candidate.

[2054] Step 9:

[2055] After completing their tour, users provide ratings and feedback through a dedicated app. The device sends the feedback data to the server, which analyzes it and uses it to improve the system. The input is feedback data, and the output is improved system information.

[2056] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2058] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2059] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2060] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2061] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2062] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2063] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2064] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings su...

Claims

1. The traveler's device, a visual device connected to the terminal; camera means for capturing information within the field of view of said visual device; a translation means for translating the information acquired by the camera means into multiple languages; a display means for displaying the translated information on a visual device; a voice output means for outputting the translated information as voice; A system including:

2. 2. The system according to claim 1, further comprising a recommendation unit that records a visit history of a traveler and presents next tourist spot candidates based on the visit history.

3. 10. The system of claim 1, further comprising a feedback means for collecting traveler ratings and feedback and improving the system based on the collected data.

Images