system

Abstract

Provide a system. A means for uploading images taken by a terminal to a cloud server; A means for analyzing the image received by the cloud server and detecting a specific item; A cloud server generates guide information for the detected item; means for transmitting guide information generated by the cloud server to the terminal; means for displaying and audibly reproducing the guide information received by the terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern travelers need to quickly and easily obtain detailed information about their destinations, but traditional guidebooks and guide displays lack real-timeness and flexibility. Furthermore, referring to multiple sources to obtain information about specific locations or items can be cumbersome and detract from the travel experience. Furthermore, language barriers make understanding the information even more difficult. To solve these problems, a system combining image recognition technology and generative AI technology is needed. [Means for solving the problem]

[0005] The present invention provides a system that uploads images taken by a terminal to a cloud server, and the cloud server analyzes the received images to detect specific items. It also provides a system that includes a means for the cloud server to generate guide information for the detected items and transmit this guide information to the terminal. Furthermore, by displaying and playing back the guide information received by the terminal as audio, travelers can obtain detailed guide information in real time based on the images they have taken, enabling quick and easy access to information.

[0006] A "terminal" is an electronic device that allows a user to take images and communicate with a cloud server. Examples include smartphones and tablets.

[0007] A "cloud server" is a computer system that receives images uploaded from terminals, analyzes them, and sends the generated guide information.

[0008] "Means for uploading images to a cloud server" refers to the functions and procedures for sending image data from a terminal to a cloud server.

[0009] "Means for analyzing images and detecting specific items" refers to image recognition technology or algorithms that process received images and identify specific objects or locations.

[0010] "Means for generating guide information" refers to artificial intelligence technology or a natural language generation engine for automatically creating explanatory text and guide information about detected items.

[0011] "Means for transmitting guide information to the terminal" refers to communication means or protocols for sending the generated guide information as data from the cloud server to the terminal.

[0012] "Means for displaying and audibly reproducing received guide information" refers to the function of visually displaying the guide information received by the terminal on the screen and providing audible guidance to the user using voice synthesis technology.

[0013] An "image recognition engine" refers to a computer program or system that uses image analysis algorithms to identify specific objects or locations in images.

[0014] A "natural language generation engine" refers to an artificial intelligence technology that automatically generates text in a format that is easy for humans to understand based on specified information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that automatically provides guide information based on images taken by travelers, and specific embodiments thereof will be described below.

[0037] 1. Image capture and upload

[0038] Image Capture

[0039] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0040] Device: Captured images are temporarily stored on the device.

[0041] Uploading an image

[0042] Device: The saved images are uploaded to the cloud server via the network, which checks the network connection and manages the transaction.

[0043] 2. Item detection using image recognition

[0044] Image analysis

[0045] Server: The cloud server analyzes the received image data, specifically using an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements in the image and detect specific items (e.g., the Great Buddha).

[0046] Server: Receives the response from the image recognition engine and proceeds to the next step based on the detection results.

[0047] 3. Generating and Providing Guide Information

[0048] Guide information generation

[0049] Server: Requests detailed guide information about the detected item from the generative AI. The generative AI (e.g., a natural language generation engine) generates useful guide information for the user based on previously learned knowledge about the detected item (e.g., the Great Buddha).

[0050] Server: The generated guide information is returned to the server in text format and, if necessary, sent to a voice generation AI (e.g., a voice synthesis engine) where it is also generated as voice data.

[0051] Providing guide information

[0052] Server: The generated guide information and audio data are sent to the device.

[0053] Device: The device displays the received guide information on the user interface so that the user can check it. The device also uses a voice playback engine to provide the audio data to the user as audio guidance.

[0054] Specific examples

[0055] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[0056] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0057] Server: The server receives the image and uses an image recognition engine such as Google® Vision API to detect "Label: Great Buddha".

[0058] Server: Based on the detection results, it sends a request to the OpenAI (registered trademark) natural language generation engine to "generate guide information about the Great Buddha of Kamakura."

[0059] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0060] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0061] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0065] Device: Captured images are temporarily stored in the device's internal storage.

[0066] Step 2:

[0067] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[0068] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0069] Step 3:

[0070] Server: Prepares the cloud server to analyze the received image data.

[0071] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[0072] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[0073] Step 4:

[0074] Server: Based on the detected item (e.g., Great Buddha), it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[0075] Server: The generative AI generates guide information and returns the text data to the cloud server.

[0076] Step 5:

[0077] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0078] Server: Sends the generated guide information text data and audio data to the terminal.

[0079] Step 6:

[0080] Terminal: Updates the text view to display the received guide information in the user interface.

[0081] Terminal: Starts the audio playback engine to play the received audio data.

[0082] Step 7:

[0083] User: The user can check the guide information displayed on the device screen and listen to the audio guide.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Conventional tourist guide systems require users to carry a guidebook to obtain detailed information, making it difficult to obtain information in real time. Furthermore, they lack a means to provide users with fast and accurate guide information about the places they visited and the items they were interested in, resulting in an unsatisfactory tourist experience.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for a user to take an image using a terminal, means for uploading the image taken by the terminal to a cloud server, means for the cloud server to analyze the received image and detect a specific item, means for the cloud server to generate guide information for the detected item, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and playing back the guide information received by the terminal as audio. This enables the user to easily obtain detailed guide information in real time based on the captured image.

[0089] "User" refers to a person who uses the system to take images of tourist spots or items of interest and obtain guide information based on those images.

[0090] "Terminal" refers to a mobile communication device that allows a user to take images and upload them to a cloud server, and includes smartphones, tablets, etc.

[0091] "Cloud server" refers to a server that provides computing resources to receive images uploaded from a terminal via a network, analyze them, generate guide information, and send that information to the terminal.

[0092] "Image Recognition Engine" refers to an algorithm or software that analyzes received image data and automatically identifies specific elements within the image.

[0093] "Natural Language Generation Engine" refers to an algorithm or software for automatically generating textual guide information based on specified prompts.

[0094] "Guide information" refers to information containing detailed explanations and commentary about a specific item, provided in text format or audio data.

[0095] "Display" refers to visually showing guide information on the user interface of the device.

[0096] "Playback by audio" refers to providing the generated audio data to the user audibly through a speaker built into the terminal or connected earphones, etc.

[0097] A "prompt" refers to an input sentence used to instruct a generative AI model to generate specific information (e.g., guide information about a specific item).

[0098] The present invention provides a system that allows users to take pictures of tourist spots or items of interest and obtain detailed guide information in real time based on the pictures. Specific embodiments of the system are described below.

[0099] Hardware and software used

[0100] 1. Device: Refers to a mobile communication device such as a smartphone or tablet used by a user, which takes, saves, uploads, and displays images, plays audio, etc.

[0101] 2. Cloud server: A remote server that receives images from devices via the network, analyzes them, generates information, and transmits it.

[0102] 3. Image recognition engine: Runs on a cloud server and includes an algorithm (e.g., a general-purpose image recognition algorithm) and an API (e.g., a general-purpose image recognition API) for identifying specific elements within an image.

[0103] 4. Natural Language Generation Engine: Refers to the algorithms and APIs (e.g., general-purpose natural language generation engines) used to generate guide information based on the output of an image recognition engine.

[0104] 5. Speech synthesis engine: Includes algorithms and APIs (e.g., general-purpose speech synthesis engines) for converting text-based guide information into speech data.

[0105] Details of data processing and calculation

[0106] 1. Image capture and saving:

[0107] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0108] Device: The captured image is temporarily stored in the device's internal storage.

[0109] 2. Upload your image:

[0110] Terminal: The saved images are uploaded to the cloud server via the network, where network connection is verified and transactions are managed.

[0111] 3. Image Analysis:

[0112] Server: The cloud server analyzes the received image data and uses an image recognition engine to detect specific items (e.g., the Great Buddha).

[0113] 4. Guide information generation:

[0114] Server: Sends a request to the natural language generation engine to generate detailed guide information about the detected item, which is then returned to the server in text format.

[0115] Server: If necessary, the speech synthesis engine generates the audio data.

[0116] 5. Guide information provided:

[0117] Server: The generated guide information (text and audio data) is sent to the device.

[0118] Terminal: The terminal displays the received guide information on the user interface and plays the audio data.

[0119] Specific examples

[0120] User: Consider a scenario in which a user wants to take a photo of the Great Buddha of Kamakura.

[0121] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0122] Server: The server receives the image and uses a general-purpose image recognition engine to detect "Label: Great Buddha".

[0123] Server: Based on the detection results, the server sends a prompt to the general-purpose natural language generation engine, saying, "I would like you to generate guide information about the Great Buddha of Kamakura."

[0124] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0125] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0126] Prompt Sentence Examples

[0127] "I want to generate guide information about the Great Buddha of Kamakura."

[0128] "Please provide guide information about the Eiffel Tower."

[0129] "Generate a detailed description of Kinkaku-ji Temple in Kyoto"

[0130] With the above flow, users can easily obtain detailed guide information in real time from images they have taken, without having to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1:

[0133] User: The user launches the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha). The input is the image captured by the smartphone camera, and the output is an image file "image.jpg" saved in the device's internal storage. The specific action is to launch the camera app and tap the capture button to capture the image.

[0134] Step 2:

[0135] Device: Save the captured image file "image.jpg" to the device's internal storage. In this step, the input is the image data obtained from the camera app, and the output is the saved image file "image.jpg". The specific operation is to use the storage API to save the image data to the specified directory.

[0136] Step 3:

[0137] Device: Uses Wi-Fi or mobile network to upload the saved image file to the cloud server. The input is the saved image file "image.jpg" and the output is the image data uploaded to the cloud server. The specific operation is to check the network connection and transfer the image file to the cloud server using the HTTPS protocol.

[0138] Step 4:

[0139] Server: The cloud server receives the image data sent from the device. The input is the image data sent from the device, and the output is the image data stored in the server's temporary storage. The specific operation is to receive the HTTPS request and store the data in the server's storage.

[0140] Step 5:

[0141] Server: Passes the received image data to the image recognition engine and identifies the specific item. The input is image data stored in the server's storage, and the output is the recognized item (e.g., "Great Buddha"). The specific operation is to call the image recognition engine (e.g., general-purpose image recognition API) and obtain the analysis results.

[0142] Step 6:

[0143] Server: Based on the detected item, it sends a request to the natural language generation engine to generate guide information. The input is the information about the recognized item (e.g., "Great Buddha"), and the output is the text data of the generated guide information. The specific operation is to call the natural language generation engine (e.g., general-purpose natural language generation API) and send the prompt sentence, "I would like you to generate guide information about the Great Buddha."

[0144] Step 7:

[0145] Generative AI model: The natural language generation engine generates detailed guide information based on the prompt. The input is the prompt "Please generate guide information about the Great Buddha," and the output is the guide information text data. The specific operation is to use a natural language generation algorithm to generate detailed explanations corresponding to the prompt.

[0146] Step 8:

[0147] Server: Receives the text data of the generated guide information and generates voice data using a voice synthesis engine as needed. The input is the text data of the generated guide information, and the output is voice data. Specifically, it passes the guide information to a voice synthesis engine (e.g., a general-purpose voice synthesis API) and generates voice data.

[0148] Step 9:

[0149] Server: Sends text guide information and audio data to the device via the cloud. The input is the generated text data and audio data, and the output is the data sent to the device. The specific operation is to send data to the device using a network protocol.

[0150] Step 10:

[0151] Terminal: Displays the received guide information on the user interface and plays the audio data. The input is the text data and audio data of the guide information sent from the server, and the output is display information and audio guidance that the user can check. Specifically, it uses the display API to display the text on the screen and starts the audio playback engine to play the audio data.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] In conventional brick-and-mortar shopping experiences, it has been difficult to quickly and easily obtain detailed product information, reviews, discount information, and the like. Furthermore, when users become interested in a product, the provision of real-time information is insufficient, which can discourage them from purchasing. The present invention aims to solve these problems and improve the brick-and-mortar shopping experience.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the cloud server to generate guide information and related information for the detected items, means for transmitting the guide information and related information generated by the cloud server to the terminal, and means for displaying and playing back the guide information and related information received by the terminal as audio. This allows users to obtain detailed information and related information about products in which they are interested in in real time.

[0157] "Terminal" refers to the equipment used by the user, specifically devices such as smartphones, tablets, and personal computers.

[0158] A "cloud server" is a remote server provided via the Internet, a collection of computer systems that store, process, and manage data.

[0159] An "image recognition engine" is software or algorithms used to analyze and identify objects or features in images.

[0160] A "natural language generation engine" is a technology that primarily uses AI models to generate natural language that humans can understand based on input data.

[0161] "Guide information" refers to information such as detailed descriptions, background information, and instructions for use regarding a particular item.

[0162] "Related Information" is additional information related to a particular item, including discount information, reviews, and related product recommendations.

[0163] A "display means" is a device or program that has the function of visually presenting information such as text and graphics to a user.

[0164] "Audio playback means" refers to a device or program that has the function of converting text data into audio and providing information to the user audibly.

[0165] The present invention relates to a system that uses a cloud server to provide product information in real time based on images taken by a user in a store. Specific embodiments for carrying out the present invention will be described below.

[0166] Hardware and software used

[0167] Device:

[0168] Users use smartphones and tablets, which require a camera and an internet connection.

[0169] Cloud Server:

[0170] A cloud server is a remote computer system that stores, processes, and manages data.

[0171] Image Recognition Engine:

[0172] Use an image recognition engine such as the Google Vision API to analyze and identify objects in the image.

[0173] Natural Language Generation Engine:

[0174] It uses natural language generation engines such as OpenAI's GPT-3 (registered trademark) to generate detailed guides and related information based on the detected items.

[0175] System Operation

[0176] Image capture and upload:

[0177] A user activates the camera on their smartphone and takes a photo of a product they are interested in (e.g., a watch). The image is temporarily saved on the device and then uploaded to a cloud server via the network.

[0178] Image recognition for object detection:

[0179] The cloud server uses an image recognition engine such as Google Vision API to analyze the uploaded image and identify the specific item (e.g., a watch).

[0180] Generate and provide guide information:

[0181] The cloud server requests detailed guide information and related information about the identified item (discount information, reviews, recommended related products, etc.) from OpenAI's GPT-3, a generative AI. The generated information is returned to the server in both text and audio formats.

[0182] Display guide information and play audio:

[0183] The terminal displays the guide information received from the server on the user interface and provides it as audio guidance using an audio playback engine.

[0184] Specific examples

[0185] For example, consider a scenario in which a user takes a photo of a Rolex watch in a store. The user takes a photo of the watch with their smartphone camera and uploads the image file "rolex_watch.jpg" to a cloud server. The server analyzes the image and identifies the "label: watch" using the Google Vision API. It then sends a request to OpenAI's natural language generation engine to "generate guide information about Rolex watches." The generated guide information, "Rolex watches are a luxury watch brand known for their exquisite craftsmanship and high reliability...," along with audio data, is sent to the device. The device displays the guide information, activates the audio playback engine, and provides an audio guide.

[0186] Prompt Sentence Examples

[0187] "I want you to generate guide information about Rolex watches."

[0188] Through this process, users can obtain detailed information and related information about the products they are interested in in real time, significantly improving the shopping experience.

[0189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0190] Step 1:

[0191] A user activates the smartphone camera and takes a photo of an item of interest (e.g., a watch).

[0192] Input: Image data taken by the user

[0193] Output: Image file temporarily saved on the smartphone

[0194] Step 2:

[0195] The device then uploads the temporarily saved images to a cloud server via the Internet, where network connection and error checks are also performed.

[0196] Input: Temporarily saved image file

[0197] Output: Image file uploaded to the cloud server

[0198] Step 3:

[0199] The server then sends the received image file to an image recognition engine such as Google Vision API, which analyzes the object in the image. For example, the object in the image may be identified as a "watch."

[0200] Input: Image file on cloud server

[0201] Output: Label as image recognition result (e.g., clock)

[0202] Step 4:

[0203] The server sends a request to OpenAI's natural language generation engine to generate guide and related information based on the identified item, including the label.

[0204] Input: Label of image recognition result (e.g., clock)

[0205] Output: Guide information and related information (text format) generated by a natural language generation engine

[0206] Step 5:

[0207] The server transmits the generated guide information and related information to a voice synthesis engine, causing it to generate voice data.

[0208] Input: Guide information and related information from a natural language generation engine

[0209] Output: Audio data (audio file)

[0210] Step 6:

[0211] The server transmits the generated guide information, related information, and audio data to the terminal.

[0212] Input: Guide information and related information (text format), audio data

[0213] Output: Data sent to the terminal

[0214] Step 7:

[0215] The terminal displays the received guide information and related information on the user interface and plays them aloud using a voice playback engine, allowing the user to visually and audibly check the details of the product.

[0216] Input: Guide information and audio data received from the server

[0217] Output: Displaying information and playing audio to the user

[0218] This series of steps allows users to obtain detailed information and related information about products they are interested in in real time, improving their shopping experience.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[0221] 1. Image capture and upload

[0222] Image Capture

[0223] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0224] Device: Captured images are temporarily stored in the device's internal storage.

[0225] Uploading an image

[0226] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[0227] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0228] 2. Item detection using image recognition

[0229] Image analysis

[0230] Server: Prepares the cloud server to analyze the received image data.

[0231] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[0232] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[0233] 3. Emotion recognition and guidance information adjustment

[0234] Emotion recognition

[0235] Device: The device's camera and sensors are used to recognize the user's face and voice, and to recognize the user's emotions. This process is carried out using an emotion engine.

[0236] Cloud server: The user's emotions may be recognized from the results of image analysis.

[0237] 4. Generating and Providing Guide Information

[0238] Guide information generation

[0239] Server: Requests a generative AI (e.g., a natural language generation engine) to generate guide information based on the detected item (e.g., the Great Buddha) and the user's recognized emotion. For example, if the user is surprised, the information is made more detailed, and if the user is relaxed, the information is made more concise.

[0240] Server: The generative AI generates guide information and returns the text data to the cloud server.

[0241] Providing guide information

[0242] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0243] Server: Sends the generated guide information text data and audio data to the terminal.

[0244] 5. Display and playback

[0245] Terminal: Updates the text view to display the received guide information in the user interface.

[0246] Terminal: Starts the audio playback engine to play the received audio data.

[0247] Specific examples

[0248] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[0249] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0250] Server: The server receives the image and uses an image recognition engine to detect "Label: Great Buddha".

[0251] Device: At the same time, the device camera captures the user's facial expression to recognize the user's emotion, and uses the emotion engine to recognize "User emotion: Surprise".

[0252] Server: Based on the detection results and the recognized emotions, the server sends a request to the generative AI to generate detailed guidance information for surprised users.

[0253] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0254] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0255] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook, and the information is provided in a way that adapts to the user's emotions, resulting in a more personalized travel experience.

[0256] The processing flow will be explained below.

[0257] Understood. The process flow is explained below step by step.

[0258] Step 1:

[0259] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0260] Device: The captured image is temporarily saved to the smartphone's internal storage.

[0261] Step 2:

[0262] On your device: Check your network connection to upload saved images to the cloud server.

[0263] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0264] Step 3:

[0265] Server: The cloud server prepares to analyze the received image data.

[0266] Server: Passes image data to an image recognition engine (e.g., an image recognition algorithm) to identify elements within the image.

[0267] Server: Receives the response from the image recognition engine and detects a specific item (e.g., Great Buddha) based on the analysis results.

[0268] Step 4:

[0269] Device: The device's camera and sensors are used to recognize the user's face and perform voice analysis, and an emotion engine is used to recognize the user's emotions.

[0270] Device: Sends the recognized emotion data to the cloud server.

[0271] Step 5:

[0272] Server: Based on the detected item (e.g., Great Buddha) and the emotion data sent from the device, it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[0273] Server: The generative AI generates guide information adapted to the user's emotions and returns the text data to the cloud server.

[0274] Step 6:

[0275] Server: Passes the guide information text data to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0276] Server: Sends the generated guide information text data and audio data to the terminal.

[0277] Step 7:

[0278] Terminal: Updates the text view to display the received guide information in the user interface.

[0279] Terminal: Starts the audio playback engine to play the received audio data.

[0280] Step 8:

[0281] User: Users can check the guide information displayed on the device screen and listen to the audio guide. The guide information is personalized to the user's emotions, enriching the travel experience.

[0282] By performing each step with specific examples, the system of the present invention quickly provides detailed guide information based on images taken by the user, and by providing this information in a manner that adapts to the user's emotions, it enables quick and easy access to information and a more personalized travel experience.

[0283] Example 2

[0284] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Conventional systems provide guide information based on images taken by the user, but they are unable to adjust the information according to the user's emotions. This results in a uniform user experience, making it difficult to provide personalized information tailored to individual needs. Furthermore, there is a lack of a method for providing guide information suited to the user in real time.

[0286] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting a specific item by analyzing an image, means for recognizing a user's emotion, and means for generating guide information based on the detected item and the recognized user's emotion. This makes it possible to provide guide information adapted to the user's emotion in real time based on an image taken by the user.

[0287] A "terminal" is a device that is directly operated by a user, and is a device that captures images, transmits data to a cloud server, displays guide information, and plays audio.

[0288] A "cloud server" is a server that can be remotely accessed via the Internet, and is a device that performs image analysis, generates guide information, and transmits data.

[0289] An "image recognition algorithm" is a computational approach and method for detecting specific items or elements in image data.

[0290] "Guide information" is information that provides detailed explanations and guidance for detected items, and is provided in text and audio formats.

[0291] A "natural language generation engine" is a system for automatically generating human-understandable natural language text based on input data.

[0292] A "speech generation engine" is a system for converting text data into speech data.

[0293] "User's emotions" refer to the user's psychological state, which is recognized using the user's facial expressions and voice analysis.

[0294] "Image data" refers to digital information such as still images and videos taken by a device.

[0295] An "API request" is a standardized request for an application to communicate with another system, such as a cloud server.

[0296] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[0297] First, a user launches the camera app on their smartphone and takes a photo of a tourist spot or an item of interest (e.g., the Great Buddha). The captured image is temporarily saved in the device's internal storage. At this time, the device checks for a network connection and prepares to upload the captured image to the cloud server. The saved image is then sent to the cloud server via an API request.

[0298] Next, the cloud server prepares the received image data for analysis. Specifically, it performs preprocessing such as image resizing and format conversion. Then, the cloud server analyzes the image data using an image recognition algorithm (e.g., TENSORFLOW (registered trademark) or OpenCV) to detect a specific item (e.g., the Great Buddha).

[0299] To recognize a user's emotions, the device uses a camera or sensors to capture the user's facial expressions and voice. This processing is often performed using an emotion engine (e.g., Microsoft® Azure® Face API). The cloud server may also perform additional image analysis to recognize the user's emotions.

[0300] The cloud server requests the generation AI (e.g., GPT-4 (registered trademark)) to generate guide information based on the detected items and the user's recognized emotions. For example, the system is adjusted to provide detailed information if the user is surprised and concise information if the user is relaxed. The generation AI generates the guide information and returns the text data to the cloud server.

[0301] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data. The cloud server then sends this text data and voice data to the device.

[0302] Finally, the device displays the received guidance information on the user interface, updates the text view, and starts the audio playback engine to play the received audio data, providing audio guidance.

[0303] Specific examples

[0304] For example, consider a scenario where a user takes a photo of the Great Buddha of Kamakura. The user takes out their smartphone and takes a photo in front of the Great Buddha of Kamakura. The captured image file, "kamakura_buddha.jpg," is saved on the device and automatically uploaded to the cloud server.

[0305] The cloud server uses an image recognition engine to identify "Label: Great Buddha" and saves the result. At the same time, the device's camera captures the user's facial expression, and the emotion engine recognizes "User emotion: Surprise." The cloud server then sends a request to the generation AI to "generate detailed guide information for surprised users."

[0306] The generation AI generates the guide information text, "The Great Buddha of Kamakura is a gigantic Buddha statue approximately 13.35 meters tall, built during the Kamakura period..." and returns it to the cloud server. The cloud server then passes this text to the voice generation AI, which generates voice data. The cloud server then sends the generated text and voice data to the device.

[0307] Guide information is displayed on the device, and the voice playback engine provides the guide audibly. This means users do not need to carry a guidebook, and can obtain detailed guide information in real time from captured images. Furthermore, the information is provided in a way that adapts to the user's emotions, realizing a more personalized travel experience.

[0308] Prompt Sentence Examples

[0309] Examples of prompts for a generative AI model might include:

[0310] "Generate detailed guide information based on the user's emotion of surprise about an image of the Great Buddha taken by the user."

[0311] "Based on the image recognition results, provide concise guidance information when the user is relaxed."

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] A user takes a photo of a tourist spot or an item of interest (e.g., a Great Buddha) with their smartphone, which causes the smartphone's camera app to generate image data.

[0315] Input: User takes a photo through the camera app.

[0316] Output: The generated image data (e.g. "kamakura_buddha.jpg").

[0317] Step 2:

[0318] The device temporarily stores the captured image data in its internal storage, which saves the image file for subsequent processing.

[0319] Input: Captured image data.

[0320] Output: Image files saved in internal storage.

[0321] Step 3:

[0322] The device checks for a network connection and prepares to upload the saved images to the cloud server, making the image data available for transmission.

[0323] Input: Image files stored in the internal storage.

[0324] Output: Image data ready to be sent to the cloud server.

[0325] Step 4:

[0326] The device makes an API request and sends image data to the cloud server, which then receives the image data.

[0327] Input: Image data for the API request.

[0328] Output: Image data sent to the cloud server.

[0329] Step 5:

[0330] The cloud server prepares the received image data for analysis, specifically by performing preprocessing such as image resizing and format conversion.

[0331] Input: Image data sent to the cloud server.

[0332] Output: Preprocessed image data.

[0333] Step 6:

[0334] The cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[0335] Input: Preprocessed image data.

[0336] Output: Item information as the analysis result (e.g. "Label: Great Buddha").

[0337] Step 7:

[0338] The device captures the user's facial expressions and voice using a camera and sensors, and recognizes the user's emotions using an emotion engine (e.g., Microsoft Azure Face API).

[0339] Input: User data captured by the device's camera or sensors.

[0340] Output: User emotion information (e.g., "User emotion: Surprise").

[0341] Step 8:

[0342] If necessary, the cloud server will recognize the user's emotions from the image analysis results.

[0343] Input: Item information as analysis results and user image data.

[0344] Output: Additional recognized user emotion information.

[0345] Step 9:

[0346] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the recognized user emotions.

[0347] Input: Item information and user emotion information.

[0348] Output: Request data for the generating AI.

[0349] Step 10:

[0350] The generation AI generates guide information and returns the text data to the cloud server.

[0351] Input: Request data for the generating AI.

[0352] Output: Text data of the generated guide information.

[0353] Step 11:

[0354] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data.

[0355] Input: Text data of the generated guide information.

[0356] Output: The generated audio data.

[0357] Step 12:

[0358] The cloud server transmits the generated text data and voice data to the device, which then becomes ready to receive the data.

[0359] Input: Generated text and audio data.

[0360] Output: Text and audio data sent to the device.

[0361] Step 13:

[0362] The terminal displays the received guide information on the user interface and updates the text view.

[0363] Input: Text data sent to the terminal.

[0364] Output: The updated text view.

[0365] Step 14:

[0366] The terminal starts an audio playback engine to play back the audio data received, and provides audio guidance.

[0367] Input: Audio data sent to the device.

[0368] Output: The audio description played.

[0369] This allows users to obtain detailed and personalized guide information in real time.

[0370] (Application example 2)

[0371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] Currently, image analysis and information provision systems on the market do not adequately provide personalized information that takes user emotions into account. As a result, they are unable to provide detailed information tailored to the user's interests and emotions, leaving room for improvement in the user experience. Furthermore, because conventional systems cannot adjust the content of information based on the user's emotional state, they may not provide the most appropriate information for the user.

[0373] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the terminal to recognize the user's emotions, means for the cloud server to generate guide information based on the detected items and the recognized user's emotions, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and audibly playing back the guide information received by the terminal. This makes it possible to provide personalized information according to the user's emotions.

[0374] A "terminal" is an electronic device used by a user, which has a camera function, takes pictures, and communicates with a cloud server.

[0375] A "cloud server" is a remote computer system that receives image data via a network, analyzes the image data, detects items, and generates and transmits guide information.

[0376] An "image recognition engine" is software or algorithms that analyze received image data and identify specific items within the image.

[0377] A "natural language generation engine" is software or an algorithm for automatically generating text for guide information based on specific items and user emotion data.

[0378] "Emotion recognition means" refers to equipment and software that analyzes the user's facial expressions and voice to determine their emotional state at that time.

[0379] "Guide information" is information provided in a user-friendly format, including explanations, usage instructions, and interesting information about detected items.

[0380] The "audio playback means" refers to software and hardware for playing back the generated guide information as audio.

[0381] "Personalized information provision" refers to information provided in a form optimized to suit the individual needs and emotional state of the user.

[0382] This invention relates to a system that automatically provides guide information based on images taken by a user. It also has a function to recognize the user's emotions and adjust the content of the guide information. A specific embodiment of this system is described below.

[0383] First, a user uses the device's camera to take a picture of a specific item (e.g., a product). The image is temporarily stored in the device's internal storage and then uploaded to a cloud server. The device and cloud server communicate via the Internet.

[0384] The cloud server analyzes the received image data and uses an image recognition engine, such as a general-purpose image recognition algorithm, to detect specific items in the image.

[0385] At the same time, the device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotions. This process uses an emotion recognition engine to determine the user's emotional state (e.g., excitement, surprise, relaxation, etc.).

[0386] The cloud server generates guide information using a natural language generation engine based on the detected items and the user's recognized emotions. For example, if it recognizes that the user has an excited expression when looking at a certain product, guide information including a detailed description of the product and how to use it is generated.

[0387] The generated guide information is sent from the cloud server to the device. The device then provides the received guide information to the user both as text and as audio. A text-to-speech engine is used for audio playback.

[0388] Specific examples

[0389] A user is wearing smart glasses while walking through a physical store. If the user looks at a new sneaker and shows an excited expression, the cloud server recognizes the product and generates a detailed product description based on the user's excitement. For example, the user is provided with information such as, "These sneakers are made with the latest technology and provide excellent foot support. This product is popular among many athletes."

[0390] Example prompts for generative AI models

[0391] "A user is looking at the product 'Sneakers'. The user's emotion is 'Excited'. Please generate detailed guide information about this product."

[0392] The system allows users to receive real-time and personalized information, significantly improving the shopping experience.

[0393] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0394] Step 1:

[0395] A user takes an image of a specific item using the device's camera. The input is the image data captured by the device's camera, and the output is an image file temporarily stored in the device's internal storage.

[0396] Step 2:

[0397] The device uploads the saved image file to the cloud server. This also checks the network connection. The input is the image data saved in step 1, and the output is the image data uploaded to the cloud server.

[0398] Step 3:

[0399] The cloud server analyzes the received image data and uses an image recognition engine to detect specific items. The input is the image data uploaded to the cloud server, and the output is the identification information of the detected item. Specifically, the cloud server passes the image data to the image recognition engine, and the algorithm analyzes the elements in the image.

[0400] Step 4:

[0401] To recognize a user's emotions, a device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice. The input is the user's facial expression and voice data captured by the camera and microphone, and the output is the user's emotional state analyzed by an emotion recognition engine. Specifically, the device passes the sensor data to the emotion recognition engine, and the algorithm determines the emotional state.

[0402] Step 5:

[0403] The cloud server uses a natural language generation engine to generate guide information based on the detected items and the recognized user emotions. The input is the identification information of the detected items and the recognized user emotional state, and the output is text data of the generated guide information. In specific operations, the cloud server passes various data to the natural language generation engine to generate optimal guide information.

[0404] Step 6:

[0405] The cloud server sends the generated guide information to the terminal. The input is the text data of the generated guide information, and the output is the text data of the guide information sent to the terminal.

[0406] Step 7:

[0407] The terminal displays the received guide information as text and plays it back aloud. A speech synthesis engine is used for the audio playback. The input is the text data of the guide information sent in step 6, and the output is the displayed text information and the audio playback of the guide information. Specifically, the terminal displays the received text data on the screen and starts the speech synthesis engine to play the audio.

[0408] Through this series of operations, users can receive detailed information about products they saw in a physical store in real time, in a form that is appropriately adjusted to their emotional state.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0410] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0412] [Second embodiment]

[0413] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0414] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0415] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0417] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0419] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0420] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0421] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0424] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0425] The present invention relates to a system that automatically provides guide information based on images taken by travelers, and specific embodiments thereof will be described below.

[0426] 1. Image capture and upload

[0427] Image Capture

[0428] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0429] Device: Captured images are temporarily stored on the device.

[0430] Uploading an image

[0431] Device: The saved images are uploaded to the cloud server via the network, which checks the network connection and manages the transaction.

[0432] 2. Item detection using image recognition

[0433] Image analysis

[0434] Server: The cloud server analyzes the received image data, specifically using an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements in the image and detect specific items (e.g., the Great Buddha).

[0435] Server: Receives the response from the image recognition engine and proceeds to the next step based on the detection results.

[0436] 3. Generating and Providing Guide Information

[0437] Guide information generation

[0438] Server: Requests detailed guide information about the detected item from the generative AI. The generative AI (e.g., a natural language generation engine) generates useful guide information for the user based on previously learned knowledge about the detected item (e.g., the Great Buddha).

[0439] Server: The generated guide information is returned to the server in text format and, if necessary, sent to a voice generation AI (e.g., a voice synthesis engine) where it is also generated as voice data.

[0440] Providing guide information

[0441] Server: The generated guide information and audio data are sent to the device.

[0442] Device: The device displays the received guide information on the user interface so that the user can check it. The device also uses a voice playback engine to provide the audio data to the user as audio guidance.

[0443] Specific examples

[0444] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[0445] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0446] Server: The server receives the image and uses an image recognition engine such as Google Vision API to detect "Label: Great Buddha".

[0447] Server: Based on the detection results, it sends a request to OpenAI's natural language generation engine to "generate guide information about the Great Buddha of Kamakura."

[0448] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0449] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0450] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0451] The processing flow will be explained below.

[0452] Step 1:

[0453] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0454] Device: Captured images are temporarily stored in the device's internal storage.

[0455] Step 2:

[0456] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[0457] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0458] Step 3:

[0459] Server: Prepares the cloud server to analyze the received image data.

[0460] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[0461] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[0462] Step 4:

[0463] Server: Based on the detected item (e.g., Great Buddha), it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[0464] Server: The generative AI generates guide information and returns the text data to the cloud server.

[0465] Step 5:

[0466] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0467] Server: Sends the generated guide information text data and audio data to the terminal.

[0468] Step 6:

[0469] Terminal: Updates the text view to display the received guide information in the user interface.

[0470] Terminal: Starts the audio playback engine to play the received audio data.

[0471] Step 7:

[0472] User: The user can check the guide information displayed on the device screen and listen to the audio guide.

[0473] Example 1

[0474] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] Conventional tourist guide systems require users to carry a guidebook to obtain detailed information, making it difficult to obtain information in real time. Furthermore, they lack a means to provide users with fast and accurate guide information about the places they visited and the items they were interested in, resulting in an unsatisfactory tourist experience.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0477] In this invention, the server includes means for a user to take an image using a terminal, means for uploading the image taken by the terminal to a cloud server, means for the cloud server to analyze the received image and detect a specific item, means for the cloud server to generate guide information for the detected item, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and playing back the guide information received by the terminal as audio. This enables the user to easily obtain detailed guide information in real time based on the captured image.

[0478] "User" refers to a person who uses the system to take images of tourist spots or items of interest and obtain guide information based on those images.

[0479] "Terminal" refers to a mobile communication device that allows a user to take images and upload them to a cloud server, and includes smartphones, tablets, etc.

[0480] "Cloud server" refers to a server that provides computing resources to receive images uploaded from a terminal via a network, analyze them, generate guide information, and send that information to the terminal.

[0481] "Image Recognition Engine" refers to an algorithm or software that analyzes received image data and automatically identifies specific elements within the image.

[0482] "Natural Language Generation Engine" refers to an algorithm or software for automatically generating textual guide information based on specified prompts.

[0483] "Guide information" refers to information containing detailed explanations and commentary about a specific item, provided in text format or audio data.

[0484] "Display" refers to visually showing guide information on the user interface of the device.

[0485] "Playback by audio" refers to providing the generated audio data to the user audibly through a speaker built into the terminal or connected earphones, etc.

[0486] A "prompt" refers to an input sentence used to instruct a generative AI model to generate specific information (e.g., guide information about a specific item).

[0487] The present invention provides a system that allows users to take pictures of tourist spots or items of interest and obtain detailed guide information in real time based on the pictures. Specific embodiments of the system are described below.

[0488] Hardware and software used

[0489] 1. Device: Refers to a mobile communication device such as a smartphone or tablet used by a user, which takes, saves, uploads, and displays images, plays audio, etc.

[0490] 2. Cloud server: A remote server that receives images from devices via the network, analyzes them, generates information, and transmits it.

[0491] 3. Image recognition engine: Runs on a cloud server and includes an algorithm (e.g., a general-purpose image recognition algorithm) and an API (e.g., a general-purpose image recognition API) for identifying specific elements within an image.

[0492] 4. Natural Language Generation Engine: Refers to the algorithms and APIs (e.g., general-purpose natural language generation engines) used to generate guide information based on the output of an image recognition engine.

[0493] 5. Speech synthesis engine: Includes algorithms and APIs (e.g., general-purpose speech synthesis engines) for converting text-based guide information into speech data.

[0494] Details of data processing and calculation

[0495] 1. Image capture and saving:

[0496] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0497] Device: The captured image is temporarily stored in the device's internal storage.

[0498] 2. Upload your image:

[0499] Terminal: The saved images are uploaded to the cloud server via the network, where network connection is verified and transactions are managed.

[0500] 3. Image Analysis:

[0501] Server: The cloud server analyzes the received image data and uses an image recognition engine to detect specific items (e.g., the Great Buddha).

[0502] 4. Guide information generation:

[0503] Server: Sends a request to the natural language generation engine to generate detailed guide information about the detected item, which is then returned to the server in text format.

[0504] Server: If necessary, the speech synthesis engine generates the audio data.

[0505] 5. Guide information provided:

[0506] Server: The generated guide information (text and audio data) is sent to the device.

[0507] Terminal: The terminal displays the received guide information on the user interface and plays the audio data.

[0508] Specific examples

[0509] User: Consider a scenario in which a user wants to take a photo of the Great Buddha of Kamakura.

[0510] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0511] Server: The server receives the image and uses a general-purpose image recognition engine to detect "Label: Great Buddha".

[0512] Server: Based on the detection results, the server sends a prompt to the general-purpose natural language generation engine, saying, "I would like you to generate guide information about the Great Buddha of Kamakura."

[0513] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0514] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0515] Prompt Sentence Examples

[0516] "I want to generate guide information about the Great Buddha of Kamakura."

[0517] "Please provide guide information about the Eiffel Tower."

[0518] "Generate a detailed description of Kinkaku-ji Temple in Kyoto"

[0519] With the above flow, users can easily obtain detailed guide information in real time from images they have taken, without having to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0520] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0521] Step 1:

[0522] User: The user launches the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha). The input is the image captured by the smartphone camera, and the output is an image file "image.jpg" saved in the device's internal storage. The specific action is to launch the camera app and tap the capture button to capture the image.

[0523] Step 2:

[0524] Device: Save the captured image file "image.jpg" to the device's internal storage. In this step, the input is the image data obtained from the camera app, and the output is the saved image file "image.jpg". The specific operation is to use the storage API to save the image data to the specified directory.

[0525] Step 3:

[0526] Device: Uses Wi-Fi or mobile network to upload the saved image file to the cloud server. The input is the saved image file "image.jpg" and the output is the image data uploaded to the cloud server. The specific operation is to check the network connection and transfer the image file to the cloud server using the HTTPS protocol.

[0527] Step 4:

[0528] Server: The cloud server receives the image data sent from the device. The input is the image data sent from the device, and the output is the image data stored in the server's temporary storage. The specific operation is to receive the HTTPS request and store the data in the server's storage.

[0529] Step 5:

[0530] Server: Passes the received image data to the image recognition engine and identifies the specific item. The input is image data stored in the server's storage, and the output is the recognized item (e.g., "Great Buddha"). The specific operation is to call the image recognition engine (e.g., general-purpose image recognition API) and obtain the analysis results.

[0531] Step 6:

[0532] Server: Based on the detected item, it sends a request to the natural language generation engine to generate guide information. The input is the information about the recognized item (e.g., "Great Buddha"), and the output is the text data of the generated guide information. The specific operation is to call the natural language generation engine (e.g., general-purpose natural language generation API) and send the prompt sentence, "I would like you to generate guide information about the Great Buddha."

[0533] Step 7:

[0534] Generative AI model: The natural language generation engine generates detailed guide information based on the prompt. The input is the prompt "Please generate guide information about the Great Buddha," and the output is the guide information text data. The specific operation is to use a natural language generation algorithm to generate detailed explanations corresponding to the prompt.

[0535] Step 8:

[0536] Server: Receives the text data of the generated guide information and generates voice data using a voice synthesis engine as needed. The input is the text data of the generated guide information, and the output is voice data. Specifically, it passes the guide information to a voice synthesis engine (e.g., a general-purpose voice synthesis API) and generates voice data.

[0537] Step 9:

[0538] Server: Sends text guide information and audio data to the device via the cloud. The input is the generated text data and audio data, and the output is the data sent to the device. The specific operation is to send data to the device using a network protocol.

[0539] Step 10:

[0540] Terminal: Displays the received guide information on the user interface and plays the audio data. The input is the text data and audio data of the guide information sent from the server, and the output is display information and audio guidance that the user can check. Specifically, it uses the display API to display the text on the screen and starts the audio playback engine to play the audio data.

[0541] (Application example 1)

[0542] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0543] In conventional brick-and-mortar shopping experiences, it has been difficult to quickly and easily obtain detailed product information, reviews, discount information, and the like. Furthermore, when users become interested in a product, the provision of real-time information is insufficient, which can discourage them from purchasing. The present invention aims to solve these problems and improve the brick-and-mortar shopping experience.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0545] In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the cloud server to generate guide information and related information for the detected items, means for transmitting the guide information and related information generated by the cloud server to the terminal, and means for displaying and playing back the guide information and related information received by the terminal as audio. This allows users to obtain detailed information and related information about products in which they are interested in in real time.

[0546] "Terminal" refers to the equipment used by the user, specifically devices such as smartphones, tablets, and personal computers.

[0547] A "cloud server" is a remote server provided via the Internet, a collection of computer systems that store, process, and manage data.

[0548] An "image recognition engine" is software or algorithms used to analyze and identify objects or features in images.

[0549] A "natural language generation engine" is a technology that primarily uses AI models to generate natural language that humans can understand based on input data.

[0550] "Guide information" refers to information such as detailed descriptions, background information, and instructions for use regarding a particular item.

[0551] "Related Information" is additional information related to a particular item, including discount information, reviews, and related product recommendations.

[0552] A "display means" is a device or program that has the function of visually presenting information such as text and graphics to a user.

[0553] "Audio playback means" refers to a device or program that has the function of converting text data into audio and providing information to the user audibly.

[0554] The present invention relates to a system that uses a cloud server to provide product information in real time based on images taken by a user in a store. Specific embodiments for carrying out the present invention will be described below.

[0555] Hardware and software used

[0556] Device:

[0557] Users use smartphones and tablets, which require a camera and an internet connection.

[0558] Cloud Server:

[0559] A cloud server is a remote computer system that stores, processes, and manages data.

[0560] Image Recognition Engine:

[0561] Use an image recognition engine such as the Google Vision API to analyze and identify objects in the image.

[0562] Natural Language Generation Engine:

[0563] It uses natural language generation engines such as OpenAI's GPT-3 to generate detailed guides and related information based on the detected items.

[0564] System Operation

[0565] Image capture and upload:

[0566] A user activates the camera on their smartphone and takes a photo of a product they are interested in (e.g., a watch). The image is temporarily saved on the device and then uploaded to a cloud server via the network.

[0567] Image recognition for object detection:

[0568] The cloud server uses an image recognition engine such as Google Vision API to analyze the uploaded image and identify the specific item (e.g., a watch).

[0569] Generate and provide guide information:

[0570] The cloud server requests detailed guide information and related information about the identified item (discount information, reviews, recommended related products, etc.) from OpenAI's GPT-3, a generative AI. The generated information is returned to the server in both text and audio formats.

[0571] Display guide information and play audio:

[0572] The terminal displays the guide information received from the server on the user interface and provides it as audio guidance using an audio playback engine.

[0573] Specific examples

[0574] For example, consider a scenario in which a user takes a photo of a Rolex watch in a store. The user takes a photo of the watch with their smartphone camera and uploads the image file "rolex_watch.jpg" to a cloud server. The server analyzes the image and identifies the "label: watch" using the Google Vision API. It then sends a request to OpenAI's natural language generation engine to "generate guide information about Rolex watches." The generated guide information, "Rolex watches are a luxury watch brand known for their exquisite craftsmanship and high reliability...," along with audio data, is sent to the device. The device displays the guide information, activates the audio playback engine, and provides an audio guide.

[0575] Prompt Sentence Examples

[0576] "I want you to generate guide information about Rolex watches."

[0577] Through this process, users can obtain detailed information and related information about the products they are interested in in real time, significantly improving the shopping experience.

[0578] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0579] Step 1:

[0580] A user activates the smartphone camera and takes a photo of an item of interest (e.g., a watch).

[0581] Input: Image data taken by the user

[0582] Output: Image file temporarily saved on the smartphone

[0583] Step 2:

[0584] The device then uploads the temporarily saved images to a cloud server via the Internet, where network connection and error checks are also performed.

[0585] Input: Temporarily saved image file

[0586] Output: Image file uploaded to the cloud server

[0587] Step 3:

[0588] The server then sends the received image file to an image recognition engine such as Google Vision API, which analyzes the object in the image. For example, the object in the image may be identified as a "watch."

[0589] Input: Image file on cloud server

[0590] Output: Label as image recognition result (e.g., clock)

[0591] Step 4:

[0592] The server sends a request to OpenAI's natural language generation engine to generate guide and related information based on the identified item, including the label.

[0593] Input: Label of image recognition result (e.g., clock)

[0594] Output: Guide information and related information (text format) generated by a natural language generation engine

[0595] Step 5:

[0596] The server transmits the generated guide information and related information to a voice synthesis engine, causing it to generate voice data.

[0597] Input: Guide information and related information from a natural language generation engine

[0598] Output: Audio data (audio file)

[0599] Step 6:

[0600] The server transmits the generated guide information, related information, and audio data to the terminal.

[0601] Input: Guide information and related information (text format), audio data

[0602] Output: Data sent to the terminal

[0603] Step 7:

[0604] The terminal displays the received guide information and related information on the user interface and plays them aloud using a voice playback engine, allowing the user to visually and audibly check the details of the product.

[0605] Input: Guide information and audio data received from the server

[0606] Output: Displaying information and playing audio to the user

[0607] This series of steps allows users to obtain detailed information and related information about products they are interested in in real time, improving their shopping experience.

[0608] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0609] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[0610] 1. Image capture and upload

[0611] Image Capture

[0612] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0613] Device: Captured images are temporarily stored in the device's internal storage.

[0614] Uploading an image

[0615] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[0616] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0617] 2. Item detection using image recognition

[0618] Image analysis

[0619] Server: Prepares the cloud server to analyze the received image data.

[0620] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[0621] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[0622] 3. Emotion recognition and guidance information adjustment

[0623] Emotion recognition

[0624] Device: The device's camera and sensors are used to recognize the user's face and voice, and to recognize the user's emotions. This process is carried out using an emotion engine.

[0625] Cloud server: The user's emotions may be recognized from the results of image analysis.

[0626] 4. Generating and Providing Guide Information

[0627] Guide information generation

[0628] Server: Requests a generative AI (e.g., a natural language generation engine) to generate guide information based on the detected item (e.g., the Great Buddha) and the user's recognized emotion. For example, if the user is surprised, the information is made more detailed, and if the user is relaxed, the information is made more concise.

[0629] Server: The generative AI generates guide information and returns the text data to the cloud server.

[0630] Providing guide information

[0631] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0632] Server: Sends the generated guide information text data and audio data to the terminal.

[0633] 5. Display and playback

[0634] Terminal: Updates the text view to display the received guide information in the user interface.

[0635] Terminal: Starts the audio playback engine to play the received audio data.

[0636] Specific examples

[0637] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[0638] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0639] Server: The server receives the image and uses an image recognition engine to detect "Label: Great Buddha".

[0640] Device: At the same time, the device camera captures the user's facial expression to recognize the user's emotion, and uses the emotion engine to recognize "User emotion: Surprise".

[0641] Server: Based on the detection results and the recognized emotions, the server sends a request to the generative AI to generate detailed guidance information for surprised users.

[0642] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0643] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0644] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook, and the information is provided in a way that adapts to the user's emotions, resulting in a more personalized travel experience.

[0645] The processing flow will be explained below.

[0646] Understood. The process flow is explained below step by step.

[0647] Step 1:

[0648] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0649] Device: The captured image is temporarily saved to the smartphone's internal storage.

[0650] Step 2:

[0651] On your device: Check your network connection to upload saved images to the cloud server.

[0652] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0653] Step 3:

[0654] Server: The cloud server prepares to analyze the received image data.

[0655] Server: Passes image data to an image recognition engine (e.g., an image recognition algorithm) to identify elements within the image.

[0656] Server: Receives the response from the image recognition engine and detects a specific item (e.g., Great Buddha) based on the analysis results.

[0657] Step 4:

[0658] Device: The device's camera and sensors are used to recognize the user's face and perform voice analysis, and an emotion engine is used to recognize the user's emotions.

[0659] Device: Sends the recognized emotion data to the cloud server.

[0660] Step 5:

[0661] Server: Based on the detected item (e.g., Great Buddha) and the emotion data sent from the device, it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[0662] Server: The generative AI generates guide information adapted to the user's emotions and returns the text data to the cloud server.

[0663] Step 6:

[0664] Server: Passes the guide information text data to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0665] Server: Sends the generated guide information text data and audio data to the terminal.

[0666] Step 7:

[0667] Terminal: Updates the text view to display the received guide information in the user interface.

[0668] Terminal: Starts the audio playback engine to play the received audio data.

[0669] Step 8:

[0670] User: Users can check the guide information displayed on the device screen and listen to the audio guide. The guide information is personalized to the user's emotions, enriching the travel experience.

[0671] By performing each step with specific examples, the system of the present invention quickly provides detailed guide information based on images taken by the user, and by providing this information in a manner that adapts to the user's emotions, it enables quick and easy access to information and a more personalized travel experience.

[0672] Example 2

[0673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0674] Conventional systems provide guide information based on images taken by the user, but they are unable to adjust the information according to the user's emotions. This results in a uniform user experience, making it difficult to provide personalized information tailored to individual needs. Furthermore, there is a lack of a method for providing guide information suited to the user in real time.

[0675] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting a specific item by analyzing an image, means for recognizing a user's emotion, and means for generating guide information based on the detected item and the recognized user's emotion. This makes it possible to provide guide information adapted to the user's emotion in real time based on an image taken by the user.

[0676] A "terminal" is a device that is directly operated by a user, and is a device that captures images, transmits data to a cloud server, displays guide information, and plays audio.

[0677] A "cloud server" is a server that can be remotely accessed via the Internet, and is a device that performs image analysis, generates guide information, and transmits data.

[0678] An "image recognition algorithm" is a computational approach and method for detecting specific items or elements in image data.

[0679] "Guide information" is information that provides detailed explanations and guidance for detected items, and is provided in text and audio formats.

[0680] A "natural language generation engine" is a system for automatically generating human-understandable natural language text based on input data.

[0681] A "speech generation engine" is a system for converting text data into speech data.

[0682] "User's emotions" refer to the user's psychological state, which is recognized using the user's facial expressions and voice analysis.

[0683] "Image data" refers to digital information such as still images and videos taken by a device.

[0684] An "API request" is a standardized request for an application to communicate with another system, such as a cloud server.

[0685] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[0686] First, a user launches the camera app on their smartphone and takes a photo of a tourist spot or an item of interest (e.g., the Great Buddha). The captured image is temporarily saved in the device's internal storage. At this time, the device checks for a network connection and prepares to upload the captured image to the cloud server. The saved image is then sent to the cloud server via an API request.

[0687] Next, the cloud server prepares the received image data for analysis. Specifically, it performs preprocessing such as image resizing and format conversion. Then, the cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[0688] To recognize a user's emotions, the device uses a camera or sensors to capture the user's facial expressions and voice. This processing is often performed using an emotion engine (e.g., Microsoft Azure Face API). A cloud server may also perform additional image analysis to recognize the user's emotions.

[0689] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the user's recognized emotions. For example, it is adjusted to provide detailed information if the user is surprised and concise information if the user is relaxed. The generation AI generates the guide information and returns the text data to the cloud server.

[0690] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data. The cloud server then sends this text data and voice data to the device.

[0691] Finally, the device displays the received guidance information on the user interface, updates the text view, and starts the audio playback engine to play the received audio data, providing audio guidance.

[0692] Specific examples

[0693] For example, consider a scenario where a user takes a photo of the Great Buddha of Kamakura. The user takes out their smartphone and takes a photo in front of the Great Buddha of Kamakura. The captured image file, "kamakura_buddha.jpg," is saved on the device and automatically uploaded to the cloud server.

[0694] The cloud server uses an image recognition engine to identify "Label: Great Buddha" and saves the result. At the same time, the device's camera captures the user's facial expression, and the emotion engine recognizes "User emotion: Surprise." The cloud server then sends a request to the generation AI to "generate detailed guide information for surprised users."

[0695] The generation AI generates the guide information text, "The Great Buddha of Kamakura is a gigantic Buddha statue approximately 13.35 meters tall, built during the Kamakura period..." and returns it to the cloud server. The cloud server then passes this text to the voice generation AI, which generates voice data. The cloud server then sends the generated text and voice data to the device.

[0696] Guide information is displayed on the device, and the voice playback engine provides the guide audibly. This means users do not need to carry a guidebook, and can obtain detailed guide information in real time from captured images. Furthermore, the information is provided in a way that adapts to the user's emotions, realizing a more personalized travel experience.

[0697] Prompt Sentence Examples

[0698] Examples of prompts for a generative AI model might include:

[0699] "Generate detailed guide information based on the user's emotion of surprise about an image of the Great Buddha taken by the user."

[0700] "Based on the image recognition results, provide concise guidance information when the user is relaxed."

[0701] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0702] Step 1:

[0703] A user takes a photo of a tourist spot or an item of interest (e.g., a Great Buddha) with their smartphone, which causes the smartphone's camera app to generate image data.

[0704] Input: User takes a photo through the camera app.

[0705] Output: The generated image data (e.g. "kamakura_buddha.jpg").

[0706] Step 2:

[0707] The device temporarily stores the captured image data in its internal storage, which saves the image file for subsequent processing.

[0708] Input: Captured image data.

[0709] Output: Image files saved in internal storage.

[0710] Step 3:

[0711] The device checks for a network connection and prepares to upload the saved images to the cloud server, making the image data available for transmission.

[0712] Input: Image files stored in the internal storage.

[0713] Output: Image data ready to be sent to the cloud server.

[0714] Step 4:

[0715] The device makes an API request and sends image data to the cloud server, which then receives the image data.

[0716] Input: Image data for the API request.

[0717] Output: Image data sent to the cloud server.

[0718] Step 5:

[0719] The cloud server prepares the received image data for analysis, specifically by performing preprocessing such as image resizing and format conversion.

[0720] Input: Image data sent to the cloud server.

[0721] Output: Preprocessed image data.

[0722] Step 6:

[0723] The cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[0724] Input: Preprocessed image data.

[0725] Output: Item information as the analysis result (e.g. "Label: Great Buddha").

[0726] Step 7:

[0727] The device captures the user's facial expressions and voice using a camera and sensors, and recognizes the user's emotions using an emotion engine (e.g., Microsoft Azure Face API).

[0728] Input: User data captured by the device's camera or sensors.

[0729] Output: User emotion information (e.g., "User emotion: Surprise").

[0730] Step 8:

[0731] If necessary, the cloud server will recognize the user's emotions from the image analysis results.

[0732] Input: Item information as analysis results and user image data.

[0733] Output: Additional recognized user emotion information.

[0734] Step 9:

[0735] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the recognized user emotions.

[0736] Input: Item information and user emotion information.

[0737] Output: Request data for the generating AI.

[0738] Step 10:

[0739] The generation AI generates guide information and returns the text data to the cloud server.

[0740] Input: Request data for the generating AI.

[0741] Output: Text data of the generated guide information.

[0742] Step 11:

[0743] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data.

[0744] Input: Text data of the generated guide information.

[0745] Output: The generated audio data.

[0746] Step 12:

[0747] The cloud server transmits the generated text data and voice data to the device, which then becomes ready to receive the data.

[0748] Input: Generated text and audio data.

[0749] Output: Text and audio data sent to the device.

[0750] Step 13:

[0751] The terminal displays the received guide information on the user interface and updates the text view.

[0752] Input: Text data sent to the terminal.

[0753] Output: The updated text view.

[0754] Step 14:

[0755] The terminal starts an audio playback engine to play back the audio data received, and provides audio guidance.

[0756] Input: Audio data sent to the device.

[0757] Output: The audio description played.

[0758] This allows users to obtain detailed and personalized guide information in real time.

[0759] (Application example 2)

[0760] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0761] Currently, image analysis and information provision systems on the market do not adequately provide personalized information that takes user emotions into account. As a result, they are unable to provide detailed information tailored to the user's interests and emotions, leaving room for improvement in the user experience. Furthermore, because conventional systems cannot adjust the content of information based on the user's emotional state, they may not provide the most appropriate information for the user.

[0762] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the terminal to recognize the user's emotions, means for the cloud server to generate guide information based on the detected items and the recognized user's emotions, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and audibly playing back the guide information received by the terminal. This makes it possible to provide personalized information according to the user's emotions.

[0763] A "terminal" is an electronic device used by a user, which has a camera function, takes pictures, and communicates with a cloud server.

[0764] A "cloud server" is a remote computer system that receives image data via a network, analyzes the image data, detects items, and generates and transmits guide information.

[0765] An "image recognition engine" is software or algorithms that analyze received image data and identify specific items within the image.

[0766] A "natural language generation engine" is software or an algorithm for automatically generating text for guide information based on specific items and user emotion data.

[0767] "Emotion recognition means" refers to equipment and software that analyzes the user's facial expressions and voice to determine their emotional state at that time.

[0768] "Guide information" is information provided in a user-friendly format, including explanations, usage instructions, and interesting information about detected items.

[0769] The "audio playback means" refers to software and hardware for playing back the generated guide information as audio.

[0770] "Personalized information provision" refers to information provided in a form optimized to suit the individual needs and emotional state of the user.

[0771] This invention relates to a system that automatically provides guide information based on images taken by a user. It also has a function to recognize the user's emotions and adjust the content of the guide information. A specific embodiment of this system is described below.

[0772] First, a user uses the device's camera to take a picture of a specific item (e.g., a product). The image is temporarily stored in the device's internal storage and then uploaded to a cloud server. The device and cloud server communicate via the Internet.

[0773] The cloud server analyzes the received image data and uses an image recognition engine, such as a general-purpose image recognition algorithm, to detect specific items in the image.

[0774] At the same time, the device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotions. This process uses an emotion recognition engine to determine the user's emotional state (e.g., excitement, surprise, relaxation, etc.).

[0775] The cloud server generates guide information using a natural language generation engine based on the detected items and the user's recognized emotions. For example, if it recognizes that the user has an excited expression when looking at a certain product, guide information including a detailed description of the product and how to use it is generated.

[0776] The generated guide information is sent from the cloud server to the device. The device then provides the received guide information to the user both as text and as audio. A text-to-speech engine is used for audio playback.

[0777] Specific examples

[0778] A user is wearing smart glasses while walking through a physical store. If the user looks at a new sneaker and shows an excited expression, the cloud server recognizes the product and generates a detailed product description based on the user's excitement. For example, the user is provided with information such as, "These sneakers are made with the latest technology and provide excellent foot support. This product is popular among many athletes."

[0779] Example prompts for generative AI models

[0780] "A user is looking at the product 'Sneakers'. The user's emotion is 'Excited'. Please generate detailed guide information about this product."

[0781] The system allows users to receive real-time and personalized information, significantly improving the shopping experience.

[0782] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0783] Step 1:

[0784] A user takes an image of a specific item using the device's camera. The input is the image data captured by the device's camera, and the output is an image file temporarily stored in the device's internal storage.

[0785] Step 2:

[0786] The device uploads the saved image file to the cloud server. This also checks the network connection. The input is the image data saved in step 1, and the output is the image data uploaded to the cloud server.

[0787] Step 3:

[0788] The cloud server analyzes the received image data and uses an image recognition engine to detect specific items. The input is the image data uploaded to the cloud server, and the output is the identification information of the detected item. Specifically, the cloud server passes the image data to the image recognition engine, and the algorithm analyzes the elements in the image.

[0789] Step 4:

[0790] To recognize a user's emotions, a device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice. The input is the user's facial expression and voice data captured by the camera and microphone, and the output is the user's emotional state analyzed by an emotion recognition engine. Specifically, the device passes the sensor data to the emotion recognition engine, and the algorithm determines the emotional state.

[0791] Step 5:

[0792] The cloud server uses a natural language generation engine to generate guide information based on the detected items and the recognized user emotions. The input is the identification information of the detected items and the recognized user emotional state, and the output is text data of the generated guide information. In specific operations, the cloud server passes various data to the natural language generation engine to generate optimal guide information.

[0793] Step 6:

[0794] The cloud server sends the generated guide information to the terminal. The input is the text data of the generated guide information, and the output is the text data of the guide information sent to the terminal.

[0795] Step 7:

[0796] The terminal displays the received guide information as text and plays it back aloud. A speech synthesis engine is used for the audio playback. The input is the text data of the guide information sent in step 6, and the output is the displayed text information and the audio playback of the guide information. Specifically, the terminal displays the received text data on the screen and starts the speech synthesis engine to play the audio.

[0797] Through this series of operations, users can receive detailed information about products they saw in a physical store in real time, in a form that is appropriately adjusted to their emotional state.

[0798] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0799] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0800] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0801] [Third embodiment]

[0802] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0803] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0804] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0805] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0806] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0807] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0808] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0809] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0810] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0811] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0812] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0813] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0814] The present invention relates to a system that automatically provides guide information based on images taken by travelers, and specific embodiments thereof will be described below.

[0815] 1. Image capture and upload

[0816] Image Capture

[0817] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0818] Device: Captured images are temporarily stored on the device.

[0819] Uploading an image

[0820] Device: The saved images are uploaded to the cloud server via the network, which checks the network connection and manages the transaction.

[0821] 2. Item detection using image recognition

[0822] Image analysis

[0823] Server: The cloud server analyzes the received image data, specifically using an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements in the image and detect specific items (e.g., the Great Buddha).

[0824] Server: Receives the response from the image recognition engine and proceeds to the next step based on the detection results.

[0825] 3. Generating and Providing Guide Information

[0826] Guide information generation

[0827] Server: Requests detailed guide information about the detected item from the generative AI. The generative AI (e.g., a natural language generation engine) generates useful guide information for the user based on previously learned knowledge about the detected item (e.g., the Great Buddha).

[0828] Server: The generated guide information is returned to the server in text format and, if necessary, sent to a voice generation AI (e.g., a voice synthesis engine) where it is also generated as voice data.

[0829] Providing guide information

[0830] Server: The generated guide information and audio data are sent to the device.

[0831] Device: The device displays the received guide information on the user interface so that the user can check it. The device also uses a voice playback engine to provide the audio data to the user as audio guidance.

[0832] Specific examples

[0833] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[0834] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0835] Server: The server receives the image and uses an image recognition engine such as Google Vision API to detect "Label: Great Buddha".

[0836] Server: Based on the detection results, it sends a request to OpenAI's natural language generation engine to "generate guide information about the Great Buddha of Kamakura."

[0837] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0838] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0839] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0840] The processing flow will be explained below.

[0841] Step 1:

[0842] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0843] Device: Captured images are temporarily stored in the device's internal storage.

[0844] Step 2:

[0845] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[0846] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[0847] Step 3:

[0848] Server: Prepares the cloud server to analyze the received image data.

[0849] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[0850] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[0851] Step 4:

[0852] Server: Based on the detected item (e.g., Great Buddha), it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[0853] Server: The generative AI generates guide information and returns the text data to the cloud server.

[0854] Step 5:

[0855] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[0856] Server: Sends the generated guide information text data and audio data to the terminal.

[0857] Step 6:

[0858] Terminal: Updates the text view to display the received guide information in the user interface.

[0859] Terminal: Starts the audio playback engine to play the received audio data.

[0860] Step 7:

[0861] User: The user can check the guide information displayed on the device screen and listen to the audio guide.

[0862] Example 1

[0863] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0864] Conventional tourist guide systems require users to carry a guidebook to obtain detailed information, making it difficult to obtain information in real time. Furthermore, they lack a means to provide users with fast and accurate guide information about the places they visited and the items they were interested in, resulting in an unsatisfactory tourist experience.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0866] In this invention, the server includes means for a user to take an image using a terminal, means for uploading the image taken by the terminal to a cloud server, means for the cloud server to analyze the received image and detect a specific item, means for the cloud server to generate guide information for the detected item, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and playing back the guide information received by the terminal as audio. This enables the user to easily obtain detailed guide information in real time based on the captured image.

[0867] "User" refers to a person who uses the system to take images of tourist spots or items of interest and obtain guide information based on those images.

[0868] "Terminal" refers to a mobile communication device that allows a user to take images and upload them to a cloud server, and includes smartphones, tablets, etc.

[0869] "Cloud server" refers to a server that provides computing resources to receive images uploaded from a terminal via a network, analyze them, generate guide information, and send that information to the terminal.

[0870] "Image Recognition Engine" refers to an algorithm or software that analyzes received image data and automatically identifies specific elements within the image.

[0871] "Natural Language Generation Engine" refers to an algorithm or software for automatically generating textual guide information based on specified prompts.

[0872] "Guide information" refers to information containing detailed explanations and commentary about a specific item, provided in text format or audio data.

[0873] "Display" refers to visually showing guide information on the user interface of the device.

[0874] "Playback by audio" refers to providing the generated audio data to the user audibly through a speaker built into the terminal or connected earphones, etc.

[0875] A "prompt" refers to an input sentence used to instruct a generative AI model to generate specific information (e.g., guide information about a specific item).

[0876] The present invention provides a system that allows users to take pictures of tourist spots or items of interest and obtain detailed guide information in real time based on the pictures. Specific embodiments of the system are described below.

[0877] Hardware and software used

[0878] 1. Device: Refers to a mobile communication device such as a smartphone or tablet used by a user, which takes, saves, uploads, and displays images, plays audio, etc.

[0879] 2. Cloud server: A remote server that receives images from devices via the network, analyzes them, generates information, and transmits it.

[0880] 3. Image recognition engine: Runs on a cloud server and includes an algorithm (e.g., a general-purpose image recognition algorithm) and an API (e.g., a general-purpose image recognition API) for identifying specific elements within an image.

[0881] 4. Natural Language Generation Engine: Refers to the algorithms and APIs (e.g., general-purpose natural language generation engines) used to generate guide information based on the output of an image recognition engine.

[0882] 5. Speech synthesis engine: Includes algorithms and APIs (e.g., general-purpose speech synthesis engines) for converting text-based guide information into speech data.

[0883] Details of data processing and calculation

[0884] 1. Image capture and saving:

[0885] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[0886] Device: The captured image is temporarily stored in the device's internal storage.

[0887] 2. Upload your image:

[0888] Terminal: The saved images are uploaded to the cloud server via the network, where network connection is verified and transactions are managed.

[0889] 3. Image Analysis:

[0890] Server: The cloud server analyzes the received image data and uses an image recognition engine to detect specific items (e.g., the Great Buddha).

[0891] 4. Guide information generation:

[0892] Server: Sends a request to the natural language generation engine to generate detailed guide information about the detected item, which is then returned to the server in text format.

[0893] Server: If necessary, the speech synthesis engine generates the audio data.

[0894] 5. Guide information provided:

[0895] Server: The generated guide information (text and audio data) is sent to the device.

[0896] Terminal: The terminal displays the received guide information on the user interface and plays the audio data.

[0897] Specific examples

[0898] User: Consider a scenario in which a user wants to take a photo of the Great Buddha of Kamakura.

[0899] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[0900] Server: The server receives the image and uses a general-purpose image recognition engine to detect "Label: Great Buddha".

[0901] Server: Based on the detection results, the server sends a prompt to the general-purpose natural language generation engine, saying, "I would like you to generate guide information about the Great Buddha of Kamakura."

[0902] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[0903] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[0904] Prompt Sentence Examples

[0905] "I want to generate guide information about the Great Buddha of Kamakura."

[0906] "Please provide guide information about the Eiffel Tower."

[0907] "Generate a detailed description of Kinkaku-ji Temple in Kyoto"

[0908] With the above flow, users can easily obtain detailed guide information in real time from images they have taken, without having to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[0909] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0910] Step 1:

[0911] User: The user launches the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha). The input is the image captured by the smartphone camera, and the output is an image file "image.jpg" saved in the device's internal storage. The specific action is to launch the camera app and tap the capture button to capture the image.

[0912] Step 2:

[0913] Device: Save the captured image file "image.jpg" to the device's internal storage. In this step, the input is the image data obtained from the camera app, and the output is the saved image file "image.jpg". The specific operation is to use the storage API to save the image data to the specified directory.

[0914] Step 3:

[0915] Device: Uses Wi-Fi or mobile network to upload the saved image file to the cloud server. The input is the saved image file "image.jpg" and the output is the image data uploaded to the cloud server. The specific operation is to check the network connection and transfer the image file to the cloud server using the HTTPS protocol.

[0916] Step 4:

[0917] Server: The cloud server receives the image data sent from the device. The input is the image data sent from the device, and the output is the image data stored in the server's temporary storage. The specific operation is to receive the HTTPS request and store the data in the server's storage.

[0918] Step 5:

[0919] Server: Passes the received image data to the image recognition engine and identifies the specific item. The input is image data stored in the server's storage, and the output is the recognized item (e.g., "Great Buddha"). The specific operation is to call the image recognition engine (e.g., general-purpose image recognition API) and obtain the analysis results.

[0920] Step 6:

[0921] Server: Based on the detected item, it sends a request to the natural language generation engine to generate guide information. The input is the information about the recognized item (e.g., "Great Buddha"), and the output is the text data of the generated guide information. The specific operation is to call the natural language generation engine (e.g., general-purpose natural language generation API) and send the prompt sentence, "I would like you to generate guide information about the Great Buddha."

[0922] Step 7:

[0923] Generative AI model: The natural language generation engine generates detailed guide information based on the prompt. The input is the prompt "Please generate guide information about the Great Buddha," and the output is the guide information text data. The specific operation is to use a natural language generation algorithm to generate detailed explanations corresponding to the prompt.

[0924] Step 8:

[0925] Server: Receives the text data of the generated guide information and generates voice data using a voice synthesis engine as needed. The input is the text data of the generated guide information, and the output is voice data. Specifically, it passes the guide information to a voice synthesis engine (e.g., a general-purpose voice synthesis API) and generates voice data.

[0926] Step 9:

[0927] Server: Sends text guide information and audio data to the device via the cloud. The input is the generated text data and audio data, and the output is the data sent to the device. The specific operation is to send data to the device using a network protocol.

[0928] Step 10:

[0929] Terminal: Displays the received guide information on the user interface and plays the audio data. The input is the text data and audio data of the guide information sent from the server, and the output is display information and audio guidance that the user can check. Specifically, it uses the display API to display the text on the screen and starts the audio playback engine to play the audio data.

[0930] (Application example 1)

[0931] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] In conventional brick-and-mortar shopping experiences, it has been difficult to quickly and easily obtain detailed product information, reviews, discount information, and the like. Furthermore, when users become interested in a product, the provision of real-time information is insufficient, which can discourage them from purchasing. The present invention aims to solve these problems and improve the brick-and-mortar shopping experience.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0934] In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the cloud server to generate guide information and related information for the detected items, means for transmitting the guide information and related information generated by the cloud server to the terminal, and means for displaying and playing back the guide information and related information received by the terminal as audio. This allows users to obtain detailed information and related information about products in which they are interested in in real time.

[0935] "Terminal" refers to the equipment used by the user, specifically devices such as smartphones, tablets, and personal computers.

[0936] A "cloud server" is a remote server provided via the Internet, a collection of computer systems that store, process, and manage data.

[0937] An "image recognition engine" is software or algorithms used to analyze and identify objects or features in images.

[0938] A "natural language generation engine" is a technology that primarily uses AI models to generate natural language that humans can understand based on input data.

[0939] "Guide information" refers to information such as detailed descriptions, background information, and instructions for use regarding a particular item.

[0940] "Related Information" is additional information related to a particular item, including discount information, reviews, and related product recommendations.

[0941] A "display means" is a device or program that has the function of visually presenting information such as text and graphics to a user.

[0942] "Audio playback means" refers to a device or program that has the function of converting text data into audio and providing information to the user audibly.

[0943] The present invention relates to a system that uses a cloud server to provide product information in real time based on images taken by a user in a store. Specific embodiments for carrying out the present invention will be described below.

[0944] Hardware and software used

[0945] Device:

[0946] Users use smartphones and tablets, which require a camera and an internet connection.

[0947] Cloud Server:

[0948] A cloud server is a remote computer system that stores, processes, and manages data.

[0949] Image Recognition Engine:

[0950] Use an image recognition engine such as the Google Vision API to analyze and identify objects in the image.

[0951] Natural Language Generation Engine:

[0952] It uses natural language generation engines such as OpenAI's GPT-3 to generate detailed guides and related information based on the detected items.

[0953] System Operation

[0954] Image capture and upload:

[0955] A user activates the camera on their smartphone and takes a photo of a product they are interested in (e.g., a watch). The image is temporarily saved on the device and then uploaded to a cloud server via the network.

[0956] Image recognition for object detection:

[0957] The cloud server uses an image recognition engine such as Google Vision API to analyze the uploaded image and identify the specific item (e.g., a watch).

[0958] Generate and provide guide information:

[0959] The cloud server requests detailed guide information and related information about the identified item (discount information, reviews, recommended related products, etc.) from OpenAI's GPT-3, a generative AI. The generated information is returned to the server in both text and audio formats.

[0960] Display guide information and play audio:

[0961] The terminal displays the guide information received from the server on the user interface and provides it as audio guidance using an audio playback engine.

[0962] Specific examples

[0963] For example, consider a scenario in which a user takes a photo of a Rolex watch in a store. The user takes a photo of the watch with their smartphone camera and uploads the image file "rolex_watch.jpg" to a cloud server. The server analyzes the image and identifies the "label: watch" using the Google Vision API. It then sends a request to OpenAI's natural language generation engine to "generate guide information about Rolex watches." The generated guide information, "Rolex watches are a luxury watch brand known for their exquisite craftsmanship and high reliability...," along with audio data, is sent to the device. The device displays the guide information, activates the audio playback engine, and provides an audio guide.

[0964] Prompt Sentence Examples

[0965] "I want you to generate guide information about Rolex watches."

[0966] Through this process, users can obtain detailed information and related information about the products they are interested in in real time, significantly improving the shopping experience.

[0967] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0968] Step 1:

[0969] A user activates the smartphone camera and takes a photo of an item of interest (e.g., a watch).

[0970] Input: Image data taken by the user

[0971] Output: Image file temporarily saved on the smartphone

[0972] Step 2:

[0973] The device then uploads the temporarily saved images to a cloud server via the Internet, where network connection and error checks are also performed.

[0974] Input: Temporarily saved image file

[0975] Output: Image file uploaded to the cloud server

[0976] Step 3:

[0977] The server then sends the received image file to an image recognition engine such as Google Vision API, which analyzes the object in the image. For example, the object in the image may be identified as a "watch."

[0978] Input: Image file on cloud server

[0979] Output: Label as image recognition result (e.g., clock)

[0980] Step 4:

[0981] The server sends a request to OpenAI's natural language generation engine to generate guide and related information based on the identified item, including the label.

[0982] Input: Label of image recognition result (e.g., clock)

[0983] Output: Guide information and related information (text format) generated by a natural language generation engine

[0984] Step 5:

[0985] The server transmits the generated guide information and related information to a voice synthesis engine, causing it to generate voice data.

[0986] Input: Guide information and related information from a natural language generation engine

[0987] Output: Audio data (audio file)

[0988] Step 6:

[0989] The server transmits the generated guide information, related information, and audio data to the terminal.

[0990] Input: Guide information and related information (text format), audio data

[0991] Output: Data sent to the terminal

[0992] Step 7:

[0993] The terminal displays the received guide information and related information on the user interface and plays them aloud using a voice playback engine, allowing the user to visually and audibly check the details of the product.

[0994] Input: Guide information and audio data received from the server

[0995] Output: Displaying information and playing audio to the user

[0996] This series of steps allows users to obtain detailed information and related information about products they are interested in in real time, improving their shopping experience.

[0997] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0998] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[0999] 1. Image capture and upload

[1000] Image Capture

[1001] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1002] Device: Captured images are temporarily stored in the device's internal storage.

[1003] Uploading an image

[1004] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[1005] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[1006] 2. Item detection using image recognition

[1007] Image analysis

[1008] Server: Prepares the cloud server to analyze the received image data.

[1009] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[1010] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[1011] 3. Emotion recognition and guidance information adjustment

[1012] Emotion recognition

[1013] Device: The device's camera and sensors are used to recognize the user's face and voice, and to recognize the user's emotions. This process is carried out using an emotion engine.

[1014] Cloud server: The user's emotions may be recognized from the results of image analysis.

[1015] 4. Generating and Providing Guide Information

[1016] Guide information generation

[1017] Server: Requests a generative AI (e.g., a natural language generation engine) to generate guide information based on the detected item (e.g., the Great Buddha) and the user's recognized emotion. For example, if the user is surprised, the information is made more detailed, and if the user is relaxed, the information is made more concise.

[1018] Server: The generative AI generates guide information and returns the text data to the cloud server.

[1019] Providing guide information

[1020] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[1021] Server: Sends the generated guide information text data and audio data to the terminal.

[1022] 5. Display and playback

[1023] Terminal: Updates the text view to display the received guide information in the user interface.

[1024] Terminal: Starts the audio playback engine to play the received audio data.

[1025] Specific examples

[1026] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[1027] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[1028] Server: The server receives the image and uses an image recognition engine to detect "Label: Great Buddha".

[1029] Device: At the same time, the device camera captures the user's facial expression to recognize the user's emotion, and uses the emotion engine to recognize "User emotion: Surprise".

[1030] Server: Based on the detection results and the recognized emotions, the server sends a request to the generative AI to generate detailed guidance information for surprised users.

[1031] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[1032] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[1033] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook, and the information is provided in a way that adapts to the user's emotions, resulting in a more personalized travel experience.

[1034] The processing flow will be explained below.

[1035] Understood. The process flow is explained below step by step.

[1036] Step 1:

[1037] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1038] Device: The captured image is temporarily saved to the smartphone's internal storage.

[1039] Step 2:

[1040] On your device: Check your network connection to upload saved images to the cloud server.

[1041] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[1042] Step 3:

[1043] Server: The cloud server prepares to analyze the received image data.

[1044] Server: Passes image data to an image recognition engine (e.g., an image recognition algorithm) to identify elements within the image.

[1045] Server: Receives the response from the image recognition engine and detects a specific item (e.g., Great Buddha) based on the analysis results.

[1046] Step 4:

[1047] Device: The device's camera and sensors are used to recognize the user's face and perform voice analysis, and an emotion engine is used to recognize the user's emotions.

[1048] Device: Sends the recognized emotion data to the cloud server.

[1049] Step 5:

[1050] Server: Based on the detected item (e.g., Great Buddha) and the emotion data sent from the device, it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[1051] Server: The generative AI generates guide information adapted to the user's emotions and returns the text data to the cloud server.

[1052] Step 6:

[1053] Server: Passes the guide information text data to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[1054] Server: Sends the generated guide information text data and audio data to the terminal.

[1055] Step 7:

[1056] Terminal: Updates the text view to display the received guide information in the user interface.

[1057] Terminal: Starts the audio playback engine to play the received audio data.

[1058] Step 8:

[1059] User: Users can check the guide information displayed on the device screen and listen to the audio guide. The guide information is personalized to the user's emotions, enriching the travel experience.

[1060] By performing each step with specific examples, the system of the present invention quickly provides detailed guide information based on images taken by the user, and by providing this information in a manner that adapts to the user's emotions, it enables quick and easy access to information and a more personalized travel experience.

[1061] Example 2

[1062] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1063] Conventional systems provide guide information based on images taken by the user, but they are unable to adjust the information according to the user's emotions. This results in a uniform user experience, making it difficult to provide personalized information tailored to individual needs. Furthermore, there is a lack of a method for providing guide information suited to the user in real time.

[1064] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting a specific item by analyzing an image, means for recognizing a user's emotion, and means for generating guide information based on the detected item and the recognized user's emotion. This makes it possible to provide guide information adapted to the user's emotion in real time based on an image taken by the user.

[1065] A "terminal" is a device that is directly operated by a user, and is a device that captures images, transmits data to a cloud server, displays guide information, and plays audio.

[1066] A "cloud server" is a server that can be remotely accessed via the Internet, and is a device that performs image analysis, generates guide information, and transmits data.

[1067] An "image recognition algorithm" is a computational approach and method for detecting specific items or elements in image data.

[1068] "Guide information" is information that provides detailed explanations and guidance for detected items, and is provided in text and audio formats.

[1069] A "natural language generation engine" is a system for automatically generating human-understandable natural language text based on input data.

[1070] A "speech generation engine" is a system for converting text data into speech data.

[1071] "User's emotions" refer to the user's psychological state, which is recognized using the user's facial expressions and voice analysis.

[1072] "Image data" refers to digital information such as still images and videos taken by a device.

[1073] An "API request" is a standardized request for an application to communicate with another system, such as a cloud server.

[1074] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[1075] First, a user launches the camera app on their smartphone and takes a photo of a tourist spot or an item of interest (e.g., the Great Buddha). The captured image is temporarily saved in the device's internal storage. At this time, the device checks for a network connection and prepares to upload the captured image to the cloud server. The saved image is then sent to the cloud server via an API request.

[1076] Next, the cloud server prepares the received image data for analysis. Specifically, it performs preprocessing such as image resizing and format conversion. Then, the cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[1077] To recognize a user's emotions, the device uses a camera or sensors to capture the user's facial expressions and voice. This processing is often performed using an emotion engine (e.g., Microsoft Azure Face API). A cloud server may also perform additional image analysis to recognize the user's emotions.

[1078] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the user's recognized emotions. For example, it is adjusted to provide detailed information if the user is surprised and concise information if the user is relaxed. The generation AI generates the guide information and returns the text data to the cloud server.

[1079] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data. The cloud server then sends this text data and voice data to the device.

[1080] Finally, the device displays the received guidance information on the user interface, updates the text view, and starts the audio playback engine to play the received audio data, providing audio guidance.

[1081] Specific examples

[1082] For example, consider a scenario where a user takes a photo of the Great Buddha of Kamakura. The user takes out their smartphone and takes a photo in front of the Great Buddha of Kamakura. The captured image file, "kamakura_buddha.jpg," is saved on the device and automatically uploaded to the cloud server.

[1083] The cloud server uses an image recognition engine to identify "Label: Great Buddha" and saves the result. At the same time, the device's camera captures the user's facial expression, and the emotion engine recognizes "User emotion: Surprise." The cloud server then sends a request to the generation AI to "generate detailed guide information for surprised users."

[1084] The generation AI generates the guide information text, "The Great Buddha of Kamakura is a gigantic Buddha statue approximately 13.35 meters tall, built during the Kamakura period..." and returns it to the cloud server. The cloud server then passes this text to the voice generation AI, which generates voice data. The cloud server then sends the generated text and voice data to the device.

[1085] Guide information is displayed on the device, and the voice playback engine provides the guide audibly. This means users do not need to carry a guidebook, and can obtain detailed guide information in real time from captured images. Furthermore, the information is provided in a way that adapts to the user's emotions, realizing a more personalized travel experience.

[1086] Prompt Sentence Examples

[1087] Examples of prompts for a generative AI model might include:

[1088] "Generate detailed guide information based on the user's emotion of surprise about an image of the Great Buddha taken by the user."

[1089] "Based on the image recognition results, provide concise guidance information when the user is relaxed."

[1090] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1091] Step 1:

[1092] A user takes a photo of a tourist spot or an item of interest (e.g., a Great Buddha) with their smartphone, which causes the smartphone's camera app to generate image data.

[1093] Input: User takes a photo through the camera app.

[1094] Output: The generated image data (e.g. "kamakura_buddha.jpg").

[1095] Step 2:

[1096] The device temporarily stores the captured image data in its internal storage, which saves the image file for subsequent processing.

[1097] Input: Captured image data.

[1098] Output: Image files saved in internal storage.

[1099] Step 3:

[1100] The device checks for a network connection and prepares to upload the saved images to the cloud server, making the image data available for transmission.

[1101] Input: Image files stored in the internal storage.

[1102] Output: Image data ready to be sent to the cloud server.

[1103] Step 4:

[1104] The device makes an API request and sends image data to the cloud server, which then receives the image data.

[1105] Input: Image data for the API request.

[1106] Output: Image data sent to the cloud server.

[1107] Step 5:

[1108] The cloud server prepares the received image data for analysis, specifically by performing preprocessing such as image resizing and format conversion.

[1109] Input: Image data sent to the cloud server.

[1110] Output: Preprocessed image data.

[1111] Step 6:

[1112] The cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[1113] Input: Preprocessed image data.

[1114] Output: Item information as the analysis result (e.g. "Label: Great Buddha").

[1115] Step 7:

[1116] The device captures the user's facial expressions and voice using a camera and sensors, and recognizes the user's emotions using an emotion engine (e.g., Microsoft Azure Face API).

[1117] Input: User data captured by the device's camera or sensors.

[1118] Output: User emotion information (e.g., "User emotion: Surprise").

[1119] Step 8:

[1120] If necessary, the cloud server will recognize the user's emotions from the image analysis results.

[1121] Input: Item information as analysis results and user image data.

[1122] Output: Additional recognized user emotion information.

[1123] Step 9:

[1124] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the recognized user emotions.

[1125] Input: Item information and user emotion information.

[1126] Output: Request data for the generating AI.

[1127] Step 10:

[1128] The generation AI generates guide information and returns the text data to the cloud server.

[1129] Input: Request data for the generating AI.

[1130] Output: Text data of the generated guide information.

[1131] Step 11:

[1132] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data.

[1133] Input: Text data of the generated guide information.

[1134] Output: The generated audio data.

[1135] Step 12:

[1136] The cloud server transmits the generated text data and voice data to the device, which then becomes ready to receive the data.

[1137] Input: Generated text and audio data.

[1138] Output: Text and audio data sent to the device.

[1139] Step 13:

[1140] The terminal displays the received guide information on the user interface and updates the text view.

[1141] Input: Text data sent to the terminal.

[1142] Output: The updated text view.

[1143] Step 14:

[1144] The terminal starts an audio playback engine to play back the audio data received, and provides audio guidance.

[1145] Input: Audio data sent to the device.

[1146] Output: The audio description played.

[1147] This allows users to obtain detailed and personalized guide information in real time.

[1148] (Application example 2)

[1149] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1150] Currently, image analysis and information provision systems on the market do not adequately provide personalized information that takes user emotions into account. As a result, they are unable to provide detailed information tailored to the user's interests and emotions, leaving room for improvement in the user experience. Furthermore, because conventional systems cannot adjust the content of information based on the user's emotional state, they may not provide the most appropriate information for the user.

[1151] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the terminal to recognize the user's emotions, means for the cloud server to generate guide information based on the detected items and the recognized user's emotions, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and audibly playing back the guide information received by the terminal. This makes it possible to provide personalized information according to the user's emotions.

[1152] A "terminal" is an electronic device used by a user, which has a camera function, takes pictures, and communicates with a cloud server.

[1153] A "cloud server" is a remote computer system that receives image data via a network, analyzes the image data, detects items, and generates and transmits guide information.

[1154] An "image recognition engine" is software or algorithms that analyze received image data and identify specific items within the image.

[1155] A "natural language generation engine" is software or an algorithm for automatically generating text for guide information based on specific items and user emotion data.

[1156] "Emotion recognition means" refers to equipment and software that analyzes the user's facial expressions and voice to determine their emotional state at that time.

[1157] "Guide information" is information provided in a user-friendly format, including explanations, usage instructions, and interesting information about detected items.

[1158] The "audio playback means" refers to software and hardware for playing back the generated guide information as audio.

[1159] "Personalized information provision" refers to information provided in a form optimized to suit the individual needs and emotional state of the user.

[1160] This invention relates to a system that automatically provides guide information based on images taken by a user. It also has a function to recognize the user's emotions and adjust the content of the guide information. A specific embodiment of this system is described below.

[1161] First, a user uses the device's camera to take a picture of a specific item (e.g., a product). The image is temporarily stored in the device's internal storage and then uploaded to a cloud server. The device and cloud server communicate via the Internet.

[1162] The cloud server analyzes the received image data and uses an image recognition engine, such as a general-purpose image recognition algorithm, to detect specific items in the image.

[1163] At the same time, the device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotions. This process uses an emotion recognition engine to determine the user's emotional state (e.g., excitement, surprise, relaxation, etc.).

[1164] The cloud server generates guide information using a natural language generation engine based on the detected items and the user's recognized emotions. For example, if it recognizes that the user has an excited expression when looking at a certain product, guide information including a detailed description of the product and how to use it is generated.

[1165] The generated guide information is sent from the cloud server to the device. The device then provides the received guide information to the user both as text and as audio. A text-to-speech engine is used for audio playback.

[1166] Specific examples

[1167] A user is wearing smart glasses while walking through a physical store. If the user looks at a new sneaker and shows an excited expression, the cloud server recognizes the product and generates a detailed product description based on the user's excitement. For example, the user is provided with information such as, "These sneakers are made with the latest technology and provide excellent foot support. This product is popular among many athletes."

[1168] Example prompts for generative AI models

[1169] "A user is looking at the product 'Sneakers'. The user's emotion is 'Excited'. Please generate detailed guide information about this product."

[1170] The system allows users to receive real-time and personalized information, significantly improving the shopping experience.

[1171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1172] Step 1:

[1173] A user takes an image of a specific item using the device's camera. The input is the image data captured by the device's camera, and the output is an image file temporarily stored in the device's internal storage.

[1174] Step 2:

[1175] The device uploads the saved image file to the cloud server. This also checks the network connection. The input is the image data saved in step 1, and the output is the image data uploaded to the cloud server.

[1176] Step 3:

[1177] The cloud server analyzes the received image data and uses an image recognition engine to detect specific items. The input is the image data uploaded to the cloud server, and the output is the identification information of the detected item. Specifically, the cloud server passes the image data to the image recognition engine, and the algorithm analyzes the elements in the image.

[1178] Step 4:

[1179] To recognize a user's emotions, a device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice. The input is the user's facial expression and voice data captured by the camera and microphone, and the output is the user's emotional state analyzed by an emotion recognition engine. Specifically, the device passes the sensor data to the emotion recognition engine, and the algorithm determines the emotional state.

[1180] Step 5:

[1181] The cloud server uses a natural language generation engine to generate guide information based on the detected items and the recognized user emotions. The input is the identification information of the detected items and the recognized user emotional state, and the output is text data of the generated guide information. In specific operations, the cloud server passes various data to the natural language generation engine to generate optimal guide information.

[1182] Step 6:

[1183] The cloud server sends the generated guide information to the terminal. The input is the text data of the generated guide information, and the output is the text data of the guide information sent to the terminal.

[1184] Step 7:

[1185] The terminal displays the received guide information as text and plays it back aloud. A speech synthesis engine is used for the audio playback. The input is the text data of the guide information sent in step 6, and the output is the displayed text information and the audio playback of the guide information. Specifically, the terminal displays the received text data on the screen and starts the speech synthesis engine to play the audio.

[1186] Through this series of operations, users can receive detailed information about products they saw in a physical store in real time, in a form that is appropriately adjusted to their emotional state.

[1187] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1188] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1189] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1190] [Fourth embodiment]

[1191] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1192] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1193] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1194] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1195] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1196] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1197] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1198] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1199] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1200] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1201] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1202] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1203] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1204] The present invention relates to a system that automatically provides guide information based on images taken by travelers, and specific embodiments thereof will be described below.

[1205] 1. Image capture and upload

[1206] Image Capture

[1207] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1208] Device: Captured images are temporarily stored on the device.

[1209] Uploading an image

[1210] Device: The saved images are uploaded to the cloud server via the network, which checks the network connection and manages the transaction.

[1211] 2. Item detection using image recognition

[1212] Image analysis

[1213] Server: The cloud server analyzes the received image data, specifically using an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements in the image and detect specific items (e.g., the Great Buddha).

[1214] Server: Receives the response from the image recognition engine and proceeds to the next step based on the detection results.

[1215] 3. Generating and Providing Guide Information

[1216] Guide information generation

[1217] Server: Requests detailed guide information about the detected item from the generative AI. The generative AI (e.g., a natural language generation engine) generates useful guide information for the user based on previously learned knowledge about the detected item (e.g., the Great Buddha).

[1218] Server: The generated guide information is returned to the server in text format and, if necessary, sent to a voice generation AI (e.g., a voice synthesis engine) where it is also generated as voice data.

[1219] Providing guide information

[1220] Server: The generated guide information and audio data are sent to the device.

[1221] Device: The device displays the received guide information on the user interface so that the user can check it. The device also uses a voice playback engine to provide the audio data to the user as audio guidance.

[1222] Specific examples

[1223] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[1224] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[1225] Server: The server receives the image and uses an image recognition engine such as Google Vision API to detect "Label: Great Buddha".

[1226] Server: Based on the detection results, it sends a request to OpenAI's natural language generation engine to "generate guide information about the Great Buddha of Kamakura."

[1227] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[1228] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[1229] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[1230] The processing flow will be explained below.

[1231] Step 1:

[1232] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1233] Device: Captured images are temporarily stored in the device's internal storage.

[1234] Step 2:

[1235] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[1236] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[1237] Step 3:

[1238] Server: Prepares the cloud server to analyze the received image data.

[1239] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[1240] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[1241] Step 4:

[1242] Server: Based on the detected item (e.g., Great Buddha), it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[1243] Server: The generative AI generates guide information and returns the text data to the cloud server.

[1244] Step 5:

[1245] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[1246] Server: Sends the generated guide information text data and audio data to the terminal.

[1247] Step 6:

[1248] Terminal: Updates the text view to display the received guide information in the user interface.

[1249] Terminal: Starts the audio playback engine to play the received audio data.

[1250] Step 7:

[1251] User: The user can check the guide information displayed on the device screen and listen to the audio guide.

[1252] Example 1

[1253] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1254] Conventional tourist guide systems require users to carry a guidebook to obtain detailed information, making it difficult to obtain information in real time. Furthermore, they lack a means to provide users with fast and accurate guide information about the places they visited and the items they were interested in, resulting in an unsatisfactory tourist experience.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1256] In this invention, the server includes means for a user to take an image using a terminal, means for uploading the image taken by the terminal to a cloud server, means for the cloud server to analyze the received image and detect a specific item, means for the cloud server to generate guide information for the detected item, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and playing back the guide information received by the terminal as audio. This enables the user to easily obtain detailed guide information in real time based on the captured image.

[1257] "User" refers to a person who uses the system to take images of tourist spots or items of interest and obtain guide information based on those images.

[1258] "Terminal" refers to a mobile communication device that allows a user to take images and upload them to a cloud server, and includes smartphones, tablets, etc.

[1259] "Cloud server" refers to a server that provides computing resources to receive images uploaded from a terminal via a network, analyze them, generate guide information, and send that information to the terminal.

[1260] "Image Recognition Engine" refers to an algorithm or software that analyzes received image data and automatically identifies specific elements within the image.

[1261] "Natural Language Generation Engine" refers to an algorithm or software for automatically generating textual guide information based on specified prompts.

[1262] "Guide information" refers to information containing detailed explanations and commentary about a specific item, provided in text format or audio data.

[1263] "Display" refers to visually showing guide information on the user interface of the device.

[1264] "Playback by audio" refers to providing the generated audio data to the user audibly through a speaker built into the terminal or connected earphones, etc.

[1265] A "prompt" refers to an input sentence used to instruct a generative AI model to generate specific information (e.g., guide information about a specific item).

[1266] The present invention provides a system that allows users to take pictures of tourist spots or items of interest and obtain detailed guide information in real time based on the pictures. Specific embodiments of the system are described below.

[1267] Hardware and software used

[1268] 1. Device: Refers to a mobile communication device such as a smartphone or tablet used by a user, which takes, saves, uploads, and displays images, plays audio, etc.

[1269] 2. Cloud server: A remote server that receives images from devices via the network, analyzes them, generates information, and transmits it.

[1270] 3. Image recognition engine: Runs on a cloud server and includes an algorithm (e.g., a general-purpose image recognition algorithm) and an API (e.g., a general-purpose image recognition API) for identifying specific elements within an image.

[1271] 4. Natural Language Generation Engine: Refers to the algorithms and APIs (e.g., general-purpose natural language generation engines) used to generate guide information based on the output of an image recognition engine.

[1272] 5. Speech synthesis engine: Includes algorithms and APIs (e.g., general-purpose speech synthesis engines) for converting text-based guide information into speech data.

[1273] Details of data processing and calculation

[1274] 1. Image capture and saving:

[1275] User: The user activates the smartphone camera and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1276] Device: The captured image is temporarily stored in the device's internal storage.

[1277] 2. Upload your image:

[1278] Terminal: The saved images are uploaded to the cloud server via the network, where network connection is verified and transactions are managed.

[1279] 3. Image Analysis:

[1280] Server: The cloud server analyzes the received image data and uses an image recognition engine to detect specific items (e.g., the Great Buddha).

[1281] 4. Guide information generation:

[1282] Server: Sends a request to the natural language generation engine to generate detailed guide information about the detected item, which is then returned to the server in text format.

[1283] Server: If necessary, the speech synthesis engine generates the audio data.

[1284] 5. Guide information provided:

[1285] Server: The generated guide information (text and audio data) is sent to the device.

[1286] Terminal: The terminal displays the received guide information on the user interface and plays the audio data.

[1287] Specific examples

[1288] User: Consider a scenario in which a user wants to take a photo of the Great Buddha of Kamakura.

[1289] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[1290] Server: The server receives the image and uses a general-purpose image recognition engine to detect "Label: Great Buddha".

[1291] Server: Based on the detection results, the server sends a prompt to the general-purpose natural language generation engine, saying, "I would like you to generate guide information about the Great Buddha of Kamakura."

[1292] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[1293] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[1294] Prompt Sentence Examples

[1295] "I want to generate guide information about the Great Buddha of Kamakura."

[1296] "Please provide guide information about the Eiffel Tower."

[1297] "Generate a detailed description of Kinkaku-ji Temple in Kyoto"

[1298] With the above flow, users can easily obtain detailed guide information in real time from images they have taken, without having to carry a guidebook. This system improves the tourist experience and makes it easier to obtain information.

[1299] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1300] Step 1:

[1301] User: The user launches the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha). The input is the image captured by the smartphone camera, and the output is an image file "image.jpg" saved in the device's internal storage. The specific action is to launch the camera app and tap the capture button to capture the image.

[1302] Step 2:

[1303] Device: Save the captured image file "image.jpg" to the device's internal storage. In this step, the input is the image data obtained from the camera app, and the output is the saved image file "image.jpg". The specific operation is to use the storage API to save the image data to the specified directory.

[1304] Step 3:

[1305] Device: Uses Wi-Fi or mobile network to upload the saved image file to the cloud server. The input is the saved image file "image.jpg" and the output is the image data uploaded to the cloud server. The specific operation is to check the network connection and transfer the image file to the cloud server using the HTTPS protocol.

[1306] Step 4:

[1307] Server: The cloud server receives the image data sent from the device. The input is the image data sent from the device, and the output is the image data stored in the server's temporary storage. The specific operation is to receive the HTTPS request and store the data in the server's storage.

[1308] Step 5:

[1309] Server: Passes the received image data to the image recognition engine and identifies the specific item. The input is image data stored in the server's storage, and the output is the recognized item (e.g., "Great Buddha"). The specific operation is to call the image recognition engine (e.g., general-purpose image recognition API) and obtain the analysis results.

[1310] Step 6:

[1311] Server: Based on the detected item, it sends a request to the natural language generation engine to generate guide information. The input is the information about the recognized item (e.g., "Great Buddha"), and the output is the text data of the generated guide information. The specific operation is to call the natural language generation engine (e.g., general-purpose natural language generation API) and send the prompt sentence, "I would like you to generate guide information about the Great Buddha."

[1312] Step 7:

[1313] Generative AI model: The natural language generation engine generates detailed guide information based on the prompt. The input is the prompt "Please generate guide information about the Great Buddha," and the output is the guide information text data. The specific operation is to use a natural language generation algorithm to generate detailed explanations corresponding to the prompt.

[1314] Step 8:

[1315] Server: Receives the text data of the generated guide information and generates voice data using a voice synthesis engine as needed. The input is the text data of the generated guide information, and the output is voice data. Specifically, it passes the guide information to a voice synthesis engine (e.g., a general-purpose voice synthesis API) and generates voice data.

[1316] Step 9:

[1317] Server: Sends text guide information and audio data to the device via the cloud. The input is the generated text data and audio data, and the output is the data sent to the device. The specific operation is to send data to the device using a network protocol.

[1318] Step 10:

[1319] Terminal: Displays the received guide information on the user interface and plays the audio data. The input is the text data and audio data of the guide information sent from the server, and the output is display information and audio guidance that the user can check. Specifically, it uses the display API to display the text on the screen and starts the audio playback engine to play the audio data.

[1320] (Application example 1)

[1321] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] In conventional brick-and-mortar shopping experiences, it has been difficult to quickly and easily obtain detailed product information, reviews, discount information, and the like. Furthermore, when users become interested in a product, the provision of real-time information is insufficient, which can discourage them from purchasing. The present invention aims to solve these problems and improve the brick-and-mortar shopping experience.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1324] In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the cloud server to generate guide information and related information for the detected items, means for transmitting the guide information and related information generated by the cloud server to the terminal, and means for displaying and playing back the guide information and related information received by the terminal as audio. This allows users to obtain detailed information and related information about products in which they are interested in in real time.

[1325] "Terminal" refers to the equipment used by the user, specifically devices such as smartphones, tablets, and personal computers.

[1326] A "cloud server" is a remote server provided via the Internet, a collection of computer systems that store, process, and manage data.

[1327] An "image recognition engine" is software or algorithms used to analyze and identify objects or features in images.

[1328] A "natural language generation engine" is a technology that primarily uses AI models to generate natural language that humans can understand based on input data.

[1329] "Guide information" refers to information such as detailed descriptions, background information, and instructions for use regarding a particular item.

[1330] "Related Information" is additional information related to a particular item, including discount information, reviews, and related product recommendations.

[1331] A "display means" is a device or program that has the function of visually presenting information such as text and graphics to a user.

[1332] "Audio playback means" refers to a device or program that has the function of converting text data into audio and providing information to the user audibly.

[1333] The present invention relates to a system that uses a cloud server to provide product information in real time based on images taken by a user in a store. Specific embodiments for carrying out the present invention will be described below.

[1334] Hardware and software used

[1335] Device:

[1336] Users use smartphones and tablets, which require a camera and an internet connection.

[1337] Cloud Server:

[1338] A cloud server is a remote computer system that stores, processes, and manages data.

[1339] Image Recognition Engine:

[1340] Use an image recognition engine such as the Google Vision API to analyze and identify objects in the image.

[1341] Natural Language Generation Engine:

[1342] It uses natural language generation engines such as OpenAI's GPT-3 to generate detailed guides and related information based on the detected items.

[1343] System Operation

[1344] Image capture and upload:

[1345] A user activates the camera on their smartphone and takes a photo of a product they are interested in (e.g., a watch). The image is temporarily saved on the device and then uploaded to a cloud server via the network.

[1346] Image recognition for object detection:

[1347] The cloud server uses an image recognition engine such as Google Vision API to analyze the uploaded image and identify the specific item (e.g., a watch).

[1348] Generate and provide guide information:

[1349] The cloud server requests detailed guide information and related information about the identified item (discount information, reviews, recommended related products, etc.) from OpenAI's GPT-3, a generative AI. The generated information is returned to the server in both text and audio formats.

[1350] Display guide information and play audio:

[1351] The terminal displays the guide information received from the server on the user interface and provides it as audio guidance using an audio playback engine.

[1352] Specific examples

[1353] For example, consider a scenario in which a user takes a photo of a Rolex watch in a store. The user takes a photo of the watch with their smartphone camera and uploads the image file "rolex_watch.jpg" to a cloud server. The server analyzes the image and identifies the "label: watch" using the Google Vision API. It then sends a request to OpenAI's natural language generation engine to "generate guide information about Rolex watches." The generated guide information, "Rolex watches are a luxury watch brand known for their exquisite craftsmanship and high reliability...," along with audio data, is sent to the device. The device displays the guide information, activates the audio playback engine, and provides an audio guide.

[1354] Prompt Sentence Examples

[1355] "I want you to generate guide information about Rolex watches."

[1356] Through this process, users can obtain detailed information and related information about the products they are interested in in real time, significantly improving the shopping experience.

[1357] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1358] Step 1:

[1359] A user activates the smartphone camera and takes a photo of an item of interest (e.g., a watch).

[1360] Input: Image data taken by the user

[1361] Output: Image file temporarily saved on the smartphone

[1362] Step 2:

[1363] The device then uploads the temporarily saved images to a cloud server via the Internet, where network connection and error checks are also performed.

[1364] Input: Temporarily saved image file

[1365] Output: Image file uploaded to the cloud server

[1366] Step 3:

[1367] The server then sends the received image file to an image recognition engine such as Google Vision API, which analyzes the object in the image. For example, the object in the image may be identified as a "watch."

[1368] Input: Image file on cloud server

[1369] Output: Label as image recognition result (e.g., clock)

[1370] Step 4:

[1371] The server sends a request to OpenAI's natural language generation engine to generate guide and related information based on the identified item, including the label.

[1372] Input: Label of image recognition result (e.g., clock)

[1373] Output: Guide information and related information (text format) generated by a natural language generation engine

[1374] Step 5:

[1375] The server transmits the generated guide information and related information to a voice synthesis engine, causing it to generate voice data.

[1376] Input: Guide information and related information from a natural language generation engine

[1377] Output: Audio data (audio file)

[1378] Step 6:

[1379] The server transmits the generated guide information, related information, and audio data to the terminal.

[1380] Input: Guide information and related information (text format), audio data

[1381] Output: Data sent to the terminal

[1382] Step 7:

[1383] The terminal displays the received guide information and related information on the user interface and plays them aloud using a voice playback engine, allowing the user to visually and audibly check the details of the product.

[1384] Input: Guide information and audio data received from the server

[1385] Output: Displaying information and playing audio to the user

[1386] This series of steps allows users to obtain detailed information and related information about products they are interested in in real time, improving their shopping experience.

[1387] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1388] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[1389] 1. Image capture and upload

[1390] Image Capture

[1391] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1392] Device: Captured images are temporarily stored in the device's internal storage.

[1393] Uploading an image

[1394] Device: Prepares to upload stored images to the cloud server, including checking network connectivity.

[1395] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[1396] 2. Item detection using image recognition

[1397] Image analysis

[1398] Server: Prepares the cloud server to analyze the received image data.

[1399] Server: Passes image data to an image recognition engine (e.g., a general-purpose image recognition algorithm) to identify elements within the image.

[1400] Server: Receives the results from the image recognition engine and detects a specific item (e.g., the Great Buddha).

[1401] 3. Emotion recognition and guidance information adjustment

[1402] Emotion recognition

[1403] Device: The device's camera and sensors are used to recognize the user's face and voice, and to recognize the user's emotions. This process is carried out using an emotion engine.

[1404] Cloud server: The user's emotions may be recognized from the results of image analysis.

[1405] 4. Generating and Providing Guide Information

[1406] Guide information generation

[1407] Server: Requests a generative AI (e.g., a natural language generation engine) to generate guide information based on the detected item (e.g., the Great Buddha) and the user's recognized emotion. For example, if the user is surprised, the information is made more detailed, and if the user is relaxed, the information is made more concise.

[1408] Server: The generative AI generates guide information and returns the text data to the cloud server.

[1409] Providing guide information

[1410] Server: Passes the guide information text to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[1411] Server: Sends the generated guide information text data and audio data to the terminal.

[1412] 5. Display and playback

[1413] Terminal: Updates the text view to display the received guide information in the user interface.

[1414] Terminal: Starts the audio playback engine to play the received audio data.

[1415] Specific examples

[1416] User: For example, consider a scenario in which a user takes a photo of the Great Buddha of Kamakura.

[1417] Device: The captured image file "kamakura_buddha.jpg" is saved on the device and automatically uploaded to the cloud server.

[1418] Server: The server receives the image and uses an image recognition engine to detect "Label: Great Buddha".

[1419] Device: At the same time, the device camera captures the user's facial expression to recognize the user's emotion, and uses the emotion engine to recognize "User emotion: Surprise".

[1420] Server: Based on the detection results and the recognized emotions, the server sends a request to the generative AI to generate detailed guidance information for surprised users.

[1421] Server: The generated guide information, "The Great Buddha of Kamakura is a huge Buddha statue approximately 13.35 meters tall and was erected during the Kamakura period...", along with the corresponding audio data, is sent to the device.

[1422] Device: Guide information is displayed on the device, the audio playback engine is activated, and audio guides are also provided.

[1423] This series of steps allows users to easily obtain detailed guide information in real time from images they have taken, without the need to carry a guidebook, and the information is provided in a way that adapts to the user's emotions, resulting in a more personalized travel experience.

[1424] The processing flow will be explained below.

[1425] Understood. The process flow is explained below step by step.

[1426] Step 1:

[1427] User: The user opens the camera app on their smartphone and takes a photo of a tourist spot or item of interest (e.g., the Great Buddha).

[1428] Device: The captured image is temporarily saved to the smartphone's internal storage.

[1429] Step 2:

[1430] On your device: Check your network connection to upload saved images to the cloud server.

[1431] Device: Creates an API request to the server and sends the request containing the image data to the cloud server.

[1432] Step 3:

[1433] Server: The cloud server prepares to analyze the received image data.

[1434] Server: Passes image data to an image recognition engine (e.g., an image recognition algorithm) to identify elements within the image.

[1435] Server: Receives the response from the image recognition engine and detects a specific item (e.g., Great Buddha) based on the analysis results.

[1436] Step 4:

[1437] Device: The device's camera and sensors are used to recognize the user's face and perform voice analysis, and an emotion engine is used to recognize the user's emotions.

[1438] Device: Sends the recognized emotion data to the cloud server.

[1439] Step 5:

[1440] Server: Based on the detected item (e.g., Great Buddha) and the emotion data sent from the device, it requests a generative AI (e.g., a natural language generation engine) to generate guide information.

[1441] Server: The generative AI generates guide information adapted to the user's emotions and returns the text data to the cloud server.

[1442] Step 6:

[1443] Server: Passes the guide information text data to a voice generation AI (e.g., a voice synthesis engine) to generate voice data.

[1444] Server: Sends the generated guide information text data and audio data to the terminal.

[1445] Step 7:

[1446] Terminal: Updates the text view to display the received guide information in the user interface.

[1447] Terminal: Starts the audio playback engine to play the received audio data.

[1448] Step 8:

[1449] User: Users can check the guide information displayed on the device screen and listen to the audio guide. The guide information is personalized to the user's emotions, enriching the travel experience.

[1450] By performing each step with specific examples, the system of the present invention quickly provides detailed guide information based on images taken by the user, and by providing this information in a manner that adapts to the user's emotions, it enables quick and easy access to information and a more personalized travel experience.

[1451] Example 2

[1452] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1453] Conventional systems provide guide information based on images taken by the user, but they are unable to adjust the information according to the user's emotions. This results in a uniform user experience, making it difficult to provide personalized information tailored to individual needs. Furthermore, there is a lack of a method for providing guide information suited to the user in real time.

[1454] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting a specific item by analyzing an image, means for recognizing a user's emotion, and means for generating guide information based on the detected item and the recognized user's emotion. This makes it possible to provide guide information adapted to the user's emotion in real time based on an image taken by the user.

[1455] A "terminal" is a device that is directly operated by a user, and is a device that captures images, transmits data to a cloud server, displays guide information, and plays audio.

[1456] A "cloud server" is a server that can be remotely accessed via the Internet, and is a device that performs image analysis, generates guide information, and transmits data.

[1457] An "image recognition algorithm" is a computational approach and method for detecting specific items or elements in image data.

[1458] "Guide information" is information that provides detailed explanations and guidance for detected items, and is provided in text and audio formats.

[1459] A "natural language generation engine" is a system for automatically generating human-understandable natural language text based on input data.

[1460] A "speech generation engine" is a system for converting text data into speech data.

[1461] "User's emotions" refer to the user's psychological state, which is recognized using the user's facial expressions and voice analysis.

[1462] "Image data" refers to digital information such as still images and videos taken by a device.

[1463] An "API request" is a standardized request for an application to communicate with another system, such as a cloud server.

[1464] The present invention relates to a system that automatically provides guide information based on images taken by a user, and further has a function of recognizing the user's emotions and adjusting the content of the guide information. Specific embodiments of the system are described below.

[1465] First, a user launches the camera app on their smartphone and takes a photo of a tourist spot or an item of interest (e.g., the Great Buddha). The captured image is temporarily saved in the device's internal storage. At this time, the device checks for a network connection and prepares to upload the captured image to the cloud server. The saved image is then sent to the cloud server via an API request.

[1466] Next, the cloud server prepares the received image data for analysis. Specifically, it performs preprocessing such as image resizing and format conversion. Then, the cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[1467] To recognize a user's emotions, the device uses a camera or sensors to capture the user's facial expressions and voice. This processing is often performed using an emotion engine (e.g., Microsoft Azure Face API). A cloud server may also perform additional image analysis to recognize the user's emotions.

[1468] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the user's recognized emotions. For example, it is adjusted to provide detailed information if the user is surprised and concise information if the user is relaxed. The generation AI generates the guide information and returns the text data to the cloud server.

[1469] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data. The cloud server then sends this text data and voice data to the device.

[1470] Finally, the device displays the received guidance information on the user interface, updates the text view, and starts the audio playback engine to play the received audio data, providing audio guidance.

[1471] Specific examples

[1472] For example, consider a scenario where a user takes a photo of the Great Buddha of Kamakura. The user takes out their smartphone and takes a photo in front of the Great Buddha of Kamakura. The captured image file, "kamakura_buddha.jpg," is saved on the device and automatically uploaded to the cloud server.

[1473] The cloud server uses an image recognition engine to identify "Label: Great Buddha" and saves the result. At the same time, the device's camera captures the user's facial expression, and the emotion engine recognizes "User emotion: Surprise." The cloud server then sends a request to the generation AI to "generate detailed guide information for surprised users."

[1474] The generation AI generates the guide information text, "The Great Buddha of Kamakura is a gigantic Buddha statue approximately 13.35 meters tall, built during the Kamakura period..." and returns it to the cloud server. The cloud server then passes this text to the voice generation AI, which generates voice data. The cloud server then sends the generated text and voice data to the device.

[1475] Guide information is displayed on the device, and the voice playback engine provides the guide audibly. This means users do not need to carry a guidebook, and can obtain detailed guide information in real time from captured images. Furthermore, the information is provided in a way that adapts to the user's emotions, realizing a more personalized travel experience.

[1476] Prompt Sentence Examples

[1477] Examples of prompts for a generative AI model might include:

[1478] "Generate detailed guide information based on the user's emotion of surprise about an image of the Great Buddha taken by the user."

[1479] "Based on the image recognition results, provide concise guidance information when the user is relaxed."

[1480] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1481] Step 1:

[1482] A user takes a photo of a tourist spot or an item of interest (e.g., a Great Buddha) with their smartphone, which causes the smartphone's camera app to generate image data.

[1483] Input: User takes a photo through the camera app.

[1484] Output: The generated image data (e.g. "kamakura_buddha.jpg").

[1485] Step 2:

[1486] The device temporarily stores the captured image data in its internal storage, which saves the image file for subsequent processing.

[1487] Input: Captured image data.

[1488] Output: Image files saved in internal storage.

[1489] Step 3:

[1490] The device checks for a network connection and prepares to upload the saved images to the cloud server, making the image data available for transmission.

[1491] Input: Image files stored in the internal storage.

[1492] Output: Image data ready to be sent to the cloud server.

[1493] Step 4:

[1494] The device makes an API request and sends image data to the cloud server, which then receives the image data.

[1495] Input: Image data for the API request.

[1496] Output: Image data sent to the cloud server.

[1497] Step 5:

[1498] The cloud server prepares the received image data for analysis, specifically by performing preprocessing such as image resizing and format conversion.

[1499] Input: Image data sent to the cloud server.

[1500] Output: Preprocessed image data.

[1501] Step 6:

[1502] The cloud server analyzes the image data using an image recognition algorithm (e.g., TensorFlow or OpenCV) to detect a specific item (e.g., the Great Buddha).

[1503] Input: Preprocessed image data.

[1504] Output: Item information as the analysis result (e.g. "Label: Great Buddha").

[1505] Step 7:

[1506] The device captures the user's facial expressions and voice using a camera and sensors, and recognizes the user's emotions using an emotion engine (e.g., Microsoft Azure Face API).

[1507] Input: User data captured by the device's camera or sensors.

[1508] Output: User emotion information (e.g., "User emotion: Surprise").

[1509] Step 8:

[1510] If necessary, the cloud server will recognize the user's emotions from the image analysis results.

[1511] Input: Item information as analysis results and user image data.

[1512] Output: Additional recognized user emotion information.

[1513] Step 9:

[1514] The cloud server requests the generation AI (e.g., GPT-4) to generate guide information based on the detected items and the recognized user emotions.

[1515] Input: Item information and user emotion information.

[1516] Output: Request data for the generating AI.

[1517] Step 10:

[1518] The generation AI generates guide information and returns the text data to the cloud server.

[1519] Input: Request data for the generating AI.

[1520] Output: Text data of the generated guide information.

[1521] Step 11:

[1522] The cloud server passes the generated text data of the guide information to a voice generation AI (e.g., Google Text-to-Speech), which generates voice data.

[1523] Input: Text data of the generated guide information.

[1524] Output: The generated audio data.

[1525] Step 12:

[1526] The cloud server transmits the generated text data and voice data to the device, which then becomes ready to receive the data.

[1527] Input: Generated text and audio data.

[1528] Output: Text and audio data sent to the device.

[1529] Step 13:

[1530] The terminal displays the received guide information on the user interface and updates the text view.

[1531] Input: Text data sent to the terminal.

[1532] Output: The updated text view.

[1533] Step 14:

[1534] The terminal starts an audio playback engine to play back the audio data received, and provides audio guidance.

[1535] Input: Audio data sent to the device.

[1536] Output: The audio description played.

[1537] This allows users to obtain detailed and personalized guide information in real time.

[1538] (Application example 2)

[1539] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1540] Currently, image analysis and information provision systems on the market do not adequately provide personalized information that takes user emotions into account. As a result, they are unable to provide detailed information tailored to the user's interests and emotions, leaving room for improvement in the user experience. Furthermore, because conventional systems cannot adjust the content of information based on the user's emotional state, they may not provide the most appropriate information for the user.

[1541] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by the terminal to the cloud server, means for the cloud server to analyze the received images and detect specific items, means for the terminal to recognize the user's emotions, means for the cloud server to generate guide information based on the detected items and the recognized user's emotions, means for transmitting the guide information generated by the cloud server to the terminal, and means for displaying and audibly playing back the guide information received by the terminal. This makes it possible to provide personalized information according to the user's emotions.

[1542] A "terminal" is an electronic device used by a user, which has a camera function, takes pictures, and communicates with a cloud server.

[1543] A "cloud server" is a remote computer system that receives image data via a network, analyzes the image data, detects items, and generates and transmits guide information.

[1544] An "image recognition engine" is software or algorithms that analyze received image data and identify specific items within the image.

[1545] A "natural language generation engine" is software or an algorithm for automatically generating text for guide information based on specific items and user emotion data.

[1546] "Emotion recognition means" refers to equipment and software that analyzes the user's facial expressions and voice to determine their emotional state at that time.

[1547] "Guide information" is information provided in a user-friendly format, including explanations, usage instructions, and interesting information about detected items.

[1548] The "audio playback means" refers to software and hardware for playing back the generated guide information as audio.

[1549] "Personalized information provision" refers to information provided in a form optimized to suit the individual needs and emotional state of the user.

[1550] This invention relates to a system that automatically provides guide information based on images taken by a user. It also has a function to recognize the user's emotions and adjust the content of the guide information. A specific embodiment of this system is described below.

[1551] First, a user uses the device's camera to take a picture of a specific item (e.g., a product). The image is temporarily stored in the device's internal storage and then uploaded to a cloud server. The device and cloud server communicate via the Internet.

[1552] The cloud server analyzes the received image data and uses an image recognition engine, such as a general-purpose image recognition algorithm, to detect specific items in the image.

[1553] At the same time, the device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotions. This process uses an emotion recognition engine to determine the user's emotional state (e.g., excitement, surprise, relaxation, etc.).

[1554] The cloud server generates guide information using a natural language generation engine based on the detected items and the user's recognized emotions. For example, if it recognizes that the user has an excited expression when looking at a certain product, guide information including a detailed description of the product and how to use it is generated.

[1555] The generated guide information is sent from the cloud server to the device. The device then provides the received guide information to the user both as text and as audio. A text-to-speech engine is used for audio playback.

[1556] Specific examples

[1557] A user is wearing smart glasses while walking through a physical store. If the user looks at a new sneaker and shows an excited expression, the cloud server recognizes the product and generates a detailed product description based on the user's excitement. For example, the user is provided with information such as, "These sneakers are made with the latest technology and provide excellent foot support. This product is popular among many athletes."

[1558] Example prompts for generative AI models

[1559] "A user is looking at the product 'Sneakers'. The user's emotion is 'Excited'. Please generate detailed guide information about this product."

[1560] The system allows users to receive real-time and personalized information, significantly improving the shopping experience.

[1561] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1562] Step 1:

[1563] A user takes an image of a specific item using the device's camera. The input is the image data captured by the device's camera, and the output is an image file temporarily stored in the device's internal storage.

[1564] Step 2:

[1565] The device uploads the saved image file to the cloud server. This also checks the network connection. The input is the image data saved in step 1, and the output is the image data uploaded to the cloud server.

[1566] Step 3:

[1567] The cloud server analyzes the received image data and uses an image recognition engine to detect specific items. The input is the image data uploaded to the cloud server, and the output is the identification information of the detected item. Specifically, the cloud server passes the image data to the image recognition engine, and the algorithm analyzes the elements in the image.

[1568] Step 4:

[1569] To recognize a user's emotions, a device uses sensors such as a camera and microphone to analyze the user's facial expressions and voice. The input is the user's facial expression and voice data captured by the camera and microphone, and the output is the user's emotional state analyzed by an emotion recognition engine. Specifically, the device passes the sensor data to the emotion recognition engine, and the algorithm determines the emotional state.

[1570] Step 5:

[1571] The cloud server uses a natural language generation engine to generate guide information based on the detected items and the recognized user emotions. The input is the identification information of the detected items and the recognized user emotional state, and the output is text data of the generated guide information. In specific operations, the cloud server passes various data to the natural language generation engine to generate optimal guide information.

[1572] Step 6:

[1573] The cloud server sends the generated guide information to the terminal. The input is the text data of the generated guide information, and the output is the text data of the guide information sent to the terminal.

[1574] Step 7:

[1575] The terminal displays the received guide information as text and plays it back aloud. A speech synthesis engine is used for the audio playback. The input is the text data of the guide information sent in step 6, and the output is the displayed text information and the audio playback of the guide information. Specifically, the terminal displays the received text data on the screen and starts the speech synthesis engine to play the audio.

[1576] Through this series of operations, users can receive detailed information about products they saw in a physical store in real time, in a form that is appropriately adjusted to their emotional state.

[1577] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1578] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1579] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1580] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1581] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1582] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1583] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1584] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1585] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1586] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1587] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1588] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1589] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1590] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1591] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1592] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1593] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1594] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1595] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1596] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1597] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1598] The following is further disclosed regarding the above embodiment.

[1599] (Claim 1)

[1600] A means for uploading images taken by the device to a cloud server;

[1601] A means for analyzing the image received by the cloud server and detecting a specific item;

[1602] A cloud server generates guide information for the detected item;

[1603] means for transmitting guide information generated by the cloud server to the terminal;

[1604] means for displaying and audibly reproducing the guide information received by the terminal;

[1605] A system including:

[1606] (Claim 2)

[1607] 10. The system of claim 1, wherein the cloud server further comprises means for using an image recognition engine to detect the particular item.

[1608] (Claim 3)

[1609] 10. The system of claim 1, wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information.

[1610] "Example 1"

[1611] (Claim 1)

[1612] A means for a user to take an image using the terminal;

[1613] A means for uploading images taken by the device to a cloud server;

[1614] A means for analyzing the image received by the cloud server and detecting a specific item;

[1615] A cloud server generates guide information for the detected item;

[1616] means for transmitting guide information generated by the cloud server to the terminal;

[1617] means for displaying and audibly reproducing the guide information received by the terminal;

[1618] A system including:

[1619] (Claim 2)

[1620] 10. The system of claim 1, wherein the cloud server further comprises means for using an image recognition engine to detect the particular item.

[1621] (Claim 3)

[1622] 10. The system of claim 1, wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information.

[1623] "Application Example 1"

[1624] (Claim 1)

[1625] A means for uploading images taken by the device to a cloud server;

[1626] A means for analyzing the image received by the cloud server and detecting a specific item;

[1627] A cloud server generates guide information and related information about the detected item;

[1628] means for transmitting the guide information and related information generated by the cloud server to the terminal;

[1629] means for displaying and audibly reproducing the guide information and related information received by the terminal;

[1630] A system including:

[1631] (Claim 2)

[1632] 10. The system of claim 1, wherein the cloud server further comprises means for using an image recognition engine to detect the particular item.

[1633] (Claim 3)

[1634] 10. The system of claim 1, wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information and related information.

[1635] "Example 2: Combining Emotion Engines"

[1636] (Claim 1)

[1637] A means for uploading images taken by the device to a cloud server;

[1638] A means for analyzing the image received by the cloud server and detecting a specific item;

[1639] A means for the terminal to recognize the user's emotion;

[1640] A cloud server generates guide information based on the detected items and the recognized user's emotions;

[1641] means for transmitting guide information generated by the cloud server to the terminal;

[1642] means for displaying and audibly reproducing the guide information received by the terminal;

[1643] A system including:

[1644] (Claim 2)

[1645] 10. The system of claim 1, wherein the cloud server further comprises means for using an image recognition algorithm to detect the particular item.

[1646] (Claim 3)

[1647] 10. The system of claim 1, wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information.

[1648] (Claim 4)

[1649] 10. The system of claim 1, wherein the cloud server further comprises means for using a voice generation engine to generate the voice data.

[1650] "Application example 2 when combining emotion engines"

[1651] (Claim 1)

[1652] A means for uploading images taken by the device to a cloud server;

[1653] A means for analyzing the image received by the cloud server and detecting a specific item;

[1654] A means for the terminal to recognize the user's emotion;

[1655] A cloud server generates guide information based on the detected items and the recognized user's emotions;

[1656] means for transmitting guide information generated by the cloud server to the terminal;

[1657] means for displaying and audibly reproducing the guide information received by the terminal;

[1658] A system including:

[1659] (Claim 2)

[1660] 10. The system of claim 1, wherein the cloud server further comprises means for using an image recognition engine to detect the particular item.

[1661] (Claim 3)

[1662] 10. The system of claim 1, wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information. [Explanation of symbols]

[1663] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for uploading images taken by the device to a cloud server; A means for analyzing the image received by the cloud server and detecting a specific item; A cloud server generates guide information for the detected item; means for transmitting guide information generated by the cloud server to the terminal; means for displaying and audibly reproducing the guide information received by the terminal; A system including:

2. The system of claim 1 , wherein the cloud server further comprises means for using an image recognition engine to detect the particular item.

3. The system of claim 1 , wherein the cloud server further comprises means for using a natural language generation engine to generate the guide information.

Images