System

Abstract

A system is provided.SOLUTION: A wearable device worn by a user, comprising: means for capturing an image of an object in a field of view by a camera; means for identifying an object in the captured image by a local AI module; means for transmitting image analysis information to an external server; means for displaying information obtained from the external server on a lens of one eye; and means for reading the obtained information aloud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, it is difficult to obtain necessary information without using one's hands while out and about or at work. In particular, there is a need to obtain and understand information instantly, such as detailed information about historical buildings when traveling or translating foreign language signs and conversations. However, current technology only provides limited means of accessing information without using one's hands without significantly obstructing one's view, and a user-friendly method is needed. [Means for solving the problem]

[0005] The present invention is intended to solve the above-mentioned problems, and provides the following means by means of a wearable device worn by a user.

[0006] A means of capturing images of objects within view using a camera

[0007] A means of identifying objects in captured images by a local AI module

[0008] A means of sending image analysis information to an external server

[0009] A method for displaying information obtained from an external server on one lens

[0010] A means of reading out acquired information aloud

[0011] This allows users to obtain necessary information in real time without obstructing their field of vision or using their hands.In addition, by including a means for the local AI module to recognize characters in the field of view and send the character data to a server, and a means for the external server to translate the received character data into the language set by the user and send the translation results to the wearable device, it becomes easy to translate signs and conversations in foreign languages.

[0012] "User" refers to an individual who wears a wearable device and obtains information.

[0013] A "wearable device" refers to an electronic device that can be attached to the body and worn by the user during daily life or work.

[0014] "Camera" refers to an on-board imaging device for capturing images of objects within its field of view.

[0015] "Field of view" refers to the range that can be seen through a wearable device worn by a user.

[0016] "Image" refers to the visual data of objects in view captured by a camera.

[0017] "Local AI module" refers to an artificial intelligence processing unit installed within a wearable device to perform image analysis and object recognition.

[0018] An "object" refers to an identifiable object or character that exists within the field of view.

[0019] "Identification" refers to the process by which the local AI module recognizes a specific object from a captured image and identifies its type and characteristics.

[0020] An "external server" refers to a remote computer system that receives information sent from a wearable device and performs analytical processing and provides information.

[0021] "Analysis Information" refers to data about an object that has been analyzed by a local AI module.

[0022] "Monocular lens" refers to a transparent or translucent display mounted on a wearable device to display information in the field of view of one of the user's eyes.

[0023] "Display" refers to the visual presentation of information such as text or images on the lens of one eye.

[0024] "Reading aloud" refers to the act of providing text information to a user as audio.

[0025] "Character recognition" refers to the technology of capturing characters within the field of view as images and converting them into text data.

[0026] "Translation" refers to the process of converting text data into another language.

[0027] "Translation result" refers to the text data obtained through the translation process. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0032] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0033] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0034] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0036] [First embodiment]

[0037] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0038] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0041] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0044] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0047] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0048] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time and acquires, displays, and reads out related information. The following describes in detail the embodiments of the present invention.

[0050] System Configuration

[0051] The wearable device of the present invention is composed of the following main components:

[0052] 1. Camera

[0053] Capture images of objects that come into view.

[0054] 2. Local AI module

[0055] Analyze the captured image and identify the object.

[0056] 3. Communication Module

[0057] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[0058] 4. Display

[0059] Information is displayed on one lens.

[0060] 5. Text-to-speech module

[0061] The acquired information is provided to the user by voice.

[0062] 6. Power Supply Unit

[0063] It supplies the necessary power to the entire wearable device.

[0064] Initial Setup

[0065] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0066] Image capture and analysis

[0067] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[0068] Identifying and transmitting information

[0069] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[0070] Obtaining and Displaying Information

[0071] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the visual information to the user as voice.

[0072] Specific examples

[0073] Obtaining information on historical buildings

[0074] The user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to retrieve detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud.

[0075] Foreign language sign translation

[0076] When a user sees a sign while traveling abroad, the camera captures an image of the sign, and the local AI module performs character recognition. The character data is sent to an external server via the communication module, and the server translates the received character data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user.

[0077] As described above, the present invention provides a user-friendly means of acquiring information by acquiring, displaying, and reading out information in real time without using hands and without significantly obstructing the field of vision.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[0081] Step 2:

[0082] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[0083] Step 3:

[0084] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[0085] Step 4:

[0086] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[0087] Step 5:

[0088] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[0089] Step 6:

[0090] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[0091] Step 7:

[0092] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[0093] Step 8:

[0094] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[0095] Step 9:

[0096] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the Eiffel Tower, who designed it, and the year it was built.

[0097] Step 10:

[0098] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as audio.

[0099] Step 11:

[0100] The user reads the information displayed on one lens or listens to the audio guidance to gather information.

[0101] Specific examples

[0102] Obtaining information on historical buildings

[0103] Step 1:

[0104] The user stands in front of the historical building and turns it on.

[0105] Step 2:

[0106] The device begins its initialization sequence, powering up the camera and sensors.

[0107] Step 3:

[0108] The device connects to the server and goes through an authentication process to establish a secure connection.

[0109] Step 4:

[0110] The device captures images of the building with its camera and sends them to the AI ​​module.

[0111] Step 5:

[0112] The terminal sends the identification result to an external server, which retrieves the information from a database.

[0113] Step 6:

[0114] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[0115] Step 7:

[0116] The device reads out information about the building aloud, and the user confirms the information.

[0117] Translation of foreign language signs

[0118] Step 1:

[0119] A user spots a sign in a foreign city and turns on the device.

[0120] Step 2:

[0121] The device will initialize and start up the camera and sensors.

[0122] Step 3:

[0123] The device connects to the server and goes through an authentication process to establish a secure connection.

[0124] Step 4:

[0125] The device captures an image of the sign and performs character recognition (OCR).

[0126] Step 5:

[0127] The terminal transmits character data to the server, and the server translates the received character data.

[0128] Step 6:

[0129] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[0130] Step 7:

[0131] The device reads out the translation results aloud and the user confirms them.

[0132] The above is a detailed description of the specific processing steps for carrying out the present invention.

[0133] Example 1

[0134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] In modern society, instant access to information is crucial. However, in many situations, visual or auditory information acquisition is often difficult. Travelers and people with visual impairments in particular have limited means of efficiently acquiring and understanding information about their surroundings. Furthermore, there are few ways to acquire information without using their hands, which is inconvenient for users. Therefore, there is a demand for systems that can acquire, display, and read information in real time without using hands and without obstructing the view.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0137] In this invention, the server includes means for receiving and analyzing object identification information, means for searching a database based on the analysis results, and means for transmitting the corresponding information to the wearable device, thereby enabling real-time identification of objects within the field of view and acquisition, display, and reading out of their associated information.

[0138] A "wearable device" is a small electronic device that can be worn by a user and has the function of capturing, analyzing, displaying, and reading out information.

[0139] A "camera" is a photographic device that captures images of objects within its field of view.

[0140] A "local AI module" is an artificial intelligence module installed inside the device that has the ability to analyze and identify objects in captured images.

[0141] An "external server" is a remote server connected via communication from the wearable device, which receives object identification information and searches for information using a database.

[0142] "Object identification information" is information about an object that has been analyzed and identified by a local AI module.

[0143] A "monocular display" is a display device placed on one lens of a wearable device, which visually presents acquired information to the user.

[0144] The "voice reading means" is a device that has the function of converting acquired information into voice and providing it to the user in voice.

[0145] "Character recognition" is the process of analyzing character data in an image and converting it into character data.

[0146] The "translation means" has the function of converting received text data into a specified language.

[0147] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail an embodiment of the present invention.

[0148] System Configuration

[0149] The wearable device of the present invention is composed of the following main components:

[0150] 1. Camera

[0151] The camera captures images of objects within its field of view. The camera has high resolution and a fast shutter speed.

[0152] 2. Local AI module

[0153] The local AI module analyzes captured images and identifies objects. This module uses TensorFlow, a machine learning framework, to run image recognition models.

[0154] 3. Communication Module

[0155] The communication module transmits information about the identified objects to an external server and retrieves related information from the external server, using Wi-Fi or cellular networks.

[0156] 4. Display

[0157] The display displays information on the lens of one eye, and by using a transparent OLED display, it is possible to display information without obstructing the view.

[0158] 5. Text-to-speech module

[0159] The text-to-speech module presents the retrieved information to the user audibly, utilizing the latest speech synthesis technology to generate natural, easy-to-listen-to voices.

[0160] 6. Power Supply Unit

[0161] The power supply unit provides the necessary power for the entire wearable device and is equipped with a large-capacity battery that can be used for a long time.

[0162] Initial Setup

[0163] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0164] Image capture and analysis

[0165] The camera captures objects in its field of view at regular intervals, and the captured image data is sent to a local AI module on the device. The local AI module uses TensorFlow to analyze the image and identify the object.

[0166] Identifying and transmitting information

[0167] The device's local AI module sends information about the identified object to an external server via a communications module. The information sent here includes the characteristics and unique information of the identified object. The external server then searches a database based on the received information to retrieve related information.

[0168] Acquiring, displaying, and reading out information

[0169] The information acquired from the external server is sent to the device via the communication module. The device displays the information on the one-eye display. At the same time, the voice reading module is activated, providing the user with the visual information as voice.

[0170] Specific examples

[0171] Obtaining information on historical buildings

[0172] When a user stands in front of a historical building, the camera captures an image of the building. A local AI module analyzes the image and sends the building's identification information to a server. The external server then searches a database to obtain the building's history, designer, and construction year, which are then sent to the device. The device then displays this information on one of the lenses and reads it aloud using a voice reading function.

[0173] Example prompt:

[0174] "Please tell me the history of this building, who designed it, and when it was built."

[0175] Foreign language sign translation

[0176] When a user sees a sign while traveling abroad, the camera captures an image of the sign. The local AI module performs character recognition and sends the text data to an external server. The external server translates the received text data and sends the translation results to the device. The device then displays the translation result on one of the lenses and conveys it to the user using a voice readout function.

[0177] Example prompt:

[0178] "Please translate the contents of this sign."

[0179] As described above, the present invention provides a means for obtaining information in real time and providing it to the user visually and audibly.

[0180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0181] Step 1: Initial Setup

[0182] When the terminal turns on the wearable device, the device automatically starts an initialization sequence. This initialization sequence starts up the camera and sensors, and the communication module establishes a connection to an external server. The user's authentication information is sent from the terminal to the server, and the server performs authentication. The input during initial setup is the user's authentication information and the terminal's device information, and the output is connection confirmation and the authentication result.

[0183] Specific behavior:

[0184] The device will power on and the system will automatically begin the initialization sequence.

[0185] The camera and sensors are activated and the communication module is connected to the network.

[0186] The authentication information is sent to the server for authentication.

[0187] Step 2: Image capture and transmission

[0188] The device's camera captures objects within its field of view at regular intervals. The captured image data is sent to a local AI module within the device. The input is the captured image data, and the output is the analysis result by the local AI module.

[0189] Specific behavior:

[0190] The camera takes an image of an object that comes into view.

[0191] The captured image data is sent to the local AI module.

[0192] Step 3: Object Identification

[0193] The local AI module uses TensorFlow to analyze images and recognize objects. The analyzed data is sent to the communication module as object identification information. The input is the image data sent to the local AI module, and the output is object identification information.

[0194] Specific behavior:

[0195] The local AI module receives the image data and analyzes it using TensorFlow.

[0196] As a result of the analysis, identification information of the object is generated.

[0197] Step 4: Submit your information

[0198] The communication module of the terminal transmits the information of the identified object to an external server, the input being the object identification information, and the output being the transmission result to the external server.

[0199] Specific behavior:

[0200] The object identification information is transmitted to an external server through a communication module.

[0201] The server receives the information and prepares a database search.

[0202] Step 5: Searching and retrieving information

[0203] The server searches the database based on the received object information to obtain related information. The input is object identification information, and the output is related information.

[0204] Specific behavior:

[0205] The server searches the database and extracts the relevant information.

[0206] The acquired information is then ready to be sent to the terminal.

[0207] Step 6: Receiving and displaying information

[0208] The device receives the information sent from the server and displays it on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. The input is the relevant information sent from the server, and the output is the display and voice output.

[0209] Specific behavior:

[0210] The terminal displays the acquired information on the display.

[0211] The text-to-speech module reads the information aloud as a synthesized voice.

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] Existing wearable devices are capable of identifying objects within their field of view and acquiring and displaying information, but they lack the ability to acquire and display real-time product information in physical stores, as well as voice guidance. This makes it difficult for users to quickly and efficiently acquire product information in physical stores. The present invention solves these issues by providing a system that acquires, displays, and provides voice guidance in real time to improve the shopping experience in physical stores.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server is a wearable device worn by a user, and includes means for capturing images of objects in the user's field of view using a camera, means for identifying objects in the captured images using a local AI module, means for transmitting image analysis information to an external server, means for displaying information acquired from the external server on one of the lenses, means for reading out the acquired information aloud, and means for displaying and reading out information about target products in real time. This allows a user to identify products in a physical store, obtain detailed information about them in real time, and efficiently obtain information through their eyes and ears.

[0217] A "wearable device worn by a user" is a device that is intended to be worn by a user and has the ability to identify objects in its field of view in real time and provide related information on the object in display and audio format.

[0218] A "camera" is a device that has the function of capturing an image of an object within its field of view, and constitutes part of a wearable device.

[0219] A "local AI module" is an artificial intelligence module that has the ability to analyze captured images and identify objects within the images.

[0220] An "external server" is a server whose role is to receive image analysis information sent from a wearable device, search for necessary information based on that information, and send the acquired information to the wearable device.

[0221] The "means for displaying on a lens of one eye" is a means having a function of displaying acquired information on a lens attached to one eye of the user.

[0222] The "means for reading out acquired information by voice" is a means having a function for converting information acquired from an external server into voice and providing the information to the user by voice.

[0223] "Means for displaying in real time and reading out aloud" refers to means that has the function of instantly displaying information about the identified product on a display and providing guidance by voice.

[0224] The present invention is a system that uses a wearable device worn by a user to identify products within the user's field of view in real time, acquire and display related information, and provide audio guidance. Specific embodiments are described in detail below.

[0225] System Configuration

[0226] The system mainly consists of the following components:

[0227] 1. Wearable devices

[0228] Camera: Captures images of objects in view.

[0229] Local AI module: Identifies objects in captured images.

[0230] Communication module: Sends image analysis information to an external server and receives information obtained from the server.

[0231] Display: Displays information on one lens.

[0232] Voice reading module: Provides the retrieved information to the user by voice.

[0233] Power supply unit: Provides the necessary power for the entire wearable device.

[0234] 2. External Server

[0235] Database: Stores detailed information about the retrieved objects.

[0236] AI analysis module: Searches the database based on image analysis information and generates related information.

[0237] Communication module: Sends and receives information to and from the wearable device.

[0238] Hardware and Software Details

[0239] Camera: Uses the OpenCV library for image capture.

[0240] Local AI module: Implements a machine learning model for object classification using TensorFlow and PyTorch.

[0241] Communication module: Communicates with the server using the HTTP protocol, and if possible, a RESTful API.

[0242] Display and voice reading: The acquired information is presented visually and audibly. Visual information is displayed on the display and voice reading is performed using gTTS (Google Text-to-Speech).

[0243] External server: Use cloud services such as AWS and Google Cloud to perform database and AI analysis.

[0244] System Operation

[0245] 1. Image capture and object identification

[0246] As a user walks around the store wearing the wearable device, the camera captures images of objects that come into view, which are then analyzed by a local AI module to identify the objects.

[0247] 2. Sending and receiving information

[0248] The information of the identified object is sent to an external server through a communication module, where an AI analysis module searches a database based on the received information to obtain related product information (price, discounts, reviews, etc.).

[0249] 3. Provision of Information

[0250] The information obtained from the external server is sent back to the wearable device via the communication module. The product information is displayed on one lens, and the voice reading module reads the information aloud.

[0251] Specific examples

[0252] When a user picks up an apple in a physical store, the camera recognizes it and the local AI module identifies it. The information is sent to an external server, which retrieves details such as the apple's price, discounts, and reviews and sends them to the wearable device. The details are displayed on the monocular display, and a voice readout function relays the information to the user.

[0253] Prompt Sentence Examples

[0254] You are an AI modeling engineer designing an application for a system that uses a wearable device worn by the user to identify products in a physical store in real time and retrieve, display, and read out related information. Please write a description of the system that, when the user holds a product in front of them, displays detailed information about the product on the display and reads out the information aloud, clearly stating the specific steps and technologies used.

[0255] ---

[0256] Technologies used: OpenCV, TensorFlow, PyTorch, Google Text-to-Speech (gTTS), playsound

[0257] ---

[0258] Please provide a detailed description of each step: input image capture, object identification, information acquisition, information display, and voice reading.

[0259] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0260] Step 1:

[0261] Image Capture

[0262] While the user is wearing the wearable device, the camera captures images of objects in its field of view at regular intervals. The input is the image in the field of view, and the output is the captured image file. In this step, the OpenCV library is used to acquire images from the camera. Specifically, the camera is started, the image is captured frame by frame, and a specific frame is saved as an image.

[0263] Step 2:

[0264] Object Identification

[0265] The captured image is sent to a local AI module, which uses TensorFlow or PyTorch to identify objects within the image. The input is the captured image file, and the output is information about the identified object. Specifically, the image is input to the AI ​​model, and the model analyzes the features within the image to identify the object. The identification results are output as text data.

[0266] Step 3:

[0267] Image analysis information transmission

[0268] The information about the identified object is sent to an external server via a communication module. The input is the information about the identified object, and the output is the result of sending it to the server (success or failure). Specifically, the information about the identified object is sent to the server as a POST request using the HTTP protocol. At this time, the information about the identified object is encoded in JSON format.

[0269] Step 4:

[0270] Information Search

[0271] The server searches the database based on the received image analysis information to obtain related product information. The input is the received image analysis information, and the output is related product information. Specifically, the AI ​​analysis module on the server side queries the database to obtain detailed information such as the price, discount, and reviews of the relevant product. This information is encoded in JSON format.

[0272] Step 5:

[0273] Information transmission

[0274] The server sends the acquired product information to the wearable device via the communication module. The input is the acquired product information, and the output is the transmission result (success or failure) to the device. Specifically, the product information is returned as an HTTP response, and the communication module receives the response.

[0275] Step 6:

[0276] Information display

[0277] The wearable device displays the received information on one lens of the eye. The input is the received product information, and the output is the visual information displayed on the display. Specifically, the product information is formatted and displayed on the one-eye display.

[0278] Step 7:

[0279] Text-to-speech

[0280] The acquired product information is provided to the user by voice. The input is the received product information, and the output is audio information. Specifically, the product information is converted into audio using gTTS (Google Text-to-Speech) and provided to the user through a voice reading module. At this time, the text data is converted into audio data, and the audio data is played back.

[0281] Through the above steps, a user can identify a product in a physical store, obtain detailed information about the product in real time, and efficiently obtain information through visual and auditory means.

[0282] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0283] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[0284] System Configuration

[0285] The wearable device of the present invention is composed of the following main components:

[0286] 1. Camera

[0287] Capture images of objects that come into view.

[0288] 2. Local AI module

[0289] Analyze the captured image and identify the object.

[0290] 3. Communication Module

[0291] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[0292] 4. Display

[0293] Information is displayed on one lens.

[0294] 5. Text-to-speech module

[0295] The acquired information is provided to the user by voice.

[0296] 6. Emotion Engine

[0297] Recognizes emotions by analyzing the user's tone of voice and facial expressions.

[0298] 7. Power Supply Unit

[0299] It supplies the necessary power to the entire wearable device.

[0300] Initial Setup

[0301] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0302] Image capture and analysis

[0303] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[0304] Identifying and transmitting information

[0305] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[0306] Obtaining and Displaying Information

[0307] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the visual information to the user as voice.

[0308] Emotion Engine Operation

[0309] The emotion engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. The recognized emotional information is reflected in the content displayed and the tone and content of the voice reading.

[0310] Specific examples

[0311] Obtaining information on historical buildings

[0312] The user stands in front of a historical building, and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. A single lens displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, it adjusts the tone and content of the reading to make it more detailed and interesting.

[0313] Foreign language sign translation

[0314] When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communications module, which then translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, it displays or reads out supplementary information to make the translation explanation more detailed and understandable.

[0315] As described above, this invention provides a user-friendly means of information acquisition by acquiring, displaying, and reading out information in real time without significantly obstructing the field of vision and without using the user's hands. Furthermore, by incorporating an emotion engine, it is possible to provide information customized to the user's emotions.

[0316] The processing flow will be explained below.

[0317] Step 1:

[0318] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[0319] Step 2:

[0320] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[0321] Step 3:

[0322] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[0323] Step 4:

[0324] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[0325] Step 5:

[0326] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[0327] Step 6:

[0328] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[0329] Step 7:

[0330] The emotion engine analyzes the user's tone of voice and facial expressions in real time to generate emotion data, which is reflected in the overall system behavior.

[0331] Step 8:

[0332] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[0333] Step 9:

[0334] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[0335] Step 10:

[0336] The server references the emotional data and customizes the information it retrieves, for example, selecting detailed information if the user is excited, or a clear explanation if the user is confused.

[0337] Step 11:

[0338] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the building, its designer, and the year it was built.

[0339] Step 12:

[0340] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as voice in a regulated tone.

[0341] Step 13:

[0342] The user gathers information by reading the information displayed on one lens or listening to the audio guidance. The emotion engine continuously monitors the user's reactions and adjusts the displayed information or audio guidance as needed.

[0343] Specific examples

[0344] Obtaining information on historical buildings

[0345] Step 1:

[0346] The user stands in front of the historical building and turns it on.

[0347] Step 2:

[0348] The device begins its initialization sequence, powering up the camera and sensors.

[0349] Step 3:

[0350] The device connects to the server and goes through an authentication process to establish a secure connection.

[0351] Step 4:

[0352] The device captures images of the building with its camera and sends them to a local AI module.

[0353] Step 5:

[0354] The terminal sends the identification result to an external server, which retrieves the information from a database.

[0355] Step 6:

[0356] The emotion engine detects the user's excitement and the server selects detailed and interesting information.

[0357] Step 7:

[0358] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[0359] Step 8:

[0360] The device reads out information about the building aloud, and the user confirms the information.

[0361] Translation of foreign language signs

[0362] Step 1:

[0363] A user spots a sign in a foreign city and turns on the device.

[0364] Step 2:

[0365] The device will initialize and start up the camera and sensors.

[0366] Step 3:

[0367] The device connects to the server and goes through an authentication process to establish a secure connection.

[0368] Step 4:

[0369] The device captures an image of the sign and performs character recognition (OCR).

[0370] Step 5:

[0371] The terminal transmits character data to the server, and the server translates the received character data.

[0372] Step 6:

[0373] The emotion engine detects the user's confusion, and the server selects an easy-to-understand translation explanation.

[0374] Step 7:

[0375] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[0376] Step 8:

[0377] The device reads out the translation results aloud and the user confirms them.

[0378] The above is a detailed description of the specific processing steps for carrying out the present invention.

[0379] Example 2

[0380] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0381] Wearable devices already exist that can identify objects in their field of view in real time and instantly provide relevant information to users. However, this information is rarely appropriately customized to reflect the user's situation or emotions. In particular, the information provided is uniform and cannot be tailored to the user's interests or emotions, resulting in incomplete information utilization and a poor user experience. Therefore, there is a need for wearable devices that can customize information based on the user's emotions and provide it more effectively.

[0382] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0383] In this invention, the server is a wearable device worn by a user, and includes: means for capturing images of objects within the user's field of view using a camera; means for identifying objects in the captured images using a local AI module; means for transmitting image analysis information to an external server; means for displaying information acquired from the external server on one of the lenses; means for reading out the acquired information aloud; and means including an emotion engine for recognizing the user's emotions and adjusting the display content and the tone of the reading out loud based on the emotions. This enables the wearable device worn by the user to appropriately customize the content of the information provided and the tone of the reading out loud according to the user's emotions.

[0384] A "user" is an individual who wears a wearable device and uses its services, or an entity that represents that individual.

[0385] A "wearable device" is an information processing device that is worn by the user on the body and is equipped with a camera, display, local AI module, communication module, voice reading module, emotion engine, etc.

[0386] A "camera" is a photographic device for capturing images of objects within the user's field of view.

[0387] The "local AI module" is an artificial intelligence processing device that identifies objects in captured images and extracts their features.

[0388] "Object" refers to a specific object or textual information in an image captured by a camera.

[0389] A "communication module" is a communication device for transmitting and receiving data between a wearable device and an external server.

[0390] An "external server" is an information processing device that receives information sent from a wearable device, searches a database based on that information, and obtains and returns related information.

[0391] A "database" is a collection of information stored in an external server, and is a data structure for searching and retrieving information based on specific conditions.

[0392] A "display" is a display device that visually displays captured information and information obtained from a server to a user.

[0393] The "voice reading module" is a device that converts acquired information into voice and provides it to the user.

[0394] The "emotion engine" is an artificial intelligence processing device that analyzes the user's tone of voice and facial expressions to recognize the user's emotions.

[0395] "Authentication information" is personal identification information required when a user accesses a wearable device, and is information for authenticating the user.

[0396] An "initialization sequence" is a series of processes that a wearable device goes through when it starts up, launching each component and making the system ready for use.

[0397] "Field of view" refers to the area that a user can see directly with the naked eye and that is captured by the camera of a wearable device.

[0398] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[0399] System Configuration

[0400] The wearable device of the present invention is composed of the following main components:

[0401] 1. Camera

[0402] The camera captures images of objects within the user's field of view and is high-resolution and capable of capturing footage in real time.

[0403] 2. Local AI module

[0404] The local AI module uses artificial intelligence algorithms based on TensorFlow and PyTorch to analyze images captured by the camera, thereby identifying objects in the image and extracting their features.

[0405] 3. Communication Module

[0406] The communication module is a device for transmitting and receiving data between the wearable device and an external server. It uses Wi-Fi or mobile data communication to send the identified information to the external server and receive the information returned from the server.

[0407] 4. Display

[0408] The display is a device for displaying information on the lens of one eye, and by using a transparent AR (augmented reality) display, information can be viewed without obstructing the field of vision.

[0409] 5. Text-to-speech module

[0410] The voice reading module is a device that provides acquired information to the user by voice, and uses APIs such as Google Text-to-Speech and Amazon Polly.

[0411] 6. Emotion Engine

[0412] The emotion engine uses the Microsoft Emotion API and Affectiva to analyze the user's tone of voice and facial expressions, recognizing the user's emotions based on the analysis results and adjusting the information and tone provided in real time.

[0413] 7. Power Supply Unit

[0414] The power supply unit is a device that supplies the necessary power to the entire wearable device. It is battery-powered and can be used for a long time.

[0415] Initial Setup

[0416] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates them. If authentication is successful, the system is ready for use.

[0417] Image capture and analysis

[0418] The camera captures objects in the user's field of view at regular intervals and sends the images to a local AI module, which analyzes the captured images and identifies the objects. TensorFlow and PyTorch are used to extract object features and generate classification results.

[0419] Identifying and transmitting information

[0420] Information about identified objects is sent to an external server via a communication module. The server searches a database based on the received information to retrieve related information. The server uses common server software such as Apache or Nginx.

[0421] Obtaining and Displaying Information

[0422] The information obtained from the server is sent to the device via the communication module and displayed on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. This voice reading process uses Google Text-to-Speech and Amazon Polly.

[0423] Emotion Engine Operation

[0424] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading. This allows the information provided to be appropriately customized to match the user's emotions.

[0425] Examples and prompts

[0426] 1. Specific examples of acquiring information on historical buildings

[0427] Example of operation: A user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image to identify the building, and the identification information is sent to an external server. The server searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, year of construction, etc., and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[0428] Prompt statement:

[0429] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[0430] 2. Examples of translating foreign language signs

[0431] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[0432] Prompt statement:

[0433] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[0434] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0435] Specific processing steps of the system

[0436] Step 1:

[0437] Initial Setup

[0438] Input: Power on, user credentials

[0439] Specific operation: When a user powers on a wearable device, the device begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection to an external server. The device then sends the user's authentication information to the server, which then verifies the authentication information and performs authentication.

[0440] Data processing and calculation: The terminal sends authentication information, and the server checks the authentication information in the database.

[0441] Output: Authentication success or failure, system ready for use

[0442] Step 2:

[0443] Image capture and analysis

[0444] Input: Objects in the user's field of view

[0445] Specific operation: The device camera captures images of objects within the user's field of view at regular intervals, and the captured images are sent to the local AI module.

[0446] Data processing and calculation: The local AI module uses TensorFlow or PyTorch to analyze images, identify objects, and extract their features.

[0447] Output: Object identity

[0448] Step 3:

[0449] Identifying and transmitting information

[0450] Input: Object identity

[0451] Specific operation: The information of the identified object is sent to an external server through a communication module.

[0452] Data processing and calculation: Information is assembled into data packets and prepared for transmission to the server.

[0453] Output: Identification information sent to the server

[0454] Step 4:

[0455] Obtaining and Displaying Information

[0456] Input: The identity received by the server

[0457] Specific operation: The server searches the database based on the received identification information, retrieves relevant information, and returns that information to the terminal via the communication module.

[0458] Data processing and calculation: The server performs database searches and collects relevant information.

[0459] Output: Retrieved information (details of historical buildings, etc.)

[0460] Step 5:

[0461] Display on the screen and read aloud

[0462] Input: Retrieved information

[0463] Specific operation: The device displays the information received from the server on the one-eye display. At the same time, the voice reading module is activated to provide the user with the visual information by voice.

[0464] Data processing and calculation: Generates display data for the display and prepares for voice synthesis.

[0465] Output: Display of visual information, provision of audio information

[0466] Step 6:

[0467] Emotion Engine Operation

[0468] Input: User's tone of voice and facial expressions

[0469] How it works: The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize their emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading.

[0470] Data processing and calculation: Algorithms analyze voice tone and facial expressions to identify emotional states.

[0471] Output: Customize information based on emotion (adjust tone, add details, etc.)

[0472] Examples and prompts

[0473] 1. Specific examples of acquiring information on historical buildings

[0474] Example of operation: A user stands in front of a historical building, the device captures an image of the building, and the local AI module identifies the building. The identification information is sent to an external server, which searches a database to obtain detailed information. The information is displayed in one of the lenses and read aloud using a text-to-speech function. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[0475] Prompt statement:

[0476] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[0477] 2. Examples of translating foreign language signs

[0478] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[0479] Prompt statement:

[0480] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[0481] (Application example 2)

[0482] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0483] While conventional wearable devices were capable of identifying objects in the field of view and obtaining information, they lacked the ability to adjust the information display and voice readout content according to the user's emotions, resulting in the problem of providing uniform information. Furthermore, when used in a physical store, there was a need for devices to obtain detailed information about products the user sees in real time and provide information that responds to the user's emotions.

[0484] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0485] In this invention, the server includes means for acquiring and displaying prices, reviews, and campaign information for products that the user has seen, means for reading out the acquired information aloud, and means for adjusting the displayed content and the read-out content in accordance with the user's emotions, thereby enabling the user to instantly acquire detailed information about products that the user has seen in a store and providing information flexibly in accordance with the user's emotions.

[0486] A "camera" is a device for capturing images of objects within its field of view.

[0487] A "local AI module" is a module that implements artificial intelligence to identify objects in captured images.

[0488] An "external server" is a server on a network that receives image analysis information and provides related information.

[0489] The "single lens" is a display lens attached to a wearable device, and is a means for displaying acquired information.

[0490] The "voice reading module" is a module for providing acquired information to the user by voice.

[0491] The "emotion engine" is a system that analyzes the user's tone of voice and facial expressions to recognize emotions.

[0492] A "physical store" is a retail store that sells merchandise and that customers can physically visit.

[0493] "Price" is information indicating the price of the product sold at the store.

[0494] A "review" is information that indicates the evaluation or impression that a customer has given about a product.

[0495] "Campaign information" is sales promotion information such as discounts and special offers offered at stores.

[0496] "User's emotion" refers to the emotional state of the user as analyzed from their tone of voice and facial expressions.

[0497] "Information display" refers to displaying acquired information in the user's field of vision.

[0498] "Flexible information provision" refers to appropriately adjusting the display and voice reading of information according to the user's emotions and situation.

[0499] This invention is a system that uses a wearable device to acquire information about objects within the user's field of view in real time and provides information according to the user's emotions. It is particularly specialized for acquiring and displaying product information in physical stores. The configuration and operation of this system are described in detail below.

[0500] System configuration

[0501] 1. Hardware:

[0502] Smart glasses: Includes a camera, display, voice reading module, and communication module.

[0503] Camera: Captures images of objects in view.

[0504] Display: Displays information on one lens.

[0505] Voice reading module: Reads the acquired information aloud.

[0506] Communication module: Sends and receives data to and from external servers.

[0507] 2. Software:

[0508] Local AI module: Identifies objects in images using TensorFlow, OpenCV, etc.

[0509] Emotion recognition engine: Recognizes user emotions using Amazon Rekognition, IBM Watson, etc.

[0510] Database: Manage product information using MySQL or MongoDB.

[0511] How it works

[0512] 1. Image capture:

[0513] The camera in the smart glasses captures the products that come into view, and this image data is sent to a local AI module.

[0514] 2. Image Analysis:

[0515] The local AI module analyzes the image data and identifies the product, and the identification information is sent to an external server via the communication module.

[0516] 3. Information acquisition:

[0517] The external server searches a database based on the identification information to obtain relevant product information (price, reviews, campaign information, etc.).

[0518] 4. Information display and reading:

[0519] The acquired information is displayed on the smart glasses' display, and a voice reading module provides it to the user aloud.

[0520] 5. Emotion recognition:

[0521] The emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotions, and adjusts the content displayed and read aloud based on the recognized emotion.

[0522] Specific examples

[0523] 1. Get detailed product information:

[0524] For example, if a user wearing smart glasses walks through a brick-and-mortar store and stops in front of a shelf of sweets, the camera captures the specific sweet. The local AI module identifies the sweet and sends the identification information to an external server. The server then searches a product database to obtain the price, reviews, and current promotion information, which it then sends to the smart glasses. This information is displayed on the display, and a voice announces, "This sweet has received top reviews. It's currently on a 50% off promotion." If the user looks confused, the emotion recognition engine detects this and provides additional information, such as, "This sweet is milk chocolate. It's low in sugar and recommended for health-conscious people."

[0525] 2. Example prompt:

[0526] "Identify specific objects in images captured by the camera and search for information related to those objects. Based on the identification results, provide detailed information about the relevant products, and recognize emotions from the user's facial expressions and voice to adjust the information displayed and the voice read out."

[0527] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0528] Step 1:

[0529] Image Capture

[0530] Subject: Device

[0531] Input: Objects in the user's field of view

[0532] Output: Image data of the object

[0533] Description: A user wears smart glasses and a camera captures images of objects in their field of view. The camera captures image data of products and objects in the user's field of view.

[0534] Step 2:

[0535] Object Identification

[0536] Subject: Device

[0537] Input: Image data

[0538] Output: Identification information

[0539] Description: The captured image data is sent to a local AI module, which uses TensorFlow and OpenCV to identify objects in the image and extract identifying information.

[0540] Step 3:

[0541] Sending image analysis information

[0542] Subject: Device

[0543] Input: Identification information

[0544] Output: Data sent to external server

[0545] Description: Identified object information is sent to an external server via the communication module. Identification information includes product name and category.

[0546] Step 4:

[0547] Information acquisition

[0548] Subject: Server

[0549] Input: Identification information

[0550] Output: Product details

[0551] Description: The external server searches the database based on the received identification information and retrieves relevant product information (price, reviews, campaign information, etc.).

[0552] Step 5:

[0553] Information transmission

[0554] Subject: Server

[0555] Input: Product details

[0556] Output: Data sent to the terminal

[0557] Description: The acquired product details are sent to the smart glasses through the communication module. The transmitted data includes the product details.

[0558] Step 6:

[0559] Information display and reading

[0560] Subject: Device

[0561] Input: Product details

[0562] Output: Display and voice reading

[0563] Description: The retrieved product information is displayed on the device's display, and the voice reading module provides the information aloud. The display shows the price, reviews, and campaign information, and the voice reads out, "This candy has received excellent reviews. It's currently on sale for 50% off."

[0564] Step 7:

[0565] emotion recognition

[0566] Subject: Device

[0567] Input: User's voice tone and facial expression data

[0568] Output: Emotional information

[0569] Description: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. Emotions can be recognized as interest or confusion, for example.

[0570] Step 8:

[0571] Information adjustment

[0572] Subject: Device

[0573] Input: Emotion information

[0574] Output: Adjusted display and speech content

[0575] Description: The system adjusts the content displayed and spoken based on the recognized emotion information. If the user is confused, additional information will be displayed on the display and spoken aloud: "This candy is milk chocolate. It is low in sugar and recommended for health-conscious people."

[0576] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0577] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0578] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0579] [Second embodiment]

[0580] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0581] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0582] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0583] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0584] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0585] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0586] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0587] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0588] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0589] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0590] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0591] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0592] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time and acquires, displays, and reads out related information. The following describes in detail the embodiments of the present invention.

[0593] System Configuration

[0594] The wearable device of the present invention is composed of the following main components:

[0595] 1. Camera

[0596] Capture images of objects that come into view.

[0597] 2. Local AI module

[0598] Analyze the captured image and identify the object.

[0599] 3. Communication Module

[0600] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[0601] 4. Display

[0602] Information is displayed on one lens.

[0603] 5. Text-to-speech module

[0604] The acquired information is provided to the user by voice.

[0605] 6. Power Supply Unit

[0606] It supplies the necessary power to the entire wearable device.

[0607] Initial Setup

[0608] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0609] Image capture and analysis

[0610] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[0611] Identifying and transmitting information

[0612] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[0613] Obtaining and Displaying Information

[0614] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the visual information to the user as voice.

[0615] Specific examples

[0616] Obtaining information on historical buildings

[0617] The user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to retrieve detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud.

[0618] Foreign language sign translation

[0619] When a user sees a sign while traveling abroad, the camera captures an image of the sign, and the local AI module performs character recognition. The character data is sent to an external server via the communication module, and the server translates the received character data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user.

[0620] As described above, the present invention provides a user-friendly means of acquiring information by acquiring, displaying, and reading out information in real time without using hands and without significantly obstructing the field of vision.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[0624] Step 2:

[0625] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[0626] Step 3:

[0627] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[0628] Step 4:

[0629] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[0630] Step 5:

[0631] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[0632] Step 6:

[0633] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[0634] Step 7:

[0635] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[0636] Step 8:

[0637] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[0638] Step 9:

[0639] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the Eiffel Tower, who designed it, and the year it was built.

[0640] Step 10:

[0641] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as audio.

[0642] Step 11:

[0643] The user reads the information displayed on one lens or listens to the audio guidance to gather information.

[0644] Specific examples

[0645] Obtaining information on historical buildings

[0646] Step 1:

[0647] The user stands in front of the historical building and turns it on.

[0648] Step 2:

[0649] The device begins its initialization sequence, powering up the camera and sensors.

[0650] Step 3:

[0651] The device connects to the server and goes through an authentication process to establish a secure connection.

[0652] Step 4:

[0653] The device captures images of the building with its camera and sends them to the AI ​​module.

[0654] Step 5:

[0655] The terminal sends the identification result to an external server, which retrieves the information from a database.

[0656] Step 6:

[0657] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[0658] Step 7:

[0659] The device reads out information about the building aloud, and the user confirms the information.

[0660] Translation of foreign language signs

[0661] Step 1:

[0662] A user spots a sign in a foreign city and turns on the device.

[0663] Step 2:

[0664] The device will initialize and start up the camera and sensors.

[0665] Step 3:

[0666] The device connects to the server and goes through an authentication process to establish a secure connection.

[0667] Step 4:

[0668] The device captures an image of the sign and performs character recognition (OCR).

[0669] Step 5:

[0670] The terminal transmits character data to the server, and the server translates the received character data.

[0671] Step 6:

[0672] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[0673] Step 7:

[0674] The device reads out the translation results aloud and the user confirms them.

[0675] The above is a detailed description of the specific processing steps for carrying out the present invention.

[0676] Example 1

[0677] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] In modern society, instant access to information is crucial. However, in many situations, visual or auditory information acquisition is often difficult. Travelers and people with visual impairments in particular have limited means of efficiently acquiring and understanding information about their surroundings. Furthermore, there are few ways to acquire information without using their hands, which is inconvenient for users. Therefore, there is a demand for systems that can acquire, display, and read information in real time without using hands and without obstructing the view.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0680] In this invention, the server includes means for receiving and analyzing object identification information, means for searching a database based on the analysis results, and means for transmitting the corresponding information to the wearable device, thereby enabling real-time identification of objects within the field of view and acquisition, display, and reading out of their associated information.

[0681] A "wearable device" is a small electronic device that can be worn by a user and has the function of capturing, analyzing, displaying, and reading out information.

[0682] A "camera" is a photographic device that captures images of objects within its field of view.

[0683] A "local AI module" is an artificial intelligence module installed inside the device that has the ability to analyze and identify objects in captured images.

[0684] An "external server" is a remote server connected via communication from the wearable device, which receives object identification information and searches for information using a database.

[0685] "Object identification information" is information about an object that has been analyzed and identified by a local AI module.

[0686] A "monocular display" is a display device placed on one lens of a wearable device, which visually presents acquired information to the user.

[0687] The "voice reading means" is a device that has the function of converting acquired information into voice and providing it to the user in voice.

[0688] "Character recognition" is the process of analyzing character data in an image and converting it into character data.

[0689] The "translation means" has the function of converting received text data into a specified language.

[0690] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail an embodiment of the present invention.

[0691] System Configuration

[0692] The wearable device of the present invention is composed of the following main components:

[0693] 1. Camera

[0694] The camera captures images of objects within its field of view. The camera has high resolution and a fast shutter speed.

[0695] 2. Local AI module

[0696] The local AI module analyzes captured images and identifies objects. This module uses TensorFlow, a machine learning framework, to run image recognition models.

[0697] 3. Communication Module

[0698] The communication module transmits information about the identified objects to an external server and retrieves related information from the external server, using Wi-Fi or cellular networks.

[0699] 4. Display

[0700] The display displays information on the lens of one eye, and by using a transparent OLED display, it is possible to display information without obstructing the view.

[0701] 5. Text-to-speech module

[0702] The text-to-speech module presents the retrieved information to the user audibly, utilizing the latest speech synthesis technology to generate natural, easy-to-listen-to voices.

[0703] 6. Power Supply Unit

[0704] The power supply unit provides the necessary power for the entire wearable device and is equipped with a large-capacity battery that can be used for a long time.

[0705] Initial Setup

[0706] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0707] Image capture and analysis

[0708] The camera captures objects in its field of view at regular intervals, and the captured image data is sent to a local AI module on the device. The local AI module uses TensorFlow to analyze the image and identify the object.

[0709] Identifying and transmitting information

[0710] The device's local AI module sends information about the identified object to an external server via a communications module. The information sent here includes the characteristics and unique information of the identified object. The external server then searches a database based on the received information to retrieve related information.

[0711] Acquiring, displaying, and reading out information

[0712] The information acquired from the external server is sent to the device via the communication module. The device displays the information on the one-eye display. At the same time, the voice reading module is activated, providing the user with the visual information as voice.

[0713] Specific examples

[0714] Obtaining information on historical buildings

[0715] When a user stands in front of a historical building, the camera captures an image of the building. A local AI module analyzes the image and sends the building's identification information to a server. The external server then searches a database to obtain the building's history, designer, and construction year, which are then sent to the device. The device then displays this information on one of the lenses and reads it aloud using a voice reading function.

[0716] Example prompt:

[0717] "Please tell me the history of this building, who designed it, and when it was built."

[0718] Foreign language sign translation

[0719] When a user sees a sign while traveling abroad, the camera captures an image of the sign. The local AI module performs character recognition and sends the text data to an external server. The external server translates the received text data and sends the translation results to the device. The device then displays the translation result on one of the lenses and conveys it to the user using a voice readout function.

[0720] Example prompt:

[0721] "Please translate the contents of this sign."

[0722] As described above, the present invention provides a means for obtaining information in real time and providing it to the user visually and audibly.

[0723] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0724] Step 1: Initial Setup

[0725] When the terminal turns on the wearable device, the device automatically starts an initialization sequence. This initialization sequence starts up the camera and sensors, and the communication module establishes a connection to an external server. The user's authentication information is sent from the terminal to the server, and the server performs authentication. The input during initial setup is the user's authentication information and the terminal's device information, and the output is connection confirmation and the authentication result.

[0726] Specific behavior:

[0727] The device will power on and the system will automatically begin the initialization sequence.

[0728] The camera and sensors are activated and the communication module is connected to the network.

[0729] The authentication information is sent to the server for authentication.

[0730] Step 2: Image capture and transmission

[0731] The device's camera captures objects within its field of view at regular intervals. The captured image data is sent to a local AI module within the device. The input is the captured image data, and the output is the analysis result by the local AI module.

[0732] Specific behavior:

[0733] The camera takes an image of an object that comes into view.

[0734] The captured image data is sent to the local AI module.

[0735] Step 3: Object Identification

[0736] The local AI module uses TensorFlow to analyze images and recognize objects. The analyzed data is sent to the communication module as object identification information. The input is the image data sent to the local AI module, and the output is object identification information.

[0737] Specific behavior:

[0738] The local AI module receives the image data and analyzes it using TensorFlow.

[0739] As a result of the analysis, identification information of the object is generated.

[0740] Step 4: Submit your information

[0741] The communication module of the terminal transmits the information of the identified object to an external server, the input being the object identification information, and the output being the transmission result to the external server.

[0742] Specific behavior:

[0743] The object identification information is transmitted to an external server through a communication module.

[0744] The server receives the information and prepares a database search.

[0745] Step 5: Searching and retrieving information

[0746] The server searches the database based on the received object information to obtain related information. The input is object identification information, and the output is related information.

[0747] Specific behavior:

[0748] The server searches the database and extracts the relevant information.

[0749] The acquired information is then ready to be sent to the terminal.

[0750] Step 6: Receiving and displaying information

[0751] The device receives the information sent from the server and displays it on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. The input is the relevant information sent from the server, and the output is the display and voice output.

[0752] Specific behavior:

[0753] The terminal displays the acquired information on the display.

[0754] The text-to-speech module reads the information aloud as a synthesized voice.

[0755] (Application example 1)

[0756] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] Existing wearable devices are capable of identifying objects within their field of view and acquiring and displaying information, but they lack the ability to acquire and display real-time product information in physical stores, as well as voice guidance. This makes it difficult for users to quickly and efficiently acquire product information in physical stores. The present invention solves these issues by providing a system that acquires, displays, and provides voice guidance in real time to improve the shopping experience in physical stores.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0759] In this invention, the server is a wearable device worn by a user, and includes means for capturing images of objects in the user's field of view using a camera, means for identifying objects in the captured images using a local AI module, means for transmitting image analysis information to an external server, means for displaying information acquired from the external server on one of the lenses, means for reading out the acquired information aloud, and means for displaying and reading out information about target products in real time. This allows a user to identify products in a physical store, obtain detailed information about them in real time, and efficiently obtain information through their eyes and ears.

[0760] A "wearable device worn by a user" is a device that is intended to be worn by a user and has the ability to identify objects in its field of view in real time and provide related information on the object in display and audio format.

[0761] A "camera" is a device that has the function of capturing an image of an object within its field of view, and constitutes part of a wearable device.

[0762] A "local AI module" is an artificial intelligence module that has the ability to analyze captured images and identify objects within the images.

[0763] An "external server" is a server whose role is to receive image analysis information sent from a wearable device, search for necessary information based on that information, and send the acquired information to the wearable device.

[0764] The "means for displaying on a lens of one eye" is a means having a function of displaying acquired information on a lens attached to one eye of the user.

[0765] The "means for reading out acquired information by voice" is a means having a function for converting information acquired from an external server into voice and providing the information to the user by voice.

[0766] "Means for displaying in real time and reading out aloud" refers to means that has the function of instantly displaying information about the identified product on a display and providing guidance by voice.

[0767] The present invention is a system that uses a wearable device worn by a user to identify products within the user's field of view in real time, acquire and display related information, and provide audio guidance. Specific embodiments are described in detail below.

[0768] System Configuration

[0769] The system mainly consists of the following components:

[0770] 1. Wearable devices

[0771] Camera: Captures images of objects in view.

[0772] Local AI module: Identifies objects in captured images.

[0773] Communication module: Sends image analysis information to an external server and receives information obtained from the server.

[0774] Display: Displays information on one lens.

[0775] Voice reading module: Provides the retrieved information to the user by voice.

[0776] Power supply unit: Provides the necessary power for the entire wearable device.

[0777] 2. External Server

[0778] Database: Stores detailed information about the retrieved objects.

[0779] AI analysis module: Searches the database based on image analysis information and generates related information.

[0780] Communication module: Sends and receives information to and from the wearable device.

[0781] Hardware and Software Details

[0782] Camera: Uses the OpenCV library for image capture.

[0783] Local AI module: Implements a machine learning model for object classification using TensorFlow and PyTorch.

[0784] Communication module: Communicates with the server using the HTTP protocol, and if possible, a RESTful API.

[0785] Display and voice reading: The acquired information is presented visually and audibly. Visual information is displayed on the display and voice reading is performed using gTTS (Google Text-to-Speech).

[0786] External server: Use cloud services such as AWS and Google Cloud to perform database and AI analysis.

[0787] System Operation

[0788] 1. Image capture and object identification

[0789] As a user walks around the store wearing the wearable device, the camera captures images of objects that come into view, which are then analyzed by a local AI module to identify the objects.

[0790] 2. Sending and receiving information

[0791] The information of the identified object is sent to an external server through a communication module, where an AI analysis module searches a database based on the received information to obtain related product information (price, discounts, reviews, etc.).

[0792] 3. Provision of Information

[0793] The information obtained from the external server is sent back to the wearable device via the communication module. The product information is displayed on one lens, and the voice reading module reads the information aloud.

[0794] Specific examples

[0795] When a user picks up an apple in a physical store, the camera recognizes it and the local AI module identifies it. The information is sent to an external server, which retrieves details such as the apple's price, discounts, and reviews and sends them to the wearable device. The details are displayed on the monocular display, and a voice readout function relays the information to the user.

[0796] Prompt Sentence Examples

[0797] You are an AI modeling engineer designing an application for a system that uses a wearable device worn by the user to identify products in a physical store in real time and retrieve, display, and read out related information. Please write a description of the system that, when the user holds a product in front of them, displays detailed information about the product on the display and reads out the information aloud, clearly stating the specific steps and technologies used.

[0798] ---

[0799] Technologies used: OpenCV, TensorFlow, PyTorch, Google Text-to-Speech (gTTS), playsound

[0800] ---

[0801] Please provide a detailed description of each step: input image capture, object identification, information acquisition, information display, and voice reading.

[0802] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0803] Step 1:

[0804] Image Capture

[0805] While the user is wearing the wearable device, the camera captures images of objects in its field of view at regular intervals. The input is the image in the field of view, and the output is the captured image file. In this step, the OpenCV library is used to acquire images from the camera. Specifically, the camera is started, the image is captured frame by frame, and a specific frame is saved as an image.

[0806] Step 2:

[0807] Object Identification

[0808] The captured image is sent to a local AI module, which uses TensorFlow or PyTorch to identify objects within the image. The input is the captured image file, and the output is information about the identified object. Specifically, the image is input to the AI ​​model, and the model analyzes the features within the image to identify the object. The identification results are output as text data.

[0809] Step 3:

[0810] Image analysis information transmission

[0811] The information about the identified object is sent to an external server via a communication module. The input is the information about the identified object, and the output is the result of sending it to the server (success or failure). Specifically, the information about the identified object is sent to the server as a POST request using the HTTP protocol. At this time, the information about the identified object is encoded in JSON format.

[0812] Step 4:

[0813] Information Search

[0814] The server searches the database based on the received image analysis information to obtain related product information. The input is the received image analysis information, and the output is related product information. Specifically, the AI ​​analysis module on the server side queries the database to obtain detailed information such as the price, discount, and reviews of the relevant product. This information is encoded in JSON format.

[0815] Step 5:

[0816] Information transmission

[0817] The server sends the acquired product information to the wearable device via the communication module. The input is the acquired product information, and the output is the transmission result (success or failure) to the device. Specifically, the product information is returned as an HTTP response, and the communication module receives the response.

[0818] Step 6:

[0819] Information display

[0820] The wearable device displays the received information on one lens of the eye. The input is the received product information, and the output is the visual information displayed on the display. Specifically, the product information is formatted and displayed on the one-eye display.

[0821] Step 7:

[0822] Text-to-speech

[0823] The acquired product information is provided to the user by voice. The input is the received product information, and the output is audio information. Specifically, the product information is converted into audio using gTTS (Google Text-to-Speech) and provided to the user through a voice reading module. At this time, the text data is converted into audio data, and the audio data is played back.

[0824] Through the above steps, a user can identify a product in a physical store, obtain detailed information about the product in real time, and efficiently obtain information through visual and auditory means.

[0825] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0826] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[0827] System Configuration

[0828] The wearable device of the present invention is composed of the following main components:

[0829] 1. Camera

[0830] Capture images of objects that come into view.

[0831] 2. Local AI module

[0832] Analyze the captured image and identify the object.

[0833] 3. Communication Module

[0834] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[0835] 4. Display

[0836] Information is displayed on one lens.

[0837] 5. Text-to-speech module

[0838] The acquired information is provided to the user by voice.

[0839] 6. Emotion Engine

[0840] Recognizes emotions by analyzing the user's tone of voice and facial expressions.

[0841] 7. Power Supply Unit

[0842] It supplies the necessary power to the entire wearable device.

[0843] Initial Setup

[0844] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[0845] Image capture and analysis

[0846] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[0847] Identifying and transmitting information

[0848] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[0849] Obtaining and Displaying Information

[0850] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the visual information to the user as voice.

[0851] Emotion Engine Operation

[0852] The emotion engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. The recognized emotional information is reflected in the content displayed and the tone and content of the voice reading.

[0853] Specific examples

[0854] Obtaining information on historical buildings

[0855] The user stands in front of a historical building, and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. A single lens displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, it adjusts the tone and content of the reading to make it more detailed and interesting.

[0856] Foreign language sign translation

[0857] When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communications module, which then translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, it displays or reads out supplementary information to make the translation explanation more detailed and understandable.

[0858] As described above, this invention provides a user-friendly means of information acquisition by acquiring, displaying, and reading out information in real time without significantly obstructing the field of vision and without using the user's hands. Furthermore, by incorporating an emotion engine, it is possible to provide information customized to the user's emotions.

[0859] The processing flow will be explained below.

[0860] Step 1:

[0861] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[0862] Step 2:

[0863] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[0864] Step 3:

[0865] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[0866] Step 4:

[0867] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[0868] Step 5:

[0869] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[0870] Step 6:

[0871] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[0872] Step 7:

[0873] The emotion engine analyzes the user's tone of voice and facial expressions in real time to generate emotion data, which is reflected in the overall system behavior.

[0874] Step 8:

[0875] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[0876] Step 9:

[0877] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[0878] Step 10:

[0879] The server references the emotional data and customizes the information it retrieves, for example, selecting detailed information if the user is excited, or a clear explanation if the user is confused.

[0880] Step 11:

[0881] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the building, its designer, and the year it was built.

[0882] Step 12:

[0883] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as voice in a regulated tone.

[0884] Step 13:

[0885] The user gathers information by reading the information displayed on one lens or listening to the audio guidance. The emotion engine continuously monitors the user's reactions and adjusts the displayed information or audio guidance as needed.

[0886] Specific examples

[0887] Obtaining information on historical buildings

[0888] Step 1:

[0889] The user stands in front of the historical building and turns it on.

[0890] Step 2:

[0891] The device begins its initialization sequence, powering up the camera and sensors.

[0892] Step 3:

[0893] The device connects to the server and goes through an authentication process to establish a secure connection.

[0894] Step 4:

[0895] The device captures images of the building with its camera and sends them to a local AI module.

[0896] Step 5:

[0897] The terminal sends the identification result to an external server, which retrieves the information from a database.

[0898] Step 6:

[0899] The emotion engine detects the user's excitement and the server selects detailed and interesting information.

[0900] Step 7:

[0901] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[0902] Step 8:

[0903] The device reads out information about the building aloud, and the user confirms the information.

[0904] Translation of foreign language signs

[0905] Step 1:

[0906] A user spots a sign in a foreign city and turns on the device.

[0907] Step 2:

[0908] The device will initialize and start up the camera and sensors.

[0909] Step 3:

[0910] The device connects to the server and goes through an authentication process to establish a secure connection.

[0911] Step 4:

[0912] The device captures an image of the sign and performs character recognition (OCR).

[0913] Step 5:

[0914] The terminal transmits character data to the server, and the server translates the received character data.

[0915] Step 6:

[0916] The emotion engine detects the user's confusion, and the server selects an easy-to-understand translation explanation.

[0917] Step 7:

[0918] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[0919] Step 8:

[0920] The device reads out the translation results aloud and the user confirms them.

[0921] The above is a detailed description of the specific processing steps for carrying out the present invention.

[0922] Example 2

[0923] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0924] Wearable devices already exist that can identify objects in their field of view in real time and instantly provide relevant information to users. However, this information is rarely appropriately customized to reflect the user's situation or emotions. In particular, the information provided is uniform and cannot be tailored to the user's interests or emotions, resulting in incomplete information utilization and a poor user experience. Therefore, there is a need for wearable devices that can customize information based on the user's emotions and provide it more effectively.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0926] In this invention, the server is a wearable device worn by a user, and includes: means for capturing images of objects within the user's field of view using a camera; means for identifying objects in the captured images using a local AI module; means for transmitting image analysis information to an external server; means for displaying information acquired from the external server on one of the lenses; means for reading out the acquired information aloud; and means including an emotion engine for recognizing the user's emotions and adjusting the display content and the tone of the reading out loud based on the emotions. This enables the wearable device worn by the user to appropriately customize the content of the information provided and the tone of the reading out loud according to the user's emotions.

[0927] A "user" is an individual who wears a wearable device and uses its services, or an entity that represents that individual.

[0928] A "wearable device" is an information processing device that is worn by the user on the body and is equipped with a camera, display, local AI module, communication module, voice reading module, emotion engine, etc.

[0929] A "camera" is a photographic device for capturing images of objects within the user's field of view.

[0930] The "local AI module" is an artificial intelligence processing device that identifies objects in captured images and extracts their features.

[0931] "Object" refers to a specific object or textual information in an image captured by a camera.

[0932] A "communication module" is a communication device for transmitting and receiving data between a wearable device and an external server.

[0933] An "external server" is an information processing device that receives information sent from a wearable device, searches a database based on that information, and obtains and returns related information.

[0934] A "database" is a collection of information stored in an external server, and is a data structure for searching and retrieving information based on specific conditions.

[0935] A "display" is a display device that visually displays captured information and information obtained from a server to a user.

[0936] The "voice reading module" is a device that converts acquired information into voice and provides it to the user.

[0937] The "emotion engine" is an artificial intelligence processing device that analyzes the user's tone of voice and facial expressions to recognize the user's emotions.

[0938] "Authentication information" is personal identification information required when a user accesses a wearable device, and is information for authenticating the user.

[0939] An "initialization sequence" is a series of processes that a wearable device goes through when it starts up, launching each component and making the system ready for use.

[0940] "Field of view" refers to the area that a user can see directly with the naked eye and that is captured by the camera of a wearable device.

[0941] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[0942] System Configuration

[0943] The wearable device of the present invention is composed of the following main components:

[0944] 1. Camera

[0945] The camera captures images of objects within the user's field of view and is high-resolution and capable of capturing footage in real time.

[0946] 2. Local AI module

[0947] The local AI module uses artificial intelligence algorithms based on TensorFlow and PyTorch to analyze images captured by the camera, thereby identifying objects in the image and extracting their features.

[0948] 3. Communication Module

[0949] The communication module is a device for transmitting and receiving data between the wearable device and an external server. It uses Wi-Fi or mobile data communication to send the identified information to the external server and receive the information returned from the server.

[0950] 4. Display

[0951] The display is a device for displaying information on the lens of one eye, and by using a transparent AR (augmented reality) display, information can be viewed without obstructing the field of vision.

[0952] 5. Text-to-speech module

[0953] The voice reading module is a device that provides acquired information to the user by voice, and uses APIs such as Google Text-to-Speech and Amazon Polly.

[0954] 6. Emotion Engine

[0955] The emotion engine uses the Microsoft Emotion API and Affectiva to analyze the user's tone of voice and facial expressions, recognizing the user's emotions based on the analysis results and adjusting the information and tone provided in real time.

[0956] 7. Power Supply Unit

[0957] The power supply unit is a device that supplies the necessary power to the entire wearable device. It is battery-powered and can be used for a long time.

[0958] Initial Setup

[0959] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates them. If authentication is successful, the system is ready for use.

[0960] Image capture and analysis

[0961] The camera captures objects in the user's field of view at regular intervals and sends the images to a local AI module, which analyzes the captured images and identifies the objects. TensorFlow and PyTorch are used to extract object features and generate classification results.

[0962] Identifying and transmitting information

[0963] Information about identified objects is sent to an external server via a communication module. The server searches a database based on the received information to retrieve related information. The server uses common server software such as Apache or Nginx.

[0964] Obtaining and Displaying Information

[0965] The information obtained from the server is sent to the device via the communication module and displayed on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. This voice reading process uses Google Text-to-Speech and Amazon Polly.

[0966] Emotion Engine Operation

[0967] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading. This allows the information provided to be appropriately customized to match the user's emotions.

[0968] Examples and prompts

[0969] 1. Specific examples of acquiring information on historical buildings

[0970] Example of operation: A user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image to identify the building, and the identification information is sent to an external server. The server searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, year of construction, etc., and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[0971] Prompt statement:

[0972] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[0973] 2. Examples of translating foreign language signs

[0974] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[0975] Prompt statement:

[0976] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[0977] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0978] Specific processing steps of the system

[0979] Step 1:

[0980] Initial Setup

[0981] Input: Power on, user credentials

[0982] Specific operation: When a user powers on a wearable device, the device begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection to an external server. The device then sends the user's authentication information to the server, which then verifies the authentication information and performs authentication.

[0983] Data processing and calculation: The terminal sends authentication information, and the server checks the authentication information in the database.

[0984] Output: Authentication success or failure, system ready for use

[0985] Step 2:

[0986] Image capture and analysis

[0987] Input: Objects in the user's field of view

[0988] Specific operation: The device camera captures images of objects within the user's field of view at regular intervals, and the captured images are sent to the local AI module.

[0989] Data processing and calculation: The local AI module uses TensorFlow or PyTorch to analyze images, identify objects, and extract their features.

[0990] Output: Object identity

[0991] Step 3:

[0992] Identifying and transmitting information

[0993] Input: Object identity

[0994] Specific operation: The information of the identified object is sent to an external server through a communication module.

[0995] Data processing and calculation: Information is assembled into data packets and prepared for transmission to the server.

[0996] Output: Identification information sent to the server

[0997] Step 4:

[0998] Obtaining and Displaying Information

[0999] Input: The identity received by the server

[1000] Specific operation: The server searches the database based on the received identification information, retrieves relevant information, and returns that information to the terminal via the communication module.

[1001] Data processing and calculation: The server performs database searches and collects relevant information.

[1002] Output: Retrieved information (details of historical buildings, etc.)

[1003] Step 5:

[1004] Display on the screen and read aloud

[1005] Input: Retrieved information

[1006] Specific operation: The device displays the information received from the server on the one-eye display. At the same time, the voice reading module is activated to provide the user with the visual information by voice.

[1007] Data processing and calculation: Generates display data for the display and prepares for voice synthesis.

[1008] Output: Display of visual information, provision of audio information

[1009] Step 6:

[1010] Emotion Engine Operation

[1011] Input: User's tone of voice and facial expressions

[1012] How it works: The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize their emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading.

[1013] Data processing and calculation: Algorithms analyze voice tone and facial expressions to identify emotional states.

[1014] Output: Customize information based on emotion (adjust tone, add details, etc.)

[1015] Examples and prompts

[1016] 1. Specific examples of acquiring information on historical buildings

[1017] Example of operation: A user stands in front of a historical building, the device captures an image of the building, and the local AI module identifies the building. The identification information is sent to an external server, which searches a database to obtain detailed information. The information is displayed in one of the lenses and read aloud using a text-to-speech function. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[1018] Prompt statement:

[1019] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[1020] 2. Examples of translating foreign language signs

[1021] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[1022] Prompt statement:

[1023] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] While conventional wearable devices were capable of identifying objects in the field of view and obtaining information, they lacked the ability to adjust the information display and voice readout content according to the user's emotions, resulting in the problem of providing uniform information. Furthermore, when used in a physical store, there was a need for devices to obtain detailed information about products the user sees in real time and provide information that responds to the user's emotions.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1028] In this invention, the server includes means for acquiring and displaying prices, reviews, and campaign information for products that the user has seen, means for reading out the acquired information aloud, and means for adjusting the displayed content and the read-out content in accordance with the user's emotions, thereby enabling the user to instantly acquire detailed information about products that the user has seen in a store and providing information flexibly in accordance with the user's emotions.

[1029] A "camera" is a device for capturing images of objects within its field of view.

[1030] A "local AI module" is a module that implements artificial intelligence to identify objects in captured images.

[1031] An "external server" is a server on a network that receives image analysis information and provides related information.

[1032] The "single lens" is a display lens attached to a wearable device, and is a means for displaying acquired information.

[1033] The "voice reading module" is a module for providing acquired information to the user by voice.

[1034] The "emotion engine" is a system that analyzes the user's tone of voice and facial expressions to recognize emotions.

[1035] A "physical store" is a retail store that sells merchandise and that customers can physically visit.

[1036] "Price" is information indicating the price of the product sold at the store.

[1037] A "review" is information that indicates the evaluation or impression that a customer has given about a product.

[1038] "Campaign information" is sales promotion information such as discounts and special offers offered at stores.

[1039] "User's emotion" refers to the emotional state of the user as analyzed from their tone of voice and facial expressions.

[1040] "Information display" refers to displaying acquired information in the user's field of vision.

[1041] "Flexible information provision" refers to appropriately adjusting the display and voice reading of information according to the user's emotions and situation.

[1042] This invention is a system that uses a wearable device to acquire information about objects within the user's field of view in real time and provides information according to the user's emotions. It is particularly specialized for acquiring and displaying product information in physical stores. The configuration and operation of this system are described in detail below.

[1043] System configuration

[1044] 1. Hardware:

[1045] Smart glasses: Includes a camera, display, voice reading module, and communication module.

[1046] Camera: Captures images of objects in view.

[1047] Display: Displays information on one lens.

[1048] Voice reading module: Reads the acquired information aloud.

[1049] Communication module: Sends and receives data to and from external servers.

[1050] 2. Software:

[1051] Local AI module: Identifies objects in images using TensorFlow, OpenCV, etc.

[1052] Emotion recognition engine: Recognizes user emotions using Amazon Rekognition, IBM Watson, etc.

[1053] Database: Manage product information using MySQL or MongoDB.

[1054] How it works

[1055] 1. Image capture:

[1056] The camera in the smart glasses captures the products that come into view, and this image data is sent to a local AI module.

[1057] 2. Image Analysis:

[1058] The local AI module analyzes the image data and identifies the product, and the identification information is sent to an external server via the communication module.

[1059] 3. Information acquisition:

[1060] The external server searches a database based on the identification information to obtain relevant product information (price, reviews, campaign information, etc.).

[1061] 4. Information display and reading:

[1062] The acquired information is displayed on the smart glasses' display, and a voice reading module provides it to the user aloud.

[1063] 5. Emotion recognition:

[1064] The emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotions, and adjusts the content displayed and read aloud based on the recognized emotion.

[1065] Specific examples

[1066] 1. Get detailed product information:

[1067] For example, if a user wearing smart glasses walks through a brick-and-mortar store and stops in front of a shelf of sweets, the camera captures the specific sweet. The local AI module identifies the sweet and sends the identification information to an external server. The server then searches a product database to obtain the price, reviews, and current promotion information, which it then sends to the smart glasses. This information is displayed on the display, and a voice announces, "This sweet has received top reviews. It's currently on a 50% off promotion." If the user looks confused, the emotion recognition engine detects this and provides additional information, such as, "This sweet is milk chocolate. It's low in sugar and recommended for health-conscious people."

[1068] 2. Example prompt:

[1069] "Identify specific objects in images captured by the camera and search for information related to those objects. Based on the identification results, provide detailed information about the relevant products, and recognize emotions from the user's facial expressions and voice to adjust the information displayed and the voice read out."

[1070] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1071] Step 1:

[1072] Image Capture

[1073] Subject: Device

[1074] Input: Objects in the user's field of view

[1075] Output: Image data of the object

[1076] Description: A user wears smart glasses and a camera captures images of objects in their field of view. The camera captures image data of products and objects in the user's field of view.

[1077] Step 2:

[1078] Object Identification

[1079] Subject: Device

[1080] Input: Image data

[1081] Output: Identification information

[1082] Description: The captured image data is sent to a local AI module, which uses TensorFlow and OpenCV to identify objects in the image and extract identifying information.

[1083] Step 3:

[1084] Sending image analysis information

[1085] Subject: Device

[1086] Input: Identification information

[1087] Output: Data sent to external server

[1088] Description: Identified object information is sent to an external server via the communication module. Identification information includes product name and category.

[1089] Step 4:

[1090] Information acquisition

[1091] Subject: Server

[1092] Input: Identification information

[1093] Output: Product details

[1094] Description: The external server searches the database based on the received identification information and retrieves relevant product information (price, reviews, campaign information, etc.).

[1095] Step 5:

[1096] Information transmission

[1097] Subject: Server

[1098] Input: Product details

[1099] Output: Data sent to the terminal

[1100] Description: The acquired product details are sent to the smart glasses through the communication module. The transmitted data includes the product details.

[1101] Step 6:

[1102] Information display and reading

[1103] Subject: Device

[1104] Input: Product details

[1105] Output: Display and voice reading

[1106] Description: The retrieved product information is displayed on the device's display, and the voice reading module provides the information aloud. The display shows the price, reviews, and campaign information, and the voice reads out, "This candy has received excellent reviews. It's currently on sale for 50% off."

[1107] Step 7:

[1108] emotion recognition

[1109] Subject: Device

[1110] Input: User's voice tone and facial expression data

[1111] Output: Emotional information

[1112] Description: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. Emotions can be recognized as interest or confusion, for example.

[1113] Step 8:

[1114] Information adjustment

[1115] Subject: Device

[1116] Input: Emotion information

[1117] Output: Adjusted display and speech content

[1118] Description: The system adjusts the content displayed and spoken based on the recognized emotion information. If the user is confused, additional information will be displayed on the display and spoken aloud: "This candy is milk chocolate. It is low in sugar and recommended for health-conscious people."

[1119] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1120] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1121] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1122] [Third embodiment]

[1123] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1124] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1125] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1126] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1127] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1128] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1129] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1130] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1131] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1133] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1134] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1135] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail the embodiments of the present invention.

[1136] System Configuration

[1137] The wearable device of the present invention is composed of the following main components:

[1138] 1. Camera

[1139] Capture images of objects that come into view.

[1140] 2. Local AI module

[1141] Analyze the captured image and identify the object.

[1142] 3. Communication Module

[1143] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[1144] 4. Display

[1145] Information is displayed on one lens.

[1146] 5. Text-to-speech module

[1147] The acquired information is presented to the user by voice.

[1148] 6. Power Supply Unit

[1149] It supplies the necessary power to the entire wearable device.

[1150] Initial Setup

[1151] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1152] Image capture and analysis

[1153] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[1154] Identifying and transmitting information

[1155] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[1156] Obtaining and Displaying Information

[1157] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the user with the visual information as voice.

[1158] Specific examples

[1159] Obtaining information on historical buildings

[1160] The user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud.

[1161] Foreign language sign translation

[1162] When a user sees a sign while traveling abroad, the camera captures an image of the sign, and the local AI module performs character recognition. The character data is sent to an external server via the communication module, and the server translates the received character data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user.

[1163] As described above, the present invention provides a user-friendly means of acquiring information by acquiring, displaying, and reading out information in real time without using hands and without significantly obstructing the field of vision.

[1164] The processing flow will be explained below.

[1165] Step 1:

[1166] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[1167] Step 2:

[1168] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[1169] Step 3:

[1170] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[1171] Step 4:

[1172] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[1173] Step 5:

[1174] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[1175] Step 6:

[1176] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[1177] Step 7:

[1178] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[1179] Step 8:

[1180] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[1181] Step 9:

[1182] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the Eiffel Tower, who designed it, and the year it was built.

[1183] Step 10:

[1184] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as audio.

[1185] Step 11:

[1186] The user reads the information displayed on one lens or listens to the audio guidance to gather information.

[1187] Specific examples

[1188] Obtaining information on historical buildings

[1189] Step 1:

[1190] The user stands in front of the historical building and turns it on.

[1191] Step 2:

[1192] The device begins its initialization sequence, powering up the camera and sensors.

[1193] Step 3:

[1194] The device connects to the server and goes through an authentication process to establish a secure connection.

[1195] Step 4:

[1196] The device captures images of the building with its camera and sends them to the AI ​​module.

[1197] Step 5:

[1198] The terminal sends the identification result to an external server, which retrieves the information from a database.

[1199] Step 6:

[1200] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[1201] Step 7:

[1202] The device reads out information about the building aloud, and the user confirms the information.

[1203] Translation of foreign language signs

[1204] Step 1:

[1205] A user spots a sign in a foreign city and turns on the device.

[1206] Step 2:

[1207] The device will initialize and start up the camera and sensors.

[1208] Step 3:

[1209] The device connects to the server and goes through an authentication process to establish a secure connection.

[1210] Step 4:

[1211] The device captures an image of the sign and performs character recognition (OCR).

[1212] Step 5:

[1213] The terminal transmits character data to the server, and the server translates the received character data.

[1214] Step 6:

[1215] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[1216] Step 7:

[1217] The device reads out the translation results aloud and the user confirms them.

[1218] The above is a detailed description of the specific processing steps for carrying out the present invention.

[1219] Example 1

[1220] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1221] In modern society, instant access to information is crucial. However, in many situations, visual or auditory information acquisition is often difficult. Travelers and people with visual impairments in particular have limited means of efficiently acquiring and understanding information about their surroundings. Furthermore, there are few ways to acquire information without using their hands, which is inconvenient for users. Therefore, there is a demand for systems that can acquire, display, and read information in real time without using hands and without obstructing the view.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1223] In this invention, the server includes means for receiving and analyzing object identification information, means for searching a database based on the analysis results, and means for transmitting the corresponding information to the wearable device, thereby enabling real-time identification of objects within the field of view and acquisition, display, and reading out of their associated information.

[1224] A "wearable device" is a small electronic device that can be worn by a user and has the function of capturing, analyzing, displaying, and reading out information.

[1225] A "camera" is a photographic device that captures images of objects within its field of view.

[1226] A "local AI module" is an artificial intelligence module installed inside the device that has the ability to analyze and identify objects in captured images.

[1227] An "external server" is a remote server connected via communication from the wearable device, which receives object identification information and searches for information using a database.

[1228] "Object identification information" is information about an object that has been analyzed and identified by a local AI module.

[1229] A "monocular display" is a display device placed on one lens of a wearable device, which visually presents acquired information to the user.

[1230] The "voice reading means" is a device that has the function of converting acquired information into voice and providing it to the user in voice.

[1231] "Character recognition" is the process of analyzing character data in an image and converting it into character data.

[1232] The "translation means" has the function of converting received text data into a specified language.

[1233] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail an embodiment of the present invention.

[1234] System Configuration

[1235] The wearable device of the present invention is composed of the following main components:

[1236] 1. Camera

[1237] The camera captures images of objects within its field of view. The camera has high resolution and a fast shutter speed.

[1238] 2. Local AI module

[1239] The local AI module analyzes captured images and identifies objects. This module uses TensorFlow, a machine learning framework, to run image recognition models.

[1240] 3. Communication Module

[1241] The communication module transmits information about the identified objects to an external server and retrieves related information from the external server, using Wi-Fi or cellular networks.

[1242] 4. Display

[1243] The display displays information on the lens of one eye, and by using a transparent OLED display, it is possible to display information without obstructing the view.

[1244] 5. Text-to-speech module

[1245] The text-to-speech module presents the retrieved information to the user audibly, utilizing the latest speech synthesis technology to generate natural, easy-to-listen-to voices.

[1246] 6. Power Supply Unit

[1247] The power supply unit provides the necessary power for the entire wearable device and is equipped with a large-capacity battery that can be used for a long time.

[1248] Initial Setup

[1249] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1250] Image capture and analysis

[1251] The camera captures objects in its field of view at regular intervals, and the captured image data is sent to a local AI module on the device. The local AI module uses TensorFlow to analyze the image and identify the object.

[1252] Identifying and transmitting information

[1253] The device's local AI module sends information about the identified object to an external server via a communications module. The information sent here includes the characteristics and unique information of the identified object. The external server then searches a database based on the received information to retrieve related information.

[1254] Acquiring, displaying, and reading out information

[1255] The information acquired from the external server is sent to the device via the communication module. The device displays the information on the one-eye display. At the same time, the voice reading module is activated, providing the user with the visual information as voice.

[1256] Specific examples

[1257] Obtaining information on historical buildings

[1258] When a user stands in front of a historical building, the camera captures an image of the building. A local AI module analyzes the image and sends the building's identification information to a server. The external server then searches a database to obtain the building's history, designer, and construction year, which are then sent to the device. The device then displays this information on one of the lenses and reads it aloud using a voice reading function.

[1259] Example prompt:

[1260] "Please tell me the history of this building, who designed it, and when it was built."

[1261] Foreign language sign translation

[1262] When a user sees a sign while traveling abroad, the camera captures an image of the sign. The local AI module performs character recognition and sends the text data to an external server. The external server translates the received text data and sends the translation results to the device. The device then displays the translation result on one of the lenses and conveys it to the user using a voice readout function.

[1263] Example prompt:

[1264] "Please translate the contents of this sign."

[1265] As described above, the present invention provides a means for obtaining information in real time and providing it to the user visually and audibly.

[1266] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1267] Step 1: Initial Setup

[1268] When the terminal turns on the wearable device, the device automatically starts an initialization sequence. This initialization sequence starts up the camera and sensors, and the communication module establishes a connection to an external server. The user's authentication information is sent from the terminal to the server, and the server performs authentication. The input during initial setup is the user's authentication information and the terminal's device information, and the output is connection confirmation and the authentication result.

[1269] Specific behavior:

[1270] The device will power on and the system will automatically begin the initialization sequence.

[1271] The camera and sensors are activated and the communication module is connected to the network.

[1272] The authentication information is sent to the server for authentication.

[1273] Step 2: Image capture and transmission

[1274] The device's camera captures objects within its field of view at regular intervals. The captured image data is sent to a local AI module within the device. The input is the captured image data, and the output is the analysis result by the local AI module.

[1275] Specific behavior:

[1276] The camera takes an image of an object that comes into view.

[1277] The captured image data is sent to the local AI module.

[1278] Step 3: Object Identification

[1279] The local AI module uses TensorFlow to analyze images and recognize objects. The analyzed data is sent to the communication module as object identification information. The input is the image data sent to the local AI module, and the output is object identification information.

[1280] Specific behavior:

[1281] The local AI module receives the image data and analyzes it using TensorFlow.

[1282] As a result of the analysis, identification information of the object is generated.

[1283] Step 4: Submit your information

[1284] The communication module of the terminal transmits the information of the identified object to an external server, the input being the object identification information, and the output being the transmission result to the external server.

[1285] Specific behavior:

[1286] The object identification information is transmitted to an external server through a communication module.

[1287] The server receives the information and prepares a database search.

[1288] Step 5: Searching and retrieving information

[1289] The server searches the database based on the received object information to obtain related information. The input is object identification information, and the output is related information.

[1290] Specific behavior:

[1291] The server searches the database and extracts the relevant information.

[1292] The acquired information is then ready to be sent to the terminal.

[1293] Step 6: Receiving and displaying information

[1294] The device receives the information sent from the server and displays it on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. The input is the relevant information sent from the server, and the output is the display and voice output.

[1295] Specific behavior:

[1296] The terminal displays the acquired information on the display.

[1297] The text-to-speech module reads the information aloud as a synthesized voice.

[1298] (Application example 1)

[1299] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1300] Existing wearable devices are capable of identifying objects within their field of view and acquiring and displaying information, but they lack the ability to acquire and display real-time product information in physical stores, as well as voice guidance. This makes it difficult for users to quickly and efficiently acquire product information in physical stores. The present invention solves these issues by providing a system that acquires, displays, and provides voice guidance in real time to improve the shopping experience in physical stores.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1302] In this invention, the server is a wearable device worn by a user, and includes means for capturing images of objects in the user's field of view using a camera, means for identifying objects in the captured images using a local AI module, means for transmitting image analysis information to an external server, means for displaying information acquired from the external server on one of the lenses, means for reading out the acquired information aloud, and means for displaying and reading out information about target products in real time. This allows a user to identify products in a physical store, obtain detailed information about them in real time, and efficiently obtain information through their eyes and ears.

[1303] A "wearable device worn by a user" is a device that is intended to be worn by a user and has the ability to identify objects in its field of view in real time and provide related information on the object in display and audio format.

[1304] A "camera" is a device that has the function of capturing an image of an object within its field of view, and constitutes part of a wearable device.

[1305] A "local AI module" is an artificial intelligence module that has the ability to analyze captured images and identify objects within the images.

[1306] An "external server" is a server whose role is to receive image analysis information sent from a wearable device, search for necessary information based on that information, and send the acquired information to the wearable device.

[1307] The "means for displaying on a lens of one eye" is a means having a function of displaying acquired information on a lens attached to one eye of the user.

[1308] The "means for reading out acquired information by voice" is a means having a function for converting information acquired from an external server into voice and providing the information to the user by voice.

[1309] "Means for displaying in real time and reading out aloud" refers to means that has the function of instantly displaying information about the identified product on a display and providing guidance by voice.

[1310] The present invention is a system that uses a wearable device worn by a user to identify products within the user's field of view in real time, acquire and display related information, and provide audio guidance. Specific embodiments are described in detail below.

[1311] System Configuration

[1312] The system mainly consists of the following components:

[1313] 1. Wearable devices

[1314] Camera: Captures images of objects in view.

[1315] Local AI module: Identifies objects in captured images.

[1316] Communication module: Sends image analysis information to an external server and receives information obtained from the server.

[1317] Display: Displays information on one lens.

[1318] Voice reading module: Provides the retrieved information to the user by voice.

[1319] Power supply unit: Provides the necessary power for the entire wearable device.

[1320] 2. External Server

[1321] Database: Stores detailed information about the retrieved objects.

[1322] AI analysis module: Searches the database based on image analysis information and generates related information.

[1323] Communication module: Sends and receives information to and from the wearable device.

[1324] Hardware and Software Details

[1325] Camera: Uses the OpenCV library for image capture.

[1326] Local AI module: Implements a machine learning model for object classification using TensorFlow and PyTorch.

[1327] Communication module: Communicates with the server using the HTTP protocol, and if possible, a RESTful API.

[1328] Display and voice reading: The acquired information is presented visually and audibly. Visual information is displayed on the display and voice reading is performed using gTTS (Google Text-to-Speech).

[1329] External server: Use cloud services such as AWS and Google Cloud to perform database and AI analysis.

[1330] System Operation

[1331] 1. Image capture and object identification

[1332] As a user walks around the store wearing the wearable device, the camera captures images of objects that come into view, which are then analyzed by a local AI module to identify the objects.

[1333] 2. Sending and receiving information

[1334] The information of the identified object is sent to an external server through a communication module, where an AI analysis module searches a database based on the received information to obtain related product information (price, discounts, reviews, etc.).

[1335] 3. Provision of Information

[1336] The information obtained from the external server is sent back to the wearable device via the communication module. The product information is displayed on one lens, and the voice reading module reads the information aloud.

[1337] Specific examples

[1338] When a user picks up an apple in a physical store, the camera recognizes it and the local AI module identifies it. The information is sent to an external server, which retrieves details such as the apple's price, discounts, and reviews and sends them to the wearable device. The details are displayed on the monocular display, and a voice readout function relays the information to the user.

[1339] Prompt Sentence Examples

[1340] You are an AI modeling engineer designing an application for a system that uses a wearable device worn by the user to identify products in a physical store in real time and retrieve, display, and read out related information. Please write a description of the system that, when the user holds a product in front of them, displays detailed information about the product on the display and reads out the information aloud, clearly stating the specific steps and technologies used.

[1341] ---

[1342] Technologies used: OpenCV, TensorFlow, PyTorch, Google Text-to-Speech (gTTS), playsound

[1343] ---

[1344] Please provide a detailed description of each step: input image capture, object identification, information acquisition, information display, and voice reading.

[1345] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1346] Step 1:

[1347] Image Capture

[1348] While the user is wearing the wearable device, the camera captures images of objects in its field of view at regular intervals. The input is the image in the field of view, and the output is the captured image file. In this step, the OpenCV library is used to acquire images from the camera. Specifically, the camera is started, the image is captured frame by frame, and a specific frame is saved as an image.

[1349] Step 2:

[1350] Object Identification

[1351] The captured image is sent to a local AI module, which uses TensorFlow or PyTorch to identify objects within the image. The input is the captured image file, and the output is information about the identified object. Specifically, the image is input to the AI ​​model, and the model analyzes the features within the image to identify the object. The identification results are output as text data.

[1352] Step 3:

[1353] Image analysis information transmission

[1354] The information about the identified object is sent to an external server via a communication module. The input is the information about the identified object, and the output is the result of sending it to the server (success or failure). Specifically, the information about the identified object is sent to the server as a POST request using the HTTP protocol. At this time, the information about the identified object is encoded in JSON format.

[1355] Step 4:

[1356] Information Search

[1357] The server searches the database based on the received image analysis information to obtain related product information. The input is the received image analysis information, and the output is related product information. Specifically, the AI ​​analysis module on the server side queries the database to obtain detailed information such as the price, discount, and reviews of the relevant product. This information is encoded in JSON format.

[1358] Step 5:

[1359] Information transmission

[1360] The server sends the acquired product information to the wearable device via the communication module. The input is the acquired product information, and the output is the transmission result (success or failure) to the device. Specifically, the product information is returned as an HTTP response, and the communication module receives the response.

[1361] Step 6:

[1362] Information display

[1363] The wearable device displays the received information on one lens of the eye. The input is the received product information, and the output is the visual information displayed on the display. Specifically, the product information is formatted and displayed on the one-eye display.

[1364] Step 7:

[1365] Text-to-speech

[1366] The acquired product information is provided to the user by voice. The input is the received product information, and the output is audio information. Specifically, the product information is converted into audio using gTTS (Google Text-to-Speech) and provided to the user through a voice reading module. At this time, the text data is converted into audio data, and the audio data is played back.

[1367] Through the above steps, a user can identify a product in a physical store, obtain detailed information about the product in real time, and efficiently obtain information through visual and auditory means.

[1368] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1369] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[1370] System Configuration

[1371] The wearable device of the present invention is composed of the following main components:

[1372] 1. Camera

[1373] Capture images of objects that come into view.

[1374] 2. Local AI module

[1375] Analyze the captured image and identify the object.

[1376] 3. Communication Module

[1377] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[1378] 4. Display

[1379] Information is displayed on one lens.

[1380] 5. Text-to-speech module

[1381] The acquired information is provided to the user by voice.

[1382] 6. Emotion Engine

[1383] Recognizes emotions by analyzing the user's tone of voice and facial expressions.

[1384] 7. Power Supply Unit

[1385] It supplies the necessary power to the entire wearable device.

[1386] Initial Setup

[1387] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1388] Image capture and analysis

[1389] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[1390] Identifying and transmitting information

[1391] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[1392] Obtaining and Displaying Information

[1393] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the visual information to the user as voice.

[1394] Emotion Engine Operation

[1395] The emotion engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. The recognized emotional information is reflected in the content displayed and the tone and content of the voice reading.

[1396] Specific examples

[1397] Obtaining information on historical buildings

[1398] The user stands in front of a historical building, and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. A single lens displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, it adjusts the tone and content of the reading to make it more detailed and interesting.

[1399] Foreign language sign translation

[1400] When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communications module, which then translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, it displays or reads out supplementary information to make the translation explanation more detailed and understandable.

[1401] As described above, this invention provides a user-friendly means of information acquisition by acquiring, displaying, and reading out information in real time without significantly obstructing the field of vision and without using the user's hands. Furthermore, by incorporating an emotion engine, it is possible to provide information customized to the user's emotions.

[1402] The processing flow will be explained below.

[1403] Step 1:

[1404] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[1405] Step 2:

[1406] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[1407] Step 3:

[1408] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[1409] Step 4:

[1410] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[1411] Step 5:

[1412] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[1413] Step 6:

[1414] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[1415] Step 7:

[1416] The emotion engine analyzes the user's tone of voice and facial expressions in real time to generate emotion data, which is reflected in the overall system behavior.

[1417] Step 8:

[1418] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[1419] Step 9:

[1420] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[1421] Step 10:

[1422] The server references the emotional data and customizes the information it retrieves, for example, selecting detailed information if the user is excited, or a clear explanation if the user is confused.

[1423] Step 11:

[1424] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the building, its designer, and the year it was built.

[1425] Step 12:

[1426] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as voice in a regulated tone.

[1427] Step 13:

[1428] The user gathers information by reading the information displayed on one lens or listening to the audio guidance. The emotion engine continuously monitors the user's reactions and adjusts the displayed information or audio guidance as needed.

[1429] Specific examples

[1430] Obtaining information on historical buildings

[1431] Step 1:

[1432] The user stands in front of the historical building and turns it on.

[1433] Step 2:

[1434] The device begins its initialization sequence, powering up the camera and sensors.

[1435] Step 3:

[1436] The device connects to the server and goes through an authentication process to establish a secure connection.

[1437] Step 4:

[1438] The device captures images of the building with its camera and sends them to a local AI module.

[1439] Step 5:

[1440] The terminal sends the identification result to an external server, which retrieves the information from a database.

[1441] Step 6:

[1442] The emotion engine detects the user's excitement and the server selects detailed and interesting information.

[1443] Step 7:

[1444] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[1445] Step 8:

[1446] The device reads out information about the building aloud, and the user confirms the information.

[1447] Translation of foreign language signs

[1448] Step 1:

[1449] A user spots a sign in a foreign city and turns on the device.

[1450] Step 2:

[1451] The device will initialize and start up the camera and sensors.

[1452] Step 3:

[1453] The device connects to the server and goes through an authentication process to establish a secure connection.

[1454] Step 4:

[1455] The device captures an image of the sign and performs character recognition (OCR).

[1456] Step 5:

[1457] The terminal transmits character data to the server, and the server translates the received character data.

[1458] Step 6:

[1459] The emotion engine detects the user's confusion, and the server selects an easy-to-understand translation explanation.

[1460] Step 7:

[1461] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[1462] Step 8:

[1463] The device reads out the translation results aloud and the user confirms them.

[1464] The above is a detailed description of the specific processing steps for carrying out the present invention.

[1465] Example 2

[1466] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1467] Wearable devices already exist that can identify objects in their field of view in real time and instantly provide relevant information to users. However, this information is rarely appropriately customized to reflect the user's situation or emotions. In particular, the information provided is uniform and cannot be tailored to the user's interests or emotions, resulting in incomplete information utilization and a poor user experience. Therefore, there is a need for wearable devices that can customize information based on the user's emotions and provide it more effectively.

[1468] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1469] In this invention, the server is a wearable device worn by a user, and includes: means for capturing images of objects within the user's field of view using a camera; means for identifying objects in the captured images using a local AI module; means for transmitting image analysis information to an external server; means for displaying information acquired from the external server on one of the lenses; means for reading out the acquired information aloud; and means including an emotion engine for recognizing the user's emotions and adjusting the display content and the tone of the reading out loud based on the emotions. This enables the wearable device worn by the user to appropriately customize the content of the information provided and the tone of the reading out loud according to the user's emotions.

[1470] A "user" is an individual who wears a wearable device and uses its services, or an entity that represents that individual.

[1471] A "wearable device" is an information processing device that is worn by the user on the body and is equipped with a camera, display, local AI module, communication module, voice reading module, emotion engine, etc.

[1472] A "camera" is a photographic device for capturing images of objects within the user's field of view.

[1473] The "local AI module" is an artificial intelligence processing device that identifies objects in captured images and extracts their features.

[1474] "Object" refers to a specific object or textual information in an image captured by a camera.

[1475] A "communication module" is a communication device for transmitting and receiving data between a wearable device and an external server.

[1476] An "external server" is an information processing device that receives information sent from a wearable device, searches a database based on that information, and obtains and returns related information.

[1477] A "database" is a collection of information stored in an external server, and is a data structure for searching and retrieving information based on specific conditions.

[1478] A "display" is a display device that visually displays captured information and information obtained from a server to a user.

[1479] The "voice reading module" is a device that converts acquired information into voice and provides it to the user.

[1480] The "emotion engine" is an artificial intelligence processing device that analyzes the user's tone of voice and facial expressions to recognize the user's emotions.

[1481] "Authentication information" is personal identification information required when a user accesses a wearable device, and is information for authenticating the user.

[1482] An "initialization sequence" is a series of processes that a wearable device goes through when it starts up, launching each component and making the system ready for use.

[1483] "Field of view" refers to the area that a user can see directly with the naked eye and that is captured by the camera of a wearable device.

[1484] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[1485] System Configuration

[1486] The wearable device of the present invention is composed of the following main components:

[1487] 1. Camera

[1488] The camera captures images of objects within the user's field of view and is high-resolution and capable of capturing footage in real time.

[1489] 2. Local AI module

[1490] The local AI module uses artificial intelligence algorithms based on TensorFlow and PyTorch to analyze images captured by the camera, thereby identifying objects in the image and extracting their features.

[1491] 3. Communication Module

[1492] The communication module is a device for transmitting and receiving data between the wearable device and an external server. It uses Wi-Fi or mobile data communication to send the identified information to the external server and receive the information returned from the server.

[1493] 4. Display

[1494] The display is a device for displaying information on the lens of one eye, and by using a transparent AR (augmented reality) display, information can be viewed without obstructing the field of vision.

[1495] 5. Text-to-speech module

[1496] The voice reading module is a device that provides acquired information to the user by voice, and uses APIs such as Google Text-to-Speech and Amazon Polly.

[1497] 6. Emotion Engine

[1498] The emotion engine uses the Microsoft Emotion API and Affectiva to analyze the user's tone of voice and facial expressions, recognizing the user's emotions based on the analysis results and adjusting the information and tone provided in real time.

[1499] 7. Power Supply Unit

[1500] The power supply unit is a device that supplies the necessary power to the entire wearable device. It is battery-powered and can be used for a long time.

[1501] Initial Setup

[1502] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates them. If authentication is successful, the system is ready for use.

[1503] Image capture and analysis

[1504] The camera captures objects in the user's field of view at regular intervals and sends the images to a local AI module, which analyzes the captured images and identifies the objects. TensorFlow and PyTorch are used to extract object features and generate classification results.

[1505] Identifying and transmitting information

[1506] Information about identified objects is sent to an external server via a communication module. The server searches a database based on the received information to retrieve related information. The server uses common server software such as Apache or Nginx.

[1507] Obtaining and Displaying Information

[1508] The information obtained from the server is sent to the device via the communication module and displayed on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. This voice reading process uses Google Text-to-Speech and Amazon Polly.

[1509] Emotion Engine Operation

[1510] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading. This allows the information provided to be appropriately customized to match the user's emotions.

[1511] Examples and prompts

[1512] 1. Specific examples of acquiring information on historical buildings

[1513] Example of operation: A user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image to identify the building, and the identification information is sent to an external server. The server searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, year of construction, etc., and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[1514] Prompt statement:

[1515] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[1516] 2. Examples of translating foreign language signs

[1517] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[1518] Prompt statement:

[1519] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[1520] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1521] Specific processing steps of the system

[1522] Step 1:

[1523] Initial Setup

[1524] Input: Power on, user credentials

[1525] Specific operation: When a user powers on a wearable device, the device begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection to an external server. The device then sends the user's authentication information to the server, which then verifies the authentication information and performs authentication.

[1526] Data processing and calculation: The terminal sends authentication information, and the server checks the authentication information in the database.

[1527] Output: Authentication success or failure, system ready for use

[1528] Step 2:

[1529] Image capture and analysis

[1530] Input: Objects in the user's field of view

[1531] Specific operation: The device camera captures images of objects within the user's field of view at regular intervals, and the captured images are sent to the local AI module.

[1532] Data processing and calculation: The local AI module uses TensorFlow or PyTorch to analyze images, identify objects, and extract their features.

[1533] Output: Object identity

[1534] Step 3:

[1535] Identifying and transmitting information

[1536] Input: Object identity

[1537] Specific operation: The information of the identified object is sent to an external server through a communication module.

[1538] Data processing and calculation: Information is assembled into data packets and prepared for transmission to the server.

[1539] Output: Identification information sent to the server

[1540] Step 4:

[1541] Obtaining and Displaying Information

[1542] Input: The identity received by the server

[1543] Specific operation: The server searches the database based on the received identification information, retrieves relevant information, and returns that information to the terminal via the communication module.

[1544] Data processing and calculation: The server performs database searches and collects relevant information.

[1545] Output: Retrieved information (details of historical buildings, etc.)

[1546] Step 5:

[1547] Display on the screen and read aloud

[1548] Input: Retrieved information

[1549] Specific operation: The device displays the information received from the server on the one-eye display. At the same time, the voice reading module is activated to provide the user with the visual information by voice.

[1550] Data processing and calculation: Generates display data for the display and prepares for voice synthesis.

[1551] Output: Display of visual information, provision of audio information

[1552] Step 6:

[1553] Emotion Engine Operation

[1554] Input: User's tone of voice and facial expressions

[1555] How it works: The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize their emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading.

[1556] Data processing and calculation: Algorithms analyze voice tone and facial expressions to identify emotional states.

[1557] Output: Customize information based on emotion (adjust tone, add details, etc.)

[1558] Examples and prompts

[1559] 1. Specific examples of acquiring information on historical buildings

[1560] Example of operation: A user stands in front of a historical building, the device captures an image of the building, and the local AI module identifies the building. The identification information is sent to an external server, which searches a database to obtain detailed information. The information is displayed in one of the lenses and read aloud using a text-to-speech function. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[1561] Prompt statement:

[1562] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add some particularly interesting facts, as the user seems excited about it."

[1563] 2. Examples of translating foreign language signs

[1564] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[1565] Prompt statement:

[1566] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[1567] (Application example 2)

[1568] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1569] While conventional wearable devices were capable of identifying objects in the field of view and obtaining information, they lacked the ability to adjust the information display and voice readout content according to the user's emotions, resulting in the problem of providing uniform information. Furthermore, when used in a physical store, there was a need for devices to obtain detailed information about products the user sees in real time and provide information that responds to the user's emotions.

[1570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1571] In this invention, the server includes means for acquiring and displaying prices, reviews, and campaign information for products that the user has seen, means for reading out the acquired information aloud, and means for adjusting the displayed content and the read-out content in accordance with the user's emotions, thereby enabling the user to instantly acquire detailed information about products that the user has seen in a store and providing information flexibly in accordance with the user's emotions.

[1572] A "camera" is a device for capturing images of objects within its field of view.

[1573] A "local AI module" is a module that implements artificial intelligence to identify objects in captured images.

[1574] An "external server" is a server on a network that receives image analysis information and provides related information.

[1575] The "single lens" is a display lens attached to a wearable device, and is a means for displaying acquired information.

[1576] The "voice reading module" is a module for providing acquired information to the user by voice.

[1577] The "emotion engine" is a system that analyzes the user's tone of voice and facial expressions to recognize emotions.

[1578] A "physical store" is a retail store that sells merchandise and that customers can physically visit.

[1579] "Price" is information indicating the price of the product sold at the store.

[1580] A "review" is information that indicates the evaluation or impression that a customer has given about a product.

[1581] "Campaign information" is sales promotion information such as discounts and special offers offered at stores.

[1582] "User's emotion" refers to the emotional state of the user as analyzed from their tone of voice and facial expressions.

[1583] "Information display" refers to displaying acquired information in the user's field of vision.

[1584] "Flexible information provision" refers to appropriately adjusting the display and voice reading of information according to the user's emotions and situation.

[1585] This invention is a system that uses a wearable device to acquire information about objects within the user's field of view in real time and provides information according to the user's emotions. It is particularly specialized for acquiring and displaying product information in physical stores. The configuration and operation of this system are described in detail below.

[1586] System configuration

[1587] 1. Hardware:

[1588] Smart glasses: Includes a camera, display, voice reading module, and communication module.

[1589] Camera: Captures images of objects in view.

[1590] Display: Displays information on one lens.

[1591] Voice reading module: Reads the acquired information aloud.

[1592] Communication module: Sends and receives data to and from external servers.

[1593] 2. Software:

[1594] Local AI module: Identifies objects in images using TensorFlow, OpenCV, etc.

[1595] Emotion recognition engine: Recognizes user emotions using Amazon Rekognition, IBM Watson, etc.

[1596] Database: Manage product information using MySQL or MongoDB.

[1597] How it works

[1598] 1. Image capture:

[1599] The camera in the smart glasses captures the products that come into view, and this image data is sent to a local AI module.

[1600] 2. Image Analysis:

[1601] The local AI module analyzes the image data and identifies the product, and the identification information is sent to an external server via the communication module.

[1602] 3. Information acquisition:

[1603] The external server searches a database based on the identification information to obtain relevant product information (price, reviews, campaign information, etc.).

[1604] 4. Information display and reading:

[1605] The acquired information is displayed on the smart glasses' display, and a voice reading module provides it to the user aloud.

[1606] 5. Emotion recognition:

[1607] The emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotions, and adjusts the content displayed and read aloud based on the recognized emotion.

[1608] Specific examples

[1609] 1. Get detailed product information:

[1610] For example, if a user wearing smart glasses walks through a brick-and-mortar store and stops in front of a shelf of sweets, the camera captures the specific sweet. The local AI module identifies the sweet and sends the identification information to an external server. The server then searches a product database to obtain the price, reviews, and current promotion information, which it then sends to the smart glasses. This information is displayed on the display, and a voice announces, "This sweet has received top reviews. It's currently on a 50% off promotion." If the user looks confused, the emotion recognition engine detects this and provides additional information, such as, "This sweet is milk chocolate. It's low in sugar and recommended for health-conscious people."

[1611] 2. Example prompt:

[1612] "Identify specific objects in images captured by the camera and search for information related to those objects. Based on the identification results, provide detailed information about the relevant products, and recognize emotions from the user's facial expressions and voice to adjust the information displayed and the voice read out."

[1613] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1614] Step 1:

[1615] Image Capture

[1616] Subject: Device

[1617] Input: Objects in the user's field of view

[1618] Output: Image data of the object

[1619] Description: A user wears smart glasses and a camera captures images of objects in their field of view. The camera captures image data of products and objects in the user's field of view.

[1620] Step 2:

[1621] Object Identification

[1622] Subject: Device

[1623] Input: Image data

[1624] Output: Identification information

[1625] Description: The captured image data is sent to a local AI module, which uses TensorFlow and OpenCV to identify objects in the image and extract identifying information.

[1626] Step 3:

[1627] Sending image analysis information

[1628] Subject: Device

[1629] Input: Identification information

[1630] Output: Data sent to external server

[1631] Description: Identified object information is sent to an external server via the communication module. Identification information includes product name and category.

[1632] Step 4:

[1633] Information acquisition

[1634] Subject: Server

[1635] Input: Identification information

[1636] Output: Product details

[1637] Description: The external server searches the database based on the received identification information and retrieves relevant product information (price, reviews, campaign information, etc.).

[1638] Step 5:

[1639] Information transmission

[1640] Subject: Server

[1641] Input: Product details

[1642] Output: Data sent to the terminal

[1643] Description: The acquired product details are sent to the smart glasses through the communication module. The transmitted data includes the product details.

[1644] Step 6:

[1645] Information display and reading

[1646] Subject: Device

[1647] Input: Product details

[1648] Output: Display and voice reading

[1649] Description: The retrieved product information is displayed on the device's display, and the voice reading module provides the information aloud. The display shows the price, reviews, and campaign information, and the voice reads out, "This candy has received excellent reviews. It's currently on sale for 50% off."

[1650] Step 7:

[1651] emotion recognition

[1652] Subject: Device

[1653] Input: User's voice tone and facial expression data

[1654] Output: Emotional information

[1655] Description: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. Emotions can be recognized as interest or confusion, for example.

[1656] Step 8:

[1657] Information adjustment

[1658] Subject: Device

[1659] Input: Emotion information

[1660] Output: Adjusted display and speech content

[1661] Description: The system adjusts the content displayed and spoken based on the recognized emotion information. If the user is confused, additional information will be displayed on the display and spoken aloud: "This candy is milk chocolate. It is low in sugar and recommended for health-conscious people."

[1662] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1663] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1664] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1665] [Fourth embodiment]

[1666] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1667] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1668] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1669] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1670] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1671] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1672] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1673] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1674] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1675] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1676] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1677] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1678] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1679] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail the embodiments of the present invention.

[1680] System Configuration

[1681] The wearable device of the present invention is composed of the following main components:

[1682] 1. Camera

[1683] Capture images of objects that come into view.

[1684] 2. Local AI module

[1685] Analyze the captured image and identify the object.

[1686] 3. Communication Module

[1687] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[1688] 4. Display

[1689] Information is displayed on one lens.

[1690] 5. Text-to-speech module

[1691] The acquired information is presented to the user by voice.

[1692] 6. Power Supply Unit

[1693] It supplies the necessary power to the entire wearable device.

[1694] Initial Setup

[1695] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1696] Image capture and analysis

[1697] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[1698] Identifying and transmitting information

[1699] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[1700] Obtaining and Displaying Information

[1701] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the user with the visual information as voice.

[1702] Specific examples

[1703] Obtaining information on historical buildings

[1704] The user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud.

[1705] Foreign language sign translation

[1706] When a user sees a sign while traveling abroad, the camera captures an image of the sign, and the local AI module performs character recognition. The character data is sent to an external server via the communication module, and the server translates the received character data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user.

[1707] As described above, the present invention provides a user-friendly means of acquiring information by acquiring, displaying, and reading out information in real time without using hands and without significantly obstructing the field of vision.

[1708] The processing flow will be explained below.

[1709] Step 1:

[1710] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[1711] Step 2:

[1712] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[1713] Step 3:

[1714] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[1715] Step 4:

[1716] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[1717] Step 5:

[1718] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[1719] Step 6:

[1720] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[1721] Step 7:

[1722] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[1723] Step 8:

[1724] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[1725] Step 9:

[1726] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the Eiffel Tower, who designed it, and the year it was built.

[1727] Step 10:

[1728] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as audio.

[1729] Step 11:

[1730] The user reads the information displayed on one lens or listens to the audio guidance to gather information.

[1731] Specific examples

[1732] Obtaining information on historical buildings

[1733] Step 1:

[1734] The user stands in front of the historical building and turns it on.

[1735] Step 2:

[1736] The device begins its initialization sequence, powering up the camera and sensors.

[1737] Step 3:

[1738] The device connects to the server and goes through an authentication process to establish a secure connection.

[1739] Step 4:

[1740] The device captures images of the building with its camera and sends them to the AI ​​module.

[1741] Step 5:

[1742] The terminal sends the identification result to an external server, which retrieves the information from a database.

[1743] Step 6:

[1744] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[1745] Step 7:

[1746] The device reads out information about the building aloud, and the user confirms the information.

[1747] Translation of foreign language signs

[1748] Step 1:

[1749] A user spots a sign in a foreign city and turns on the device.

[1750] Step 2:

[1751] The device will initialize and start up the camera and sensors.

[1752] Step 3:

[1753] The device connects to the server and goes through an authentication process to establish a secure connection.

[1754] Step 4:

[1755] The device captures an image of the sign and performs character recognition (OCR).

[1756] Step 5:

[1757] The terminal transmits character data to the server, and the server translates the received character data.

[1758] Step 6:

[1759] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[1760] Step 7:

[1761] The device reads out the translation results aloud and the user confirms them.

[1762] The above is a detailed description of the specific processing steps for carrying out the present invention.

[1763] Example 1

[1764] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1765] In modern society, instant access to information is crucial. However, in many situations, visual or auditory information acquisition is often difficult. Travelers and people with visual impairments in particular have limited means of efficiently acquiring and understanding information about their surroundings. Furthermore, there are few ways to acquire information without using their hands, which is inconvenient for users. Therefore, there is a demand for systems that can acquire, display, and read information in real time without using hands and without obstructing the view.

[1766] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1767] In this invention, the server includes means for receiving and analyzing object identification information, means for searching a database based on the analysis results, and means for transmitting the corresponding information to the wearable device, thereby enabling real-time identification of objects within the field of view and acquisition, display, and reading out of their associated information.

[1768] A "wearable device" is a small electronic device that can be worn by a user and has the function of capturing, analyzing, displaying, and reading out information.

[1769] A "camera" is a photographic device that captures images of objects within its field of view.

[1770] A "local AI module" is an artificial intelligence module installed inside the device that has the ability to analyze and identify objects in captured images.

[1771] An "external server" is a remote server connected via communication from the wearable device, which receives object identification information and searches for information using a database.

[1772] "Object identification information" is information about an object that has been analyzed and identified by a local AI module.

[1773] A "monocular display" is a display device placed on one lens of a wearable device, which visually presents acquired information to the user.

[1774] The "voice reading means" is a device that has the function of converting acquired information into voice and providing it to the user in voice.

[1775] "Character recognition" is the process of analyzing character data in an image and converting it into character data.

[1776] The "translation means" has the function of converting received text data into a specified language.

[1777] The present invention is a system that uses a wearable device worn by a user to identify objects within the user's field of view in real time, and acquires, displays, and reads out related information. The following describes in detail an embodiment of the present invention.

[1778] System Configuration

[1779] The wearable device of the present invention is composed of the following main components:

[1780] 1. Camera

[1781] The camera captures images of objects within its field of view. The camera has high resolution and a fast shutter speed.

[1782] 2. Local AI module

[1783] The local AI module analyzes captured images and identifies objects. This module uses TensorFlow, a machine learning framework, to run image recognition models.

[1784] 3. Communication Module

[1785] The communication module transmits information about the identified objects to an external server and retrieves related information from the external server, using Wi-Fi or cellular networks.

[1786] 4. Display

[1787] The display displays information on the lens of one eye, and by using a transparent OLED display, it is possible to display information without obstructing the field of view.

[1788] 5. Text-to-speech module

[1789] The text-to-speech module presents the retrieved information to the user audibly, utilizing the latest speech synthesis technology to generate natural, easy-to-listen-to voices.

[1790] 6. Power Supply Unit

[1791] The power supply unit provides the necessary power for the entire wearable device and is equipped with a large-capacity battery that can be used for a long time.

[1792] Initial Setup

[1793] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1794] Image capture and analysis

[1795] The camera captures objects in its field of view at regular intervals, and the captured image data is sent to a local AI module on the device. The local AI module uses TensorFlow to analyze the image and identify the object.

[1796] Identifying and transmitting information

[1797] The device's local AI module sends information about the identified object to an external server via a communications module. The information sent here includes the characteristics and unique information of the identified object. The external server then searches a database based on the received information to retrieve related information.

[1798] Acquiring, displaying, and reading out information

[1799] The information acquired from the external server is sent to the device via the communication module. The device displays the information on the one-eye display. At the same time, the voice reading module is activated, providing the user with the visual information as voice.

[1800] Specific examples

[1801] Obtaining information on historical buildings

[1802] When a user stands in front of a historical building, the camera captures an image of the building. A local AI module analyzes the image and sends the building's identification information to a server. The external server then searches a database to obtain the building's history, designer, and construction year, which are then sent to the device. The device then displays this information on one of the lenses and reads it aloud using a voice reading function.

[1803] Example prompt:

[1804] "Please tell me the history of this building, who designed it, and when it was built."

[1805] Foreign language sign translation

[1806] When a user sees a sign while traveling abroad, the camera captures an image of the sign. The local AI module performs character recognition and sends the text data to an external server. The external server translates the received text data and sends the translation results to the device. The device then displays the translation result on one of the lenses and conveys it to the user using a voice readout function.

[1807] Example prompt:

[1808] "Please translate the contents of this sign."

[1809] As described above, the present invention provides a means for obtaining information in real time and providing it to the user visually and audibly.

[1810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1811] Step 1: Initial Setup

[1812] When the terminal turns on the wearable device, the device automatically starts an initialization sequence. This initialization sequence starts up the camera and sensors, and the communication module establishes a connection to an external server. The user's authentication information is sent from the terminal to the server, and the server performs authentication. The input during initial setup is the user's authentication information and the terminal's device information, and the output is connection confirmation and the authentication result.

[1813] Specific behavior:

[1814] The device will power on and the system will automatically begin the initialization sequence.

[1815] The camera and sensors are activated and the communication module is connected to the network.

[1816] The authentication information is sent to the server for authentication.

[1817] Step 2: Image capture and transmission

[1818] The device's camera captures objects within its field of view at regular intervals. The captured image data is sent to a local AI module within the device. The input is the captured image data, and the output is the analysis result by the local AI module.

[1819] Specific behavior:

[1820] The camera takes an image of an object that comes into view.

[1821] The captured image data is sent to the local AI module.

[1822] Step 3: Object Identification

[1823] The local AI module uses TensorFlow to analyze images and recognize objects. The analyzed data is sent to the communication module as object identification information. The input is the image data sent to the local AI module, and the output is object identification information.

[1824] Specific behavior:

[1825] The local AI module receives the image data and analyzes it using TensorFlow.

[1826] As a result of the analysis, identification information of the object is generated.

[1827] Step 4: Submit your information

[1828] The communication module of the terminal transmits the information of the identified object to an external server, the input being the object identification information, and the output being the transmission result to the external server.

[1829] Specific behavior:

[1830] The object identification information is transmitted to an external server through a communication module.

[1831] The server receives the information and prepares a database search.

[1832] Step 5: Searching and retrieving information

[1833] The server searches the database based on the received object information to obtain related information. The input is object identification information, and the output is related information.

[1834] Specific behavior:

[1835] The server searches the database and extracts the relevant information.

[1836] The acquired information is then ready to be sent to the terminal.

[1837] Step 6: Receiving and displaying information

[1838] The device receives the information sent from the server and displays it on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as voice. The input is the relevant information sent from the server, and the output is the display and voice output.

[1839] Specific behavior:

[1840] The terminal displays the acquired information on the display.

[1841] The text-to-speech module reads the information aloud as a synthesized voice.

[1842] (Application example 1)

[1843] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1844] Existing wearable devices are capable of identifying objects within their field of view and acquiring and displaying information, but they lack the ability to acquire and display real-time product information in physical stores, as well as voice guidance. This makes it difficult for users to quickly and efficiently acquire product information in physical stores. The present invention solves these issues by providing a system that acquires, displays, and provides voice guidance in real time to improve the shopping experience in physical stores.

[1845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1846] In this invention, the server is a wearable device worn by a user, and includes means for capturing images of objects in the user's field of view using a camera, means for identifying objects in the captured images using a local AI module, means for transmitting image analysis information to an external server, means for displaying information acquired from the external server on one lens, means for reading out the acquired information aloud, and means for displaying and reading out information about target products in real time. This allows a user to identify products in a physical store, obtain detailed information about them in real time, and efficiently obtain information through their eyes and ears.

[1847] A "wearable device worn by a user" is a device that is intended to be worn by a user and has the ability to identify objects in its field of view in real time and provide related information on the object in display and audio format.

[1848] A "camera" is a device that has the function of capturing an image of an object within its field of view, and constitutes part of a wearable device.

[1849] A "local AI module" is an artificial intelligence module that has the ability to analyze captured images and identify objects within the images.

[1850] An "external server" is a server whose role is to receive image analysis information sent from a wearable device, search for necessary information based on that information, and send the acquired information to the wearable device.

[1851] The "means for displaying on a lens of one eye" is a means having a function of displaying acquired information on a lens attached to one eye of the user.

[1852] The "means for reading out acquired information by voice" is a means having a function for converting information acquired from an external server into voice and providing the information to the user by voice.

[1853] "Means for displaying in real time and reading out aloud" refers to means that has the function of instantly displaying information about the identified product on a display and providing guidance by voice.

[1854] The present invention is a system that uses a wearable device worn by a user to identify products within the user's field of view in real time, acquire and display related information, and provide audio guidance. Specific embodiments are described in detail below.

[1855] System Configuration

[1856] The system mainly consists of the following components:

[1857] 1. Wearable devices

[1858] Camera: Captures images of objects in view.

[1859] Local AI module: Identifies objects in captured images.

[1860] Communication module: Sends image analysis information to an external server and receives information obtained from the server.

[1861] Display: Displays information on one lens.

[1862] Voice reading module: Provides the retrieved information to the user by voice.

[1863] Power supply unit: Provides the necessary power for the entire wearable device.

[1864] 2. External Server

[1865] Database: Stores detailed information about the retrieved objects.

[1866] AI analysis module: Searches the database based on image analysis information and generates related information.

[1867] Communication module: Sends and receives information to and from the wearable device.

[1868] Hardware and Software Details

[1869] Camera: Uses the OpenCV library for image capture.

[1870] Local AI module: Implements a machine learning model for object classification using TensorFlow and PyTorch.

[1871] Communication module: Communicates with the server using the HTTP protocol, and if possible, a RESTful API.

[1872] Display and voice reading: The acquired information is presented visually and audibly. Visual information is displayed on the display and voice reading is performed using gTTS (Google Text-to-Speech).

[1873] External server: Use cloud services such as AWS and Google Cloud to perform database and AI analysis.

[1874] System Operation

[1875] 1. Image capture and object identification

[1876] As a user walks around the store wearing the wearable device, the camera captures images of objects that come into view, which are then analyzed by a local AI module to identify the objects.

[1877] 2. Sending and receiving information

[1878] The information of the identified object is sent to an external server through a communication module, where an AI analysis module searches a database based on the received information to obtain related product information (price, discounts, reviews, etc.).

[1879] 3. Provision of Information

[1880] The information obtained from the external server is sent back to the wearable device via the communication module. The product information is displayed on one lens, and the voice reading module reads the information aloud.

[1881] Specific examples

[1882] When a user picks up an apple in a physical store, the camera recognizes it and the local AI module identifies it. The information is sent to an external server, which retrieves details such as the apple's price, discounts, and reviews and sends them to the wearable device. The details are displayed on the monocular display, and a voice readout function relays the information to the user.

[1883] Prompt Sentence Examples

[1884] You are an AI modeling engineer designing an application for a system that uses a wearable device worn by the user to identify products in a physical store in real time and retrieve, display, and read out related information. Please write a description of the system that, when the user holds a product in front of them, displays detailed information about the product on the display and reads out the information aloud, clearly stating the specific steps and technologies used.

[1885] ---

[1886] Technologies used: OpenCV, TensorFlow, PyTorch, Google Text-to-Speech (gTTS), playsound

[1887] ---

[1888] Please provide a detailed description of each step: input image capture, object identification, information acquisition, information display, and voice reading.

[1889] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1890] Step 1:

[1891] Image Capture

[1892] While the user is wearing the wearable device, the camera captures images of objects in its field of view at regular intervals. The input is the image in the field of view, and the output is the captured image file. In this step, the OpenCV library is used to acquire images from the camera. Specifically, the camera is started, the image is captured frame by frame, and a specific frame is saved as an image.

[1893] Step 2:

[1894] Object Identification

[1895] The captured image is sent to a local AI module, which uses TensorFlow or PyTorch to identify objects within the image. The input is the captured image file, and the output is information about the identified object. Specifically, the image is input to the AI ​​model, and the model analyzes the features within the image to identify the object. The identification results are output as text data.

[1896] Step 3:

[1897] Image analysis information transmission

[1898] The information about the identified object is sent to an external server via a communication module. The input is the information about the identified object, and the output is the result of sending it to the server (success or failure). Specifically, the information about the identified object is sent to the server as a POST request using the HTTP protocol. At this time, the information about the identified object is encoded in JSON format.

[1899] Step 4:

[1900] Information Search

[1901] The server searches the database based on the received image analysis information to obtain related product information. The input is the received image analysis information, and the output is related product information. Specifically, the AI ​​analysis module on the server side queries the database to obtain detailed information such as the price, discount, and reviews of the relevant product. This information is encoded in JSON format.

[1902] Step 5:

[1903] Information transmission

[1904] The server sends the acquired product information to the wearable device via the communication module. The input is the acquired product information, and the output is the transmission result (success or failure) to the device. Specifically, the product information is returned as an HTTP response, and the communication module receives the response.

[1905] Step 6:

[1906] Information display

[1907] The wearable device displays the received information on one lens of the eye. The input is the received product information, and the output is the visual information displayed on the display. Specifically, the product information is formatted and displayed on the one-eye display.

[1908] Step 7:

[1909] Text-to-speech

[1910] The acquired product information is provided to the user by voice. The input is the received product information, and the output is audio information. Specifically, the product information is converted into audio using gTTS (Google Text-to-Speech) and provided to the user through a voice reading module. At this time, the text data is converted into audio data, and the audio data is played back.

[1911] Through the above steps, a user can identify a product in a physical store, obtain detailed information about the product in real time, and efficiently obtain information through visual and auditory means.

[1912] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1913] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[1914] System Configuration

[1915] The wearable device of the present invention is composed of the following main components:

[1916] 1. Camera

[1917] Capture images of objects that come into view.

[1918] 2. Local AI module

[1919] Analyze the captured image and identify the object.

[1920] 3. Communication Module

[1921] The object identification information is transmitted to an external server, and the information obtained from the server is received.

[1922] 4. Display

[1923] Information is displayed on one lens.

[1924] 5. Text-to-speech module

[1925] The acquired information is presented to the user by voice.

[1926] 6. Emotion Engine

[1927] Recognizes emotions by analyzing the user's tone of voice and facial expressions.

[1928] 7. Power Supply Unit

[1929] It supplies the necessary power to the entire wearable device.

[1930] Initial Setup

[1931] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence: the camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's authentication information to the server, which then authenticates the user.

[1932] Image capture and analysis

[1933] The camera captures objects in its field of view at regular intervals and sends the images to a local AI module, which analyzes the objects in the captured images, extracts their features, and identifies them.

[1934] Identifying and transmitting information

[1935] The information of the identified object is transmitted to an external server through a communication module, and the server searches a database based on the received information to obtain related information.

[1936] Obtaining and Displaying Information

[1937] The information obtained from the server is sent to the device via the communication module and displayed on one lens, while the voice reading module is activated to provide the user with the visual information as voice.

[1938] Emotion Engine Operation

[1939] The emotion engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. The recognized emotional information is reflected in the content displayed and the tone and content of the voice reading.

[1940] Specific examples

[1941] Obtaining information on historical buildings

[1942] The user stands in front of a historical building, and the device captures an image of the building. A local AI module analyzes the image and sends the building's identification information to an external server. The server then searches a database to obtain detailed information about the building and sends it to the device. A single lens displays the building's history, designer, and year of construction, and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, it adjusts the tone and content of the reading to make it more detailed and interesting.

[1943] Foreign language sign translation

[1944] When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communications module, which then translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, it displays or reads out supplementary information to make the translation explanation more detailed and understandable.

[1945] As described above, this invention provides a user-friendly means of information acquisition by acquiring, displaying, and reading out information in real time without significantly obstructing the field of vision and without using the user's hands. Furthermore, by incorporating an emotion engine, it is possible to provide information customized to the user's emotions.

[1946] The processing flow will be explained below.

[1947] Step 1:

[1948] The user puts on the wearable device and turns it on. When the power button on the device is pressed, the system begins an initialization sequence.

[1949] Step 2:

[1950] During the device initialization sequence, the camera and sensors are activated. Internal software checks the status of the camera and sensors to ensure they are working properly.

[1951] Step 3:

[1952] The device attempts to connect to an external server via the communication module and sends authentication information to the server using the user's ID and password.

[1953] Step 4:

[1954] The server verifies the received authentication information and, if authentication is successful, a secure connection is established.

[1955] Step 5:

[1956] The device uses the camera at regular intervals to capture images of the field of view, for example, every second, and stores the data in its internal memory.

[1957] Step 6:

[1958] The device sends the captured image to a local AI module for analysis, which then recognizes objects in the image and extracts specific features (such as the shape of a building or the text on a sign).

[1959] Step 7:

[1960] The emotion engine analyzes the user's tone of voice and facial expressions in real time to generate emotion data, which is reflected in the overall system behavior.

[1961] Step 8:

[1962] The device sends the information identified by the AI ​​module to an external server. For example, it sends data including "images of buildings" and "recognized names" to the server.

[1963] Step 9:

[1964] The server searches the database based on the received information to obtain the appropriate information, and retrieves the corresponding information (e.g., detailed information about historical buildings) from the database.

[1965] Step 10:

[1966] The server references the emotional data and customizes the information it retrieves, for example, selecting detailed information if the user is excited, or a clear explanation if the user is confused.

[1967] Step 11:

[1968] The server sends the acquired information to the terminal as a data packet, which may include information such as the history of the building, its designer, and the year it was built.

[1969] Step 12:

[1970] The device displays the received information on one lens, and at the same time, the voice reading module provides this information to the user as voice in a regulated tone.

[1971] Step 13:

[1972] The user gathers information by reading the information displayed on one lens or listening to the audio guidance. The emotion engine continuously monitors the user's reactions and adjusts the displayed information or audio guidance as needed.

[1973] Specific examples

[1974] Obtaining information on historical buildings

[1975] Step 1:

[1976] The user stands in front of the historical building and turns it on.

[1977] Step 2:

[1978] The device begins its initialization sequence, powering up the camera and sensors.

[1979] Step 3:

[1980] The device connects to the server and goes through an authentication process to establish a secure connection.

[1981] Step 4:

[1982] The device captures images of the building with its camera and sends them to a local AI module.

[1983] Step 5:

[1984] The terminal sends the identification result to an external server, which retrieves the information from a database.

[1985] Step 6:

[1986] The emotion engine detects the user's excitement and the server selects detailed and interesting information.

[1987] Step 7:

[1988] The server transmits information data to the terminal, and the terminal displays the information on the lens of one eye.

[1989] Step 8:

[1990] The device reads out information about the building aloud, and the user confirms the information.

[1991] Translation of foreign language signs

[1992] Step 1:

[1993] A user spots a sign in a foreign city and turns on the device.

[1994] Step 2:

[1995] The device will initialize and start up the camera and sensors.

[1996] Step 3:

[1997] The device connects to the server and goes through an authentication process to establish a secure connection.

[1998] Step 4:

[1999] The device captures an image of the sign and performs character recognition (OCR).

[2000] Step 5:

[2001] The terminal transmits character data to the server, and the server translates the received character data.

[2002] Step 6:

[2003] The emotion engine detects the user's confusion, and the server selects an easy-to-understand translation explanation.

[2004] Step 7:

[2005] The server sends the translation result to the terminal, which then displays the translation result on the lens of one eye.

[2006] Step 8:

[2007] The device reads out the translation results aloud and the user confirms them.

[2008] The above is a detailed description of the specific processing steps for carrying out the present invention.

[2009] Example 2

[2010] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2011] Wearable devices already exist that can identify objects in their field of view in real time and instantly provide relevant information to users. However, this information is rarely appropriately customized to reflect the user's situation or emotions. In particular, the information provided is uniform and cannot be tailored to the user's interests or emotions, resulting in incomplete information utilization and a poor user experience. Therefore, there is a need for wearable devices that can customize information based on the user's emotions and provide it more effectively.

[2012] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2013] In this invention, the server is a wearable device worn by a user, and includes: means for capturing images of objects within the user's field of view using a camera; means for identifying objects in the captured images using a local AI module; means for transmitting image analysis information to an external server; means for displaying information acquired from the external server on one of the lenses; means for reading out the acquired information aloud; and means including an emotion engine for recognizing the user's emotions and adjusting the display content and the tone of the reading out loud based on the emotions. This enables the wearable device worn by the user to appropriately customize the content of the information provided and the tone of the reading out loud according to the user's emotions.

[2014] A "user" is an individual who wears a wearable device and uses its services, or an entity that represents that individual.

[2015] A "wearable device" is an information processing device that is worn by the user on the body and is equipped with a camera, display, local AI module, communication module, voice reading module, emotion engine, etc.

[2016] A "camera" is a photographic device for capturing images of objects within the user's field of view.

[2017] The "local AI module" is an artificial intelligence processing device that identifies objects in captured images and extracts their features.

[2018] "Object" refers to a specific object or textual information in an image captured by a camera.

[2019] A "communication module" is a communication device for transmitting and receiving data between a wearable device and an external server.

[2020] An "external server" is an information processing device that receives information sent from a wearable device, searches a database based on that information, and obtains and returns related information.

[2021] A "database" is a collection of information stored in an external server, and is a data structure for searching and retrieving information based on specific conditions.

[2022] A "display" is a display device that visually displays captured information and information obtained from a server to a user.

[2023] The "voice reading module" is a device that converts acquired information into voice and provides it to the user.

[2024] The "emotion engine" is an artificial intelligence processing device that analyzes the user's tone of voice and facial expressions to recognize the user's emotions.

[2025] "Authentication information" is personal identification information required when a user accesses a wearable device, and is information for authenticating the user.

[2026] An "initialization sequence" is a series of processes that a wearable device goes through when it starts up, launching each component and making the system ready for use.

[2027] "Field of view" refers to the area that a user can see directly with the naked eye and that is captured by the camera of a wearable device.

[2028] The present invention is a system that uses a wearable device worn by the user to identify objects within the user's field of view in real time and acquire, display, and read out related information. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system can adjust the displayed information and the content of the voice readout to match the user's emotions. The following describes in detail the embodiments of the present invention.

[2029] System Configuration

[2030] The wearable device of the present invention is composed of the following main components:

[2031] 1. Camera

[2032] The camera captures images of objects within the user's field of view and is high-resolution and capable of capturing video in real time.

[2033] 2. Local AI module

[2034] The local AI module uses artificial intelligence algorithms based on TensorFlow and PyTorch to analyze images captured by the camera, thereby identifying objects in the image and extracting their features.

[2035] 3. Communication Module

[2036] The communication module is a device for transmitting and receiving data between the wearable device and an external server. It uses Wi-Fi or mobile data communication to send the identified information to the external server and receive the information returned from the server.

[2037] 4. Display

[2038] The display is a device for displaying information on the lens of one eye, and by using a transparent AR (augmented reality) display, information can be viewed without obstructing the field of vision.

[2039] 5. Text-to-speech module

[2040] The voice reading module is a device that provides acquired information to the user by voice, and uses APIs such as Google Text-to-Speech and Amazon Polly.

[2041] 6. Emotion Engine

[2042] The emotion engine uses the Microsoft Emotion API and Affectiva to analyze the user's tone of voice and facial expressions, recognizing the user's emotions based on the analysis results and adjusting the information and tone provided in real time.

[2043] 7. Power Supply Unit

[2044] The power supply unit is a device that supplies the necessary power to the entire wearable device. It is battery-powered and can be used for a long time.

[2045] Initial Setup

[2046] When a user puts on a wearable device and turns it on, the device automatically begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection with an external server. The device sends the user's credentials to the server, which then authenticates them. If authentication is successful, the system is ready for use.

[2047] Image capture and analysis

[2048] The camera captures objects in the user's field of view at regular intervals and sends the images to a local AI module, which analyzes the captured images and identifies the objects. TensorFlow and PyTorch are used to extract object features and generate classification results.

[2049] Identifying and transmitting information

[2050] Information about identified objects is sent to an external server via a communication module. The server then searches a database based on the received information to retrieve related information. The server uses common server software such as Apache or Nginx.

[2051] Obtaining and Displaying Information

[2052] The information obtained from the server is sent to the device via the communication module and displayed on one of the lenses. At the same time, the voice reading module is activated and provides the visual information to the user as audio. This voice reading process uses Google Text-to-Speech and Amazon Polly.

[2053] Emotion Engine Operation

[2054] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading. This allows the information provided to be appropriately customized to match the user's emotions.

[2055] Examples and prompts

[2056] 1. Specific examples of acquiring information on historical buildings

[2057] Example of operation: A user stands in front of a historical building and the device captures an image of the building. A local AI module analyzes the image to identify the building, and the identification information is sent to an external server. The server searches a database to obtain detailed information about the building and sends it to the device. The lens in one eye displays the building's history, designer, year of construction, etc., and a voice reading function reads this information aloud. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[2058] Prompt statement:

[2059] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add any particularly interesting facts, as the user seems excited about it."

[2060] 2. Examples of translating foreign language signs

[2061] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[2062] Prompt statement:

[2063] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[2064] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2065] Specific processing steps of the system

[2066] Step 1:

[2067] Initial Setup

[2068] Input: Power on, user credentials

[2069] Specific operation: When a user powers on a wearable device, the device begins an initialization sequence. The camera and sensors start up, and the communication module establishes a connection to an external server. The device then sends the user's authentication information to the server, which then verifies the authentication information and performs authentication.

[2070] Data processing and calculation: The terminal sends authentication information, and the server checks the authentication information in the database.

[2071] Output: Authentication success or failure, system ready for use

[2072] Step 2:

[2073] Image capture and analysis

[2074] Input: Objects in the user's field of view

[2075] Specific operation: The device camera captures images of objects within the user's field of view at regular intervals, and the captured images are sent to the local AI module.

[2076] Data processing and calculation: The local AI module uses TensorFlow or PyTorch to analyze images, identify objects, and extract their features.

[2077] Output: Object identity

[2078] Step 3:

[2079] Identifying and transmitting information

[2080] Input: Object identity

[2081] Specific operation: The information of the identified object is sent to an external server through a communication module.

[2082] Data processing and calculation: Information is assembled into data packets and prepared for transmission to the server.

[2083] Output: Identification information sent to the server

[2084] Step 4:

[2085] Obtaining and Displaying Information

[2086] Input: The identification information received by the server

[2087] Specific operation: The server searches the database based on the received identification information, retrieves relevant information, and returns that information to the terminal via the communication module.

[2088] Data processing and calculation: The server performs database searches and collects relevant information.

[2089] Output: Retrieved information (details of historical buildings, etc.)

[2090] Step 5:

[2091] Display on the screen and read aloud

[2092] Input: Retrieved information

[2093] Specific operation: The device displays the information received from the server on one of the displays. At the same time, the voice reading module is activated to provide the user with the visual information by voice.

[2094] Data processing and calculation: Generates display data for the display and prepares for voice synthesis.

[2095] Output: Display of visual information, provision of audio information

[2096] Step 6:

[2097] Emotion Engine Operation

[2098] Input: User's tone of voice and facial expressions

[2099] How it works: The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize their emotions. The recognition results are reflected in the content of the information provided and the tone of the voice reading.

[2100] Data processing and calculation: Algorithms analyze voice tone and facial expressions to identify emotional states.

[2101] Output: Customize information based on emotion (adjust tone, add details, etc.)

[2102] Examples and prompts

[2103] 1. Specific examples of acquiring information on historical buildings

[2104] Example of operation: A user stands in front of a historical building, the device captures an image of the building, and the local AI module identifies the building. The identification information is sent to an external server, which searches a database to obtain detailed information. The obtained information is displayed in one lens and read aloud using a text-to-speech function. If the emotion engine detects the user's excitement, the tone of the reading is adjusted to be more detailed and interesting.

[2105] Prompt statement:

[2106] "The user has taken a picture of the structure in front of them. Please provide more information about the structure, its history, who designed it, and when it was built. Please add any particularly interesting facts, as the user seems excited about it."

[2107] 2. Examples of translating foreign language signs

[2108] Example of operation: When a user sees a sign while traveling abroad, the camera captures an image of the sign and the local AI module performs character recognition. The text data is sent to an external server via the communication module, and the server translates the received text data. The translation result is sent to the device and displayed on one of the lenses, while the voice reading module conveys the translation result to the user. If the emotion engine detects user confusion, supplementary information is displayed or read aloud to make the translation explanation more detailed and understandable.

[2109] Prompt statement:

[2110] "A user has taken an image of a sign in a foreign country. Please provide a translation of what is written on the sign. The user seems confused, so please provide a more detailed and clearer explanation."

[2111] (Application example 2)

[2112] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2113] While conventional wearable devices were capable of identifying objects in the field of view and obtaining information, they lacked the ability to adjust the information display and voice readout content according to the user's emotions, resulting in the problem of providing uniform information. Furthermore, when used in a physical store, there was a need for devices to obtain detailed information about products the user sees in real time and provide information that responds to the user's emotions.

[2114] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2115] In this invention, the server includes means for acquiring and displaying prices, reviews, and campaign information for products that the user has seen, means for reading out the acquired information aloud, and means for adjusting the displayed content and the read-out content in accordance with the user's emotions, thereby enabling the user to instantly acquire detailed information about products that the user has seen in a store and providing information flexibly in accordance with the user's emotions.

[2116] A "camera" is a device for capturing images of objects within its field of view.

[2117] A "local AI module" is a module that implements artificial intelligence to identify objects in captured images.

[2118] An "external server" is a server on a network that receives image analysis information and provides related information.

[2119] The "single lens" is a display lens attached to a wearable device, and is a means for displaying acquired information.

[2120] The "voice reading module" is a module for providing acquired information to the user by voice.

[2121] The "emotion engine" is a system that analyzes the user's tone of voice and facial expressions to recognize emotions.

[2122] A "physical store" is a retail store that sells merchandise and that customers can physically visit.

[2123] "Price" is information indicating the price of the product sold at the store.

[2124] A "review" is information that indicates the evaluation or impression that a customer has given about a product.

[2125] "Campaign information" is sales promotion information such as discounts and special offers offered at stores.

[2126] "User's emotion" refers to the emotional state of the user as analyzed from their tone of voice and facial expressions.

[2127] "Information display" refers to displaying acquired information in the user's field of vision.

[2128] "Flexible information provision" refers to appropriately adjusting the display and voice reading of information according to the user's emotions and situation.

[2129] This invention is a system that uses a wearable device to acquire information about objects within the user's field of view in real time and provides information according to the user's emotions. It is particularly specialized for acquiring and displaying product information in physical stores. The configuration and operation of this system are described in detail below.

[2130] System configuration

[2131] 1. Hardware:

[2132] Smart glasses: Includes a camera, display, voice reading module, and communication module.

[2133] Camera: Captures images of objects in view.

[2134] Display: Displays information on one lens.

[2135] Voice reading module: Reads the acquired information aloud.

[2136] Communication module: Sends and receives data to and from external servers.

[2137] 2. Software:

[2138] Local AI module: Identifies objects in images using TensorFlow, OpenCV, etc.

[2139] Emotion recognition engine: Recognizes user emotions using Amazon Rekognition, IBM Watson, etc.

[2140] Database: Manage product information using MySQL or MongoDB.

[2141] How it works

[2142] 1. Image capture:

[2143] The smart glasses' camera captures items in its field of view, and this image data is sent to a local AI module.

[2144] 2. Image Analysis:

[2145] The local AI module analyzes the image data and identifies the product, and the identification information is sent to an external server via the communication module.

[2146] 3. Information acquisition:

[2147] The external server searches a database based on the identification information to obtain relevant product information (price, reviews, campaign information, etc.).

[2148] 4. Information display and reading:

[2149] The acquired information is displayed on the smart glasses' display, and a voice reading module provides it to the user aloud.

[2150] 5. Emotion recognition:

[2151] The emotion recognition engine analyzes the user's tone of voice and facial expressions to identify their emotions, and adjusts the content displayed and read aloud based on the recognized emotion.

[2152] Specific examples

[2153] 1. Get detailed product information:

[2154] For example, if a user wearing smart glasses walks through a brick-and-mortar store and stops in front of a shelf of sweets, the camera captures the specific sweet. The local AI module identifies the sweet and sends the identification information to an external server. The server then searches a product database to obtain the price, reviews, and current promotion information, which it then sends to the smart glasses. This information is displayed on the display, and a voice announces, "This sweet has received top reviews. It's currently on a 50% off sale." If the user shows a confused expression, the emotion recognition engine detects this and provides additional information, such as, "This sweet is milk chocolate. It's low in sugar and recommended for health-conscious people."

[2155] 2. Example prompt:

[2156] "Identify specific objects in images captured by the camera and search for information related to those objects. Based on the identification results, provide detailed information about the relevant products, and recognize emotions from the user's facial expressions and voice to adjust the information displayed and the voice read out."

[2157] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2158] Step 1:

[2159] Image Capture

[2160] Subject: Device

[2161] Input: Objects in the user's field of view

[2162] Output: Image data of the object

[2163] Description: A user wears smart glasses and the camera captures images of objects in their field of view. The camera captures image data of products and objects in the user's field of view.

[2164] Step 2:

[2165] Object Identification

[2166] Subject: Device

[2167] Input: Image data

[2168] Output: Identification information

[2169] Description: The captured image data is sent to a local AI module, which uses TensorFlow and OpenCV to identify objects in the image and extract identifying information.

[2170] Step 3:

[2171] Sending image analysis information

[2172] Subject: Device

[2173] Input: Identification information

[2174] Output: Data sent to external server

[2175] Description: Identified object information is sent to an external server via the communication module. Identification information includes product name and category.

[2176] Step 4:

[2177] Information acquisition

[2178] Subject: Server

[2179] Input: Identification information

[2180] Output: Product details

[2181] Description: The external server searches the database based on the received identification information and retrieves relevant product information (price, reviews, campaign information, etc.).

[2182] Step 5:

[2183] Information transmission

[2184] Subject: Server

[2185] Input: Product details

[2186] Output: Data sent to the terminal

[2187] Description: The acquired product details are sent to the smart glasses through the communication module. The transmitted data includes the product details.

[2188] Step 6:

[2189] Information display and reading

[2190] Subject: Device

[2191] Input: Product details

[2192] Output: Display and voice reading

[2193] Description: The retrieved product information is displayed on the device's display, and the voice reading module provides the information aloud. The display shows the price, reviews, and campaign information, and the voice reads out, "This candy has received excellent reviews. It's currently on sale for 50% off."

[2194] Step 7:

[2195] emotion recognition

[2196] Subject: Device

[2197] Input: User's voice tone and facial expression data

[2198] Output: Emotional information

[2199] Description: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize emotions in real time. Emotions can be recognized as interest or confusion, for example.

[2200] Step 8:

[2201] Information adjustment

[2202] Subject: Device

[2203] Input: Emotion information

[2204] Output: Adjusted display and speech content

[2205] Description: The system adjusts the content displayed and spoken based on the recognized emotion information. If the user is confused, additional information will be displayed on the display and spoken aloud: "This candy is milk chocolate. It is low in sugar and recommended for health-conscious people."

[2206] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2208] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2209] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2210] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2211] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2212] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represe...

Claims

1. A wearable device worn by a user, means for capturing images of objects in view by a camera; means for identifying objects in the captured images by a local AI module; means for transmitting image analysis information to an external server; a means for displaying information acquired from an external server on a lens of one eye; a means for reading out the acquired information aloud; A system including:

2. The system of claim 1 further comprising means for the local AI module to perform character recognition of objects in its field of view and transmit the character data to a server.

3. The system according to claim 1 , further comprising: means for translating the text data received by the external server into a language set by the user and transmitting the translation result to the wearable device.

Images