System

Abstract

A system is provided.SOLUTION: An operation assistance system for a smart device, comprising: a display device worn by a user, wherein the display device comprises a camera unit for capturing a screen of the smart device from a viewpoint of the user, a network unit for receiving a captured screen image, an image recognition unit for identifying a user interface element on the screen of the smart device from the received image, a voice input unit for receiving a voice instruction of the user, an artificial intelligence unit for analyzing the voice instruction and generating a next operation guide based on the user interface element, and a display unit for displaying the generated operation guide on the display device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people and those unfamiliar with smart device operation often find themselves confused by the device's complex controls and unable to operate them properly. This can result in them being unable to use important functions, hindering their daily lives and communication. Furthermore, even if they are taught how to operate a smart device by others, they often find it difficult to adapt to repetitive operations or new functions. To address this situation, an effective system that provides real-time support for smart device operation is needed. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. Smart glasses are provided as a display device worn by a user. The display device is equipped with a camera that captures the screen of a smart device from the user's point of view. The smart glasses have network means for transmitting the captured screen image to a server, and the server has image recognition means for identifying user interface elements on the screen of the smart device from the received image. Furthermore, the smart glasses are equipped with voice input means for receiving voice instructions from the user, and the server has generative artificial intelligence means for analyzing the voice instructions and generating an operation guide. The finally generated operation guide is visually displayed on the display of the display device, highlighting operation points and providing instructions to the user. This allows the user to easily operate the smart device while receiving clear operation guidance in real time.

[0006] A "user" is a person who wears smart glasses and operates a smart device.

[0007] A "display device" is a device, including smart glasses, for displaying information based on a user's point of view.

[0008] "Camera means" refers to a camera mounted on the smart glasses, and is a means having the function of capturing the screen of the smart device from the user's point of view.

[0009] The "network means" is a communication means for transmitting the captured image data to the server.

[0010] "Image Recognition Means" means software or algorithms running on a server that identify user interface elements on the screen of a smart device from received image data.

[0011] "Voice input means" refers to means including a microphone and related functions for receiving voice instructions from a user.

[0012] The "generative artificial intelligence means" is an artificial intelligence system that analyzes the user's voice instructions and generates a guide for the next operation to be performed.

[0013] The "display means" refers to the display part of the smart glasses, and is a means for visually presenting the generated operation guide to the user.

[0014] An "operation guide" is a guide message or visual display that instructs the user on the next operation to be performed on the smart device.

[0015] "User interface elements" are elements that can be operated by the user, such as buttons and icons displayed on the screen of a smart device. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[0038] System Configuration

[0039] The system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence.

[0040] 1. Capture and send images

[0041] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[0042] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[0043] 2. Image Recognition

[0044] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[0045] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[0046] 3. Voice input reception and question analysis

[0047] The smart glasses (device) receives user voice commands through a microphone, and this voice data is sent to a server via a voice recognition system.

[0048] The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent.

[0049] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[0050] 4. Response Generation

[0051] The server determines the next action the user should take based on the voice instructions and the results of image recognition, and generates an operation guide that includes clear visual instructions.

[0052] Example: The generated instruction guide is "Tap the Phone app icon."

[0053] 5. Guide display

[0054] The smart glasses (device) displays the generated operation guide on the display, allowing the user to receive clear instructions and perform the next operation.

[0055] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[0056] Specific examples

[0057] Example 1: Making a phone call

[0058] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0059] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0060] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0061] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0062] 5. The audio data is sent to the server and converted into text.

[0063] 6. Generative AI analyzes the instruction "Open the phone app."

[0064] 7. The server generates an operation guide saying "Tap the phone app icon."

[0065] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0066] 9. The user follows the instructions and taps the Phone app icon.

[0067] In this way, this system helps users who are unfamiliar with operating smart devices to operate them easily and intuitively.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user looks at their smartphone, the camera in the smart glasses captures the screen and saves it as image data.

[0071] Step 2:

[0072] The smart glasses (device) sends the captured image data to the server via a network connection, where it is prepared for analysis.

[0073] Step 3:

[0074] The server analyzes the received image data and identifies interface elements on the smart device screen. It uses image recognition technology (such as OCR) to detect text, buttons, icons, etc., and obtains their location information.

[0075] Step 4:

[0076] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[0077] Step 5:

[0078] The server analyzes the voice data and converts it into text. The voice data is converted into text using voice recognition technology, and the user's intent is analyzed using generative artificial intelligence.

[0079] Step 6:

[0080] The server generates an operation guide based on the analysis results. It combines the user's voice instructions and the image recognition results to generate the next operation guide (e.g., open the phone app).

[0081] Step 7:

[0082] The smart glasses (device) displays the generated operation guide on the display, visually highlighting the operation points and providing instructions that are easy for the user to understand.

[0083] Step 8:

[0084] The user operates the smart device according to the instructions on the smart glasses, for example, by tapping a specific icon according to the instructions displayed on the smart glasses.

[0085] Step 9:

[0086] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] The operation of modern information devices is becoming more complex, and there is a demand for an environment that allows elderly people and users who are unfamiliar with operating smart devices to operate them easily. In particular, many information devices have numerous icons and buttons, which can be difficult for first-time users to understand and operate. In addition, there is a need for a system that allows for more intuitive and reliable operation by simultaneously providing visual and auditory support.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes a communication means for receiving the captured screen image, an image recognition means for identifying user interface elements on the screen of the information device from the received image, and a generative artificial intelligence means for analyzing the voice instruction and generating the next operation guide based on the user interface elements. This allows even a user who is unfamiliar with operating the information device to operate it intuitively and reliably while receiving visual and voice support.

[0092] "User" refers to a person who operates an information device and uses a device that supports the operation from that point of view.

[0093] "Display device" refers to a device worn by a user that has a display function for providing visual information.

[0094] "Photographing means" means a camera or photographing device for capturing the screen of an information device from the user's point of view.

[0095] "Communication means" refers to the network technology used to transmit the captured screen image to the server, including Wi-Fi, Bluetooth, etc.

[0096] "Image recognition means" refers to technology for identifying user interface elements on the screen of an information device from images received by the server, and includes, in particular, pattern recognition algorithms and OCR technology.

[0097] The "voice input means" refers to a microphone or voice collection device for receiving voice instructions from the user, and is a means for acquiring voice data.

[0098] "Generative artificial intelligence means" refers to an artificial intelligence technology for analyzing voice instructions and generating next operation guidance based on user interface elements.

[0099] The "display means" refers to a component or device having a display function for visually presenting the generated operation guide to the user.

[0100] "Operation guide" refers to guide information that clearly indicates to the user the next operation to be performed and includes instructions that visually emphasize the operation.

[0101] "Information devices" refer to electronic devices operated by users, such as smartphones and tablets, which have many app icons and buttons.

[0102] This invention is a system that supports the operation of information devices by combining a display device worn by the user with a server. It uses smart glasses and generative artificial intelligence to enable users to easily operate information devices that they are unfamiliar with.

[0103] System Configuration

[0104] The system mainly consists of the following components:

[0105] Terminal (smart glasses): Worn by the user, it includes a camera means for capturing the screen of the information device from the user's viewpoint, a voice input means for picking up voice, and a display means for displaying the generated operation guide.

[0106] Server: Includes a communication means for receiving screen images, an image recognition means for recognizing the received images, and a generative artificial intelligence means for analyzing the user's voice instructions.

[0107] Specific actions

[0108] 1. Image capture and transmission:

[0109] The device (smart glasses) uses a camera to capture the screen of the information device from the user's perspective and transmits the image to a server via Wi-Fi or Bluetooth.

[0110] Example: When a user puts on smart glasses while viewing the home screen of their smartphone, the camera automatically captures the home screen and sends the image data to the server.

[0111] 2. Image Recognition:

[0112] The server analyzes the received image data and uses image recognition technology such as Google® Cloud Vision API to identify on-screen user interface elements (app icons and buttons).

[0113] Example: The server analyzes the image data of the home screen received and identifies app icons such as "Phone" and "Message."

[0114] 3. Voice input acceptance and conversion:

[0115] The terminal (smart glasses) picks up the user's instructions using a voice input means and transmits them to a server via a communication means.

[0116] The server converts the received voice data into text using Google Cloud Speech-to-Text technology.

[0117] For example, if a user says "I want to make a call," the smart glasses will pick up the voice and send it to the server, which will convert the voice data into text.

[0118] 4. Intent Analysis:

[0119] The server inputs the converted text data into a generative AI model (e.g., OpenAI's GPT-4®) to analyze the user's intent and identify appropriate operational procedures.

[0120] Example: A generative AI model analyzes the instruction "I want to make a call" into the specific operation "Open the phone app."

[0121] 5. Response Generation:

[0122] Based on the analysis results, the server generates an operation guide for the user, which includes specific instructions and visual highlights.

[0123] Example: Generate an action guide that says "Tap the Phone app icon" with instructions to highlight the Phone app icon in a red border.

[0124] 6. Guide display:

[0125] The terminal (smart glasses) visually presents the generated operation guide to the user on a display means.

[0126] Example: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the phone app icon will be highlighted in a red frame. The user will follow the instructions.

[0127] Specific examples and prompts

[0128] Example: A scenario for making a phone call

[0129] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0130] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0131] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0132] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0133] 5. The audio data is sent to the server and converted into text.

[0134] 6. The generative AI model analyzes the instruction "Open the phone app."

[0135] 7. The server generates an operation guide saying "Tap the phone app icon."

[0136] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0137] 9. The user follows the instructions and taps the Phone app icon.

[0138] Prompt Sentence Examples

[0139] "When a user says, 'I want to make a phone call,' generate a guide for the next operation based on the image captured by the smartglasses' camera."

[0140] As described above, the present invention provides a system that allows even a user who is unfamiliar with operating information devices to operate it easily and intuitively.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The terminal (smart glasses) captures the screen of the information device from the user's point of view with a camera and transmits the image to a server via a network means.

[0144] Input: The screen of the information device that the user is looking at

[0145] Data processing: The camera captures the screen and generates image data

[0146] Output: Image data sent to the server

[0147] Specific operation: When a user looks at the home screen of their smartphone, the camera in the smart glasses captures the screen and sends the generated image data to a server via Wi-Fi or Bluetooth.

[0148] Step 2:

[0149] The server analyzes the received image data and identifies user interface elements using image recognition technology such as the Google Cloud Vision API.

[0150] Input: Screen image sent from the device (smart glasses)

[0151] Data processing: Apply image recognition algorithms (such as Google Cloud Vision API) to identify icons and buttons on the screen.

[0152] Output: A list of identified user interface elements

[0153] Specific operation: The server processes the image data of the home screen it receives and identifies app icons such as "Phone" and "Message."

[0154] Step 3:

[0155] The device (smart glasses) receives the user's voice instructions through a microphone and transmits them to the server as voice data.

[0156] Input: User's voice command

[0157] Data processing: Collection of voice data

[0158] Output: Audio data sent to the server

[0159] Specific operation: When a user says "I want to make a call," the smart glasses pick up the voice and send the voice data to the server.

[0160] Step 4:

[0161] The server converts the received voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[0162] Input: Audio data sent from the device (smart glasses)

[0163] Data processing: Converting voice data into text using voice recognition technology

[0164] Output: Text data

[0165] Specific operation: The server converts the received voice data into text using Google Cloud Speech-to-Text. The text obtained is "I want to make a phone call."

[0166] Step 5:

[0167] The server inputs the text data into a generative AI model (such as OpenAI's GPT-4) to analyze the user's intent.

[0168] Input: Text data converted from audio data

[0169] Data processing: Analysis and intent identification with generative AI models

[0170] Output: Instructions based on the user's intent (e.g., "Open the Phone app")

[0171] Specific action: The generative AI model analyzes the instruction "I want to make a call" into the specific action of "opening the phone app."

[0172] Step 6:

[0173] The server generates specific operation guides based on the analysis results and sends them to the device (smart glasses).

[0174] Input: Operation instructions based on the user's intention

[0175] Data processing: Generate operation guides, including visual instructions and highlights

[0176] Output: Generated instructions

[0177] Specific operation: The server generates an operation guide saying "Tap the phone app icon" and instructs the phone app icon to be highlighted in a red frame.

[0178] Step 7:

[0179] The device (smart glasses) displays the generated operation guide on its own display, providing the user with clear operating instructions.

[0180] Input: Operation guide sent from the server

[0181] Data processing: Display of operation guide

[0182] Output: Operation guide displayed on the screen

[0183] Specific operation: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the icon will be highlighted in red. The user should follow the instructions and tap the phone app icon.

[0184] Through the above steps, this system provides an environment in which even users who are unfamiliar with operating information devices can operate the system easily and intuitively.

[0185] (Application example 1)

[0186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0187] Conventional smart device operation support systems have problems, such as complicated operation and difficulty in navigating and obtaining product information, especially for elderly users and users unfamiliar with device operation. This has prevented users from using the device effectively and has caused inconvenience. In addition, there has been a lack of support for checking product information in physical stores and obtaining location information within the store, which has reduced user convenience.

[0188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0189] In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a video output means for displaying the generated operation guide on the display device; and a display means for the user to obtain in-store product information and a store map. This allows the user to intuitively operate the smart device, making it easier to navigate in a physical store and obtain product information.

[0190] "Smart devices" is a general term for portable electronic devices such as mobile phones, tablets, e-book readers, and smartwatches that can communicate over a network and run a variety of applications.

[0191] A "display device" is a device worn by a user that outputs visual information, and examples of this include head-mounted displays and smart glasses.

[0192] The "photographing means" is a camera mounted on the display device, and is a device that has the function of capturing an image from the user's viewpoint.

[0193] "Communication means" refers to a network connection device for relaying captured image and audio data to a server, and includes wireless and wired communication.

[0194] "Image analysis means" refers to technology that identifies specific elements from received image data, and uses optical character recognition (OCR) or machine learning algorithms.

[0195] The "voice input means" is a device for collecting the user's voice and transmitting it to the server as voice data.

[0196] "Generative AI means" refers to an AI technology that analyzes the user's intentions based on voice input and image analysis data, and generates the next operation guide.

[0197] The "video output means" is a device for visually outputting the generated operation guide to a display device.

[0198] The "display means" is a device that provides visual information for the user to obtain in-store product information and a store map.

[0199] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[0200] System Configuration

[0201] This system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence. It also includes a display means that allows the user to obtain in-store product information and a store map.

[0202] Capture and send images

[0203] The smart glasses, which are the terminals, capture the screen of the smart device from the user's point of view. The captured image is immediately sent to the server via a communication means. For example, if a user is looking at a product shelf in a store, the camera in the smart glasses captures the image and sends it to the server.

[0204] Image Recognition

[0205] The server analyzes the received image data and identifies the user interface elements displayed on the smart device screen. For example, in a store, the server analyzes the images of product shelves and recognizes the barcodes and labels of each product.

[0206] Voice input reception and question analysis

[0207] The smart glasses, which act as a terminal, receive voice commands from the user via a microphone. This voice data is sent to a server via a voice recognition system. The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent. For example, if a user says, "I want to see a list of special sale items," the voice data is sent to the server, converted into text, and the generative artificial intelligence interprets it as the instruction to "display a list of special sale items."

[0208] Response Generation

[0209] Based on the voice instructions and image recognition results, the server determines the next action the user should take and generates an operation guide that includes clear visual instructions, such as "Please scan the barcode."

[0210] Guide Display

[0211] The smart glasses, which are the terminals, display the generated operation guide on their screens, allowing the user to receive clear instructions and perform the next operation. For example, the operation guide is displayed on the smart glasses' display, highlighting the product barcode with a red frame.

[0212] Specific examples

[0213] Example 1: In-store navigation

[0214] 1. A user wears smart glasses and walks around a store.

[0215] 2. The camera in the smart glasses captures the inside of the store and sends the images to the server.

[0216] 3. The server performs image recognition to identify the shelves and the barcodes and labels of each product.

[0217] 4. The user speaks into the microphone on the smart glasses, "Where is the detergent shelf?"

[0218] 5. The audio data is sent to the server and converted into text.

[0219] 6. The generative AI analyzes the instruction "Show the location of the detergent shelf."

[0220] 7. The server generates a navigation guide and shows the desired shelf location on a store map.

[0221] 8. Navigation guides will be displayed on the smart glasses display, allowing users to check the route to their destination.

[0222] Prompt Sentence Examples

[0223] Prompt to find sale items:

[0224] "The user says, 'I want to see a list of special offers,' and the system converts this into text and inputs the instruction 'Show me a list of special offers' as a prompt to the generative AI."

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] System program processing flow

[0227] Step 1:

[0228] The camera in the smart glasses captures images from the user's point of view. Specifically, the image capture means acquires images while the user is wearing the smart glasses and looking at the screen of a smart device or a product shelf in a store.

[0229] Input: Image from the user's perspective

[0230] Output: Captured image data

[0231] Step 2:

[0232] The captured image data is sent to the server via a communication means. Internally, the smart glasses use wireless communication to send image data to the server in real time.

[0233] Input: Captured image data

[0234] Output: Image data sent to the server

[0235] Step 3:

[0236] The server analyzes the received image data and uses image analysis methods to identify user interface elements displayed on the smart device screen or on the store shelf, and extracts information such as barcodes and labels using OCR technology and object recognition algorithms.

[0237] Input: Image data sent to the server

[0238] Output: Parsed interface elements (e.g. barcodes, labels)

[0239] Step 4:

[0240] The smart glasses' microphone accepts the user's voice commands, allowing the user to use voice input to request specific actions or information.

[0241] Input: User's voice command

[0242] Output: Audio data

[0243] Step 5:

[0244] The voice data is sent to the server via a communication means. The smart glasses send the voice data to the server in real time.

[0245] Input: Audio data

[0246] Output: Audio data sent to the server

[0247] Step 6:

[0248] The server converts the voice data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text).

[0249] Input: Audio data sent to the server

[0250] Output: Text data

[0251] Step 7:

[0252] Generative AI analyzes text data to understand user intent. In this process, the generative AI model generates appropriate responses and operational guidance based on the user's questions and requests.

[0253] Input: Text data

[0254] Output: Appropriate response or action guide (text)

[0255] Step 8:

[0256] Based on the analysis results, the server determines the next action the user should take and generates a visual operation guide, such as "Please scan the barcode."

[0257] Input: Appropriate response or instruction (text)

[0258] Output: Visual operation guide

[0259] Step 9:

[0260] The generated operation guide is sent to the smart glasses display via a communication method and displayed. The user visually confirms it and performs the next operation.

[0261] Input: Visual guide

[0262] Output: Guide displayed on the smart glasses display

[0263] Example: In-store navigation processing steps

[0264] Step 1:

[0265] As a user wears the smart glasses and walks around the store, the camera in the smart glasses captures images of the store's shelves.

[0266] Input: Image of the interior from the user's perspective

[0267] Output: Captured in-store image data

[0268] Step 2:

[0269] The captured in-store image data is sent from the smart glasses to a server.

[0270] Input: Captured in-store image data

[0271] Output: In-store image data sent to the server

[0272] Step 3:

[0273] The server analyzes the in-store image data transmitted and uses image analysis means to identify the barcodes and labels of each product.

[0274] Input: In-store image data sent to the server

[0275] Output: Identified barcode or label

[0276] Step 4:

[0277] The user speaks into the microphone on the smart glasses, asking, "Where is the detergent shelf?"

[0278] Input: User's voice command

[0279] Output: Audio data

[0280] Step 5:

[0281] The audio data is sent from the smart glasses to a server.

[0282] Input: Audio data

[0283] Output: Audio data sent to the server

[0284] Step 6:

[0285] The server converts the audio data into text.

[0286] Input: Audio data sent to the server

[0287] Output: Data converted to text

[0288] Step 7:

[0289] Generative artificial intelligence analyzes text data and generates optimal responses to user questions.

[0290] Input: Data converted to text

[0291] output: Text that serves as a navigation guide

[0292] Step 8:

[0293] Based on the analysis results, the server generates a navigation guide showing the location of shelves within the store.

[0294] Input: Text to serve as a navigation guide

[0295] Output: A visual navigation guide

[0296] Step 9:

[0297] The generated navigation guide is sent to the smartglasses display and displayed to the user.

[0298] Input: Visual navigation guide

[0299] Output: Navigation guide displayed on the smart glasses display

[0300] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0301] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses, generative artificial intelligence, and an emotion engine.

[0302] System Configuration

[0303] This system consists of smart glasses, a server, and an emotion engine. The smart glasses are equipped with a camera, a voice input device, a display, and emotion recognition sensors, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[0304] 1. Capture and send images

[0305] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[0306] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[0307] 2. Image Recognition

[0308] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[0309] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[0310] 3. Voice input reception and question analysis

[0311] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[0312] The server analyzes the voice data and converts it into text using voice recognition technology, and then uses generative artificial intelligence to analyze the user's intent.

[0313] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[0314] 4. Emotion Data Acquisition and Analysis

[0315] The smart glasses (devices) use emotion recognition sensors to obtain emotions from the user's voice and facial expressions, including voice tone analysis and facial expression recognition technology.

[0316] The server analyzes the acquired emotional data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[0317] Example: If a user becomes anxious while operating the device, the sensors in the smart glasses will detect the user's stress level from facial expressions and voice tone and send that information to the server.

[0318] 5. Response Generation

[0319] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[0320] Example: If the user is in a hurry, generate and display a more detailed and easy-to-understand operation guide.

[0321] 6. Guide display

[0322] The smart glasses (device) displays the generated operation guide on the display, emphasizing the operation points by, for example, increasing the text size or adding animation depending on the user's emotional state.

[0323] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[0324] Specific examples

[0325] Example 1: Making a phone call (when the user is in a hurry)

[0326] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0327] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0328] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0329] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0330] 5. The audio data is sent to the server and converted into text.

[0331] 6. Generative AI analyzes the instruction "Open the phone app."

[0332] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[0333] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[0334] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0335] 10. The user follows the instructions and taps the Phone app icon.

[0336] In this way, the system improves the user's operating experience by providing real-time support for smart device operation while taking into account the user's emotional state.

[0337] The processing flow will be explained below.

[0338] Step 1:

[0339] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user is looking at the home screen of their smartphone, the camera on the smart glasses captures the screen as image data.

[0340] Step 2:

[0341] The smart glasses (devices) transmit the captured image data to a server via a network connection, where the image data is stored and ready for analysis.

[0342] Step 3:

[0343] The server analyzes the received image data and identifies user interface elements (such as app icons and buttons) on the smart device screen using image recognition technology (such as OCR), and also obtains the location information of the elements.

[0344] Step 4:

[0345] The smart glasses (device) receive voice commands from the user through a microphone. When the user says something like "I want to make a phone call," the smart glasses capture the voice data and send it to a server via the network.

[0346] Step 5:

[0347] The server converts the received voice data into text using speech recognition technology. Using this text data, generative AI analyzes the user's intent. For example, the instruction "I want to make a call" is interpreted as the operation of opening the phone app.

[0348] Step 6:

[0349] The smart glasses (device) uses an emotion recognition sensor to recognize the user's emotions and acquires emotion data from the user's voice and facial expressions. This data is based on voice tone analysis and facial expression recognition technology.

[0350] Step 7:

[0351] The server analyzes the acquired emotional data and recognizes the user's emotional state, for example, identifying that the user is experiencing stress.

[0352] Step 8:

[0353] The server generates operation guidance based on the user's voice instructions and emotional data. It adjusts the way the operation guidance is presented depending on the user's emotional state. For example, if the user is in a hurry, it adjusts the operation guidance to be more thorough and easy to understand.

[0354] Step 9:

[0355] The smart glasses (device) displays the generated operation guide on the display. For example, the generated operation guide may be a detailed instruction such as "Tap the phone app icon," and the icon will be highlighted in a red frame.

[0356] Step 10:

[0357] The user operates the smart device according to the instructions of the smart glasses. For example, the user looks at the operation guide displayed on the smart glasses and taps the icon of the specified app.

[0358] Step 11:

[0359] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[0360] Example 2

[0361] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0362] Modern smart devices are extremely multifunctional, but their operation is complex and difficult to use, especially for elderly people and users who are unfamiliar with smart device operation. For this reason, there is a demand for intuitive operation and detailed operational support that responds to the user's emotional state.

[0363] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice acquisition means for receiving the user's voice instructions; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; an emotion recognition means for acquiring emotional data from the user's voice and facial expressions; an emotion analysis means for analyzing the acquired emotional data and recognizing the user's emotional state; a response generation means for adjusting and generating a presentation method of an operation guide according to the user's emotional state; and a display means for displaying the generated operation guide on the display device. This allows the user to easily operate the smart device, improving the operation experience.

[0364] "User" refers to a person who wears a device such as smart glasses and receives assistance in operating the smart device.

[0365] A "display device" refers to a device worn by a user that displays information in the field of view.

[0366] "Capture means" refers to equipment such as cameras and sensors used to capture the screen of a smart device from the user's perspective.

[0367] "Communication means" refers to the network function for transmitting captured image and audio data to a server.

[0368] "Image analysis means" refers to software or algorithms used to identify user interface elements on the screen of a smart device from received image data.

[0369] "Voice acquisition means" refers to a microphone or voice input device for receiving voice instructions from the user.

[0370] "Generative artificial intelligence means" refers to an artificial intelligence system for analyzing voice instructions and generating next operation guidance based on user interface elements.

[0371] "Emotion recognition means" refers to sensors and analysis software for acquiring emotional data from the user's voice and facial expressions.

[0372] "Emotion analysis means" refers to software or algorithms that analyze acquired emotional data and recognize the user's emotional state.

[0373] "Response generation means" refers to software or algorithms that generate and adjust the way in which operation guidance is presented depending on the user's emotional state.

[0374] The "display means" refers to a display device for visually displaying the generated operation guide to the user.

[0375] The following describes in detail an embodiment of the present invention. This system uses smart glasses, generative artificial intelligence, and an emotion engine to provide an environment in which elderly people and users unfamiliar with operating smart devices can easily operate smart devices. Specifically, this system is composed of smart glasses, a server, and an emotion engine.

[0376] Hardware and Software Configuration

[0377] Smart glasses (device)

[0378] The smart glasses incorporate a camera, a voice input device, a display, and an emotion recognition sensor. The camera captures the screen of the smart device from the user's point of view, the voice input device receives the user's voice instructions, the display shows the generated operation guide to the user, and the emotion recognition sensor captures the user's emotional state from their voice and facial expressions.

[0379] server

[0380] The server runs various software for image recognition, generative AI, and sentiment analysis. It uses OpenCV and Tesseract OCR for image recognition, Google Cloud Speech-to-Text for speech recognition, GPT-3 (registered trademark) and ChatGPT (registered trademark) for generative AI, and Affectiva and IBM Watson (registered trademark) for sentiment analysis.

[0381] Specific example of system operation

[0382] 1. Capture and image transmission: The camera in the smart glasses captures the screen of the smart device from the user's point of view and transmits the image data to the server via Wi-Fi or Bluetooth.

[0383] 2. Image Recognition: The server analyzes the image data received and identifies user interface elements on the smart device screen, such as app icons like "Phone" and "Message" displayed on the home screen.

[0384] 3. Voice input acceptance and question analysis: When a user says "I want to make a phone call" into the microphone of the smart glasses, the voice data is sent to the server. The server converts the voice data into text, and the generative AI analyzes the instruction to "open the phone app."

[0385] 4. Emotion data acquisition and analysis: The smart glasses use emotion recognition sensors to acquire emotional data from the user's voice and facial expressions, and then send the data to a server, which then uses emotion analysis software to analyze the user's emotional state.

[0386] 5. Response generation: The server generates operation guidance based on the user's voice instructions and emotional data. For example, if the user is in a hurry, it generates a polite guidance such as "Please tap the phone app icon a little more slowly."

[0387] 6. Guide display: The generated operation guide is displayed on the smart glasses display, and the font size and highlighting are adjusted as needed.

[0388] Specific examples

[0389] Example 1: Making a phone call (when the user is in a hurry)

[0390] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0391] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0392] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0393] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0394] 5. The audio data is sent to the server and converted into text.

[0395] 6. Generative AI analyzes the instruction "Open the phone app."

[0396] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[0397] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[0398] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0399] 10. The user follows the instructions and taps the Phone app icon.

[0400] This system can improve the user's operating experience by providing real-time support for operating smart devices while taking into account the user's emotional state. In this way, by combining generative AI and emotion recognition technology, it is possible to provide optimal operating guidance to users.

[0401] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0402] Step 1:

[0403] The terminal (smart glasses) captures the screen of the smart device from the user's point of view. Specifically, the camera captures the home screen of the smartphone. The input at this time is the screen of the smart device that the user is looking at, and the output is the captured image data.

[0404] Step 2:

[0405] The device (smart glasses) transmits the captured image data to the server via Wi-Fi or Bluetooth. The input is the captured image data, and the output is the image data transferred to the server.

[0406] Step 3:

[0407] The server analyzes the received image data using image analysis software (e.g., OpenCV, Tesseract OCR). The input is the image data transferred to the server, and the output is identification data for on-screen user interface elements (e.g., app icons and buttons). Specifically, the position and label of each icon are identified.

[0408] Step 4:

[0409] The user issues a voice command, such as "I want to make a call," into the microphone of the smart glasses. The input is the user's voice command, and the output is voice data.

[0410] Step 5:

[0411] The device (smart glasses) captures voice data and transmits it to the server via Wi-Fi or Bluetooth. The input is the user's voice data, and the output is the voice data transmitted to the server.

[0412] Step 6:

[0413] The server converts the voice data into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The input is the transferred voice data, and the output is text data. Specifically, the voice data "I want to make a call" is converted into the text "I want to make a call."

[0414] Step 7:

[0415] The server uses generative artificial intelligence (e.g., GPT-3, ChatGPT) to analyze user intent from text data. The input is text data, and the output is the analyzed user intent (e.g., "Open the phone app").

[0416] Step 8:

[0417] The device (smart glasses) uses an emotion recognition sensor to acquire emotional data from the user's voice and facial expressions. The input is the user's voice and facial expressions, and the output is the acquired emotional data. Specifically, the sensor measures the user's voice tone and facial expressions in real time.

[0418] Step 9:

[0419] The emotion data acquired by the device (smart glasses) is sent to the server via Wi-Fi or Bluetooth. The input is the acquired emotion data, and the output is the emotion data transferred to the server.

[0420] Step 10:

[0421] The server uses emotion analysis software (e.g., Affectiva, IBM Watson Emotion Analysis) to analyze the acquired emotion data and identify the user's emotional state, where the input is the transferred emotion data and the output is the user's emotional state (e.g., stress, irritability).

[0422] Step 11:

[0423] The server generates operation guidance based on the user's voice instructions and emotional data. The input is the analyzed user's intention and emotional state, and the output is the operation guidance. For example, if the user is in a hurry, a polite operation guidance such as "Please tap the phone app icon a little more slowly" is generated.

[0424] Step 12:

[0425] The operation guide generated by the server is sent to the device (smart glasses) via Wi-Fi or Bluetooth. The input at this time is the operation guide, and the output is the operation guide transferred to the smart glasses.

[0426] Step 13:

[0427] The device (smart glasses) displays the operation guide on the display. The input is the transferred operation guide, and the output is the displayed operation guide. Specifically, visual feedback is provided, such as the phone app icon being highlighted in a red frame.

[0428] (Application example 2)

[0429] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0430] Providing an environment where elderly people and users unfamiliar with operating smart devices can easily obtain and use information in the real world is an important challenge in modern society. In particular, obtaining product information and supporting purchasing in physical stores requires an intuitive and adaptive interface. However, conventional systems have difficulty responding while fully considering the user's emotions and operating situation. Therefore, there is a need to develop a system that provides appropriate feedback according to the user's emotional state and supports operation.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0432] In this invention, the server is a display device worn by a user, and includes: a camera means for capturing the screen of the smart device from the user's viewpoint; a network means for receiving the captured screen image; an image recognition means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a display means for displaying the generated operation guide on the display device; an emotion recognition means for acquiring the user's emotional state; and an emotion analysis means for adjusting the content of the operation guide based on the emotional state, thereby enabling adaptive and intuitive feedback according to the user's emotional state.

[0433] A "display device" is a device worn by a user to provide visual feedback.

[0434] "Camera means" refers to a camera attached to a display device to capture the screen of the smart device from the user's point of view.

[0435] "Network means" refers to the communications infrastructure used to receive the captured screen image.

[0436] "Image recognition means" refers to a technology that identifies user interface elements on the screen of a smart device from a received image.

[0437] The "voice input means" is a device such as a microphone that receives voice instructions from the user.

[0438] "Generative artificial intelligence means" refers to AI technology that analyzes voice instructions and generates the next operation guide based on the user interface elements.

[0439] The "display means" is a screen for displaying the generated operation guide on a display device.

[0440] An "emotion recognition means" is a sensor or software for acquiring the user's emotional state.

[0441] The "emotion analysis means" is a technique for adjusting the content of operation guidance based on the acquired emotional state.

[0442] MODE FOR CARRYING OUT THE INVENTION

[0443] System Configuration

[0444] To implement this invention, a display device (smart glasses) worn by a user, a server, and emotion recognition means are required. Specifically, the smart glasses incorporate a camera means, a voice input means, a display means, and an emotion recognition sensor, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[0445] Specific examples of program processing

[0446] Capture and send images

[0447] The smart glasses use a camera means to capture the screen of the smart device from the user's point of view, and the captured image is immediately transmitted to a server via a network means.

[0448] Image Recognition

[0449] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR technology).

[0450] Voice input reception and question analysis

[0451] The smart glasses receive voice commands from the user via a microphone. When the user issues a voice command, such as "I want to search for a specific product," the voice data is captured and sent to a server. The server then analyzes the voice data and converts it into text. Generative AI is used to convert the voice data into text and analyze the user's intent.

[0452] Emotion data acquisition and analysis

[0453] The smart glasses use emotion recognition sensors to capture emotions from the user's voice and facial expressions. This includes voice tone analysis and facial expression recognition technology. The server analyzes the captured emotion data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[0454] Response Generation

[0455] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[0456] Guide Display

[0457] The smart glasses display the generated operation guide on the screen, emphasizing the operation points by, for example, increasing the text size or adding animations depending on the user's emotional state.

[0458] Hardware and software used

[0459] Hardware: Smart glasses (camera, microphone, display, emotion recognition sensor), server

[0460] Software: OpenCV (camera image processing), pytesseract (OCR text recognition), speech_recognition (voice recognition), requests (API communication), emotion_recognition (emotion recognition)

[0461] Specific examples

[0462] Here is a specific example of how smart glasses can read the QR code (registered trademark) of a product that a user shows interest in at a physical store. The camera means of the smart glasses captures the QR code, and the image recognition means extracts the product code. Using this product code, the server obtains detailed information about the product. If the user issues a voice command such as "I want to find a cheaper product," the command is analyzed by generative artificial intelligence, and information about cheaper products is provided by voice. If the user becomes impatient while operating the device, the emotion recognition means detects this, and the server provides adaptive feedback such as "Please stay calm."

[0463] Prompt Sentence Examples

[0464] "If a user wants to scan a QR code and get the product price and recommended points from a voice assistant, how do you provide that feedback?"

[0465] In this way, users will receive smart product information and adaptive feedback to improve their real-world shopping experience.

[0466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0467] Step 1:

[0468] The smart glasses (terminal) use a camera to capture the screen of the smart device as seen from the user's point of view.

[0469] Input: The screen of the smart device that the user is looking at while wearing the smart glasses.

[0470] Output: The captured screen image.

[0471] Specific operation: The camera continuously takes images and generates image data.

[0472] Step 2:

[0473] The smart glasses (terminal) transmit the captured screen image to the server via the network.

[0474] Input: A captured screen image.

[0475] Output: Screen image sent to the server.

[0476] Specific operation: The captured image data is sent to the server using a network means such as Wi-Fi or Bluetooth.

[0477] Step 3:

[0478] The server analyzes the received image data and uses image recognition technology to identify user interface elements on the screen of the smart device.

[0479] Input: The submitted screen image.

[0480] Output: Identified user interface elements (e.g. app icons and buttons).

[0481] Specific operation: Image recognition software on the server (e.g., OpenCV or pytesseract) analyzes the image and extracts specific elements.

[0482] Step 4:

[0483] The smart glasses (device) accept the user's voice instructions via a microphone and capture the voice data.

[0484] Input: User's spoken command (e.g., "I would like to search for a specific product").

[0485] Output: The captured audio data.

[0486] How it works: The microphone built into the smart glasses records the user's voice and generates digital audio data.

[0487] Step 5:

[0488] The smart glasses (terminals) transmit the captured audio data to a server via a network.

[0489] Input: Captured audio data.

[0490] Output: The audio data sent to the server.

[0491] Specific operation: Audio data is transmitted to a server via a network means.

[0492] Step 6:

[0493] The server analyzes the transmitted voice data and converts it into text using voice recognition technology.

[0494] Input: The audio data sent to the server.

[0495] Output: Text data (voice instructions converted into text).

[0496] What happens: The server's speech recognition software (e.g., speech_recognition) processes the audio data and generates corresponding text data.

[0497] Step 7:

[0498] The server uses generative artificial intelligence to analyze the user's voice instructions and generate operation guides.

[0499] Input: Text data and identified user interface elements.

[0500] Output: The generated how-to guide.

[0501] What it does: Generative AI (e.g., a machine learning model) analyzes text data and generates guidelines for the actions the user wants to take.

[0502] Step 8:

[0503] The smart glasses (device) collect emotional data from the user's voice and facial expressions.

[0504] Input: The user's tone of voice and facial expressions.

[0505] Output: The obtained emotion data.

[0506] Specific operation: The emotion recognition sensor analyzes the user's tone of voice and facial expressions, and digitizes their emotional state.

[0507] Step 9:

[0508] The server analyzes the acquired emotional data and recognizes the emotional state of the user.

[0509] Input: The acquired emotion data.

[0510] Output: Parsed emotional state (e.g. stress state).

[0511] What it does: The emotion recognition engine analyzes the emotion data and identifies the user's emotional state.

[0512] Step 10:

[0513] The server adjusts the content of the operation guide based on the emotional state of the user.

[0514] Input: Parsed emotional state and generated interaction guide.

[0515] Output: Adjusted operation guide.

[0516] Specific operation: Generative AI incorporates emotional data and modifies the operation guide to suit the user.

[0517] Step 11:

[0518] The smart glasses (terminal) will display adjusted operation guides on the display.

[0519] Enter: the adjusted operating guide.

[0520] Output: Operation guide displayed on the screen.

[0521] Specific operation: The smart glasses display visually displays operation guides and draws the user's attention by adjusting the font size and color as needed.

[0522] Through these steps, users will be able to intuitively and adaptively operate smart devices and product information.

[0523] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0524] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0525] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0526] [Second embodiment]

[0527] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0528] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0529] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0530] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0531] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0532] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0533] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0534] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0535] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0536] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0537] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0538] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0539] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[0540] System Configuration

[0541] The system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence.

[0542] 1. Capture and send images

[0543] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[0544] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[0545] 2. Image Recognition

[0546] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[0547] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[0548] 3. Voice input reception and question analysis

[0549] The smart glasses (device) receives user voice commands through a microphone, and this voice data is sent to a server via a voice recognition system.

[0550] The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent.

[0551] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[0552] 4. Response Generation

[0553] The server determines the next action the user should take based on the voice instructions and the results of image recognition, and generates an operation guide that includes clear visual instructions.

[0554] Example: The generated instruction guide is "Tap the Phone app icon."

[0555] 5. Guide display

[0556] The smart glasses (device) displays the generated operation guide on the display, allowing the user to receive clear instructions and perform the next operation.

[0557] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[0558] Specific examples

[0559] Example 1: Making a phone call

[0560] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0561] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0562] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0563] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0564] 5. The audio data is sent to the server and converted into text.

[0565] 6. Generative AI analyzes the instruction "Open the phone app."

[0566] 7. The server generates an operation guide saying "Tap the phone app icon."

[0567] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0568] 9. The user follows the instructions and taps the Phone app icon.

[0569] In this way, this system helps users who are unfamiliar with operating smart devices to operate them easily and intuitively.

[0570] The processing flow will be explained below.

[0571] Step 1:

[0572] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user looks at their smartphone, the camera in the smart glasses captures the screen and saves it as image data.

[0573] Step 2:

[0574] The smart glasses (device) sends the captured image data to the server via a network connection, where it is prepared for analysis.

[0575] Step 3:

[0576] The server analyzes the received image data and identifies interface elements on the smart device screen. It uses image recognition technology (such as OCR) to detect text, buttons, icons, etc., and obtains their location information.

[0577] Step 4:

[0578] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[0579] Step 5:

[0580] The server analyzes the voice data and converts it into text. The voice data is converted into text using voice recognition technology, and the user's intent is analyzed using generative artificial intelligence.

[0581] Step 6:

[0582] The server generates an operation guide based on the analysis results. It combines the user's voice instructions and the image recognition results to generate the next operation guide (e.g., open the phone app).

[0583] Step 7:

[0584] The smart glasses (device) displays the generated operation guide on the display, visually highlighting the operation points and providing instructions that are easy for the user to understand.

[0585] Step 8:

[0586] The user operates the smart device according to the instructions on the smart glasses, for example, by tapping a specific icon according to the instructions displayed on the smart glasses.

[0587] Step 9:

[0588] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[0589] Example 1

[0590] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] The operation of modern information devices is becoming more complex, and there is a demand for an environment that allows elderly people and users who are unfamiliar with operating smart devices to operate them easily. In particular, many information devices have numerous icons and buttons, which can be difficult for first-time users to understand and operate. In addition, there is a need for a system that allows for more intuitive and reliable operation by simultaneously providing visual and auditory support.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0593] In this invention, the server includes a communication means for receiving the captured screen image, an image recognition means for identifying user interface elements on the screen of the information device from the received image, and a generative artificial intelligence means for analyzing the voice instruction and generating the next operation guide based on the user interface elements. This allows even a user who is unfamiliar with operating the information device to operate it intuitively and reliably while receiving visual and voice support.

[0594] "User" refers to a person who operates an information device and uses a device that supports the operation from that point of view.

[0595] "Display device" refers to a device worn by a user that has a display function for providing visual information.

[0596] "Photographing means" means a camera or photographing device for capturing the screen of an information device from the user's point of view.

[0597] "Communication means" refers to the network technology used to transmit the captured screen image to the server, including Wi-Fi, Bluetooth, etc.

[0598] "Image recognition means" refers to technology for identifying user interface elements on the screen of an information device from images received by the server, and includes, in particular, pattern recognition algorithms and OCR technology.

[0599] The "voice input means" refers to a microphone or voice collection device for receiving voice instructions from the user, and is a means for acquiring voice data.

[0600] "Generative artificial intelligence means" refers to an artificial intelligence technology for analyzing voice instructions and generating next operation guidance based on user interface elements.

[0601] The "display means" refers to a component or device having a display function for visually presenting the generated operation guide to the user.

[0602] "Operation guide" refers to guide information that clearly indicates to the user the next operation to be performed and includes instructions that visually emphasize the operation.

[0603] "Information devices" refer to electronic devices operated by users, such as smartphones and tablets, which have many app icons and buttons.

[0604] This invention is a system that supports the operation of information devices by combining a display device worn by the user with a server. It uses smart glasses and generative artificial intelligence to enable users to easily operate information devices that they are unfamiliar with.

[0605] System Configuration

[0606] The system mainly consists of the following components:

[0607] Terminal (smart glasses): Worn by the user, it includes a camera means for capturing the screen of the information device from the user's viewpoint, a voice input means for picking up voice, and a display means for displaying the generated operation guide.

[0608] Server: Includes a communication means for receiving screen images, an image recognition means for recognizing the received images, and a generative artificial intelligence means for analyzing the user's voice instructions.

[0609] Specific actions

[0610] 1. Image capture and transmission:

[0611] The device (smart glasses) uses a camera to capture the screen of the information device from the user's perspective and transmits the image to a server via Wi-Fi or Bluetooth.

[0612] Example: When a user puts on smart glasses while viewing the home screen of their smartphone, the camera automatically captures the home screen and sends the image data to the server.

[0613] 2. Image Recognition:

[0614] The server analyzes the received image data and uses image recognition technology such as Google Cloud Vision API to identify on-screen user interface elements (app icons and buttons).

[0615] Example: The server analyzes the image data of the home screen received and identifies app icons such as "Phone" and "Message."

[0616] 3. Voice input acceptance and conversion:

[0617] The terminal (smart glasses) picks up the user's instructions using a voice input means and transmits them to a server via a communication means.

[0618] The server converts the received voice data into text using Google Cloud Speech-to-Text technology.

[0619] For example, if a user says "I want to make a call," the smart glasses will pick up the voice and send it to the server, which will convert the voice data into text.

[0620] 4. Intent Analysis:

[0621] The server inputs the converted text data into a generative AI model (such as OpenAI's GPT-4), analyzes the user's intent, and identifies appropriate operating procedures.

[0622] Example: A generative AI model analyzes the instruction "I want to make a call" into the specific operation "Open the phone app."

[0623] 5. Response Generation:

[0624] Based on the analysis results, the server generates an operation guide for the user, which includes specific instructions and visual highlights.

[0625] Example: Generate an action guide that says "Tap the Phone app icon" with instructions to highlight the Phone app icon in a red border.

[0626] 6. Guide display:

[0627] The terminal (smart glasses) visually presents the generated operation guide to the user on a display means.

[0628] Example: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the phone app icon will be highlighted in a red frame. The user will follow the instructions.

[0629] Specific examples and prompts

[0630] Example: A scenario for making a phone call

[0631] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0632] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0633] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0634] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0635] 5. The audio data is sent to the server and converted into text.

[0636] 6. The generative AI model analyzes the instruction "Open the phone app."

[0637] 7. The server generates an operation guide saying "Tap the phone app icon."

[0638] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0639] 9. The user follows the instructions and taps the Phone app icon.

[0640] Prompt Sentence Examples

[0641] "When a user says, 'I want to make a phone call,' generate a guide for the next operation based on the image captured by the smartglasses' camera."

[0642] As described above, the present invention provides a system that allows even a user who is unfamiliar with operating information devices to operate it easily and intuitively.

[0643] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0644] Step 1:

[0645] The terminal (smart glasses) captures the screen of the information device from the user's point of view with a camera and transmits the image to a server via a network means.

[0646] Input: The screen of the information device that the user is looking at

[0647] Data processing: The camera captures the screen and generates image data

[0648] Output: Image data sent to the server

[0649] Specific operation: When a user looks at the home screen of their smartphone, the camera in the smart glasses captures the screen and sends the generated image data to a server via Wi-Fi or Bluetooth.

[0650] Step 2:

[0651] The server analyzes the received image data and identifies user interface elements using image recognition technology such as the Google Cloud Vision API.

[0652] Input: Screen image sent from the device (smart glasses)

[0653] Data processing: Apply image recognition algorithms (such as Google Cloud Vision API) to identify icons and buttons on the screen.

[0654] Output: A list of identified user interface elements

[0655] Specific operation: The server processes the image data of the home screen it receives and identifies app icons such as "Phone" and "Message."

[0656] Step 3:

[0657] The device (smart glasses) receives the user's voice instructions through a microphone and transmits them to the server as voice data.

[0658] Input: User's voice command

[0659] Data processing: Collection of voice data

[0660] Output: Audio data sent to the server

[0661] Specific operation: When a user says "I want to make a call," the smart glasses pick up the voice and send the voice data to the server.

[0662] Step 4:

[0663] The server converts the received voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[0664] Input: Audio data sent from the device (smart glasses)

[0665] Data processing: Converting voice data into text using voice recognition technology

[0666] Output: Text data

[0667] Specific operation: The server converts the received voice data into text using Google Cloud Speech-to-Text. The text obtained is "I want to make a phone call."

[0668] Step 5:

[0669] The server inputs the text data into a generative AI model (such as OpenAI's GPT-4) to analyze the user's intent.

[0670] Input: Text data converted from audio data

[0671] Data processing: Analysis and intent identification with generative AI models

[0672] Output: Instructions based on the user's intent (e.g., "Open the Phone app")

[0673] Specific action: The generative AI model analyzes the instruction "I want to make a call" into the specific action of "opening the phone app."

[0674] Step 6:

[0675] The server generates specific operation guides based on the analysis results and sends them to the device (smart glasses).

[0676] Input: Operation instructions based on the user's intention

[0677] Data processing: Generate operation guides, including visual instructions and highlights

[0678] Output: Generated instructions

[0679] Specific operation: The server generates an operation guide saying "Tap the phone app icon" and instructs the phone app icon to be highlighted in a red frame.

[0680] Step 7:

[0681] The device (smart glasses) displays the generated operation guide on its own display, providing the user with clear operating instructions.

[0682] Input: Operation guide sent from the server

[0683] Data processing: Display of operation guide

[0684] Output: Operation guide displayed on the screen

[0685] Specific operation: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the icon will be highlighted in red. The user should follow the instructions and tap the phone app icon.

[0686] Through the above steps, this system provides an environment in which even users who are unfamiliar with operating information devices can operate the system easily and intuitively.

[0687] (Application example 1)

[0688] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0689] Conventional smart device operation support systems have problems, such as complicated operation and difficulty in navigating and obtaining product information, especially for elderly users and users unfamiliar with device operation. This has prevented users from using the device effectively and has caused inconvenience. In addition, there has been a lack of support for checking product information in physical stores and obtaining location information within the store, which has reduced user convenience.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0691] In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a video output means for displaying the generated operation guide on the display device; and a display means for the user to obtain in-store product information and a store map. This allows the user to intuitively operate the smart device, making it easier to navigate in a physical store and obtain product information.

[0692] "Smart devices" is a general term for portable electronic devices such as mobile phones, tablets, e-book readers, and smartwatches that can communicate over a network and run a variety of applications.

[0693] A "display device" is a device worn by a user that outputs visual information, and examples of this include head-mounted displays and smart glasses.

[0694] The "photographing means" is a camera mounted on the display device, and is a device that has the function of capturing an image from the user's viewpoint.

[0695] "Communication means" refers to a network connection device for relaying captured image and audio data to a server, and includes wireless and wired communication.

[0696] "Image analysis means" refers to technology that identifies specific elements from received image data, and uses optical character recognition (OCR) or machine learning algorithms.

[0697] The "voice input means" is a device for collecting the user's voice and transmitting it to the server as voice data.

[0698] "Generative AI means" refers to an AI technology that analyzes the user's intentions based on voice input and image analysis data, and generates the next operation guide.

[0699] The "video output means" is a device for visually outputting the generated operation guide to a display device.

[0700] The "display means" is a device that provides visual information for the user to obtain in-store product information and a store map.

[0701] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[0702] System Configuration

[0703] This system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence. It also includes a display means that allows the user to obtain in-store product information and a store map.

[0704] Capture and send images

[0705] The smart glasses, which are the terminals, capture the screen of the smart device from the user's point of view. The captured image is immediately sent to the server via a communication means. For example, if a user is looking at a product shelf in a store, the camera in the smart glasses captures the image and sends it to the server.

[0706] Image Recognition

[0707] The server analyzes the received image data and identifies the user interface elements displayed on the smart device screen. For example, in a store, the server analyzes the images of product shelves and recognizes the barcodes and labels of each product.

[0708] Voice input reception and question analysis

[0709] The smart glasses, which act as a terminal, receive voice commands from the user via a microphone. This voice data is sent to a server via a voice recognition system. The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent. For example, if a user says, "I want to see a list of special sale items," the voice data is sent to the server, converted into text, and the generative artificial intelligence interprets it as the instruction to "display a list of special sale items."

[0710] Response Generation

[0711] Based on the voice instructions and image recognition results, the server determines the next action the user should take and generates an operation guide that includes clear visual instructions, such as "Please scan the barcode."

[0712] Guide Display

[0713] The smart glasses, which are the terminals, display the generated operation guide on their screens, allowing the user to receive clear instructions and perform the next operation. For example, the operation guide is displayed on the smart glasses' display, highlighting the product barcode with a red frame.

[0714] Specific examples

[0715] Example 1: In-store navigation

[0716] 1. A user wears smart glasses and walks around a store.

[0717] 2. The camera in the smart glasses captures the inside of the store and sends the images to the server.

[0718] 3. The server performs image recognition to identify the shelves and the barcodes and labels of each product.

[0719] 4. The user speaks into the microphone on the smart glasses, "Where is the detergent shelf?"

[0720] 5. The audio data is sent to the server and converted into text.

[0721] 6. The generative AI analyzes the instruction "Show the location of the detergent shelf."

[0722] 7. The server generates a navigation guide and shows the desired shelf location on a store map.

[0723] 8. Navigation guides will be displayed on the smart glasses display, allowing users to check the route to their destination.

[0724] Prompt Sentence Examples

[0725] Prompt to find sale items:

[0726] "The user says, 'I want to see a list of special offers,' and the system converts this into text and inputs the instruction 'Show me a list of special offers' as a prompt to the generative AI."

[0727] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0728] System program processing flow

[0729] Step 1:

[0730] The camera in the smart glasses captures images from the user's point of view. Specifically, the image capture means acquires images while the user is wearing the smart glasses and looking at the screen of a smart device or a product shelf in a store.

[0731] Input: Image from the user's perspective

[0732] Output: Captured image data

[0733] Step 2:

[0734] The captured image data is sent to the server via a communication means. Internally, the smart glasses use wireless communication to send image data to the server in real time.

[0735] Input: Captured image data

[0736] Output: Image data sent to the server

[0737] Step 3:

[0738] The server analyzes the received image data and uses image analysis methods to identify user interface elements displayed on the smart device screen or on the store shelf, and extracts information such as barcodes and labels using OCR technology and object recognition algorithms.

[0739] Input: Image data sent to the server

[0740] Output: Parsed interface elements (e.g. barcodes, labels)

[0741] Step 4:

[0742] The smart glasses' microphone accepts the user's voice commands, allowing the user to use voice input to request specific actions or information.

[0743] Input: User's voice command

[0744] Output: Audio data

[0745] Step 5:

[0746] The voice data is sent to the server via a communication means. The smart glasses send the voice data to the server in real time.

[0747] Input: Audio data

[0748] Output: Audio data sent to the server

[0749] Step 6:

[0750] The server converts the voice data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text).

[0751] Input: Audio data sent to the server

[0752] Output: Text data

[0753] Step 7:

[0754] Generative AI analyzes text data to understand user intent. In this process, the generative AI model generates appropriate responses and operational guidance based on the user's questions and requests.

[0755] Input: Text data

[0756] Output: Appropriate response or action guide (text)

[0757] Step 8:

[0758] Based on the analysis results, the server determines the next action the user should take and generates a visual operation guide, such as "Please scan the barcode."

[0759] Input: Appropriate response or instruction (text)

[0760] Output: Visual operation guide

[0761] Step 9:

[0762] The generated operation guide is sent to the smart glasses display via a communication method and displayed. The user visually confirms it and performs the next operation.

[0763] Input: Visual guide

[0764] Output: Guide displayed on the smart glasses display

[0765] Example: In-store navigation processing steps

[0766] Step 1:

[0767] As a user wears the smart glasses and walks around the store, the camera in the smart glasses captures images of the store's shelves.

[0768] Input: Image of the interior from the user's perspective

[0769] Output: Captured in-store image data

[0770] Step 2:

[0771] The captured in-store image data is sent from the smart glasses to a server.

[0772] Input: Captured in-store image data

[0773] Output: In-store image data sent to the server

[0774] Step 3:

[0775] The server analyzes the in-store image data transmitted and uses image analysis means to identify the barcodes and labels of each product.

[0776] Input: In-store image data sent to the server

[0777] Output: Identified barcode or label

[0778] Step 4:

[0779] The user speaks into the microphone on the smart glasses, asking, "Where is the detergent shelf?"

[0780] Input: User's voice command

[0781] Output: Audio data

[0782] Step 5:

[0783] The audio data is sent from the smart glasses to a server.

[0784] Input: Audio data

[0785] Output: Audio data sent to the server

[0786] Step 6:

[0787] The server converts the audio data into text.

[0788] Input: Audio data sent to the server

[0789] Output: Data converted to text

[0790] Step 7:

[0791] Generative artificial intelligence analyzes text data and generates optimal responses to user questions.

[0792] Input: Data converted to text

[0793] output: Text that serves as a navigation guide

[0794] Step 8:

[0795] Based on the analysis results, the server generates a navigation guide showing the location of shelves within the store.

[0796] Input: Text to serve as a navigation guide

[0797] Output: A visual navigation guide

[0798] Step 9:

[0799] The generated navigation guide is sent to the smartglasses display and displayed to the user.

[0800] Input: Visual navigation guide

[0801] Output: Navigation guide displayed on the smart glasses display

[0802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0803] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses, generative artificial intelligence, and an emotion engine.

[0804] System Configuration

[0805] This system consists of smart glasses, a server, and an emotion engine. The smart glasses are equipped with a camera, a voice input device, a display, and emotion recognition sensors, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[0806] 1. Capture and send images

[0807] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[0808] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[0809] 2. Image Recognition

[0810] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[0811] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[0812] 3. Voice input reception and question analysis

[0813] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[0814] The server analyzes the voice data and converts it into text using voice recognition technology, and then uses generative artificial intelligence to analyze the user's intent.

[0815] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[0816] 4. Emotion Data Acquisition and Analysis

[0817] The smart glasses (devices) use emotion recognition sensors to obtain emotions from the user's voice and facial expressions, including voice tone analysis and facial expression recognition technology.

[0818] The server analyzes the acquired emotional data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[0819] Example: If a user becomes anxious while operating the device, the sensors in the smart glasses will detect the user's stress level from facial expressions and voice tone and send that information to the server.

[0820] 5. Response Generation

[0821] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[0822] Example: If the user is in a hurry, generate and display a more detailed and easy-to-understand operation guide.

[0823] 6. Guide display

[0824] The smart glasses (device) displays the generated operation guide on the display, emphasizing the operation points by, for example, increasing the text size or adding animation depending on the user's emotional state.

[0825] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[0826] Specific examples

[0827] Example 1: Making a phone call (when the user is in a hurry)

[0828] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0829] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0830] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0831] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0832] 5. The audio data is sent to the server and converted into text.

[0833] 6. Generative AI analyzes the instruction "Open the phone app."

[0834] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[0835] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[0836] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0837] 10. The user follows the instructions and taps the Phone app icon.

[0838] In this way, the system improves the user's operating experience by providing real-time support for smart device operation while taking into account the user's emotional state.

[0839] The processing flow will be explained below.

[0840] Step 1:

[0841] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user is looking at the home screen of their smartphone, the camera on the smart glasses captures the screen as image data.

[0842] Step 2:

[0843] The smart glasses (devices) transmit the captured image data to a server via a network connection, where the image data is stored and ready for analysis.

[0844] Step 3:

[0845] The server analyzes the received image data and identifies user interface elements (such as app icons and buttons) on the smart device screen using image recognition technology (such as OCR), and also obtains the location information of the elements.

[0846] Step 4:

[0847] The smart glasses (device) receive voice commands from the user through a microphone. When the user says something like "I want to make a phone call," the smart glasses capture the voice data and send it to a server via the network.

[0848] Step 5:

[0849] The server converts the received voice data into text using speech recognition technology. Using this text data, generative AI analyzes the user's intent. For example, the instruction "I want to make a call" is interpreted as the operation of opening the phone app.

[0850] Step 6:

[0851] The smart glasses (device) uses an emotion recognition sensor to recognize the user's emotions and acquires emotion data from the user's voice and facial expressions. This data is based on voice tone analysis and facial expression recognition technology.

[0852] Step 7:

[0853] The server analyzes the acquired emotional data and recognizes the user's emotional state, for example, identifying that the user is experiencing stress.

[0854] Step 8:

[0855] The server generates operation guidance based on the user's voice instructions and emotional data. It adjusts the way the operation guidance is presented depending on the user's emotional state. For example, if the user is in a hurry, it adjusts the operation guidance to be more thorough and easy to understand.

[0856] Step 9:

[0857] The smart glasses (device) displays the generated operation guide on the display. For example, the generated operation guide may be a detailed instruction such as "Tap the phone app icon," and the icon will be highlighted in a red frame.

[0858] Step 10:

[0859] The user operates the smart device according to the instructions of the smart glasses. For example, the user looks at the operation guide displayed on the smart glasses and taps the icon of the specified app.

[0860] Step 11:

[0861] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[0862] Example 2

[0863] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] Modern smart devices are extremely multifunctional, but their operation is complex and difficult to use, especially for elderly people and users who are unfamiliar with smart device operation. For this reason, there is a demand for intuitive operation and detailed operational support that responds to the user's emotional state.

[0865] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice acquisition means for receiving the user's voice instructions; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; an emotion recognition means for acquiring emotional data from the user's voice and facial expressions; an emotion analysis means for analyzing the acquired emotional data and recognizing the user's emotional state; a response generation means for adjusting and generating a presentation method of an operation guide according to the user's emotional state; and a display means for displaying the generated operation guide on the display device. This allows the user to easily operate the smart device, improving the operation experience.

[0866] "User" refers to a person who wears a device such as smart glasses and receives assistance in operating the smart device.

[0867] A "display device" refers to a device worn by a user that displays information in the field of view.

[0868] "Capture means" refers to equipment such as cameras and sensors used to capture the screen of a smart device from the user's perspective.

[0869] "Communication means" refers to the network function for transmitting captured image and audio data to a server.

[0870] "Image analysis means" refers to software or algorithms used to identify user interface elements on the screen of a smart device from received image data.

[0871] "Voice acquisition means" refers to a microphone or voice input device for receiving voice instructions from the user.

[0872] "Generative artificial intelligence means" refers to an artificial intelligence system for analyzing voice instructions and generating next operation guidance based on user interface elements.

[0873] "Emotion recognition means" refers to sensors and analysis software for acquiring emotional data from the user's voice and facial expressions.

[0874] "Emotion analysis means" refers to software or algorithms that analyze acquired emotional data and recognize the user's emotional state.

[0875] "Response generation means" refers to software or algorithms that generate and adjust the way in which operation guidance is presented depending on the user's emotional state.

[0876] The "display means" refers to a display device for visually displaying the generated operation guide to the user.

[0877] The following describes in detail an embodiment of the present invention. This system uses smart glasses, generative artificial intelligence, and an emotion engine to provide an environment in which elderly people and users unfamiliar with operating smart devices can easily operate smart devices. Specifically, this system is composed of smart glasses, a server, and an emotion engine.

[0878] Hardware and Software Configuration

[0879] Smart glasses (device)

[0880] The smart glasses incorporate a camera, a voice input device, a display, and an emotion recognition sensor. The camera captures the screen of the smart device from the user's point of view, the voice input device receives the user's voice instructions, the display shows the generated operation guide to the user, and the emotion recognition sensor captures the user's emotional state from their voice and facial expressions.

[0881] server

[0882] The server runs various software for image recognition, generative AI, and sentiment analysis. For image recognition, it uses OpenCV and Tesseract OCR, for speech recognition it uses Google Cloud Speech-to-Text, for generative AI it uses GPT-3 and ChatGPT, and for sentiment analysis it uses Affectiva and IBM Watson Emotion Analysis.

[0883] Specific example of system operation

[0884] 1. Capture and image transmission: The camera in the smart glasses captures the screen of the smart device from the user's point of view and transmits the image data to the server via Wi-Fi or Bluetooth.

[0885] 2. Image Recognition: The server analyzes the image data received and identifies user interface elements on the smart device screen, such as app icons like "Phone" and "Message" displayed on the home screen.

[0886] 3. Voice input acceptance and question analysis: When a user says "I want to make a phone call" into the microphone of the smart glasses, the voice data is sent to the server. The server converts the voice data into text, and the generative AI analyzes the instruction to "open the phone app."

[0887] 4. Emotion data acquisition and analysis: The smart glasses use emotion recognition sensors to acquire emotional data from the user's voice and facial expressions, and then send the data to a server, which then uses emotion analysis software to analyze the user's emotional state.

[0888] 5. Response generation: The server generates operation guidance based on the user's voice instructions and emotional data. For example, if the user is in a hurry, it generates a polite guidance such as "Please tap the phone app icon a little more slowly."

[0889] 6. Guide display: The generated operation guide is displayed on the smart glasses display, and the font size and highlighting are adjusted as needed.

[0890] Specific examples

[0891] Example 1: Making a phone call (when the user is in a hurry)

[0892] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[0893] 2. The smart glasses camera captures the home screen and sends the image to the server.

[0894] 3. The server performs image recognition to identify the user interface elements on the home screen.

[0895] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[0896] 5. The audio data is sent to the server and converted into text.

[0897] 6. Generative AI analyzes the instruction "Open the phone app."

[0898] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[0899] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[0900] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[0901] 10. The user follows the instructions and taps the Phone app icon.

[0902] This system can improve the user's operating experience by providing real-time support for operating smart devices while taking into account the user's emotional state. In this way, by combining generative AI and emotion recognition technology, it is possible to provide optimal operating guidance to users.

[0903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0904] Step 1:

[0905] The terminal (smart glasses) captures the screen of the smart device from the user's point of view. Specifically, the camera captures the home screen of the smartphone. The input at this time is the screen of the smart device that the user is looking at, and the output is the captured image data.

[0906] Step 2:

[0907] The device (smart glasses) transmits the captured image data to the server via Wi-Fi or Bluetooth. The input is the captured image data, and the output is the image data transferred to the server.

[0908] Step 3:

[0909] The server analyzes the received image data using image analysis software (e.g., OpenCV, Tesseract OCR). The input is the image data transferred to the server, and the output is identification data for on-screen user interface elements (e.g., app icons and buttons). Specifically, the position and label of each icon are identified.

[0910] Step 4:

[0911] The user issues a voice command, such as "I want to make a call," into the microphone of the smart glasses. The input is the user's voice command, and the output is voice data.

[0912] Step 5:

[0913] The device (smart glasses) captures voice data and transmits it to the server via Wi-Fi or Bluetooth. The input is the user's voice data, and the output is the voice data transmitted to the server.

[0914] Step 6:

[0915] The server converts the voice data into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The input is the transferred voice data, and the output is text data. Specifically, the voice data "I want to make a call" is converted into the text "I want to make a call."

[0916] Step 7:

[0917] The server uses generative artificial intelligence (e.g., GPT-3, ChatGPT) to analyze user intent from text data. The input is text data, and the output is the analyzed user intent (e.g., "Open the phone app").

[0918] Step 8:

[0919] The device (smart glasses) uses an emotion recognition sensor to acquire emotional data from the user's voice and facial expressions. The input is the user's voice and facial expressions, and the output is the acquired emotional data. Specifically, the sensor measures the user's voice tone and facial expressions in real time.

[0920] Step 9:

[0921] The emotion data acquired by the device (smart glasses) is sent to the server via Wi-Fi or Bluetooth. The input is the acquired emotion data, and the output is the emotion data transferred to the server.

[0922] Step 10:

[0923] The server uses emotion analysis software (e.g., Affectiva, IBM Watson Emotion Analysis) to analyze the acquired emotion data and identify the user's emotional state, where the input is the transferred emotion data and the output is the user's emotional state (e.g., stress, irritability).

[0924] Step 11:

[0925] The server generates operation guidance based on the user's voice instructions and emotional data. The input is the analyzed user's intention and emotional state, and the output is the operation guidance. For example, if the user is in a hurry, a polite operation guidance such as "Please tap the phone app icon a little more slowly" is generated.

[0926] Step 12:

[0927] The operation guide generated by the server is sent to the device (smart glasses) via Wi-Fi or Bluetooth. The input at this time is the operation guide, and the output is the operation guide transferred to the smart glasses.

[0928] Step 13:

[0929] The device (smart glasses) displays the operation guide on the display. The input is the transferred operation guide, and the output is the displayed operation guide. Specifically, visual feedback is provided, such as the phone app icon being highlighted in a red frame.

[0930] (Application example 2)

[0931] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0932] Providing an environment where elderly people and users unfamiliar with operating smart devices can easily obtain and use information in the real world is an important challenge in modern society. In particular, obtaining product information and supporting purchasing in physical stores requires an intuitive and adaptive interface. However, conventional systems have difficulty responding while fully considering the user's emotions and operating situation. Therefore, there is a need to develop a system that provides appropriate feedback according to the user's emotional state and supports operation.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0934] In this invention, the server is a display device worn by a user, and includes: a camera means for capturing the screen of the smart device from the user's viewpoint; a network means for receiving the captured screen image; an image recognition means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a display means for displaying the generated operation guide on the display device; an emotion recognition means for acquiring the user's emotional state; and an emotion analysis means for adjusting the content of the operation guide based on the emotional state, thereby enabling adaptive and intuitive feedback according to the user's emotional state.

[0935] A "display device" is a device worn by a user to provide visual feedback.

[0936] "Camera means" refers to a camera attached to a display device to capture the screen of the smart device from the user's point of view.

[0937] "Network means" refers to the communications infrastructure used to receive the captured screen image.

[0938] "Image recognition means" refers to a technology that identifies user interface elements on the screen of a smart device from a received image.

[0939] The "voice input means" is a device such as a microphone that receives voice instructions from the user.

[0940] "Generative artificial intelligence means" refers to AI technology that analyzes voice instructions and generates the next operation guide based on the user interface elements.

[0941] The "display means" is a screen for displaying the generated operation guide on a display device.

[0942] An "emotion recognition means" is a sensor or software for acquiring the user's emotional state.

[0943] The "emotion analysis means" is a technique for adjusting the content of operation guidance based on the acquired emotional state.

[0944] MODE FOR CARRYING OUT THE INVENTION

[0945] System Configuration

[0946] To implement this invention, a display device (smart glasses) worn by a user, a server, and emotion recognition means are required. Specifically, the smart glasses incorporate a camera means, a voice input means, a display means, and an emotion recognition sensor, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[0947] Specific examples of program processing

[0948] Capture and send images

[0949] The smart glasses use a camera means to capture the screen of the smart device from the user's point of view, and the captured image is immediately transmitted to a server via a network means.

[0950] Image Recognition

[0951] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR technology).

[0952] Voice input reception and question analysis

[0953] The smart glasses receive voice commands from the user via a microphone. When the user issues a voice command, such as "I want to search for a specific product," the voice data is captured and sent to a server. The server then analyzes the voice data and converts it into text. Generative AI is used to convert the voice data into text and analyze the user's intent.

[0954] Emotion data acquisition and analysis

[0955] The smart glasses use emotion recognition sensors to capture emotions from the user's voice and facial expressions. This includes voice tone analysis and facial expression recognition technology. The server analyzes the captured emotion data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[0956] Response Generation

[0957] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[0958] Guide Display

[0959] The smart glasses display the generated operation guide on the screen, emphasizing the operation points by, for example, increasing the text size or adding animations depending on the user's emotional state.

[0960] Hardware and software used

[0961] Hardware: Smart glasses (camera, microphone, display, emotion recognition sensor), server

[0962] Software: OpenCV (camera image processing), pytesseract (OCR text recognition), speech_recognition (voice recognition), requests (API communication), emotion_recognition (emotion recognition)

[0963] Specific examples

[0964] Here is a specific example of a case where smart glasses are used to read the QR code of a product that a user shows interest in at a physical store. The camera means of the smart glasses captures the QR code, and the image recognition means extracts the product code. Using this product code, the server obtains detailed information about the product. If the user issues a voice command such as "I want to find a cheaper product," the command is analyzed by generative artificial intelligence, and information about cheaper products is provided by voice. If the user becomes impatient while operating the device, the emotion recognition means detects this, and the server provides adaptive feedback such as "Please stay calm."

[0965] Prompt Sentence Examples

[0966] "If a user wants to scan a QR code and get the product price and recommended points from a voice assistant, how do you provide that feedback?"

[0967] In this way, users will receive smart product information and adaptive feedback to improve their real-world shopping experience.

[0968] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0969] Step 1:

[0970] The smart glasses (terminal) use a camera to capture the screen of the smart device as seen from the user's point of view.

[0971] Input: The screen of the smart device that the user is looking at while wearing the smart glasses.

[0972] Output: The captured screen image.

[0973] Specific operation: The camera continuously takes images and generates image data.

[0974] Step 2:

[0975] The smart glasses (terminal) transmit the captured screen image to the server via the network.

[0976] Input: A captured screen image.

[0977] Output: Screen image sent to the server.

[0978] Specific operation: The captured image data is sent to the server using a network means such as Wi-Fi or Bluetooth.

[0979] Step 3:

[0980] The server analyzes the received image data and uses image recognition technology to identify user interface elements on the screen of the smart device.

[0981] Input: The submitted screen image.

[0982] Output: Identified user interface elements (e.g. app icons and buttons).

[0983] Specific operation: Image recognition software on the server (e.g., OpenCV or pytesseract) analyzes the image and extracts specific elements.

[0984] Step 4:

[0985] The smart glasses (device) accept the user's voice instructions via a microphone and capture the voice data.

[0986] Input: User's spoken command (e.g., "I would like to search for a specific product").

[0987] Output: The captured audio data.

[0988] How it works: The microphone built into the smart glasses records the user's voice and generates digital audio data.

[0989] Step 5:

[0990] The smart glasses (terminals) transmit the captured audio data to a server via a network.

[0991] Input: Captured audio data.

[0992] Output: The audio data sent to the server.

[0993] Specific operation: Audio data is transmitted to a server via a network means.

[0994] Step 6:

[0995] The server analyzes the transmitted voice data and converts it into text using voice recognition technology.

[0996] Input: The audio data sent to the server.

[0997] Output: Text data (voice instructions converted into text).

[0998] What happens: The server's speech recognition software (e.g., speech_recognition) processes the audio data and generates corresponding text data.

[0999] Step 7:

[1000] The server uses generative artificial intelligence to analyze the user's voice instructions and generate operation guides.

[1001] Input: Text data and identified user interface elements.

[1002] Output: The generated how-to guide.

[1003] What it does: Generative AI (e.g., a machine learning model) analyzes text data and generates guidelines for the actions the user wants to take.

[1004] Step 8:

[1005] The smart glasses (device) collect emotional data from the user's voice and facial expressions.

[1006] Input: The user's tone of voice and facial expressions.

[1007] Output: The obtained emotion data.

[1008] Specific operation: The emotion recognition sensor analyzes the user's tone of voice and facial expressions, and digitizes their emotional state.

[1009] Step 9:

[1010] The server analyzes the acquired emotional data and recognizes the emotional state of the user.

[1011] Input: The acquired emotion data.

[1012] Output: Parsed emotional state (e.g. stress state).

[1013] What it does: The emotion recognition engine analyzes the emotion data and identifies the user's emotional state.

[1014] Step 10:

[1015] The server adjusts the content of the operation guide based on the emotional state of the user.

[1016] Input: Parsed emotional state and generated interaction guide.

[1017] Output: Adjusted operation guide.

[1018] Specific operation: Generative AI incorporates emotional data and modifies the operation guide to suit the user.

[1019] Step 11:

[1020] The smart glasses (terminal) will display adjusted operation guides on the display.

[1021] Enter: the adjusted operating guide.

[1022] Output: Operation guide displayed on the screen.

[1023] Specific operation: The smart glasses display visually displays operation guides and draws the user's attention by adjusting the font size and color as needed.

[1024] Through these steps, users will be able to intuitively and adaptively operate smart devices and product information.

[1025] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1026] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1027] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1028] [Third embodiment]

[1029] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1030] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1032] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1033] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1034] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1036] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1037] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1039] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1040] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1041] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[1042] System Configuration

[1043] The system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence.

[1044] 1. Capture and send images

[1045] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[1046] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[1047] 2. Image Recognition

[1048] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[1049] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[1050] 3. Voice input reception and question analysis

[1051] The smart glasses (device) receives user voice commands through a microphone, and this voice data is sent to a server via a voice recognition system.

[1052] The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent.

[1053] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[1054] 4. Response Generation

[1055] The server determines the next action the user should take based on the voice instructions and the results of image recognition, and generates an operation guide that includes clear visual instructions.

[1056] Example: The generated instruction guide is "Tap the Phone app icon."

[1057] 5. Guide display

[1058] The smart glasses (device) displays the generated operation guide on the display, allowing the user to receive clear instructions and perform the next operation.

[1059] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[1060] Specific examples

[1061] Example 1: Making a phone call

[1062] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1063] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1064] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1065] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1066] 5. The audio data is sent to the server and converted into text.

[1067] 6. Generative AI analyzes the instruction "Open the phone app."

[1068] 7. The server generates an operation guide saying "Tap the phone app icon."

[1069] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1070] 9. The user follows the instructions and taps the Phone app icon.

[1071] In this way, this system helps users who are unfamiliar with operating smart devices to operate them easily and intuitively.

[1072] The processing flow will be explained below.

[1073] Step 1:

[1074] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user looks at their smartphone, the camera in the smart glasses captures the screen and saves it as image data.

[1075] Step 2:

[1076] The smart glasses (device) sends the captured image data to the server via a network connection, where it is prepared for analysis.

[1077] Step 3:

[1078] The server analyzes the received image data and identifies interface elements on the smart device screen. It uses image recognition technology (such as OCR) to detect text, buttons, icons, etc., and obtains their location information.

[1079] Step 4:

[1080] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[1081] Step 5:

[1082] The server analyzes the voice data and converts it into text. The voice data is converted into text using voice recognition technology, and the user's intent is analyzed using generative artificial intelligence.

[1083] Step 6:

[1084] The server generates an operation guide based on the analysis results. It combines the user's voice instructions and the image recognition results to generate the next operation guide (e.g., open the phone app).

[1085] Step 7:

[1086] The smart glasses (device) displays the generated operation guide on the display, visually highlighting the operation points and providing instructions that are easy for the user to understand.

[1087] Step 8:

[1088] The user operates the smart device according to the instructions on the smart glasses, for example, by tapping a specific icon according to the instructions displayed on the smart glasses.

[1089] Step 9:

[1090] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[1091] Example 1

[1092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1093] The operation of modern information devices is becoming more complex, and there is a demand for an environment that allows elderly people and users who are unfamiliar with operating smart devices to operate them easily. In particular, many information devices have numerous icons and buttons, which can be difficult for first-time users to understand and operate. In addition, there is a need for a system that allows for more intuitive and reliable operation by simultaneously providing visual and auditory support.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1095] In this invention, the server includes a communication means for receiving the captured screen image, an image recognition means for identifying user interface elements on the screen of the information device from the received image, and a generative artificial intelligence means for analyzing the voice instruction and generating the next operation guide based on the user interface elements. This allows even a user who is unfamiliar with operating the information device to operate it intuitively and reliably while receiving visual and voice support.

[1096] "User" refers to a person who operates an information device and uses a device that supports the operation from that point of view.

[1097] "Display device" refers to a device worn by a user that has a display function for providing visual information.

[1098] "Photographing means" means a camera or photographing device for capturing the screen of an information device from the user's point of view.

[1099] "Communication means" refers to the network technology used to transmit the captured screen image to the server, including Wi-Fi, Bluetooth, etc.

[1100] "Image recognition means" refers to technology for identifying user interface elements on the screen of an information device from images received by the server, and includes, in particular, pattern recognition algorithms and OCR technology.

[1101] The "voice input means" refers to a microphone or voice collection device for receiving voice instructions from the user, and is a means for acquiring voice data.

[1102] "Generative artificial intelligence means" refers to an artificial intelligence technology for analyzing voice instructions and generating next operation guidance based on user interface elements.

[1103] The "display means" refers to a component or device having a display function for visually presenting the generated operation guide to the user.

[1104] "Operation guide" refers to guide information that clearly indicates to the user the next operation to be performed and includes instructions that visually emphasize the operation.

[1105] "Information devices" refer to electronic devices operated by users, such as smartphones and tablets, which have many app icons and buttons.

[1106] This invention is a system that supports the operation of information devices by combining a display device worn by the user with a server. It uses smart glasses and generative artificial intelligence to enable users to easily operate information devices that they are unfamiliar with.

[1107] System Configuration

[1108] The system mainly consists of the following components:

[1109] Terminal (smart glasses): Worn by the user, it includes a camera means for capturing the screen of the information device from the user's viewpoint, a voice input means for picking up voice, and a display means for displaying the generated operation guide.

[1110] Server: Includes a communication means for receiving screen images, an image recognition means for recognizing the received images, and a generative artificial intelligence means for analyzing the user's voice instructions.

[1111] Specific actions

[1112] 1. Image capture and transmission:

[1113] The device (smart glasses) uses a camera to capture the screen of the information device from the user's perspective and transmits the image to a server via Wi-Fi or Bluetooth.

[1114] Example: When a user puts on smart glasses while viewing the home screen of their smartphone, the camera automatically captures the home screen and sends the image data to the server.

[1115] 2. Image Recognition:

[1116] The server analyzes the received image data and uses image recognition technology such as Google Cloud Vision API to identify on-screen user interface elements (app icons and buttons).

[1117] Example: The server analyzes the image data of the home screen received and identifies app icons such as "Phone" and "Message."

[1118] 3. Voice input acceptance and conversion:

[1119] The terminal (smart glasses) picks up the user's instructions using a voice input means and transmits them to a server via a communication means.

[1120] The server converts the received voice data into text using Google Cloud Speech-to-Text technology.

[1121] For example, if a user says "I want to make a call," the smart glasses will pick up the voice and send it to the server, which will convert the voice data into text.

[1122] 4. Intent Analysis:

[1123] The server inputs the converted text data into a generative AI model (such as OpenAI's GPT-4), analyzes the user's intent, and identifies appropriate operating procedures.

[1124] Example: A generative AI model analyzes the instruction "I want to make a call" into the specific operation "Open the phone app."

[1125] 5. Response Generation:

[1126] Based on the analysis results, the server generates an operation guide for the user, which includes specific instructions and visual highlights.

[1127] Example: Generate an action guide that says "Tap the Phone app icon" with instructions to highlight the Phone app icon in a red border.

[1128] 6. Guide display:

[1129] The terminal (smart glasses) visually presents the generated operation guide to the user on a display means.

[1130] Example: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the phone app icon will be highlighted in a red frame. The user will follow the instructions.

[1131] Specific examples and prompts

[1132] Example: A scenario for making a phone call

[1133] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1134] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1135] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1136] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1137] 5. The audio data is sent to the server and converted into text.

[1138] 6. The generative AI model analyzes the instruction "Open the phone app."

[1139] 7. The server generates an operation guide saying "Tap the phone app icon."

[1140] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1141] 9. The user follows the instructions and taps the Phone app icon.

[1142] Prompt Sentence Examples

[1143] "When a user says, 'I want to make a phone call,' generate a guide for the next operation based on the image captured by the smartglasses' camera."

[1144] As described above, the present invention provides a system that allows even a user who is unfamiliar with operating information devices to operate it easily and intuitively.

[1145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1146] Step 1:

[1147] The terminal (smart glasses) captures the screen of the information device from the user's point of view with a camera and transmits the image to a server via a network means.

[1148] Input: The screen of the information device that the user is looking at

[1149] Data processing: The camera captures the screen and generates image data

[1150] Output: Image data sent to the server

[1151] Specific operation: When a user looks at the home screen of their smartphone, the camera in the smart glasses captures the screen and sends the generated image data to a server via Wi-Fi or Bluetooth.

[1152] Step 2:

[1153] The server analyzes the received image data and identifies user interface elements using image recognition technology such as the Google Cloud Vision API.

[1154] Input: Screen image sent from the device (smart glasses)

[1155] Data processing: Apply image recognition algorithms (such as Google Cloud Vision API) to identify icons and buttons on the screen.

[1156] Output: A list of identified user interface elements

[1157] Specific operation: The server processes the image data of the home screen it receives and identifies app icons such as "Phone" and "Message."

[1158] Step 3:

[1159] The device (smart glasses) receives the user's voice instructions through a microphone and transmits them to the server as voice data.

[1160] Input: User's voice command

[1161] Data processing: Collection of voice data

[1162] Output: Audio data sent to the server

[1163] Specific operation: When a user says "I want to make a call," the smart glasses pick up the voice and send the voice data to the server.

[1164] Step 4:

[1165] The server converts the received voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[1166] Input: Audio data sent from the device (smart glasses)

[1167] Data processing: Converting voice data into text using voice recognition technology

[1168] Output: Text data

[1169] Specific operation: The server converts the received voice data into text using Google Cloud Speech-to-Text. The text obtained is "I want to make a phone call."

[1170] Step 5:

[1171] The server inputs the text data into a generative AI model (such as OpenAI's GPT-4) to analyze the user's intent.

[1172] Input: Text data converted from audio data

[1173] Data processing: Analysis and intent identification with generative AI models

[1174] Output: Instructions based on the user's intent (e.g., "Open the Phone app")

[1175] Specific action: The generative AI model analyzes the instruction "I want to make a call" into the specific action of "opening the phone app."

[1176] Step 6:

[1177] The server generates specific operation guides based on the analysis results and sends them to the device (smart glasses).

[1178] Input: Operation instructions based on the user's intention

[1179] Data processing: Generate operation guides, including visual instructions and highlights

[1180] Output: Generated instructions

[1181] Specific operation: The server generates an operation guide saying "Tap the phone app icon" and instructs the phone app icon to be highlighted in a red frame.

[1182] Step 7:

[1183] The device (smart glasses) displays the generated operation guide on its own display, providing the user with clear operating instructions.

[1184] Input: Operation guide sent from the server

[1185] Data processing: Display of operation guide

[1186] Output: Operation guide displayed on the screen

[1187] Specific operation: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the icon will be highlighted in red. The user should follow the instructions and tap the phone app icon.

[1188] Through the above steps, this system provides an environment in which even users who are unfamiliar with operating information devices can operate the system easily and intuitively.

[1189] (Application example 1)

[1190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] Conventional smart device operation support systems have problems, such as complicated operation and difficulty in navigating and obtaining product information, especially for elderly users and users unfamiliar with device operation. This has prevented users from using the device effectively and has caused inconvenience. In addition, there has been a lack of support for checking product information in physical stores and obtaining location information within the store, which has reduced user convenience.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1193] In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a video output means for displaying the generated operation guide on the display device; and a display means for the user to obtain in-store product information and a store map. This allows the user to intuitively operate the smart device, making it easier to navigate in a physical store and obtain product information.

[1194] "Smart devices" is a general term for portable electronic devices such as mobile phones, tablets, e-book readers, and smartwatches that can communicate over a network and run a variety of applications.

[1195] A "display device" is a device worn by a user that outputs visual information, and examples of this include head-mounted displays and smart glasses.

[1196] The "photographing means" is a camera mounted on the display device, and is a device that has the function of capturing an image from the user's viewpoint.

[1197] "Communication means" refers to a network connection device for relaying captured image and audio data to a server, and includes wireless and wired communication.

[1198] "Image analysis means" refers to technology that identifies specific elements from received image data, and uses optical character recognition (OCR) or machine learning algorithms.

[1199] The "voice input means" is a device for collecting the user's voice and transmitting it to the server as voice data.

[1200] "Generative AI means" refers to an AI technology that analyzes the user's intentions based on voice input and image analysis data, and generates the next operation guide.

[1201] The "video output means" is a device for visually outputting the generated operation guide to a display device.

[1202] The "display means" is a device that provides visual information for the user to obtain in-store product information and a store map.

[1203] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[1204] System Configuration

[1205] This system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence. It also includes a display means that allows the user to obtain in-store product information and a store map.

[1206] Capture and send images

[1207] The smart glasses, which are the terminals, capture the screen of the smart device from the user's point of view. The captured image is immediately sent to the server via a communication means. For example, if a user is looking at a product shelf in a store, the camera in the smart glasses captures the image and sends it to the server.

[1208] Image Recognition

[1209] The server analyzes the received image data and identifies the user interface elements displayed on the smart device screen. For example, in a store, the server analyzes the images of product shelves and recognizes the barcodes and labels of each product.

[1210] Voice input reception and question analysis

[1211] The smart glasses, which act as a terminal, receive voice commands from the user via a microphone. This voice data is sent to a server via a voice recognition system. The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent. For example, if a user says, "I want to see a list of special sale items," the voice data is sent to the server, converted into text, and the generative artificial intelligence interprets it as the instruction to "display a list of special sale items."

[1212] Response Generation

[1213] Based on the voice instructions and image recognition results, the server determines the next action the user should take and generates an operation guide that includes clear visual instructions, such as "Please scan the barcode."

[1214] Guide Display

[1215] The smart glasses, which are the terminals, display the generated operation guide on their screens, allowing the user to receive clear instructions and perform the next operation. For example, the operation guide is displayed on the smart glasses' display, highlighting the product barcode with a red frame.

[1216] Specific examples

[1217] Example 1: In-store navigation

[1218] 1. A user wears smart glasses and walks around a store.

[1219] 2. The camera in the smart glasses captures the inside of the store and sends the images to the server.

[1220] 3. The server performs image recognition to identify the shelves and the barcodes and labels of each product.

[1221] 4. The user speaks into the microphone on the smart glasses, "Where is the detergent shelf?"

[1222] 5. The audio data is sent to the server and converted into text.

[1223] 6. The generative AI analyzes the instruction "Show the location of the detergent shelf."

[1224] 7. The server generates a navigation guide and shows the desired shelf location on a store map.

[1225] 8. Navigation guides will be displayed on the smart glasses display, allowing users to check the route to their destination.

[1226] Prompt Sentence Examples

[1227] Prompt to find sale items:

[1228] "The user says, 'I want to see a list of special offers,' and the system converts this into text and inputs the instruction 'Show me a list of special offers' as a prompt to the generative AI."

[1229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1230] System program processing flow

[1231] Step 1:

[1232] The camera in the smart glasses captures images from the user's point of view. Specifically, the image capture means acquires images while the user is wearing the smart glasses and looking at the screen of a smart device or a product shelf in a store.

[1233] Input: Image from the user's perspective

[1234] Output: Captured image data

[1235] Step 2:

[1236] The captured image data is sent to the server via a communication means. Internally, the smart glasses use wireless communication to send image data to the server in real time.

[1237] Input: Captured image data

[1238] Output: Image data sent to the server

[1239] Step 3:

[1240] The server analyzes the received image data and uses image analysis methods to identify user interface elements displayed on the smart device screen or on the store shelf, and extracts information such as barcodes and labels using OCR technology and object recognition algorithms.

[1241] Input: Image data sent to the server

[1242] Output: Parsed interface elements (e.g. barcodes, labels)

[1243] Step 4:

[1244] The smart glasses' microphone accepts the user's voice commands, allowing the user to use voice input to request specific actions or information.

[1245] Input: User's voice command

[1246] Output: Audio data

[1247] Step 5:

[1248] The voice data is sent to the server via a communication means. The smart glasses send the voice data to the server in real time.

[1249] Input: Audio data

[1250] Output: Audio data sent to the server

[1251] Step 6:

[1252] The server converts the voice data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text).

[1253] Input: Audio data sent to the server

[1254] Output: Text data

[1255] Step 7:

[1256] Generative AI analyzes text data to understand user intent. In this process, the generative AI model generates appropriate responses and operational guidance based on the user's questions and requests.

[1257] Input: Text data

[1258] Output: Appropriate response or action guide (text)

[1259] Step 8:

[1260] Based on the analysis results, the server determines the next action the user should take and generates a visual operation guide, such as "Please scan the barcode."

[1261] Input: Appropriate response or instruction (text)

[1262] Output: Visual operation guide

[1263] Step 9:

[1264] The generated operation guide is sent to the smart glasses display via a communication method and displayed. The user visually confirms it and performs the next operation.

[1265] Input: Visual guide

[1266] Output: Guide displayed on the smart glasses display

[1267] Example: In-store navigation processing steps

[1268] Step 1:

[1269] As a user wears the smart glasses and walks around the store, the camera in the smart glasses captures images of the store's shelves.

[1270] Input: Image of the interior from the user's perspective

[1271] Output: Captured in-store image data

[1272] Step 2:

[1273] The captured in-store image data is sent from the smart glasses to a server.

[1274] Input: Captured in-store image data

[1275] Output: In-store image data sent to the server

[1276] Step 3:

[1277] The server analyzes the in-store image data transmitted and uses image analysis means to identify the barcodes and labels of each product.

[1278] Input: In-store image data sent to the server

[1279] Output: Identified barcode or label

[1280] Step 4:

[1281] The user speaks into the microphone on the smart glasses, asking, "Where is the detergent shelf?"

[1282] Input: User's voice command

[1283] Output: Audio data

[1284] Step 5:

[1285] The audio data is sent from the smart glasses to a server.

[1286] Input: Audio data

[1287] Output: Audio data sent to the server

[1288] Step 6:

[1289] The server converts the audio data into text.

[1290] Input: Audio data sent to the server

[1291] Output: Data converted to text

[1292] Step 7:

[1293] Generative artificial intelligence analyzes text data and generates optimal responses to user questions.

[1294] Input: Data converted to text

[1295] output: Text that serves as a navigation guide

[1296] Step 8:

[1297] Based on the analysis results, the server generates a navigation guide showing the location of shelves within the store.

[1298] Input: Text to serve as a navigation guide

[1299] Output: A visual navigation guide

[1300] Step 9:

[1301] The generated navigation guide is sent to the smartglasses display and displayed to the user.

[1302] Input: Visual navigation guide

[1303] Output: Navigation guide displayed on the smart glasses display

[1304] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1305] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses, generative artificial intelligence, and an emotion engine.

[1306] System Configuration

[1307] This system consists of smart glasses, a server, and an emotion engine. The smart glasses are equipped with a camera, a voice input device, a display, and emotion recognition sensors, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[1308] 1. Capture and send images

[1309] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[1310] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[1311] 2. Image Recognition

[1312] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[1313] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[1314] 3. Voice input reception and question analysis

[1315] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[1316] The server analyzes the voice data and converts it into text using voice recognition technology, and then uses generative artificial intelligence to analyze the user's intent.

[1317] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[1318] 4. Emotion Data Acquisition and Analysis

[1319] The smart glasses (devices) use emotion recognition sensors to obtain emotions from the user's voice and facial expressions, including voice tone analysis and facial expression recognition technology.

[1320] The server analyzes the acquired emotional data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[1321] Example: If a user becomes anxious while operating the device, the sensors in the smart glasses will detect the user's stress level from facial expressions and voice tone and send that information to the server.

[1322] 5. Response Generation

[1323] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[1324] Example: If the user is in a hurry, generate and display a more detailed and easy-to-understand operation guide.

[1325] 6. Guide display

[1326] The smart glasses (device) displays the generated operation guide on the display, emphasizing the operation points by, for example, increasing the text size or adding animation depending on the user's emotional state.

[1327] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[1328] Specific examples

[1329] Example 1: Making a phone call (when the user is in a hurry)

[1330] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1331] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1332] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1333] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1334] 5. The audio data is sent to the server and converted into text.

[1335] 6. Generative AI analyzes the instruction "Open the phone app."

[1336] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[1337] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[1338] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1339] 10. The user follows the instructions and taps the Phone app icon.

[1340] In this way, the system improves the user's operating experience by providing real-time support for smart device operation while taking into account the user's emotional state.

[1341] The processing flow will be explained below.

[1342] Step 1:

[1343] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user is looking at the home screen of their smartphone, the camera on the smart glasses captures the screen as image data.

[1344] Step 2:

[1345] The smart glasses (devices) transmit the captured image data to a server via a network connection, where the image data is stored and ready for analysis.

[1346] Step 3:

[1347] The server analyzes the received image data and identifies user interface elements (such as app icons and buttons) on the smart device screen using image recognition technology (such as OCR), and also obtains the location information of the elements.

[1348] Step 4:

[1349] The smart glasses (device) receive voice commands from the user through a microphone. When the user says something like "I want to make a phone call," the smart glasses capture the voice data and send it to a server via the network.

[1350] Step 5:

[1351] The server converts the received voice data into text using speech recognition technology. Using this text data, generative AI analyzes the user's intent. For example, the instruction "I want to make a call" is interpreted as the operation of opening the phone app.

[1352] Step 6:

[1353] The smart glasses (device) uses an emotion recognition sensor to recognize the user's emotions and acquires emotion data from the user's voice and facial expressions. This data is based on voice tone analysis and facial expression recognition technology.

[1354] Step 7:

[1355] The server analyzes the acquired emotional data and recognizes the user's emotional state, for example, identifying that the user is experiencing stress.

[1356] Step 8:

[1357] The server generates operation guidance based on the user's voice instructions and emotional data. It adjusts the way the operation guidance is presented depending on the user's emotional state. For example, if the user is in a hurry, it adjusts the operation guidance to be more thorough and easy to understand.

[1358] Step 9:

[1359] The smart glasses (device) displays the generated operation guide on the display. For example, the generated operation guide may be a detailed instruction such as "Tap the phone app icon," and the icon will be highlighted in a red frame.

[1360] Step 10:

[1361] The user operates the smart device according to the instructions of the smart glasses. For example, the user looks at the operation guide displayed on the smart glasses and taps the icon of the specified app.

[1362] Step 11:

[1363] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[1364] Example 2

[1365] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1366] Modern smart devices are extremely multifunctional, but their operation is complex and difficult to use, especially for elderly people and users who are unfamiliar with smart device operation. For this reason, there is a demand for intuitive operation and detailed operational support that responds to the user's emotional state.

[1367] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice acquisition means for receiving the user's voice instructions; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; an emotion recognition means for acquiring emotional data from the user's voice and facial expressions; an emotion analysis means for analyzing the acquired emotional data and recognizing the user's emotional state; a response generation means for adjusting and generating a presentation method of an operation guide according to the user's emotional state; and a display means for displaying the generated operation guide on the display device. This allows the user to easily operate the smart device, improving the operation experience.

[1368] "User" refers to a person who wears a device such as smart glasses and receives assistance in operating the smart device.

[1369] A "display device" refers to a device worn by a user that displays information in the field of view.

[1370] "Capture means" refers to equipment such as cameras and sensors used to capture the screen of a smart device from the user's perspective.

[1371] "Communication means" refers to the network function for transmitting captured image and audio data to a server.

[1372] "Image analysis means" refers to software or algorithms used to identify user interface elements on the screen of a smart device from received image data.

[1373] "Voice acquisition means" refers to a microphone or voice input device for receiving voice instructions from the user.

[1374] "Generative artificial intelligence means" refers to an artificial intelligence system for analyzing voice instructions and generating next operation guidance based on user interface elements.

[1375] "Emotion recognition means" refers to sensors and analysis software for acquiring emotional data from the user's voice and facial expressions.

[1376] "Emotion analysis means" refers to software or algorithms that analyze acquired emotional data and recognize the user's emotional state.

[1377] "Response generation means" refers to software or algorithms that generate and adjust the way in which operation guidance is presented depending on the user's emotional state.

[1378] The "display means" refers to a display device for visually displaying the generated operation guide to the user.

[1379] The following describes in detail an embodiment of the present invention. This system uses smart glasses, generative artificial intelligence, and an emotion engine to provide an environment in which elderly people and users unfamiliar with operating smart devices can easily operate smart devices. Specifically, this system is composed of smart glasses, a server, and an emotion engine.

[1380] Hardware and Software Configuration

[1381] Smart glasses (device)

[1382] The smart glasses incorporate a camera, a voice input device, a display, and an emotion recognition sensor. The camera captures the screen of the smart device from the user's point of view, the voice input device receives the user's voice instructions, the display shows the generated operation guide to the user, and the emotion recognition sensor captures the user's emotional state from their voice and facial expressions.

[1383] server

[1384] The server runs various software for image recognition, generative AI, and sentiment analysis. For image recognition, it uses OpenCV and Tesseract OCR, for speech recognition it uses Google Cloud Speech-to-Text, for generative AI it uses GPT-3 and ChatGPT, and for sentiment analysis it uses Affectiva and IBM Watson Emotion Analysis.

[1385] Specific example of system operation

[1386] 1. Capture and image transmission: The camera in the smart glasses captures the screen of the smart device from the user's point of view and transmits the image data to the server via Wi-Fi or Bluetooth.

[1387] 2. Image Recognition: The server analyzes the image data received and identifies user interface elements on the smart device screen, such as app icons like "Phone" and "Message" displayed on the home screen.

[1388] 3. Voice input acceptance and question analysis: When a user says "I want to make a phone call" into the microphone of the smart glasses, the voice data is sent to the server. The server converts the voice data into text, and the generative AI analyzes the instruction to "open the phone app."

[1389] 4. Emotion data acquisition and analysis: The smart glasses use emotion recognition sensors to acquire emotional data from the user's voice and facial expressions, and then send the data to a server, which then uses emotion analysis software to analyze the user's emotional state.

[1390] 5. Response generation: The server generates operation guidance based on the user's voice instructions and emotional data. For example, if the user is in a hurry, it generates a polite guidance such as "Please tap the phone app icon a little more slowly."

[1391] 6. Guide display: The generated operation guide is displayed on the smart glasses display, and the font size and highlighting are adjusted as needed.

[1392] Specific examples

[1393] Example 1: Making a phone call (when the user is in a hurry)

[1394] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1395] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1396] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1397] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1398] 5. The audio data is sent to the server and converted into text.

[1399] 6. Generative AI analyzes the instruction "Open the phone app."

[1400] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[1401] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[1402] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1403] 10. The user follows the instructions and taps the Phone app icon.

[1404] This system can improve the user's operating experience by providing real-time support for operating smart devices while taking into account the user's emotional state. In this way, by combining generative AI and emotion recognition technology, it is possible to provide optimal operating guidance to users.

[1405] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1406] Step 1:

[1407] The terminal (smart glasses) captures the screen of the smart device from the user's point of view. Specifically, the camera captures the home screen of the smartphone. The input at this time is the screen of the smart device that the user is looking at, and the output is the captured image data.

[1408] Step 2:

[1409] The device (smart glasses) transmits the captured image data to the server via Wi-Fi or Bluetooth. The input is the captured image data, and the output is the image data transferred to the server.

[1410] Step 3:

[1411] The server analyzes the received image data using image analysis software (e.g., OpenCV, Tesseract OCR). The input is the image data transferred to the server, and the output is identification data for on-screen user interface elements (e.g., app icons and buttons). Specifically, the position and label of each icon are identified.

[1412] Step 4:

[1413] The user issues a voice command, such as "I want to make a call," into the microphone of the smart glasses. The input is the user's voice command, and the output is voice data.

[1414] Step 5:

[1415] The device (smart glasses) captures voice data and transmits it to the server via Wi-Fi or Bluetooth. The input is the user's voice data, and the output is the voice data transmitted to the server.

[1416] Step 6:

[1417] The server converts the voice data into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The input is the transferred voice data, and the output is text data. Specifically, the voice data "I want to make a call" is converted into the text "I want to make a call."

[1418] Step 7:

[1419] The server uses generative artificial intelligence (e.g., GPT-3, ChatGPT) to analyze user intent from text data. The input is text data, and the output is the analyzed user intent (e.g., "Open the phone app").

[1420] Step 8:

[1421] The device (smart glasses) uses an emotion recognition sensor to acquire emotional data from the user's voice and facial expressions. The input is the user's voice and facial expressions, and the output is the acquired emotional data. Specifically, the sensor measures the user's voice tone and facial expressions in real time.

[1422] Step 9:

[1423] The emotion data acquired by the device (smart glasses) is sent to the server via Wi-Fi or Bluetooth. The input is the acquired emotion data, and the output is the emotion data transferred to the server.

[1424] Step 10:

[1425] The server uses emotion analysis software (e.g., Affectiva, IBM Watson Emotion Analysis) to analyze the acquired emotion data and identify the user's emotional state, where the input is the transferred emotion data and the output is the user's emotional state (e.g., stress, irritability).

[1426] Step 11:

[1427] The server generates operation guidance based on the user's voice instructions and emotional data. The input is the analyzed user's intention and emotional state, and the output is the operation guidance. For example, if the user is in a hurry, a polite operation guidance such as "Please tap the phone app icon a little more slowly" is generated.

[1428] Step 12:

[1429] The operation guide generated by the server is sent to the device (smart glasses) via Wi-Fi or Bluetooth. The input at this time is the operation guide, and the output is the operation guide transferred to the smart glasses.

[1430] Step 13:

[1431] The device (smart glasses) displays the operation guide on the display. The input is the transferred operation guide, and the output is the displayed operation guide. Specifically, visual feedback is provided, such as the phone app icon being highlighted in a red frame.

[1432] (Application example 2)

[1433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1434] Providing an environment where elderly people and users unfamiliar with operating smart devices can easily obtain and use information in the real world is an important challenge in modern society. In particular, obtaining product information and supporting purchasing in physical stores requires an intuitive and adaptive interface. However, conventional systems have difficulty responding while fully considering the user's emotions and operating situation. Therefore, there is a need to develop a system that provides appropriate feedback according to the user's emotional state and supports operation.

[1435] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1436] In this invention, the server is a display device worn by a user, and includes: a camera means for capturing the screen of the smart device from the user's viewpoint; a network means for receiving the captured screen image; an image recognition means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a display means for displaying the generated operation guide on the display device; an emotion recognition means for acquiring the user's emotional state; and an emotion analysis means for adjusting the content of the operation guide based on the emotional state, thereby enabling adaptive and intuitive feedback according to the user's emotional state.

[1437] A "display device" is a device worn by a user to provide visual feedback.

[1438] "Camera means" refers to a camera attached to a display device to capture the screen of the smart device from the user's point of view.

[1439] "Network means" refers to the communications infrastructure used to receive the captured screen image.

[1440] "Image recognition means" refers to a technology that identifies user interface elements on the screen of a smart device from a received image.

[1441] The "voice input means" is a device such as a microphone that receives voice instructions from the user.

[1442] "Generative artificial intelligence means" refers to AI technology that analyzes voice instructions and generates the next operation guide based on the user interface elements.

[1443] The "display means" is a screen for displaying the generated operation guide on a display device.

[1444] An "emotion recognition means" is a sensor or software for acquiring the user's emotional state.

[1445] The "emotion analysis means" is a technique for adjusting the content of operation guidance based on the acquired emotional state.

[1446] MODE FOR CARRYING OUT THE INVENTION

[1447] System Configuration

[1448] To implement this invention, a display device (smart glasses) worn by a user, a server, and emotion recognition means are required. Specifically, the smart glasses incorporate a camera means, a voice input means, a display means, and an emotion recognition sensor, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[1449] Specific examples of program processing

[1450] Capture and send images

[1451] The smart glasses use a camera means to capture the screen of the smart device from the user's point of view, and the captured image is immediately transmitted to a server via a network means.

[1452] Image Recognition

[1453] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR technology).

[1454] Voice input reception and question analysis

[1455] The smart glasses receive voice commands from the user via a microphone. When the user issues a voice command, such as "I want to search for a specific product," the voice data is captured and sent to a server. The server then analyzes the voice data and converts it into text. Generative AI is used to convert the voice data into text and analyze the user's intent.

[1456] Emotion data acquisition and analysis

[1457] The smart glasses use emotion recognition sensors to capture emotions from the user's voice and facial expressions. This includes voice tone analysis and facial expression recognition technology. The server analyzes the captured emotion data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[1458] Response Generation

[1459] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[1460] Guide Display

[1461] The smart glasses display the generated operation guide on the screen, emphasizing the operation points by, for example, increasing the text size or adding animations depending on the user's emotional state.

[1462] Hardware and software used

[1463] Hardware: Smart glasses (camera, microphone, display, emotion recognition sensor), server

[1464] Software: OpenCV (camera image processing), pytesseract (OCR text recognition), speech_recognition (voice recognition), requests (API communication), emotion_recognition (emotion recognition)

[1465] Specific examples

[1466] Here is a specific example of a case where smart glasses are used to read the QR code of a product that a user shows interest in at a physical store. The camera means of the smart glasses captures the QR code, and the image recognition means extracts the product code. Using this product code, the server obtains detailed information about the product. If the user issues a voice command such as "I want to find a cheaper product," the command is analyzed by generative artificial intelligence, and information about cheaper products is provided by voice. If the user becomes impatient while operating the device, the emotion recognition means detects this, and the server provides adaptive feedback such as "Please stay calm."

[1467] Prompt Sentence Examples

[1468] "If a user wants to scan a QR code and get the product price and recommended points from a voice assistant, how do you provide that feedback?"

[1469] In this way, users will receive smart product information and adaptive feedback to improve their real-world shopping experience.

[1470] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1471] Step 1:

[1472] The smart glasses (terminal) use a camera to capture the screen of the smart device as seen from the user's point of view.

[1473] Input: The screen of the smart device that the user is looking at while wearing the smart glasses.

[1474] Output: The captured screen image.

[1475] Specific operation: The camera continuously takes images and generates image data.

[1476] Step 2:

[1477] The smart glasses (terminal) transmit the captured screen image to the server via the network.

[1478] Input: A captured screen image.

[1479] Output: Screen image sent to the server.

[1480] Specific operation: The captured image data is sent to the server using a network means such as Wi-Fi or Bluetooth.

[1481] Step 3:

[1482] The server analyzes the received image data and uses image recognition technology to identify user interface elements on the screen of the smart device.

[1483] Input: The submitted screen image.

[1484] Output: Identified user interface elements (e.g. app icons and buttons).

[1485] Specific operation: Image recognition software on the server (e.g., OpenCV or pytesseract) analyzes the image and extracts specific elements.

[1486] Step 4:

[1487] The smart glasses (device) accept the user's voice instructions via a microphone and capture the voice data.

[1488] Input: User's spoken command (e.g., "I would like to search for a specific product").

[1489] Output: The captured audio data.

[1490] How it works: The microphone built into the smart glasses records the user's voice and generates digital audio data.

[1491] Step 5:

[1492] The smart glasses (terminals) transmit the captured audio data to a server via a network.

[1493] Input: Captured audio data.

[1494] Output: The audio data sent to the server.

[1495] Specific operation: Audio data is transmitted to a server via a network means.

[1496] Step 6:

[1497] The server analyzes the transmitted voice data and converts it into text using voice recognition technology.

[1498] Input: The audio data sent to the server.

[1499] Output: Text data (voice instructions converted into text).

[1500] What happens: The server's speech recognition software (e.g., speech_recognition) processes the audio data and generates corresponding text data.

[1501] Step 7:

[1502] The server uses generative artificial intelligence to analyze the user's voice instructions and generate operation guides.

[1503] Input: Text data and identified user interface elements.

[1504] Output: The generated how-to guide.

[1505] What it does: Generative AI (e.g., a machine learning model) analyzes text data and generates guidelines for the actions the user wants to take.

[1506] Step 8:

[1507] The smart glasses (device) collect emotional data from the user's voice and facial expressions.

[1508] Input: The user's tone of voice and facial expressions.

[1509] Output: The obtained emotion data.

[1510] Specific operation: The emotion recognition sensor analyzes the user's tone of voice and facial expressions, and digitizes their emotional state.

[1511] Step 9:

[1512] The server analyzes the acquired emotional data and recognizes the emotional state of the user.

[1513] Input: The acquired emotion data.

[1514] Output: Parsed emotional state (e.g. stress state).

[1515] What it does: The emotion recognition engine analyzes the emotion data and identifies the user's emotional state.

[1516] Step 10:

[1517] The server adjusts the content of the operation guide based on the emotional state of the user.

[1518] Input: Parsed emotional state and generated interaction guide.

[1519] Output: Adjusted operation guide.

[1520] Specific operation: Generative AI incorporates emotional data and modifies the operation guide to suit the user.

[1521] Step 11:

[1522] The smart glasses (terminal) will display adjusted operation guides on the display.

[1523] Enter: the adjusted operating guide.

[1524] Output: Operation guide displayed on the screen.

[1525] Specific operation: The smart glasses display visually displays operation guides and draws the user's attention by adjusting the font size and color as needed.

[1526] Through these steps, users will be able to intuitively and adaptively operate smart devices and product information.

[1527] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1529] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1530] [Fourth embodiment]

[1531] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1532] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1533] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1534] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1535] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1536] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1537] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1538] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1539] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1540] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1541] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1542] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1543] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1544] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[1545] System Configuration

[1546] The system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence.

[1547] 1. Capture and send images

[1548] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[1549] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[1550] 2. Image Recognition

[1551] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[1552] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[1553] 3. Voice input reception and question analysis

[1554] The smart glasses (device) receives user voice commands through a microphone, and this voice data is sent to a server via a voice recognition system.

[1555] The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent.

[1556] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[1557] 4. Response Generation

[1558] The server determines the next action the user should take based on the voice instructions and the results of image recognition, and generates an operation guide that includes clear visual instructions.

[1559] Example: The generated instruction guide is "Tap the Phone app icon."

[1560] 5. Guide display

[1561] The smart glasses (device) displays the generated operation guide on the display, allowing the user to receive clear instructions and perform the next operation.

[1562] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[1563] Specific examples

[1564] Example 1: Making a phone call

[1565] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1566] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1567] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1568] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1569] 5. The audio data is sent to the server and converted into text.

[1570] 6. Generative AI analyzes the instruction "Open the phone app."

[1571] 7. The server generates an operation guide saying "Tap the phone app icon."

[1572] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1573] 9. The user follows the instructions and taps the Phone app icon.

[1574] In this way, this system helps users who are unfamiliar with operating smart devices to operate them easily and intuitively.

[1575] The processing flow will be explained below.

[1576] Step 1:

[1577] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user looks at their smartphone, the camera in the smart glasses captures the screen and saves it as image data.

[1578] Step 2:

[1579] The smart glasses (device) sends the captured image data to the server via a network connection, where it is prepared for analysis.

[1580] Step 3:

[1581] The server analyzes the received image data and identifies interface elements on the smart device screen. It uses image recognition technology (such as OCR) to detect text, buttons, icons, etc., and obtains their location information.

[1582] Step 4:

[1583] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[1584] Step 5:

[1585] The server analyzes the voice data and converts it into text. The voice data is converted into text using voice recognition technology, and the user's intent is analyzed using generative artificial intelligence.

[1586] Step 6:

[1587] The server generates an operation guide based on the analysis results. It combines the user's voice instructions and the image recognition results to generate the next operation guide (e.g., open the phone app).

[1588] Step 7:

[1589] The smart glasses (device) displays the generated operation guide on the display, visually highlighting the operation points and providing instructions that are easy for the user to understand.

[1590] Step 8:

[1591] The user operates the smart device according to the instructions on the smart glasses, for example, by tapping a specific icon according to the instructions displayed on the smart glasses.

[1592] Step 9:

[1593] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[1594] Example 1

[1595] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1596] The operation of modern information devices is becoming more complex, and there is a demand for an environment that allows elderly people and users who are unfamiliar with operating smart devices to operate them easily. In particular, many information devices have numerous icons and buttons, which can be difficult for first-time users to understand and operate. In addition, there is a need for a system that allows for more intuitive and reliable operation by simultaneously providing visual and auditory support.

[1597] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1598] In this invention, the server includes a communication means for receiving the captured screen image, an image recognition means for identifying user interface elements on the screen of the information device from the received image, and a generative artificial intelligence means for analyzing the voice instruction and generating the next operation guide based on the user interface elements. This allows even a user who is unfamiliar with operating the information device to operate it intuitively and reliably while receiving visual and voice support.

[1599] "User" refers to a person who operates an information device and uses a device that supports the operation from that point of view.

[1600] "Display device" refers to a device worn by a user that has a display function for providing visual information.

[1601] "Photographing means" means a camera or photographing device for capturing the screen of an information device from the user's point of view.

[1602] "Communication means" refers to the network technology used to transmit the captured screen image to the server, including Wi-Fi, Bluetooth, etc.

[1603] "Image recognition means" refers to technology for identifying user interface elements on the screen of an information device from images received by the server, and includes, in particular, pattern recognition algorithms and OCR technology.

[1604] The "voice input means" refers to a microphone or voice collection device for receiving voice instructions from the user, and is a means for acquiring voice data.

[1605] "Generative artificial intelligence means" refers to an artificial intelligence technology for analyzing voice instructions and generating next operation guidance based on user interface elements.

[1606] The "display means" refers to a component or device having a display function for visually presenting the generated operation guide to the user.

[1607] "Operation guide" refers to guide information that clearly indicates to the user the next operation to be performed and includes instructions that visually emphasize the operation.

[1608] "Information devices" refer to electronic devices operated by users, such as smartphones and tablets, which have many app icons and buttons.

[1609] This invention is a system that supports the operation of information devices by combining a display device worn by the user with a server. It uses smart glasses and generative artificial intelligence to enable users to easily operate information devices that they are unfamiliar with.

[1610] System Configuration

[1611] The system mainly consists of the following components:

[1612] Terminal (smart glasses): Worn by the user, it includes a camera means for capturing the screen of the information device from the user's viewpoint, a voice input means for picking up voice, and a display means for displaying the generated operation guide.

[1613] Server: Includes a communication means for receiving screen images, an image recognition means for recognizing the received images, and a generative artificial intelligence means for analyzing the user's voice instructions.

[1614] Specific actions

[1615] 1. Image capture and transmission:

[1616] The device (smart glasses) uses a camera to capture the screen of the information device from the user's perspective and transmits the image to a server via Wi-Fi or Bluetooth.

[1617] Example: When a user puts on smart glasses while viewing the home screen of their smartphone, the camera automatically captures the home screen and sends the image data to the server.

[1618] 2. Image Recognition:

[1619] The server analyzes the received image data and uses image recognition technology such as Google Cloud Vision API to identify on-screen user interface elements (app icons and buttons).

[1620] Example: The server analyzes the image data of the home screen received and identifies app icons such as "Phone" and "Message."

[1621] 3. Voice input acceptance and conversion:

[1622] The terminal (smart glasses) picks up the user's instructions using a voice input means and transmits them to a server via a communication means.

[1623] The server converts the received voice data into text using Google Cloud Speech-to-Text technology.

[1624] For example, if a user says "I want to make a call," the smart glasses will pick up the voice and send it to the server, which will convert the voice data into text.

[1625] 4. Intent Analysis:

[1626] The server inputs the converted text data into a generative AI model (such as OpenAI's GPT-4), analyzes the user's intent, and identifies appropriate operating procedures.

[1627] Example: A generative AI model analyzes the instruction "I want to make a call" into the specific operation "Open the phone app."

[1628] 5. Response Generation:

[1629] Based on the analysis results, the server generates an operation guide for the user, which includes specific instructions and visual highlights.

[1630] Example: Generate an action guide that says "Tap the Phone app icon" with instructions to highlight the Phone app icon in a red border.

[1631] 6. Guide display:

[1632] The terminal (smart glasses) visually presents the generated operation guide to the user on a display means.

[1633] Example: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the phone app icon will be highlighted in a red frame. The user will follow the instructions.

[1634] Specific examples and prompts

[1635] Example: A scenario for making a phone call

[1636] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1637] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1638] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1639] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1640] 5. The audio data is sent to the server and converted into text.

[1641] 6. The generative AI model analyzes the instruction "Open the phone app."

[1642] 7. The server generates an operation guide saying "Tap the phone app icon."

[1643] 8. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1644] 9. The user follows the instructions and taps the Phone app icon.

[1645] Prompt Sentence Examples

[1646] "When a user says, 'I want to make a phone call,' generate a guide for the next operation based on the image captured by the smartglasses' camera."

[1647] As described above, the present invention provides a system that allows even a user who is unfamiliar with operating information devices to operate it easily and intuitively.

[1648] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1649] Step 1:

[1650] The terminal (smart glasses) captures the screen of the information device from the user's point of view with a camera and transmits the image to a server via a network means.

[1651] Input: The screen of the information device that the user is looking at

[1652] Data processing: The camera captures the screen and generates image data

[1653] Output: Image data sent to the server

[1654] Specific operation: When a user looks at the home screen of their smartphone, the camera in the smart glasses captures the screen and sends the generated image data to a server via Wi-Fi or Bluetooth.

[1655] Step 2:

[1656] The server analyzes the received image data and identifies user interface elements using image recognition technology such as the Google Cloud Vision API.

[1657] Input: Screen image sent from the device (smart glasses)

[1658] Data processing: Apply image recognition algorithms (such as Google Cloud Vision API) to identify icons and buttons on the screen.

[1659] Output: A list of identified user interface elements

[1660] Specific operation: The server processes the image data of the home screen it receives and identifies app icons such as "Phone" and "Message."

[1661] Step 3:

[1662] The device (smart glasses) receives the user's voice instructions through a microphone and transmits them to the server as voice data.

[1663] Input: User's voice command

[1664] Data processing: Collection of voice data

[1665] Output: Audio data sent to the server

[1666] Specific operation: When a user says "I want to make a call," the smart glasses pick up the voice and send the voice data to the server.

[1667] Step 4:

[1668] The server converts the received voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[1669] Input: Audio data sent from the device (smart glasses)

[1670] Data processing: Converting voice data into text using voice recognition technology

[1671] Output: Text data

[1672] Specific operation: The server converts the received voice data into text using Google Cloud Speech-to-Text. The text obtained is "I want to make a phone call."

[1673] Step 5:

[1674] The server inputs the text data into a generative AI model (such as OpenAI's GPT-4) to analyze the user's intent.

[1675] Input: Text data converted from audio data

[1676] Data processing: Analysis and intent identification with generative AI models

[1677] Output: Instructions based on the user's intent (e.g., "Open the Phone app")

[1678] Specific action: The generative AI model analyzes the instruction "I want to make a call" into the specific action of "opening the phone app."

[1679] Step 6:

[1680] The server generates specific operation guides based on the analysis results and sends them to the device (smart glasses).

[1681] Input: Operation instructions based on the user's intention

[1682] Data processing: Generate operation guides, including visual instructions and highlights

[1683] Output: Generated instructions

[1684] Specific operation: The server generates an operation guide saying "Tap the phone app icon" and instructs the phone app icon to be highlighted in a red frame.

[1685] Step 7:

[1686] The device (smart glasses) displays the generated operation guide on its own display, providing the user with clear operating instructions.

[1687] Input: Operation guide sent from the server

[1688] Data processing: Display of operation guide

[1689] Output: Operation guide displayed on the screen

[1690] Specific operation: The smart glasses display will show an operation guide saying "Tap the phone app icon," and the icon will be highlighted in red. The user should follow the instructions and tap the phone app icon.

[1691] Through the above steps, this system provides an environment in which even users who are unfamiliar with operating information devices can operate the system easily and intuitively.

[1692] (Application example 1)

[1693] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1694] Conventional smart device operation support systems have problems, such as complicated operation and difficulty in navigating and obtaining product information, especially for elderly users and users unfamiliar with device operation. This has prevented users from using the device effectively and has caused inconvenience. In addition, there has been a lack of support for checking product information in physical stores and obtaining location information within the store, which has reduced user convenience.

[1695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1696] In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a video output means for displaying the generated operation guide on the display device; and a display means for the user to obtain in-store product information and a store map. This allows the user to intuitively operate the smart device, making it easier to navigate in a physical store and obtain product information.

[1697] "Smart devices" is a general term for portable electronic devices such as mobile phones, tablets, e-book readers, and smartwatches that can communicate over a network and run a variety of applications.

[1698] A "display device" is a device worn by a user that outputs visual information, and examples of this include head-mounted displays and smart glasses.

[1699] The "photographing means" is a camera mounted on the display device, and is a device that has the function of capturing an image from the user's viewpoint.

[1700] "Communication means" refers to a network connection device for relaying captured image and audio data to a server, and includes wireless and wired communication.

[1701] "Image analysis means" refers to technology that identifies specific elements from received image data, and uses optical character recognition (OCR) or machine learning algorithms.

[1702] The "voice input means" is a device for collecting the user's voice and transmitting it to the server as voice data.

[1703] "Generative AI means" refers to an AI technology that analyzes the user's intentions based on voice input and image analysis data, and generates the next operation guide.

[1704] The "video output means" is a device for visually outputting the generated operation guide to a display device.

[1705] The "display means" is a device that provides visual information for the user to obtain in-store product information and a store map.

[1706] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses and generative AI.

[1707] System Configuration

[1708] This system consists of smart glasses and a server. The smart glasses are equipped with a camera, voice input device, and display, and are worn by the user. The server runs various software, including image recognition and generative artificial intelligence. It also includes a display means that allows the user to obtain in-store product information and a store map.

[1709] Capture and send images

[1710] The smart glasses, which are the terminals, capture the screen of the smart device from the user's point of view. The captured image is immediately sent to the server via a communication means. For example, if a user is looking at a product shelf in a store, the camera in the smart glasses captures the image and sends it to the server.

[1711] Image Recognition

[1712] The server analyzes the received image data and identifies the user interface elements displayed on the smart device screen. For example, in a store, the server analyzes the images of product shelves and recognizes the barcodes and labels of each product.

[1713] Voice input reception and question analysis

[1714] The smart glasses, which act as a terminal, receive voice commands from the user via a microphone. This voice data is sent to a server via a voice recognition system. The server converts the voice data into text and uses generative artificial intelligence to analyze the user's intent. For example, if a user says, "I want to see a list of special sale items," the voice data is sent to the server, converted into text, and the generative artificial intelligence interprets it as the instruction to "display a list of special sale items."

[1715] Response Generation

[1716] Based on the voice instructions and image recognition results, the server determines the next action the user should take and generates an operation guide that includes clear visual instructions, such as "Please scan the barcode."

[1717] Guide Display

[1718] The smart glasses, which are the terminals, display the generated operation guide on their screens, allowing the user to receive clear instructions and perform the next operation. For example, the operation guide is displayed on the smart glasses' display, highlighting the product barcode with a red frame.

[1719] Specific examples

[1720] Example 1: In-store navigation

[1721] 1. A user wears smart glasses and walks around a store.

[1722] 2. The camera in the smart glasses captures the inside of the store and sends the images to the server.

[1723] 3. The server performs image recognition to identify the shelves and the barcodes and labels of each product.

[1724] 4. The user speaks into the microphone on the smart glasses, "Where is the detergent shelf?"

[1725] 5. The audio data is sent to the server and converted into text.

[1726] 6. The generative AI analyzes the instruction "Show the location of the detergent shelf."

[1727] 7. The server generates a navigation guide and shows the desired shelf location on a store map.

[1728] 8. Navigation guides will be displayed on the smart glasses display, allowing users to check the route to their destination.

[1729] Prompt Sentence Examples

[1730] Prompt to find sale items:

[1731] "The user says, 'I want to see a list of special offers,' and the system converts this into text and inputs the instruction 'Show me a list of special offers' as a prompt to the generative AI."

[1732] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1733] System program processing flow

[1734] Step 1:

[1735] The camera in the smart glasses captures images from the user's point of view. Specifically, the image capture means acquires images while the user is wearing the smart glasses and looking at the screen of a smart device or a product shelf in a store.

[1736] Input: Image from the user's perspective

[1737] Output: Captured image data

[1738] Step 2:

[1739] The captured image data is sent to the server via a communication means. Internally, the smart glasses use wireless communication to send image data to the server in real time.

[1740] Input: Captured image data

[1741] Output: Image data sent to the server

[1742] Step 3:

[1743] The server analyzes the received image data and uses image analysis methods to identify user interface elements displayed on the smart device screen or on the store shelf, and extracts information such as barcodes and labels using OCR technology and object recognition algorithms.

[1744] Input: Image data sent to the server

[1745] Output: Parsed interface elements (e.g. barcodes, labels)

[1746] Step 4:

[1747] The smart glasses' microphone accepts the user's voice commands, allowing the user to use voice input to request specific actions or information.

[1748] Input: User's voice command

[1749] Output: Audio data

[1750] Step 5:

[1751] The voice data is sent to the server via a communication means. The smart glasses send the voice data to the server in real time.

[1752] Input: Audio data

[1753] Output: Audio data sent to the server

[1754] Step 6:

[1755] The server converts the voice data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text).

[1756] Input: Audio data sent to the server

[1757] Output: Text data

[1758] Step 7:

[1759] Generative AI analyzes text data to understand user intent. In this process, the generative AI model generates appropriate responses and operational guidance based on the user's questions and requests.

[1760] Input: Text data

[1761] Output: Appropriate response or action guide (text)

[1762] Step 8:

[1763] Based on the analysis results, the server determines the next action the user should take and generates a visual operation guide, such as "Please scan the barcode."

[1764] Input: Appropriate response or instruction (text)

[1765] Output: Visual operation guide

[1766] Step 9:

[1767] The generated operation guide is sent to the smart glasses display via a communication method and displayed. The user visually confirms it and performs the next operation.

[1768] Input: Visual guide

[1769] Output: Guide displayed on the smart glasses display

[1770] Example: In-store navigation processing steps

[1771] Step 1:

[1772] As a user wears the smart glasses and walks around the store, the camera in the smart glasses captures images of the store's shelves.

[1773] Input: Image of the interior from the user's perspective

[1774] Output: Captured in-store image data

[1775] Step 2:

[1776] The captured in-store image data is sent from the smart glasses to a server.

[1777] Input: Captured in-store image data

[1778] Output: In-store image data sent to the server

[1779] Step 3:

[1780] The server analyzes the in-store image data transmitted and uses image analysis means to identify the barcodes and labels of each product.

[1781] Input: In-store image data sent to the server

[1782] Output: Identified barcode or label

[1783] Step 4:

[1784] The user speaks into the microphone on the smart glasses, asking, "Where is the detergent shelf?"

[1785] Input: User's voice command

[1786] Output: Audio data

[1787] Step 5:

[1788] The audio data is sent from the smart glasses to a server.

[1789] Input: Audio data

[1790] Output: Audio data sent to the server

[1791] Step 6:

[1792] The server converts the audio data into text.

[1793] Input: Audio data sent to the server

[1794] Output: Data converted to text

[1795] Step 7:

[1796] Generative artificial intelligence analyzes text data and generates optimal responses to user questions.

[1797] Input: Data converted to text

[1798] output: Text that serves as a navigation guide

[1799] Step 8:

[1800] Based on the analysis results, the server generates a navigation guide showing the location of shelves within the store.

[1801] Input: Text to serve as a navigation guide

[1802] Output: A visual navigation guide

[1803] Step 9:

[1804] The generated navigation guide is sent to the smartglasses display and displayed to the user.

[1805] Input: Visual navigation guide

[1806] Output: Navigation guide displayed on the smart glasses display

[1807] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1808] As an embodiment of the present invention, the specific program processing of the following system is explained in natural language. We provide an environment in which elderly people and users who are unfamiliar with operating smart devices can easily operate smart devices using smart glasses, generative artificial intelligence, and an emotion engine.

[1809] System Configuration

[1810] This system consists of smart glasses, a server, and an emotion engine. The smart glasses are equipped with a camera, a voice input device, a display, and emotion recognition sensors, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[1811] 1. Capture and send images

[1812] The smart glasses (terminal) capture the screen of the smart device from the user's point of view, and the captured image is immediately sent to the server via network means.

[1813] When a user is looking at the home screen of their smartphone, the camera in the smart glasses captures that screen as an image and sends it to the server.

[1814] 2. Image Recognition

[1815] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR).

[1816] The server analyzes the image of the home screen sent and identifies app icons such as "Phone" and "Message."

[1817] 3. Voice input reception and question analysis

[1818] The smart glasses (device) accepts the user's voice commands through a microphone. When the user issues a voice command such as "I want to make a phone call," the voice data is captured and sent to the server.

[1819] The server analyzes the voice data and converts it into text using voice recognition technology, and then uses generative artificial intelligence to analyze the user's intent.

[1820] Example: When a user says, "I want to make a call," the voice data is sent to a server, converted into text, and a generative artificial intelligence analyzes the instruction to "open the phone app."

[1821] 4. Emotion Data Acquisition and Analysis

[1822] The smart glasses (devices) use emotion recognition sensors to obtain emotions from the user's voice and facial expressions, including voice tone analysis and facial expression recognition technology.

[1823] The server analyzes the acquired emotional data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[1824] Example: If a user becomes anxious while operating the device, the sensors in the smart glasses will detect the user's stress level from facial expressions and voice tone and send that information to the server.

[1825] 5. Response Generation

[1826] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[1827] Example: If the user is in a hurry, generate and display a more detailed and easy-to-understand operation guide.

[1828] 6. Guide display

[1829] The smart glasses (device) displays the generated operation guide on the display, emphasizing the operation points by, for example, increasing the text size or adding animation depending on the user's emotional state.

[1830] Example: An operation guide appears on the smart glasses display, and the phone app icon is highlighted in a red frame.

[1831] Specific examples

[1832] Example 1: Making a phone call (when the user is in a hurry)

[1833] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1834] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1835] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1836] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1837] 5. The audio data is sent to the server and converted into text.

[1838] 6. Generative AI analyzes the instruction "Open the phone app."

[1839] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[1840] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[1841] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1842] 10. The user follows the instructions and taps the Phone app icon.

[1843] In this way, the system improves the user's operating experience by providing real-time support for smart device operation while taking into account the user's emotional state.

[1844] The processing flow will be explained below.

[1845] Step 1:

[1846] The smart glasses (terminal) capture the screen of the smart device from the user's point of view. When the user is looking at the home screen of their smartphone, the camera on the smart glasses captures the screen as image data.

[1847] Step 2:

[1848] The smart glasses (devices) transmit the captured image data to a server via a network connection, where the image data is stored and ready for analysis.

[1849] Step 3:

[1850] The server analyzes the received image data and identifies user interface elements (such as app icons and buttons) on the smart device screen using image recognition technology (such as OCR), and also obtains the location information of the elements.

[1851] Step 4:

[1852] The smart glasses (device) receive voice commands from the user through a microphone. When the user says something like "I want to make a phone call," the smart glasses capture the voice data and send it to a server via the network.

[1853] Step 5:

[1854] The server converts the received voice data into text using speech recognition technology. Using this text data, generative AI analyzes the user's intent. For example, the instruction "I want to make a call" is interpreted as the operation of opening the phone app.

[1855] Step 6:

[1856] The smart glasses (device) uses an emotion recognition sensor to recognize the user's emotions and acquires emotion data from the user's voice and facial expressions. This data is based on voice tone analysis and facial expression recognition technology.

[1857] Step 7:

[1858] The server analyzes the acquired emotional data and recognizes the user's emotional state, for example, identifying that the user is experiencing stress.

[1859] Step 8:

[1860] The server generates operation guidance based on the user's voice instructions and emotional data. It adjusts the way the operation guidance is presented depending on the user's emotional state. For example, if the user is in a hurry, it adjusts the operation guidance to be more thorough and easy to understand.

[1861] Step 9:

[1862] The smart glasses (device) displays the generated operation guide on the display. For example, the generated operation guide may be a detailed instruction such as "Tap the phone app icon," and the icon will be highlighted in a red frame.

[1863] Step 10:

[1864] The user operates the smart device according to the instructions of the smart glasses. For example, the user looks at the operation guide displayed on the smart glasses and taps the icon of the specified app.

[1865] Step 11:

[1866] The smart glasses (device) check the user's operations and display the next instructions as necessary. When the user's operation is completed, the device displays the next operation guide.

[1867] Example 2

[1868] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] Modern smart devices are extremely multifunctional, but their operation is complex and difficult to use, especially for elderly people and users who are unfamiliar with smart device operation. For this reason, there is a demand for intuitive operation and detailed operational support that responds to the user's emotional state.

[1870] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is a display device worn by a user, and includes: an imaging means for capturing the screen of the smart device from the user's viewpoint; a communication means for receiving the captured screen image; an image analysis means for identifying user interface elements on the screen of the smart device from the received image; a voice acquisition means for receiving the user's voice instructions; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; an emotion recognition means for acquiring emotional data from the user's voice and facial expressions; an emotion analysis means for analyzing the acquired emotional data and recognizing the user's emotional state; a response generation means for adjusting and generating a presentation method of an operation guide according to the user's emotional state; and a display means for displaying the generated operation guide on the display device. This allows the user to easily operate the smart device, improving the operation experience.

[1871] "User" refers to a person who wears a device such as smart glasses and receives assistance in operating the smart device.

[1872] A "display device" refers to a device worn by a user that displays information in the field of view.

[1873] "Capture means" refers to equipment such as cameras and sensors used to capture the screen of a smart device from the user's perspective.

[1874] "Communication means" refers to the network function for transmitting captured image and audio data to a server.

[1875] "Image analysis means" refers to software or algorithms used to identify user interface elements on the screen of a smart device from received image data.

[1876] "Voice acquisition means" refers to a microphone or voice input device for receiving voice instructions from the user.

[1877] "Generative artificial intelligence means" refers to an artificial intelligence system for analyzing voice instructions and generating next operation guidance based on user interface elements.

[1878] "Emotion recognition means" refers to sensors and analysis software for acquiring emotional data from the user's voice and facial expressions.

[1879] "Emotion analysis means" refers to software or algorithms that analyze acquired emotional data and recognize the user's emotional state.

[1880] "Response generation means" refers to software or algorithms that generate and adjust the way in which operation guidance is presented depending on the user's emotional state.

[1881] The "display means" refers to a display device for visually displaying the generated operation guide to the user.

[1882] The following describes in detail an embodiment of the present invention. This system uses smart glasses, generative artificial intelligence, and an emotion engine to provide an environment in which elderly people and users unfamiliar with operating smart devices can easily operate smart devices. Specifically, this system is composed of smart glasses, a server, and an emotion engine.

[1883] Hardware and Software Configuration

[1884] Smart glasses (device)

[1885] The smart glasses incorporate a camera, a voice input device, a display, and an emotion recognition sensor. The camera captures the screen of the smart device from the user's point of view, the voice input device receives the user's voice instructions, the display shows the generated operation guide to the user, and the emotion recognition sensor captures the user's emotional state from their voice and facial expressions.

[1886] server

[1887] The server runs various software for image recognition, generative AI, and sentiment analysis. For image recognition, it uses OpenCV and Tesseract OCR, for speech recognition it uses Google Cloud Speech-to-Text, for generative AI it uses GPT-3 and ChatGPT, and for sentiment analysis it uses Affectiva and IBM Watson Emotion Analysis.

[1888] Specific example of system operation

[1889] 1. Capture and image transmission: The camera in the smart glasses captures the screen of the smart device from the user's point of view and transmits the image data to the server via Wi-Fi or Bluetooth.

[1890] 2. Image Recognition: The server analyzes the image data received and identifies user interface elements on the smart device screen, such as app icons like "Phone" and "Message" displayed on the home screen.

[1891] 3. Voice input acceptance and question analysis: When a user says "I want to make a phone call" into the microphone of the smart glasses, the voice data is sent to the server. The server converts the voice data into text, and the generative AI analyzes the instruction to "open the phone app."

[1892] 4. Emotion data acquisition and analysis: The smart glasses use emotion recognition sensors to acquire emotional data from the user's voice and facial expressions, and then send the data to a server, which then uses emotion analysis software to analyze the user's emotional state.

[1893] 5. Response generation: The server generates operation guidance based on the user's voice instructions and emotional data. For example, if the user is in a hurry, it generates a polite guidance such as "Please tap the phone app icon a little more slowly."

[1894] 6. Guide display: The generated operation guide is displayed on the smart glasses display, and the font size and highlighting are adjusted as needed.

[1895] Specific examples

[1896] Example 1: Making a phone call (when the user is in a hurry)

[1897] 1. A user is wearing smart glasses and looking at the home screen of their smartphone.

[1898] 2. The smart glasses camera captures the home screen and sends the image to the server.

[1899] 3. The server performs image recognition to identify the user interface elements on the home screen.

[1900] 4. The user speaks into the microphone on the smart glasses, "I want to make a phone call."

[1901] 5. The audio data is sent to the server and converted into text.

[1902] 6. Generative AI analyzes the instruction "Open the phone app."

[1903] 7. At the same time, the smart glasses' emotion recognition sensor detects the user's stress level and transmits the emotion data to the server.

[1904] 8. The server adjusts the operation guide based on the emotional data and generates a polite operation guide such as "Please tap the phone app icon a little more slowly."

[1905] 9. An operation guide will appear on the smart glasses display, and the phone app icon will be highlighted in a red frame.

[1906] 10. The user follows the instructions and taps the Phone app icon.

[1907] This system can improve the user's operating experience by providing real-time support for operating smart devices while taking into account the user's emotional state. In this way, by combining generative AI and emotion recognition technology, it is possible to provide optimal operating guidance to users.

[1908] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1909] Step 1:

[1910] The terminal (smart glasses) captures the screen of the smart device from the user's point of view. Specifically, the camera captures the home screen of the smartphone. The input at this time is the screen of the smart device that the user is looking at, and the output is the captured image data.

[1911] Step 2:

[1912] The device (smart glasses) transmits the captured image data to the server via Wi-Fi or Bluetooth. The input is the captured image data, and the output is the image data transferred to the server.

[1913] Step 3:

[1914] The server analyzes the received image data using image analysis software (e.g., OpenCV, Tesseract OCR). The input is the image data transferred to the server, and the output is identification data for on-screen user interface elements (e.g., app icons and buttons). Specifically, the position and label of each icon are identified.

[1915] Step 4:

[1916] The user issues a voice command, such as "I want to make a call," into the microphone of the smart glasses. The input is the user's voice command, and the output is voice data.

[1917] Step 5:

[1918] The device (smart glasses) captures voice data and transmits it to the server via Wi-Fi or Bluetooth. The input is the user's voice data, and the output is the voice data transmitted to the server.

[1919] Step 6:

[1920] The server converts the voice data into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The input is the transferred voice data, and the output is text data. Specifically, the voice data "I want to make a call" is converted into the text "I want to make a call."

[1921] Step 7:

[1922] The server uses generative artificial intelligence (e.g., GPT-3, ChatGPT) to analyze user intent from text data. The input is text data, and the output is the analyzed user intent (e.g., "Open the phone app").

[1923] Step 8:

[1924] The device (smart glasses) uses an emotion recognition sensor to acquire emotional data from the user's voice and facial expressions. The input is the user's voice and facial expressions, and the output is the acquired emotional data. Specifically, the sensor measures the user's voice tone and facial expressions in real time.

[1925] Step 9:

[1926] The emotion data acquired by the device (smart glasses) is sent to the server via Wi-Fi or Bluetooth. The input is the acquired emotion data, and the output is the emotion data transferred to the server.

[1927] Step 10:

[1928] The server uses emotion analysis software (e.g., Affectiva, IBM Watson Emotion Analysis) to analyze the acquired emotion data and identify the user's emotional state, where the input is the transferred emotion data and the output is the user's emotional state (e.g., stress, irritability).

[1929] Step 11:

[1930] The server generates operation guidance based on the user's voice instructions and emotional data. The input is the analyzed user's intention and emotional state, and the output is the operation guidance. For example, if the user is in a hurry, a polite operation guidance such as "Please tap the phone app icon a little more slowly" is generated.

[1931] Step 12:

[1932] The operation guide generated by the server is sent to the device (smart glasses) via Wi-Fi or Bluetooth. The input at this time is the operation guide, and the output is the operation guide transferred to the smart glasses.

[1933] Step 13:

[1934] The device (smart glasses) displays the operation guide on the display. The input is the transferred operation guide, and the output is the displayed operation guide. Specifically, visual feedback is provided, such as the phone app icon being highlighted in a red frame.

[1935] (Application example 2)

[1936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1937] Providing an environment where elderly people and users unfamiliar with operating smart devices can easily obtain and use information in the real world is an important challenge in modern society. In particular, obtaining product information and supporting purchasing in physical stores requires an intuitive and adaptive interface. However, conventional systems have difficulty responding while fully considering the user's emotions and operating situation. Therefore, there is a need to develop a system that provides appropriate feedback according to the user's emotional state and supports operation.

[1938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1939] In this invention, the server is a display device worn by a user, and includes: a camera means for capturing the screen of the smart device from the user's viewpoint; a network means for receiving the captured screen image; an image recognition means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing the voice instructions and generating the next operation guide based on the user interface elements; a display means for displaying the generated operation guide on the display device; an emotion recognition means for acquiring the user's emotional state; and an emotion analysis means for adjusting the content of the operation guide based on the emotional state, thereby enabling adaptive and intuitive feedback according to the user's emotional state.

[1940] A "display device" is a device worn by a user to provide visual feedback.

[1941] "Camera means" refers to a camera attached to a display device to capture the screen of the smart device from the user's point of view.

[1942] "Network means" refers to the communications infrastructure used to receive the captured screen image.

[1943] "Image recognition means" refers to a technology that identifies user interface elements on the screen of a smart device from a received image.

[1944] The "voice input means" is a device such as a microphone that receives voice instructions from the user.

[1945] "Generative artificial intelligence means" refers to AI technology that analyzes voice instructions and generates the next operation guide based on the user interface elements.

[1946] The "display means" is a screen for displaying the generated operation guide on a display device.

[1947] An "emotion recognition means" is a sensor or software for acquiring the user's emotional state.

[1948] The "emotion analysis means" is a technique for adjusting the content of operation guidance based on the acquired emotional state.

[1949] MODE FOR CARRYING OUT THE INVENTION

[1950] System Configuration

[1951] To implement this invention, a display device (smart glasses) worn by a user, a server, and emotion recognition means are required. Specifically, the smart glasses incorporate a camera means, a voice input means, a display means, and an emotion recognition sensor, and are worn by the user. The server runs various software for image recognition, generative artificial intelligence, and emotion analysis.

[1952] Specific examples of program processing

[1953] Capture and send images

[1954] The smart glasses use a camera means to capture the screen of the smart device from the user's point of view, and the captured image is immediately transmitted to a server via a network means.

[1955] Image Recognition

[1956] The server analyzes the received image data and identifies the user interface elements (e.g., app icons and buttons) displayed on the smart device screen using image recognition technology (e.g., OCR technology).

[1957] Voice input reception and question analysis

[1958] The smart glasses receive voice commands from the user via a microphone. When the user issues a voice command, such as "I want to search for a specific product," the voice data is captured and sent to a server. The server then analyzes the voice data and converts it into text. Generative AI is used to convert the voice data into text and analyze the user's intent.

[1959] Emotion data acquisition and analysis

[1960] The smart glasses use emotion recognition sensors to capture emotions from the user's voice and facial expressions. This includes voice tone analysis and facial expression recognition technology. The server analyzes the captured emotion data and recognizes the user's emotional state. For example, if the user is feeling stressed, that data is fed back to the generative AI.

[1961] Response Generation

[1962] The server generates operation guidance based on the user's voice instructions and emotional data, and provides appropriate feedback to the user by adjusting the presentation method of the operation guidance according to the user's emotional state.

[1963] Guide Display

[1964] The smart glasses display the generated operation guide on the screen, emphasizing the operation points by, for example, increasing the text size or adding animations depending on the user's emotional state.

[1965] Hardware and software used

[1966] Hardware: Smart glasses (camera, microphone, display, emotion recognition sensor), server

[1967] Software: OpenCV (camera image processing), pytesseract (OCR text recognition), speech_recognition (voice recognition), requests (API communication), emotion_recognition (emotion recognition)

[1968] Specific examples

[1969] Here is a specific example of a case where smart glasses are used to read the QR code of a product that a user shows interest in at a physical store. The camera means of the smart glasses captures the QR code, and the image recognition means extracts the product code. Using this product code, the server obtains detailed information about the product. If the user issues a voice command such as "I want to find a cheaper product," the command is analyzed by generative artificial intelligence, and information about cheaper products is provided by voice. If the user becomes impatient while operating the device, the emotion recognition means detects this, and the server provides adaptive feedback such as "Please stay calm."

[1970] Prompt Sentence Examples

[1971] "If a user wants to scan a QR code and get the product price and recommended points from a voice assistant, how do you provide that feedback?"

[1972] In this way, users will receive smart product information and adaptive feedback to improve their real-world shopping experience.

[1973] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1974] Step 1:

[1975] The smart glasses (terminal) use a camera to capture the screen of the smart device as seen from the user's point of view.

[1976] Input: The screen of the smart device that the user is looking at while wearing the smart glasses.

[1977] Output: The captured screen image.

[1978] Specific operation: The camera continuously takes images and generates image data.

[1979] Step 2:

[1980] The smart glasses (terminal) transmit the captured screen image to the server via the network.

[1981] Input: A captured screen image.

[1982] Output: Screen image sent to the server.

[1983] Specific operation: The captured image data is sent to the server using a network means such as Wi-Fi or Bluetooth.

[1984] Step 3:

[1985] The server analyzes the received image data and uses image recognition technology to identify user interface elements on the screen of the smart device.

[1986] Input: The submitted screen image.

[1987] Output: Identified user interface elements (e.g. app icons and buttons).

[1988] Specific operation: Image recognition software on the server (e.g., OpenCV or pytesseract) analyzes the image and extracts specific elements.

[1989] Step 4:

[1990] The smart glasses (device) accept the user's voice instructions via a microphone and capture the voice data.

[1991] Input: User's spoken command (e.g., "I would like to search for a specific product").

[1992] Output: The captured audio data.

[1993] How it works: The microphone built into the smart glasses records the user's voice and generates digital audio data.

[1994] Step 5:

[1995] The smart glasses (terminals) transmit the captured audio data to a server via a network.

[1996] Input: Captured audio data.

[1997] Output: The audio data sent to the server.

[1998] Specific operation: Audio data is transmitted to a server via a network means.

[1999] Step 6:

[2000] The server analyzes the transmitted voice data and converts it into text using voice recognition technology.

[2001] Input: The audio data sent to the server.

[2002] Output: Text data (voice instructions converted into text).

[2003] What happens: The server's speech recognition software (e.g., speech_recognition) processes the audio data and generates corresponding text data.

[2004] Step 7:

[2005] The server uses generative artificial intelligence to analyze the user's voice instructions and generate operation guides.

[2006] Input: Text data and identified user interface elements.

[2007] Output: The generated how-to guide.

[2008] What it does: Generative AI (e.g., a machine learning model) analyzes text data and generates guidelines for the actions the user wants to take.

[2009] Step 8:

[2010] The smart glasses (device) collect emotional data from the user's voice and facial expressions.

[2011] Input: The user's tone of voice and facial expressions.

[2012] Output: The obtained emotion data.

[2013] Specific operation: The emotion recognition sensor analyzes the user's tone of voice and facial expressions, and digitizes their emotional state.

[2014] Step 9:

[2015] The server analyzes the acquired emotional data and recognizes the emotional state of the user.

[2016] Input: The acquired emotion data.

[2017] Output: Parsed emotional state (e.g. stress state).

[2018] What it does: The emotion recognition engine analyzes the emotion data and identifies the user's emotional state.

[2019] Step 10:

[2020] The server adjusts the content of the operation guide based on the emotional state of the user.

[2021] Input: Parsed emotional state and generated interaction guide.

[2022] Output: Adjusted operation guide.

[2023] Specific operation: Generative AI incorporates emotional data and modifies the operation guide to suit the user.

[2024] Step 11:

[2025] The smart glasses (terminal) will display adjusted operation guides on the display.

[2026] Enter: the adjusted operating guide.

[2027] Output: Operation guide displayed on the screen.

[2028] Specific operation: The smart glasses display visually displays operation guides and draws the user's attention by adjusting the font size and color as needed.

[2029] Through these steps, users will be able to intuitively and adaptively operate smart devices and product information.

[2030] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2031] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2032] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2033] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2034] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2035] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2036] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2037] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2038] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2039] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2040] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2041] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2042] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2043] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2044] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2045] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2046] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2047] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2048] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2049] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2050] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2051] The following is further disclosed regarding the above embodiment.

[2052] (Claim 1)

[2053] An operation support system for a smart device,

[2054] a camera means for capturing a screen of the smart device from the user's point of view;

[2055] network means for receiving the captured screen image;

[2056] image recognition means for identifying user interface elements on the screen of the smart device from the received image;

[2057] a voice input means for receiving voice instructions from the user;

[2058] a generative artificial intelligence means for analyzing a voice instruction and generating a next operation guide based on the user interface elements;

[2059] a display means for displaying the generated operation guide on the display device;

[2060] A system including:

[2061] (Claim 2)

[2062] The system according to claim 1, further comprising means for visually highlighting operation locations to a user in the generated operation guide.

[2063] (Claim 3)

[2064] The system according to claim 1, wherein the generative artificial intelligence means includes means for identifying an operation procedure for a smart device based on the user's voice instructions and generating a specific operation guide.

[2065] "Example 1"

[2066] (Claim 1)

[2067] a display device worn by a user, and an imaging means for capturing a screen of the information device from the viewpoint of the user;

[2068] a communication means for receiving the captured screen image;

[2069] image recognition means for identifying user interface elements on the screen of the information device from the received image;

[2070] a voice input means for receiving voice instructions from the user;

[2071] a generative artificial intelligence means for analyzing a voice instruction and generating a next operation guide based on the user interface elements;

[2072] a display means for displaying the generated operation guide on the display device;

[2073] A system including:

[2074] (Claim 2)

[2075] The system according to claim 1, further comprising means for visually highlighting operation locations to a user in the generated operation guide.

[2076] (Claim 3)

[2077] 2. The system according to claim 1, wherein the generative artificial intelligence means includes means for identifying an operating procedure for an information device based on a voice instruction from the user and generating a specific operating guide.

[2078] "Application Example 1"

[2079] (Claim 1)

[2080] An operation support system for a smart device,

[2081] a display device worn by a user, and an imaging means for capturing a screen of the smart device from the viewpoint of the user;

[2082] a communication means for receiving the captured screen image;

[2083] image analysis means for identifying user interface elements on the screen of the smart device from the received image;

[2084] a voice input means for receiving voice instructions from the user;

[2085] a generative artificial intelligence means for analyzing a voice instruction and generating a next operation guide based on the user interface elements;

[2086] a video output means for displaying the generated operation guide on the display device;

[2087] A display means for users to obtain in-store product information and in-store maps;

[2088] A system including:

[2089] (Claim 2)

[2090] The system according to claim 1, further comprising means for visually highlighting operation locations to a user in the generated operation guide.

[2091] (Claim 3)

[2092] The system according to claim 1, wherein the generative artificial intelligence means includes means for identifying an operation procedure for a smart device based on the user's voice instructions and generating a specific operation guide.

[2093] "Example 2: Combining Emotion Engines"

[2094] (Claim 1)

[2095] a display device worn by a user, and an imaging means for capturing a screen of the smart device from the viewpoint of the user;

[2096] a communication means for receiving the captured screen image;

[2097] image analysis means for identifying user interface elements on the screen of the smart device from the received image;

[2098] a voice acquisition means for receiving a voice instruction from the user;

[2099] a generative artificial intelligence means ...

Claims

1. An operation support system for a smart device, a camera means for capturing a screen of the smart device from the user's point of view; network means for receiving the captured screen image; image recognition means for identifying user interface elements on the screen of the smart device from the received image; a voice input means for receiving voice instructions from the user; a generative artificial intelligence means for analyzing a voice instruction and generating a next operation guide based on the user interface elements; a display means for displaying the generated operation guide on the display device; A system including:

2. The system according to claim 1 , further comprising means for visually highlighting operation points for the user in the generated operation guide.

3. The system according to claim 1 , wherein the generative artificial intelligence means includes means for identifying an operation procedure for a smart device based on the user's voice instruction and generating a specific operation guide.

Images