System

Abstract

A system is provided.SOLUTION: A system including means for capturing a display of a user in real time, means for transmitting the captured image to a server, means for analyzing the received image in the server and performing image recognition, means for capturing a voice of the user and transmitting voice data to the server, means for analyzing the voice data in the server and performing voice recognition, means for generating an operation instruction using an AI generated based on a result of image recognition and a result of voice recognition in the server, means for converting the generated operation instruction into voice data and providing the voice data to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems seniors face when using smartphones and other electronic devices is the sudden changes in notifications, settings, and application operations. These problems are particularly severe for seniors who cannot attend smartphone classes due to health reasons. Furthermore, since conducting smartphone classes for seniors requires significant labor costs, efficient support methods are needed. The present invention aims to provide a means to alleviate such anxiety about smartphone operation for seniors and provide efficient support. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for capturing a user's display in real time, a means for transmitting the captured image to a server, a means for analyzing the received image and performing image recognition in the server, a means for capturing the user's voice and transmitting the voice data to the server, a means for analyzing the voice data and performing voice recognition in the server, a means for generating operation instructions in the server using a generation AI based on the results of image recognition and voice recognition, and a means for converting the generated operation instructions into voice data and providing it to the user. This system can provide real-time assistance to elderly people who encounter problems when operating smartphones and other electronic devices, eliminating their anxiety about operation. It is also expected to reduce labor costs for smartphone classes.

[0006] A "display" is a display device that allows a user to visually confirm information.

[0007] "Real time" refers to a time frame in which processing and reaction occurs immediately.

[0008] "Capture" refers to the act or means of acquiring something, and in this case means capturing images or audio.

[0009] A "server" is a computing device that processes and manages data.

[0010] "Transmitting" refers to the act of sending data from one device to another.

[0011] "Image recognition" refers to the technology of analyzing and understanding specific information and patterns from captured images.

[0012] "Audio data" refers to data that has been converted from audio into digital form.

[0013] "Speech recognition" is a technology that analyzes captured voice data and converts it into text.

[0014] "Generative AI" refers to artificial intelligence that creates natural language instructions based on given information.

[0015] An "operation instruction" is information that instructs the user on what operation to perform.

[0016] "Convert to audio data" refers to the process of converting information in text format into audio format.

[0017] "User" refers to a person who uses this system to operate a smartphone or other electronic device. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures the user's display in real time, sends the image to a server, and uses image and voice recognition on the server to provide operation instructions using generative AI.

[0040] Overall system configuration

[0041] This system mainly consists of the user's devices (smart glasses and smartphone) and a central server. The smart glasses have a built-in camera and microphone, and are responsible for capturing images from the user's smartphone screen and collecting the user's voice. The server is responsible for processing the data and generating operation instructions.

[0042] Program processing overview

[0043] 1. Smart glasses capture the user's display

[0044] The user wears the smart glasses and operates a smartphone.

[0045] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[0046] 2. The server performs image recognition

[0047] It decodes the received image and uses image recognition algorithms to analyze elements on the smartphone screen.

[0048] The analysis results are temporarily stored in a database and the next step is taken.

[0049] 3. Recognize the user's voice

[0050] The smart glasses capture the user's voice with a microphone and send the voice data to a server.

[0051] The server uses a speech recognition engine to convert the voice data into text.

[0052] Analyze text data to identify the user's intent and instructions.

[0053] 4. Generative AI generates instructions

[0054] The results of image recognition and voice recognition are integrated and appropriate operating instructions are generated using generative AI (e.g., large-scale language models).

[0055] For example, if the user's intention is recognized as "I want to delete the notification," a specific instruction such as "Swipe the notification to the left" is generated.

[0056] 5. The server provides audio instructions

[0057] A text-to-speech engine (TTS) is used to convert the generated text instructions into audio data.

[0058] The generated voice data is sent to the smart glasses to provide voice instructions to the user.

[0059] Specific examples

[0060] Example 1: Deleting a notification

[0061] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0062] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0063] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0064] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[0065] 5. The server uses voice recognition to obtain the text "I want to delete this notification."

[0066] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[0067] 7. The server transmits the generated voice data to the smart glasses.

[0068] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0069] Example 2: Changing settings

[0070] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0071] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0072] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0073] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0074] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0075] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[0076] 7. The server transmits the generated voice data to the smart glasses.

[0077] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0078] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] When a user looks at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and compressed.

[0082] Step 2:

[0083] The device (smart glasses) transmits the captured images to the server in real time using a secure communication protocol (e.g., HTTPS).

[0084] Step 3:

[0085] The server decodes the received image data and analyzes it using image recognition algorithms to identify important elements on the screen (notifications, icons, buttons, etc.).

[0086] Step 4:

[0087] The device (smart glasses) captures the user's voice using a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired and sent to the server.

[0088] Step 5:

[0089] The server passes the received voice data through a speech recognition engine to convert it into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[0090] Step 6:

[0091] The server combines the analyzed image recognition results with the voice recognition results and uses generative AI to generate appropriate instructions for the user, such as "swipe left on the notification."

[0092] Step 7:

[0093] The server uses a text-to-speech (TTS) engine to convert the generated text instructions into audio data, which is then sent to the smart glasses.

[0094] Step 8:

[0095] The device (smart glasses) plays the received audio data and provides the user with audio instructions, such as "Swipe the notification left."

[0096] These are the specific processing steps of the program. Through this process, users can receive appropriate and prompt instructions based on the information displayed in real time. This can eliminate the anxiety and difficulty that seniors may have when operating smartphones and other electronic devices.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] The purpose of this project is to provide a system that solves the difficulties that elderly people face when operating smartphones and other electronic devices due to visual, hearing, and operational difficulties. In particular, for people with visual or hearing impairments, it is difficult to provide appropriate operational instructions in real time, and these operations need to be intuitive and easy for users to understand.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for capturing an image of the user's display device in real time, means for transmitting the captured image to a central processing unit, means for analyzing the received image in the central processing unit and performing image recognition, means for recording the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice recognition in the central processing unit, means for generating operation instructions in the central processing unit using a generation AI based on the results of image recognition and voice recognition, and means for converting the generated operation instructions into voice data and providing it to the user. This enables users to receive intuitive and easy-to-understand operation instructions in real time without having difficulty operating their smartphones or other electronic devices.

[0102] "User" refers to the person who operates the system, and in this system includes elderly people who use smart glasses or smartphones.

[0103] "Display device" refers to electronic devices that display visual information, such as smartphones and tablets.

[0104] The term "means for capturing images" refers to a device, such as a camera or an imaging device, that captures the screen of a display device in real time.

[0105] "Central processing unit" refers to a computing resource accessed via a network for data analysis and processing, such as a cloud server or remote server.

[0106] "Transmitting means" refers to technologies and devices for transmitting data to other devices or servers, including wireless communication modules, Wi-Fi, Bluetooth, etc.

[0107] "Means for receiving" refers to the technology and devices for obtaining data via a network.

[0108] "Image recognition" refers to the technology of analyzing captured image data and recognizing specific elements or patterns.

[0109] "Voice recognition" refers to the technology of analyzing recorded voice data and converting what is spoken into text data.

[0110] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to generate text data or operational instructions, including large-scale language models such as GPT-4 (registered trademark).

[0111] "Operation instructions" refers to information that instructs the user to perform specific actions to achieve the operation or goal desired by the user.

[0112] "Means for converting into voice data" refers to technology that converts text data into voice using a text-to-speech synthesis engine (TTS) or similar.

[0113] "Means for providing to the user" refers to devices and technologies for conveying specific operating instructions to the user in voice instructions or other forms.

[0114] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system takes a picture of the user's display device in real time, sends the image to a central processing unit, and performs image and voice recognition in the central processing unit to provide operation instructions using a generative AI.

[0115] Overall system configuration

[0116] This system mainly consists of the user's devices (smart glasses and smartphone) and a central processing unit. The smart glasses have a built-in camera and microphone, and are responsible for capturing images of the user's smartphone screen and collecting the user's voice. The central processing unit is responsible for processing data and generating operating instructions.

[0117] Hardware and Software Configuration

[0118] 1. Smart Glasses

[0119] The smart glasses are equipped with a built-in camera module, microphone, and wireless communication module, which allows them to capture the user's smartphone screen and transmit it to a central processing unit in real time.The microphone also captures the user's voice and transmits it to the central processing unit.

[0120] 2. Central Processing Unit

[0121] The server used as the central processing unit is built on a high-performance cloud infrastructure. This server uses OpenCV and TENSORFLOW® for image analysis, Google® Speech-to-Text API for voice analysis, GPT-4 for generative AI, and Amazon Polly for text-to-speech (TTS) engine.

[0122] Processing flow

[0123] 1. The user puts on the smart glasses

[0124] The user puts on the smart glasses and prepares to operate the smartphone.

[0125] 2. The device (smart glasses) captures the display of the user's smartphone.

[0126] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[0127] 3. The server processes the received image

[0128] The server decodes the received image and analyzes the elements on the screen using image recognition algorithms such as OpenCV and TensorFlow.

[0129] 4. The device (smart glasses) captures the user's voice.

[0130] The smart glasses' microphone is used to capture the user's voice in real time and send the voice data to a server.

[0131] 5. The server recognizes the voice

[0132] The server analyzes the voice data using the Google Speech-to-Text API and converts it into text data.

[0133] 6. Generative AI generates operation instructions

[0134] Based on the results of image and voice recognition, appropriate operation instructions are generated using generative AI such as GPT-4. For example, if the voice recognizes "I want to change my Wi-Fi settings," the instruction generated is "Open the Settings app."

[0135] 7. The server provides audio instructions

[0136] The generated operation instructions are converted into voice data using TTS such as Amazon Polly and sent to the smart glasses to provide voice instructions to the user.

[0137] Specific examples

[0138] Example 1: Deleting a notification

[0139] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0140] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0141] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0142] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[0143] 5. The server uses voice recognition to obtain the text "I want to dismiss this notification."

[0144] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[0145] 7. The server sends the generated voice data to the smart glasses.

[0146] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0147] Example 2: Changing settings

[0148] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0149] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0150] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0151] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0152] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0153] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[0154] 7. The server sends the generated voice data to the smart glasses.

[0155] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0156] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Program processing flow

[0159] Step 1: User puts on smart glasses

[0160] Input: User action (wearing smart glasses)

[0161] Output: Smart glasses start

[0162] Specific behavior:

[0163] When a user puts on the smart glasses, the built-in sensors in the smart glasses detect that they are being worn and activate the camera and microphone, preparing the display device for screen and audio capture.

[0164] Step 2: The device (smart glasses) captures the display of the user's smartphone.

[0165] Input: The user's smartphone display

[0166] Output: Captured image data

[0167] Specific behavior:

[0168] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals (for example, 2 frames per second), and the captured image data is compressed in real time and sent to a central processing unit (server) via a wireless communication module.

[0169] Step 3: The server processes the received image

[0170] Input: Capture image data

[0171] Output: Analysis results (element data on the screen)

[0172] Specific behavior:

[0173] The server decodes the image data received via the network and analyzes elements on the smartphone screen (icons, notifications, etc.) using image recognition algorithms such as OpenCV and TensorFlow. The analysis results are temporarily stored in a database.

[0174] Step 4: The device (smart glasses) captures the user's voice

[0175] Input: User's voice

[0176] Output: Audio data

[0177] Specific behavior:

[0178] When a user speaks operational instructions into the smart glasses, the smart glasses' built-in microphone captures the voice in real time and transmits the voice data to a server via a wireless communication module.

[0179] Step 5: The server analyzes the audio data

[0180] Input: Audio data

[0181] Output: Analysis results (text data)

[0182] Specific behavior:

[0183] The server receives the voice data and converts it into text using a speech recognition engine such as the Google Speech-to-Text API. The resulting text data is used as input for the generation AI.

[0184] Step 6: Generative AI generates operation instructions

[0185] Input: Image analysis results, audio analysis results (text data)

[0186] Output: Instruction text

[0187] Specific behavior:

[0188] The server combines the results of image and voice recognition and generates appropriate instructions using a generative AI model such as GPT-4. For example, if the text "I want to change my Wi-Fi settings" is obtained through voice recognition, the server generates the specific instruction "Open the Settings app."

[0189] Step 7: The server converts the instructions into audio data

[0190] Input: Instruction text

[0191] Output: Operation instruction voice data

[0192] Specific behavior:

[0193] The text instructions generated by the AI ​​are converted into voice data using a text-to-speech (TTS) engine such as Amazon Polly, and the generated voice data is sent to the smart glasses via the network.

[0194] Step 8: The device (smart glasses) plays back the operation instructions.

[0195] Input: Operation instruction voice data

[0196] Output: Instructions provided to the user

[0197] Specific behavior:

[0198] The smart glasses play back the audio data received from the server and give the user instructions such as "Open the Settings app," allowing the user to intuitively understand the appropriate operation.

[0199] These are the specific processing steps of this system's program. The specific operations, inputs, and outputs are clearly defined at each step, allowing for a detailed understanding of how the entire system operates.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] There is an urgent need to address the difficulties that seniors face when using smartphones and other electronic devices. Particularly in brick-and-mortar stores, seniors often experience unfamiliar operational challenges when searching for products or making payments. Complicated procedures and small screen displays can also be a visual burden. The present invention aims to provide a system that provides real-time assistance to facilitate these operations.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for periodically transmitting the captured image of the user's display to the server, means for analyzing the smart device screen using an image recognition algorithm after transmitting the captured image to the server, means for capturing the user's voice with the microphone of the smart glasses and transmitting it to the server, means for analyzing the voice data and performing voice recognition in the server, means for integrating the image recognition results and the voice recognition results and generating specific operation instructions using a generation AI, means for converting the generated operation instructions into voice data using a text-to-speech engine and providing it to the user via the smart glasses, and means for providing purchasing support and payment assistance in physical stores, which enables elderly people to efficiently purchase products in physical stores and smoothly complete payment procedures.

[0205] The "captured user display image" is the display content of the device being operated by the user captured as is and acquired as image data.

[0206] "Transmission means" refers to the technical means by which data is transferred from one device to another.

[0207] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying elements or patterns within it.

[0208] "Means for analyzing the screen of a smart device" refers to a technical means for understanding the display information of a smartphone or other digital device used by a user and extracting the necessary information.

[0209] "Capturing the user's voice with the microphone of the smart glasses" refers to using the microphone built into the smart glasses to obtain the user's speech as voice data.

[0210] "Means for performing speech recognition" refers to the technical means for analyzing captured speech data and expressing its contents in text form.

[0211] "Means for generating specific operating instructions using generative AI" refers to using artificial intelligence technology to create specific operating instructions for users based on the results of voice recognition and image recognition.

[0212] "Text-to-speech engine" refers to the technical means for converting generated text information into speech data.

[0213] "Purchase support and payment assistance in physical stores" refers to smoothly supporting the process of users selecting and purchasing products and making payments in physical stores.

[0214] This invention is a system that allows elderly people to easily purchase products in physical stores and smoothly complete payment procedures. The system uses the user's smart glasses and smartphone as the main terminals and operates in conjunction with a central server.

[0215] Overall system configuration

[0216] The system consists of user devices (smart glasses and smartphones), a central server, and various software modules.

[0217] Main hardware and software used

[0218] Hardware:

[0219] Smart glasses (built-in camera, microphone, and speaker)

[0220] Smartphone

[0221] software:

[0222] Image recognition algorithm (TensorFlow)

[0223] Speech recognition engine (Google Speech-to-Text API)

[0224] Generative AI models (GPT-3 (registered trademark) and similar large-scale language models)

[0225] Text-to-speech engine (Google Text-to-Speech API)

[0226] Program Processing Overview

[0227] 1. Capture:

[0228] When a customer (user) speaks to the smart glasses and says, "I would like to purchase this product," the camera in the smart glasses captures the screen of the POS terminal.

[0229] The captured image is compressed and sent to the server.

[0230] 2. Speech Recognition:

[0231] The microphone in the smart glasses captures the customer's voice and sends the voice data to a server.

[0232] The server uses a speech recognition engine to convert the voice data into text and identify the customer's instructions.

[0233] 3. Image Recognition and Instruction Generation:

[0234] The server decodes the received image and uses image recognition algorithms to analyze the screen elements of the POS terminal.

[0235] The image recognition results and voice recognition results are integrated, and the generative AI model generates appropriate operating instructions.

[0236] For example, an operation instruction such as "Please scan the QR code (registered trademark)" is generated.

[0237] 4. Audio guide:

[0238] The generated operation instructions are converted into voice data by a text-to-speech synthesis engine.

[0239] Voice data is sent to the smart glasses, and operational instructions are provided to the customer via voice.

[0240] Specific examples

[0241] 1. Product purchase assistance:

[0242] A customer says, "I would like to purchase this product."

[0243] The camera in the smart glasses captures the screen of the POS terminal and sends the image to the server.

[0244] The audio data is also sent to the server at the same time.

[0245] Image and voice recognition is performed on the server side, and the generation AI generates operation instructions.

[0246] Operation instructions are converted into voice and provided to the customer through smart glasses.

[0247] For example, you might hear a voice prompt saying, "Please scan the QR code."

[0248] 2. Payment Assistance:

[0249] The customer asks, "How do I pay?"

[0250] The camera in the smart glasses captures the payment terminal screen and sends the image to a server.

[0251] The received voice data is also analyzed by the server.

[0252] The generative AI creates steps and generates specific instructions, such as "insert your card."

[0253] Instructions converted into voice data are communicated to the customer through smart glasses.

[0254] Prompt Sentence Examples

[0255] Based on the user's intent of "I want to purchase this product," analyze the captured image of the POS terminal screen and generate appropriate instructions, including how to scan the QR code.

[0256] In this way, the system of the present invention provides a concrete means for assisting elderly people in purchasing and paying for products in brick-and-mortar stores.

[0257] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0258] Step 1:

[0259] The user speaks to the smart glasses, saying, "I want to buy this product." The microphone in the smart glasses captures this voice. The input is the user's speech, and the output is the captured voice data. This voice data is sent from the smart glasses to the server.

[0260] Step 2:

[0261] After capturing the user's speech, the camera on the smart glasses captures the screen of the POS terminal. The input is the screen of the POS terminal, and the output is the captured screen image. This image is also sent from the smart glasses to the server.

[0262] Step 3:

[0263] The server analyzes the received voice data. The input is the voice data, and the output is the speech recognition result in text format. This analysis is performed using a speech recognition engine (Google Speech-to-Text API).

[0264] Step 4:

[0265] The server analyzes the received image data in parallel. The input is the screen image, and the output is the analysis result to identify the screen elements. An image recognition algorithm (TensorFlow) is used to recognize the information on the POS terminal screen.

[0266] Step 5:

[0267] The server integrates the results of speech recognition and image recognition. The input is the speech recognition result and the image recognition result, and the output is the interpretation data that combines them. This determines the appropriate instruction based on the user's intention (e.g., "I want to purchase this product").

[0268] Step 6:

[0269] The server generates instructions using a generative AI model (such as GPT-3). The input is interpretation data, and the output is specific instructions (e.g., "Scan the QR code"). The generative AI determines the optimal operation procedure based on the results of voice recognition and image recognition.

[0270] Step 7:

[0271] The server converts the generated operation instructions into voice data using a text-to-speech synthesis engine (Google Text-to-Speech API). The input is the text data of the operation instructions, and the output is voice data.

[0272] Step 8:

[0273] The device (smart glasses) receives the voice data sent from the server and provides instructions to the user via the speaker. The input is voice data, and the output is voice guidance to the user. The smart glasses provide specific voice instructions such as "Please scan the QR code."

[0274] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0275] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. The system captures the user's display in real time, analyzes image and voice data on a server, and generates and provides appropriate operating instructions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions.

[0276] Overall system configuration

[0277] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server is responsible for processing the data and generating operation instructions, while the emotion engine is responsible for analyzing the user's emotions from the collected data.

[0278] Program processing overview

[0279] 1. Smart glasses capture the user's display

[0280] The user wears the smart glasses and operates a smartphone.

[0281] The camera in the smart glasses captures the screen of the user's smartphone in real time, compresses the image and sends it to a server.

[0282] 2. The server performs image recognition

[0283] The received image data is analyzed and image recognition algorithms are used to identify on-screen elements (notifications, icons, buttons, etc.) The analysis results are temporarily stored in a database.

[0284] 3. Smart glasses will capture your voice

[0285] The smart glasses capture the user's voice with a built-in microphone and send the audio data to a server.

[0286] 4. The server performs voice recognition

[0287] The received voice data is passed through a speech recognition engine to be converted into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[0288] 5. Emotion engine recognizes user emotions

[0289] The smart glasses capture the user's facial expressions and send the image data to a server, which also analyzes the voice data, and an emotion engine determines the user's emotions from the voice and facial expressions.

[0290] 6. Generative AI generates instructions

[0291] It combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operating instructions using generative AI. For example, when the user is feeling anxious, it intends to generate more polite and easy-to-understand instructions.

[0292] 7. The server provides audio instructions

[0293] The generated text instructions are converted into voice data using a text-to-speech (TTS) engine, and the generated voice data is sent to the smart glasses.

[0294] 8. Smart glasses will play voice commands

[0295] The smart glasses then play back the received audio data and provide the user with voice instructions, allowing them to perform operations while receiving real-time assistance.

[0296] Specific examples

[0297] Example 1: Deleting a notification

[0298] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0299] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[0300] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[0301] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[0302] 5. The server transmits the generated voice data to the smart glasses.

[0303] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0304] Example 2: Changing settings

[0305] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0306] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[0307] 3. The server uses an image recognition algorithm to detect icons on the home screen and in the settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[0308] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0309] 5. The server transmits the generated voice data to the smart glasses.

[0310] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0311] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[0312] The processing flow will be explained below.

[0313] Step 1:

[0314] When a user is looking at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and the image data is compressed.

[0315] Step 2:

[0316] The device (smart glasses) transmits the captured image data to the server in real time using a secure communication protocol (e.g., HTTPS).

[0317] Step 3:

[0318] The server decodes the received image data and uses image recognition algorithms to analyze on-screen elements (notifications, icons, buttons, etc.) and temporarily stores the analysis results in a database.

[0319] Step 4:

[0320] The device (smart glasses) captures the user's voice with a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired, compressed, and sent to the server.

[0321] Step 5:

[0322] The server converts the received voice data into text data through a speech recognition engine, and analyzes the converted text data to identify the user's intentions and instructions.

[0323] Step 6:

[0324] The device (smart glasses) uses a built-in camera to capture the user's facial expressions, and transmits the acquired facial expression data to a server.

[0325] Step 7:

[0326] The server runs the facial expression data through an emotion engine to analyze the user's emotions. It also determines emotions from voice data and combines them with the facial expression data to determine the user's current emotional state.

[0327] Step 8:

[0328] The server combines the results of image recognition, voice recognition, and the emotion engine. Based on all of this data, it uses generative AI to generate operational instructions. For example, if the user is feeling anxious, it generates gentle and polite instructions such as "Please remain calm and swipe left on the notification."

[0329] Step 9:

[0330] The server passes the generated text instructions through a text-to-speech engine (TTS) to convert them into voice data, which is then sent to the smart glasses.

[0331] Step 10:

[0332] The device (smart glasses) plays the received voice data and provides the user with voice instructions, allowing the user to perform operations while receiving real-time assistance.

[0333] The following is an explanation using a specific example.

[0334] Example 1: Deleting a notification

[0335] Step 1:

[0336] The user speaks to the smart glasses, saying, "I want to delete this notification."

[0337] Step 2:

[0338] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0339] Step 3:

[0340] The device (smart glasses) captures the user's voice data and sends it to the server.

[0341] Step 4:

[0342] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[0343] Step 5:

[0344] The server uses an image recognition algorithm to detect the notification on the screen.

[0345] Step 6:

[0346] The server uses voice recognition to obtain the text "I want to delete this notification."

[0347] Step 7:

[0348] The server analyzes the user's emotions using an emotion engine.

[0349] Step 8:

[0350] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate instructions such as "Swipe left on the notification," which are then converted into voice data.

[0351] Step 9:

[0352] The server transmits the generated voice data to the smart glasses.

[0353] Step 10:

[0354] The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0355] Example 2: Changing settings

[0356] Step 1:

[0357] The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0358] Step 2:

[0359] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0360] Step 3:

[0361] The device (smart glasses) captures the user's voice data and sends it to the server.

[0362] Step 4:

[0363] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[0364] Step 5:

[0365] The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0366] Step 6:

[0367] The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0368] Step 7:

[0369] The server analyzes the user's emotions using an emotion engine.

[0370] Step 8:

[0371] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0372] Step 9:

[0373] The server transmits the generated voice data to the smart glasses.

[0374] Step 10:

[0375] The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0376] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[0377] Example 2

[0378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0379] Currently, the difficulties that older people experience when operating smartphones and other electronic devices stem from the learning curve and cognitive load. Complex setting changes and notification operations, in particular, pose a significant burden for older people. Furthermore, existing operation assistance systems are unable to provide appropriate assistance that takes into account the user's emotional state. Therefore, there is a need for systems that enable older people to operate electronic devices more easily and without stress.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0381] In this invention, the server includes means for capturing a user's display device in real time, means for compressing the captured image and transmitting it to a central processing unit, means for analyzing the received image in the central processing unit and performing object recognition, means for capturing the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice conversion in the central processing unit, means for generating operation instructions in the central processing unit using generative artificial intelligence based on the results of the object recognition and the voice conversion, means for converting the generated operation instructions into voice data and providing them to the user, means for capturing the user's facial expressions and performing emotion analysis, and means for reflecting the results of the emotion analysis in the generation of operation instructions. This reduces the difficulty that elderly people experience when operating electronic devices and makes it possible to provide optimal operation assistance in real time according to the user's emotional state.

[0382] A "display device" is a part of an electronic device that allows a user to visually receive information, specifically a device that includes a screen such as a smartphone or tablet.

[0383] A "central processing unit" is a computer system that receives, analyzes, and generates instructions, and specifically includes a server or a cloud computing platform.

[0384] "Object recognition" is a technology that identifies specific objects or elements (e.g., notifications, icons, buttons, etc.) from image data and determines their presence.

[0385] "Speech conversion" is a technology that converts voice data into text data, and is used to analyze a user's intentions and instructions.

[0386] "Generative artificial intelligence" is an AI technology that generates appropriate operating instructions based on multiple data (for example, the results of image recognition, voice conversion, etc.).

[0387] "Emotion analysis" is a technology that determines a user's emotional state based on the user's voice and facial expression data.

[0388] "Operation assistance" refers to instructions and assistance functions provided in real time when a user operates an electronic device, and is assistance that makes user operations easier and more efficient.

[0389] MODE FOR CARRYING OUT THE INVENTION

[0390] This invention is a system for reducing the difficulty that elderly people have when operating smartphones and other electronic devices. This system captures the user's display device in real time, analyzes the image and voice data, and generates and provides appropriate operating instructions. Furthermore, it has the function of recognizing the user's emotions and flexibly adjusting the operating instructions based on the results.

[0391] Overall system configuration

[0392] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine that performs emotion analysis. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server processes the data and generates operation instructions, while the emotion engine analyzes the user's emotions from the collected data.

[0393] Program processing

[0394] 1. Smart glasses capture the user's display

[0395] The user wears smart glasses and operates a smartphone. The camera in the smart glasses captures the smartphone screen in real time, compresses the image data, and sends it to a server. In this case, the specific hardware used is the smart glasses, and the software used is a software module that can process image data in real time.

[0396] 2. The server performs image recognition

[0397] The server analyzes the image data received from the smart glasses and uses image recognition algorithms such as OpenCV to identify elements in the image (notifications, icons, buttons, etc.). The results are temporarily stored in a database.

[0398] 3. Smart glasses will capture your voice

[0399] The user speaks to the smart glasses, and the built-in microphone in the smart glasses captures the voice and sends the voice data to the server.

[0400] 4. The server performs voice recognition

[0401] The server converts the voice data received from the smart glasses into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), then analyzes the converted text data to determine what the user wants.

[0402] 5. Emotion engine recognizes user emotions

[0403] The user's facial expression data is captured and sent to the server, where it is analyzed together with the voice data, and the emotion engine determines the user's emotion from the voice and facial expression.

[0404] 6. Generative AI generates instructions

[0405] The generative AI model combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operational instructions. For example, if the user feels anxious, a polite and specific instruction such as "Open the Settings app" is generated. An example of a prompt sentence from this generative AI is as follows:

[0406] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[0407] 7. The server provides audio instructions

[0408] The generated operation instructions are converted into voice data using a Text-to-Speech (TTS) engine, and this voice data is sent to the smart glasses.

[0409] 8. Smart glasses will play voice commands

[0410] The smart glasses then play back the received voice data and provide real-time voice instructions to the user, allowing them to perform operations without stress.

[0411] Specific examples

[0412] Example 1: Deleting a notification

[0413] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0414] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[0415] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[0416] 4. The server combines the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[0417] 5. The server sends the generated voice data to the smart glasses.

[0418] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0419] Example 2: Changing settings

[0420] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0421] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[0422] 3. The server uses an image recognition algorithm to detect icons on the home screen and settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[0423] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0424] 5. The server sends the generated voice data to the smart glasses.

[0425] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0426] In this way, the system collects and analyzes real-time user data to help elderly people operate electronic devices safely and efficiently.

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Processing Steps

[0429] Step 1: Smart glasses capture the user's display

[0430] Input: The user wears the smart glasses and operates a smartphone.

[0431] How it works: The device (smart glasses) uses its built-in camera to capture the screen of the user's smartphone in real time.

[0432] Data processing: Compress the captured image data to reduce the data size.

[0433] Output: Send the compressed image data to the server.

[0434] Step 2: The server performs image recognition

[0435] Input: The server receives the compressed image data sent from the smart glasses.

[0436] How it works: The server uses image recognition algorithms such as OpenCV to identify elements on the image (notifications, icons, buttons, etc.).

[0437] Data processing: Analyze image data and identify and classify each element.

[0438] Output: The analysis results are temporarily saved in a database.

[0439] Step 3: Smart glasses capture user's voice

[0440] Input: The user speaks their intention to operate the smart glasses.

[0441] How it works: The device (smart glasses) uses a built-in microphone to capture the user's voice.

[0442] Data processing: Converting captured audio data into a certain format.

[0443] Output: Sends the audio data to the server.

[0444] Step 4: The server performs speech recognition

[0445] Input: The server receives the audio data sent from the smart glasses.

[0446] How it works: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio data into text.

[0447] Data processing: Convert the audio data into text data and analyze the meaning.

[0448] Output: Send the parsed text data to the appropriate processing module in the server.

[0449] Step 5: The emotion engine recognizes the user's emotion

[0450] Input: The smart glasses capture the user's facial expressions and send them to the server, as well as audio data.

[0451] Operation: The server uses an emotion engine to analyze facial and voice data to determine the user's emotional state.

[0452] Data processing: Combining facial expression analysis and voice analysis to identify the user's emotions.

[0453] Output: The results of sentiment analysis are saved in a database on the server.

[0454] Step 6: Generative AI generates instructions

[0455] Input: The results of image recognition, voice recognition, and emotion recognition are stored in a database on the server.

[0456] Action: The server integrates this data and uses a generative AI model to generate appropriate operational instructions.

[0457] Data processing: Various data are input into the generative AI model as prompt sentences to generate instruction sentences.

[0458] Output: The generated operation instructions are output as text data. This text includes specific operation methods.

[0459] Prompt Sentence Examples

[0460] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[0461] Step 7: The server provides spoken instructions

[0462] Input: Text data of operation instructions generated by the generative AI model.

[0463] How it works: The server uses a text-to-speech (TTS) engine to convert text data into audio data (e.g., Google Cloud Text-to-Speech).

[0464] Data processing: Convert text data into audio data.

[0465] Output: The generated audio data is sent to the smart glasses.

[0466] Step 8: Smart glasses play voice commands

[0467] Input: Audio data sent from the server.

[0468] Operation: The device (smart glasses) plays the received audio data and provides real-time operating instructions to the user.

[0469] Data processing: None in particular (data reproduction).

[0470] Output: Provides an environment where the user can listen to instructions and assists with operation.

[0471] (Application example 2)

[0472] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0473] Elderly people often experience difficulties when searching for products in physical stores, completing purchase procedures, and obtaining campaign information. Therefore, there is a need for support systems that allow elderly people to enjoy shopping independently and comfortably. While current technology supports elderly people using voice recognition and image recognition, there is a lack of systems that combine emotion recognition and real-time support, which has resulted in a lack of improvements in the user experience.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0475] In this invention, the server includes means for capturing a user's display in real time, means for transmitting the captured image to the server, means for analyzing the received image in the server and performing image recognition, means for capturing the user's voice and transmitting the voice data to the server, means for analyzing the voice data in the server and performing voice recognition, means for generating operation instructions in the server using a generation AI based on the results of the image recognition and the voice recognition, means for converting the generated operation instructions into voice data and providing it to the user, means for receiving a question about a specific product so that the user can search for the product in a physical store and providing voice guidance on the location of the product on the shelf, means for obtaining information on the product's barcode and purchase procedure and providing voice guidance on how to purchase, and means for obtaining campaign information such as discounted products from a database and providing voice guidance in real time. This enables elderly people to enjoy shopping independently and comfortably in physical stores.

[0476] "Capturing a user's display" means taking a real-time picture of the screen of the electronic device being used by the user.

[0477] A "server" is a central control unit that processes data and provides necessary information over a network.

[0478] "Image recognition" is a technology that analyzes captured image data and identifies specific objects, characters, symbols, etc. within it.

[0479] "Voice capture" is the process of recording a user's speech with a microphone and collecting that voice data.

[0480] "Speech recognition" is a technology that analyzes captured voice data and converts it into text data.

[0481] "Generative AI" is an artificial intelligence technology that generates appropriate output based on input data, creating operational instructions based on user questions and requests.

[0482] "Generation of operation instructions" is the process of determining appropriate actions based on the results of image recognition and voice recognition, and generating those instructions.

[0483] "Converting voice data" refers to the process of converting the generated textual instructions into voice using speech synthesis technology.

[0484] A "physical store" is a commercial facility that consumers can physically visit and where goods and services are offered.

[0485] "Product search" is the process of gathering information and providing guidance needed to locate a specific product.

[0486] "Shelf location information guidance" refers to audio notification of where specific products are located within the store.

[0487] "Barcode acquisition" refers to reading the barcode information to identify the uniqueness of the product.

[0488] "Purchase procedure guidance" refers to audio instructions explaining the steps and methods for purchasing a product.

[0489] "Providing campaign information" means informing users in real time about current discounts, special offers, and other information.

[0490] A "database" is an electronic information storage system that organizes and stores information, allowing it to be quickly searched and updated as needed.

[0491] This system aims to alleviate the difficulties that seniors face when searching for products in brick-and-mortar stores, completing purchase procedures, and obtaining campaign information. In particular, it uses smart glasses, smartphones, a central control server, and an emotion engine to help seniors enjoy shopping independently and comfortably.

[0492] Overall system configuration

[0493] The system mainly consists of the user's devices (smart glasses and smartphone), a central server, an emotion engine, and a database. The smart glasses are equipped with a camera and microphone to capture the user's voice and image. The server processes this data and generates the necessary operation instructions, which are then provided via voice.

[0494] Hardware and Software

[0495] Smart glasses: Built-in camera and microphone capture the user's voice and display. Example: A common name for smart wearable devices.

[0496] Server: Processes and analyzes data and generates operational instructions. Example: A common name is a cloud server.

[0497] Speech recognition engine: converts voice data into text. Example: speech processing software.

[0498] Image Recognition Engine: Analyzes image data and extracts information. Example: Image analysis software.

[0499] Emotion engine: Determines the user's emotions from voice and facial expression data. Example: emotion analysis software.

[0500] Database: Stores product information, store maps, and campaign information and updates them in real time. Example: Data storage system.

[0501] Specific functions and operations of the system

[0502] 1. Product search support

[0503] A user puts on smart glasses and asks, "Where is this item?"

[0504] The device (smart glasses) captures images of the store using a camera and sends them to a server. It also captures the user's voice using a microphone and sends it to the server.

[0505] The server analyzes the received data using an image recognition engine and a voice recognition engine to identify the product's location information.

[0506] The server uses generation AI to generate shelf location information as voice instructions, and then converts this into voice data using a TTS engine.

[0507] The terminal (smart glasses) guides the generated voice instructions to the user.

[0508] 2. Purchase procedure assistance

[0509] The user asks the smart glasses, "How do I purchase this product?"

[0510] The device (smart glasses) captures the user's voice and sends it to the server.

[0511] The server converts the voice data into text using a voice recognition engine and obtains information about the purchase procedure.

[0512] The server uses generative AI to generate purchase procedure instructions and converts them into voice data using a TTS engine.

[0513] The terminal (smart glasses) guides the generated voice instructions to the user.

[0514] 3. Providing campaign information

[0515] The user asks the smart glasses, "What are the deals today?"

[0516] The device (smart glasses) captures the user's voice and sends it to the server.

[0517] The server converts the voice data into text using a voice recognition engine and retrieves the latest campaign information from the database in real time.

[0518] The server uses generation AI to generate campaign information as voice instructions and converts them into voice data using a TTS engine.

[0519] The terminal (smart glasses) guides the generated voice instructions to the user.

[0520] Examples of concrete examples and prompts

[0521] Specific examples of product searches

[0522] 1. A user asks, "Where is this item?"

[0523] 2. The device (smart glasses) captures images and audio and sends them to the server.

[0524] 3. The server identifies the product's location and provides a voice message saying, "The product is on the left side of aisle 3."

[0525] Example prompt sentence:

[0526] When you ask, "Where is this product?" the system analyzes camera footage from within the store and provides voice guidance on the location of the product.

[0527] Examples of campaign information provision

[0528] 1. A user asks, "What are the discounted items today?"

[0529] 2. The device (smart glasses) captures the audio and sends it to the server.

[0530] 3. The server retrieves the latest campaign information from the database and announces, "Today's discounted items are apples and oranges."

[0531] Example prompt sentence:

[0532] When you ask, "What are the discounted items today?" the system will query the database and provide you with the latest campaign information via voice.

[0533] The system combines advanced technologies such as real-time image capture, voice recognition, and emotion recognition to help seniors shop independently and comfortably in brick-and-mortar stores.

[0534] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0535] Step 1:

[0536] The user puts on the smart glasses and asks a question.

[0537] Input: User utterance ("Where is this item?").

[0538] Output: Audio data.

[0539] Specific operation: The user asks a question into the microphone of the smart glasses. The voice is captured by the microphone of the smart glasses and collected as voice data.

[0540] Step 2:

[0541] The device (smart glasses) captures images of the inside of the store using a camera and sends them to the server along with audio data.

[0542] Input: User's voice data and store image data.

[0543] Output: Audio and image data sent to the server.

[0544] How it works: The smart glasses' camera captures images of the store in real time, while the smart glasses' microphone simultaneously collects audio data and sends both to a server.

[0545] Step 3:

[0546] The image data received by the server is analyzed using an image analysis engine.

[0547] Input: Image data sent to the server.

[0548] Output: Image analysis results (shelf and item location information).

[0549] Specific operation: The server passes the received image data to an image analysis engine (e.g., image analysis software) for analysis. It extracts the location information of shelves and products in the image and temporarily stores the analysis results.

[0550] Step 4:

[0551] The server analyzes the received voice data using a voice recognition engine.

[0552] Input: The audio data sent to the server.

[0553] Output: Text data (user question).

[0554] Specific operation: The server passes the voice data to a voice recognition engine (e.g., voice processing software) and converts the voice into text, which extracts the question asked by the user as text data.

[0555] Step 5:

[0556] The server integrates the image recognition results and voice recognition results and generates operation instructions using a generative AI model.

[0557] Input: Image analysis results and text data.

[0558] Output: Text data of operation instructions.

[0559] Specific operation: The server integrates the image analysis results and the voice recognition results, and generates appropriate operation instructions for the user's question using a generative AI model (e.g., generative AI). The generated operation instructions are saved as text data.

[0560] Step 6:

[0561] The server converts the generated operation instructions into voice data using a TTS engine.

[0562] Input: Text data of operation instructions.

[0563] Output: Audio data.

[0564] Specific operation: The server passes the text-based operation instructions to a TTS engine (e.g., speech synthesis software) and converts them into voice data, which is then sent to the smart glasses.

[0565] Step 7:

[0566] The device (smart glasses) plays the received audio data to the user.

[0567] Input: Audio data sent from the server.

[0568] Output: Audio instructions provided to the user.

[0569] Specific operation: The smart glasses play the received audio data through a speaker and provide the user with audio instructions, such as guidance on the location of product shelves and the purchase procedure.

[0570] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0572] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0573] [Second embodiment]

[0574] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0575] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0576] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0577] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0578] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0579] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0580] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0581] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0582] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0583] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0584] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0585] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0586] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures the user's display in real time, sends the image to a server, and uses image and voice recognition on the server to provide operation instructions using generative AI.

[0587] Overall system configuration

[0588] This system mainly consists of the user's devices (smart glasses and smartphone) and a central server. The smart glasses have a built-in camera and microphone, and are responsible for capturing images from the user's smartphone screen and collecting the user's voice. The server is responsible for processing the data and generating operation instructions.

[0589] Program processing overview

[0590] 1. Smart glasses capture the user's display

[0591] The user wears the smart glasses and operates a smartphone.

[0592] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[0593] 2. The server performs image recognition

[0594] It decodes the received image and uses image recognition algorithms to analyze elements on the smartphone screen.

[0595] The analysis results are temporarily stored in a database and the next step is taken.

[0596] 3. Recognize the user's voice

[0597] The smart glasses capture the user's voice with a microphone and send the voice data to a server.

[0598] The server uses a speech recognition engine to convert the voice data into text.

[0599] Analyze text data to identify the user's intent and instructions.

[0600] 4. Generative AI generates instructions

[0601] The results of image recognition and voice recognition are integrated and appropriate operating instructions are generated using generative AI (e.g., large-scale language models).

[0602] For example, if the user's intention is recognized as "I want to delete the notification," a specific instruction such as "Swipe the notification to the left" is generated.

[0603] 5. The server provides audio instructions

[0604] A text-to-speech engine (TTS) is used to convert the generated text instructions into audio data.

[0605] The generated voice data is sent to the smart glasses to provide voice instructions to the user.

[0606] Specific examples

[0607] Example 1: Deleting a notification

[0608] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0609] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0610] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0611] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[0612] 5. The server uses voice recognition to obtain the text "I want to delete this notification."

[0613] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[0614] 7. The server transmits the generated voice data to the smart glasses.

[0615] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0616] Example 2: Changing settings

[0617] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0618] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0619] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0620] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0621] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0622] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[0623] 7. The server transmits the generated voice data to the smart glasses.

[0624] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0625] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[0626] The processing flow will be explained below.

[0627] Step 1:

[0628] When a user looks at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and compressed.

[0629] Step 2:

[0630] The device (smart glasses) transmits the captured images to the server in real time using a secure communication protocol (e.g., HTTPS).

[0631] Step 3:

[0632] The server decodes the received image data and analyzes it using image recognition algorithms to identify important elements on the screen (notifications, icons, buttons, etc.).

[0633] Step 4:

[0634] The device (smart glasses) captures the user's voice using a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired and sent to the server.

[0635] Step 5:

[0636] The server passes the received voice data through a speech recognition engine to convert it into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[0637] Step 6:

[0638] The server combines the analyzed image recognition results with the voice recognition results and uses generative AI to generate appropriate instructions for the user, such as "swipe left on the notification."

[0639] Step 7:

[0640] The server uses a text-to-speech (TTS) engine to convert the generated text instructions into audio data, which is then sent to the smart glasses.

[0641] Step 8:

[0642] The device (smart glasses) plays the received audio data and provides the user with audio instructions, such as "Swipe the notification left."

[0643] These are the specific processing steps of the program. Through this process, users can receive appropriate and prompt instructions based on the information displayed in real time. This can eliminate the anxiety and difficulty that seniors may have when operating smartphones and other electronic devices.

[0644] Example 1

[0645] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] The purpose of this project is to provide a system that solves the difficulties that elderly people face when operating smartphones and other electronic devices due to visual, hearing, and operational difficulties. In particular, for people with visual or hearing impairments, it is difficult to provide appropriate operational instructions in real time, and these operations need to be intuitive and easy for users to understand.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0648] In this invention, the server includes means for capturing an image of the user's display device in real time, means for transmitting the captured image to a central processing unit, means for analyzing the received image in the central processing unit and performing image recognition, means for recording the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice recognition in the central processing unit, means for generating operation instructions in the central processing unit using a generation AI based on the results of image recognition and voice recognition, and means for converting the generated operation instructions into voice data and providing it to the user. This enables users to receive intuitive and easy-to-understand operation instructions in real time without having difficulty operating their smartphones or other electronic devices.

[0649] "User" refers to the person who operates the system, and in this system includes elderly people who use smart glasses or smartphones.

[0650] "Display device" refers to electronic devices that display visual information, such as smartphones and tablets.

[0651] The term "means for capturing images" refers to a device, such as a camera or an imaging device, that captures the screen of a display device in real time.

[0652] "Central processing unit" refers to a computing resource accessed via a network for data analysis and processing, such as a cloud server or remote server.

[0653] "Transmitting means" refers to technologies and devices for transmitting data to other devices or servers, including wireless communication modules, Wi-Fi, Bluetooth, etc.

[0654] "Means for receiving" refers to the technology and devices for obtaining data via a network.

[0655] "Image recognition" refers to the technology of analyzing captured image data and recognizing specific elements or patterns.

[0656] "Voice recognition" refers to the technology of analyzing recorded voice data and converting what is spoken into text data.

[0657] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to generate text data or operational instructions, including large-scale language models such as GPT-4.

[0658] "Operation instructions" refers to information that instructs the user to perform specific actions to achieve the operation or goal desired by the user.

[0659] "Means for converting into voice data" refers to technology that converts text data into voice using a text-to-speech synthesis engine (TTS) or similar.

[0660] "Means for providing to the user" refers to devices and technologies for conveying specific operating instructions to the user in voice instructions or other forms.

[0661] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures a user's display device in real time, sends the image to a central processing unit, and uses a generation AI to provide operating instructions by performing image and voice recognition in the central processing unit.

[0662] Overall system configuration

[0663] This system mainly consists of the user's devices (smart glasses and smartphone) and a central processing unit. The smart glasses have a built-in camera and microphone, and are responsible for capturing images of the user's smartphone screen and collecting the user's voice. The central processing unit is responsible for processing data and generating operating instructions.

[0664] Hardware and Software Configuration

[0665] 1. Smart Glasses

[0666] The smart glasses are equipped with a built-in camera module, microphone, and wireless communication module, which allows them to capture the user's smartphone screen and transmit it to a central processing unit in real time.The microphone also captures the user's voice and transmits it to the central processing unit.

[0667] 2. Central Processing Unit

[0668] The server used as the central processing unit is built on a high-performance cloud infrastructure. This server uses OpenCV and TensorFlow for image analysis, the Google Speech-to-Text API for voice analysis, GPT-4 for generative AI, and Amazon Polly for text-to-speech (TTS) engine.

[0669] Processing flow

[0670] 1. The user puts on the smart glasses

[0671] The user puts on the smart glasses and prepares to operate the smartphone.

[0672] 2. The device (smart glasses) captures the display of the user's smartphone.

[0673] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[0674] 3. The server processes the received image

[0675] The server decodes the received image and analyzes the elements on the screen using image recognition algorithms such as OpenCV and TensorFlow.

[0676] 4. The device (smart glasses) captures the user's voice.

[0677] The smart glasses' microphone is used to capture the user's voice in real time and send the voice data to a server.

[0678] 5. The server recognizes the voice

[0679] The server analyzes the voice data using the Google Speech-to-Text API and converts it into text data.

[0680] 6. Generative AI generates operation instructions

[0681] Based on the results of image and voice recognition, appropriate operation instructions are generated using generative AI such as GPT-4. For example, if the voice recognizes "I want to change my Wi-Fi settings," the instruction generated is "Open the Settings app."

[0682] 7. The server provides audio instructions

[0683] The generated operation instructions are converted into voice data using TTS such as Amazon Polly and sent to the smart glasses to provide voice instructions to the user.

[0684] Specific examples

[0685] Example 1: Deleting a notification

[0686] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0687] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0688] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0689] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[0690] 5. The server uses voice recognition to obtain the text "I want to dismiss this notification."

[0691] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[0692] 7. The server sends the generated voice data to the smart glasses.

[0693] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0694] Example 2: Changing settings

[0695] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0696] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0697] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[0698] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0699] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0700] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[0701] 7. The server sends the generated voice data to the smart glasses.

[0702] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0703] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[0704] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0705] Program processing flow

[0706] Step 1: User puts on smart glasses

[0707] Input: User action (wearing smart glasses)

[0708] Output: Smart glasses start

[0709] Specific behavior:

[0710] When a user puts on the smart glasses, the built-in sensors in the smart glasses detect that they are being worn and activate the camera and microphone, preparing the display device for screen and audio capture.

[0711] Step 2: The device (smart glasses) captures the display of the user's smartphone.

[0712] Input: The user's smartphone display

[0713] Output: Captured image data

[0714] Specific behavior:

[0715] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals (for example, 2 frames per second), and the captured image data is compressed in real time and sent to a central processing unit (server) via a wireless communication module.

[0716] Step 3: The server processes the received image

[0717] Input: Capture image data

[0718] Output: Analysis results (element data on the screen)

[0719] Specific behavior:

[0720] The server decodes the image data received via the network and analyzes elements on the smartphone screen (icons, notifications, etc.) using image recognition algorithms such as OpenCV and TensorFlow. The analysis results are temporarily stored in a database.

[0721] Step 4: The device (smart glasses) captures the user's voice

[0722] Input: User's voice

[0723] Output: Audio data

[0724] Specific behavior:

[0725] When a user speaks operational instructions into the smart glasses, the smart glasses' built-in microphone captures the voice in real time and transmits the voice data to a server via a wireless communication module.

[0726] Step 5: The server analyzes the audio data

[0727] Input: Audio data

[0728] Output: Analysis results (text data)

[0729] Specific behavior:

[0730] The server receives the voice data and converts it into text using a speech recognition engine such as the Google Speech-to-Text API. The resulting text data is used as input for the generation AI.

[0731] Step 6: Generative AI generates operation instructions

[0732] Input: Image analysis results, audio analysis results (text data)

[0733] Output: Instruction text

[0734] Specific behavior:

[0735] The server combines the results of image and voice recognition and generates appropriate instructions using a generative AI model such as GPT-4. For example, if the text "I want to change my Wi-Fi settings" is obtained through voice recognition, the server generates the specific instruction "Open the Settings app."

[0736] Step 7: The server converts the instructions into audio data

[0737] Input: Instruction text

[0738] Output: Operation instruction voice data

[0739] Specific behavior:

[0740] The text instructions generated by the AI ​​are converted into voice data using a text-to-speech (TTS) engine such as Amazon Polly, and the generated voice data is sent to the smart glasses via the network.

[0741] Step 8: The device (smart glasses) plays back the operation instructions.

[0742] Input: Operation instruction voice data

[0743] Output: Instructions provided to the user

[0744] Specific behavior:

[0745] The smart glasses play back the audio data received from the server and give the user instructions such as "Open the Settings app," allowing the user to intuitively understand the appropriate operation.

[0746] These are the specific processing steps of this system's program. The specific operations, inputs, and outputs are clearly defined at each step, allowing for a detailed understanding of how the entire system operates.

[0747] (Application example 1)

[0748] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0749] There is an urgent need to address the difficulties that seniors face when using smartphones and other electronic devices. Particularly in brick-and-mortar stores, seniors often experience unfamiliar operational challenges when searching for products or making payments. Complicated procedures and small screen displays can also be a visual burden. The present invention aims to provide a system that provides real-time assistance to facilitate these operations.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0751] In this invention, the server includes means for periodically transmitting the captured image of the user's display to the server, means for analyzing the smart device screen using an image recognition algorithm after transmitting the captured image to the server, means for capturing the user's voice with the microphone of the smart glasses and transmitting it to the server, means for analyzing the voice data and performing voice recognition in the server, means for integrating the image recognition results and the voice recognition results and generating specific operation instructions using a generation AI, means for converting the generated operation instructions into voice data using a text-to-speech engine and providing it to the user via the smart glasses, and means for providing purchasing support and payment assistance in physical stores, which enables elderly people to efficiently purchase products in physical stores and smoothly complete payment procedures.

[0752] The "captured user display image" is the display content of the device being operated by the user captured as is and acquired as image data.

[0753] "Transmission means" refers to the technical means by which data is transferred from one device to another.

[0754] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying elements or patterns within it.

[0755] "Means for analyzing the screen of a smart device" refers to a technical means for understanding the display information of a smartphone or other digital device used by a user and extracting the necessary information.

[0756] "Capturing the user's voice with the microphone of the smart glasses" refers to using the microphone built into the smart glasses to obtain the user's speech as voice data.

[0757] "Means for performing speech recognition" refers to the technical means for analyzing captured speech data and expressing its contents in text form.

[0758] "Means for generating specific operating instructions using generative AI" refers to using artificial intelligence technology to create specific operating instructions for users based on the results of voice recognition and image recognition.

[0759] "Text-to-speech engine" refers to the technical means for converting generated text information into speech data.

[0760] "Purchase support and payment assistance in physical stores" refers to smoothly supporting the process of users selecting and purchasing products and making payments in physical stores.

[0761] This invention is a system that allows elderly people to easily purchase products in physical stores and smoothly complete payment procedures. The system uses the user's smart glasses and smartphone as the main terminals and operates in conjunction with a central server.

[0762] Overall system configuration

[0763] The system consists of user devices (smart glasses and smartphones), a central server, and various software modules.

[0764] Main hardware and software used

[0765] Hardware:

[0766] Smart glasses (built-in camera, microphone, and speaker)

[0767] Smartphone

[0768] software:

[0769] Image recognition algorithm (TensorFlow)

[0770] Speech recognition engine (Google Speech-to-Text API)

[0771] Generative AI models (GPT-3 and similar large-scale language models)

[0772] Text-to-speech engine (Google Text-to-Speech API)

[0773] Program Processing Overview

[0774] 1. Capture:

[0775] When a customer (user) speaks to the smart glasses and says, "I would like to purchase this product," the camera in the smart glasses captures the screen of the POS terminal.

[0776] The captured image is compressed and sent to the server.

[0777] 2. Speech Recognition:

[0778] The microphone in the smart glasses captures the customer's voice and sends the voice data to a server.

[0779] The server uses a speech recognition engine to convert the voice data into text and identify the customer's instructions.

[0780] 3. Image Recognition and Instruction Generation:

[0781] The server decodes the received image and uses image recognition algorithms to analyze the screen elements of the POS terminal.

[0782] The image recognition results and voice recognition results are integrated, and the generative AI model generates appropriate operating instructions.

[0783] For example, generate an operation instruction such as "Scan the QR code."

[0784] 4. Audio guide:

[0785] The generated operation instructions are converted into voice data by a text-to-speech synthesis engine.

[0786] Voice data is sent to the smart glasses, and operational instructions are provided to the customer via voice.

[0787] Specific examples

[0788] 1. Product purchase assistance:

[0789] A customer says, "I would like to purchase this product."

[0790] The camera in the smart glasses captures the screen of the POS terminal and sends the image to the server.

[0791] The audio data is also sent to the server at the same time.

[0792] Image and voice recognition is performed on the server side, and the generation AI generates operation instructions.

[0793] Operation instructions are converted into voice and provided to the customer through smart glasses.

[0794] For example, you might hear a voice prompt saying, "Please scan the QR code."

[0795] 2. Payment Assistance:

[0796] The customer asks, "How do I pay?"

[0797] The camera in the smart glasses captures the screen of the payment terminal and sends the image to a server.

[0798] The received voice data is also analyzed by the server.

[0799] The generative AI creates steps and generates specific instructions, such as "insert your card."

[0800] Instructions converted into voice data are communicated to the customer through smart glasses.

[0801] Prompt Sentence Examples

[0802] Based on the user's intent of "I want to purchase this product," analyze the captured image of the POS terminal screen and generate appropriate instructions, including how to scan the QR code.

[0803] In this way, the system of the present invention provides a concrete means for assisting seniors in purchasing and paying for products in brick-and-mortar stores.

[0804] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0805] Step 1:

[0806] The user speaks to the smart glasses, saying, "I want to buy this product." The microphone in the smart glasses captures this voice. The input is the user's speech, and the output is the captured voice data. This voice data is sent from the smart glasses to the server.

[0807] Step 2:

[0808] After capturing the user's speech, the camera on the smart glasses captures the screen of the POS terminal. The input is the screen of the POS terminal, and the output is the captured screen image. This image is also sent from the smart glasses to the server.

[0809] Step 3:

[0810] The server analyzes the received voice data. The input is the voice data, and the output is the speech recognition result in text format. This analysis is performed using a speech recognition engine (Google Speech-to-Text API).

[0811] Step 4:

[0812] The server analyzes the received image data in parallel. The input is the screen image, and the output is the analysis result to identify the screen elements. An image recognition algorithm (TensorFlow) is used to recognize the information on the POS terminal screen.

[0813] Step 5:

[0814] The server integrates the results of speech recognition and image recognition. The input is the speech recognition result and the image recognition result, and the output is the interpretation data that combines them. This determines the appropriate instruction based on the user's intention (e.g., "I want to purchase this product").

[0815] Step 6:

[0816] The server generates instructions using a generative AI model (such as GPT-3). The input is interpretation data, and the output is specific instructions (e.g., "Scan the QR code"). The generative AI determines the optimal operation procedure based on the results of voice recognition and image recognition.

[0817] Step 7:

[0818] The server converts the generated operation instructions into voice data using a text-to-speech synthesis engine (Google Text-to-Speech API). The input is the text data of the operation instructions, and the output is voice data.

[0819] Step 8:

[0820] The device (smart glasses) receives the voice data sent from the server and provides instructions to the user via the speaker. The input is voice data, and the output is voice guidance to the user. The smart glasses provide specific voice instructions such as "Please scan the QR code."

[0821] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0822] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. The system captures the user's display in real time, analyzes image and voice data on a server, and generates and provides appropriate operating instructions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions.

[0823] Overall system configuration

[0824] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server is responsible for processing the data and generating operation instructions, while the emotion engine is responsible for analyzing the user's emotions from the collected data.

[0825] Program processing overview

[0826] 1. Smart glasses capture the user's display

[0827] The user wears the smart glasses and operates a smartphone.

[0828] The camera in the smart glasses captures the screen of the user's smartphone in real time, compresses the image and sends it to a server.

[0829] 2. The server performs image recognition

[0830] The received image data is analyzed and image recognition algorithms are used to identify on-screen elements (notifications, icons, buttons, etc.) The analysis results are temporarily stored in a database.

[0831] 3. Smart glasses will capture your voice

[0832] The smart glasses capture the user's voice with a built-in microphone and send the audio data to a server.

[0833] 4. The server performs voice recognition

[0834] The received voice data is passed through a speech recognition engine to be converted into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[0835] 5. Emotion engine recognizes user emotions

[0836] The smart glasses capture the user's facial expressions and send the image data to a server, which also analyzes the voice data, and an emotion engine determines the user's emotions from the voice and facial expressions.

[0837] 6. Generative AI generates instructions

[0838] It combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operating instructions using generative AI. For example, when the user is feeling anxious, it intends to generate more polite and easy-to-understand instructions.

[0839] 7. The server provides audio instructions

[0840] The generated text instructions are converted into voice data using a text-to-speech engine (TTS), and the generated voice data is sent to the smart glasses.

[0841] 8. Smart glasses will play voice commands

[0842] The smart glasses then play back the received audio data and provide the user with voice instructions, allowing them to perform operations while receiving real-time assistance.

[0843] Specific examples

[0844] Example 1: Deleting a notification

[0845] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0846] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[0847] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[0848] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[0849] 5. The server transmits the generated voice data to the smart glasses.

[0850] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0851] Example 2: Changing settings

[0852] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0853] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[0854] 3. The server uses an image recognition algorithm to detect icons on the home screen and in the settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[0855] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0856] 5. The server transmits the generated voice data to the smart glasses.

[0857] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0858] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[0859] The processing flow will be explained below.

[0860] Step 1:

[0861] When a user is looking at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and the image data is compressed.

[0862] Step 2:

[0863] The device (smart glasses) transmits the captured image data to the server in real time using a secure communication protocol (e.g., HTTPS).

[0864] Step 3:

[0865] The server decodes the received image data and uses image recognition algorithms to analyze on-screen elements (notifications, icons, buttons, etc.) and temporarily stores the analysis results in a database.

[0866] Step 4:

[0867] The device (smart glasses) captures the user's voice with a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired, compressed, and sent to the server.

[0868] Step 5:

[0869] The server converts the received voice data into text data through a speech recognition engine, and analyzes the converted text data to identify the user's intentions and instructions.

[0870] Step 6:

[0871] The device (smart glasses) uses a built-in camera to capture the user's facial expressions, and transmits the acquired facial expression data to a server.

[0872] Step 7:

[0873] The server runs the facial expression data through an emotion engine to analyze the user's emotions. It also determines emotions from voice data and combines them with the facial expression data to determine the user's current emotional state.

[0874] Step 8:

[0875] The server combines the results of image recognition, voice recognition, and the emotion engine. Based on all of this data, it uses generative AI to generate operational instructions. For example, if the user is feeling anxious, it generates gentle and polite instructions such as "Please remain calm and swipe left on the notification."

[0876] Step 9:

[0877] The server passes the generated text instructions through a text-to-speech engine (TTS) to convert them into voice data, which is then sent to the smart glasses.

[0878] Step 10:

[0879] The device (smart glasses) plays the received voice data and provides the user with voice instructions, allowing the user to perform operations while receiving real-time assistance.

[0880] The following is an explanation using a specific example.

[0881] Example 1: Deleting a notification

[0882] Step 1:

[0883] The user speaks to the smart glasses, saying, "I want to delete this notification."

[0884] Step 2:

[0885] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0886] Step 3:

[0887] The device (smart glasses) captures the user's voice data and sends it to the server.

[0888] Step 4:

[0889] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[0890] Step 5:

[0891] The server uses an image recognition algorithm to detect the notification on the screen.

[0892] Step 6:

[0893] The server uses voice recognition to obtain the text "I want to delete this notification."

[0894] Step 7:

[0895] The server analyzes the user's emotions using an emotion engine.

[0896] Step 8:

[0897] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate instructions such as "Swipe left on the notification," which are then converted into voice data.

[0898] Step 9:

[0899] The server transmits the generated voice data to the smart glasses.

[0900] Step 10:

[0901] The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0902] Example 2: Changing settings

[0903] Step 1:

[0904] The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0905] Step 2:

[0906] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[0907] Step 3:

[0908] The device (smart glasses) captures the user's voice data and sends it to the server.

[0909] Step 4:

[0910] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[0911] Step 5:

[0912] The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[0913] Step 6:

[0914] The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[0915] Step 7:

[0916] The server analyzes the user's emotions using an emotion engine.

[0917] Step 8:

[0918] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0919] Step 9:

[0920] The server transmits the generated voice data to the smart glasses.

[0921] Step 10:

[0922] The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0923] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[0924] Example 2

[0925] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] Currently, the difficulties that older people experience when operating smartphones and other electronic devices stem from the learning curve and cognitive overhead. Complex setting changes and notification operations, in particular, pose a significant burden for older people. Furthermore, existing operation assistance systems are unable to provide appropriate assistance that takes into account the user's emotional state. Therefore, there is a need for systems that enable older people to operate electronic devices more easily and without stress.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0928] In this invention, the server includes means for capturing a user's display device in real time, means for compressing the captured image and transmitting it to a central processing unit, means for analyzing the received image in the central processing unit and performing object recognition, means for capturing the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice conversion in the central processing unit, means for generating operation instructions in the central processing unit using generative artificial intelligence based on the results of the object recognition and the voice conversion, means for converting the generated operation instructions into voice data and providing them to the user, means for capturing the user's facial expressions and performing emotion analysis, and means for reflecting the results of the emotion analysis in the generation of operation instructions. This reduces the difficulty that elderly people experience when operating electronic devices and makes it possible to provide optimal operation assistance in real time according to the user's emotional state.

[0929] A "display device" is a part of an electronic device that allows a user to visually receive information, specifically a device that includes a screen such as a smartphone or tablet.

[0930] A "central processing unit" is a computer system that receives, analyzes, and generates instructions, and specifically includes a server or a cloud computing platform.

[0931] "Object recognition" is a technology that identifies specific objects or elements (e.g., notifications, icons, buttons, etc.) from image data and determines their presence.

[0932] "Speech conversion" is a technology that converts voice data into text data, and is used to analyze a user's intentions and instructions.

[0933] "Generative artificial intelligence" is an AI technology that generates appropriate operating instructions based on multiple data (for example, the results of image recognition, voice conversion, etc.).

[0934] "Emotion analysis" is a technology that determines a user's emotional state based on the user's voice and facial expression data.

[0935] "Operation assistance" refers to instructions and assistance functions provided in real time when a user operates an electronic device, and is assistance that makes user operations easier and more efficient.

[0936] MODE FOR CARRYING OUT THE INVENTION

[0937] This invention is a system for reducing the difficulty that elderly people have when operating smartphones and other electronic devices. This system captures the user's display device in real time, analyzes the image and voice data, and generates and provides appropriate operating instructions. Furthermore, it has the function of recognizing the user's emotions and flexibly adjusting the operating instructions based on the results.

[0938] Overall system configuration

[0939] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine that performs emotion analysis. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server processes the data and generates operation instructions, while the emotion engine analyzes the user's emotions from the collected data.

[0940] Program processing

[0941] 1. Smart glasses capture the user's display

[0942] The user wears smart glasses and operates a smartphone. The camera in the smart glasses captures the smartphone screen in real time, compresses the image data, and sends it to a server. In this case, the specific hardware used is the smart glasses, and the software used is a software module that can process image data in real time.

[0943] 2. The server performs image recognition

[0944] The server analyzes the image data received from the smart glasses and uses image recognition algorithms such as OpenCV to identify elements in the image (notifications, icons, buttons, etc.). The results are temporarily stored in a database.

[0945] 3. Smart glasses will capture your voice

[0946] The user speaks to the smart glasses, and the built-in microphone in the smart glasses captures the voice and sends the voice data to the server.

[0947] 4. The server performs voice recognition

[0948] The server converts the voice data received from the smart glasses into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), then analyzes the converted text data to determine what the user wants.

[0949] 5. Emotion engine recognizes user emotions

[0950] The user's facial expression data is captured and sent to the server, where it is analyzed together with the voice data, and the emotion engine determines the user's emotion from the voice and facial expression.

[0951] 6. Generative AI generates instructions

[0952] The generative AI model combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operational instructions. For example, if the user feels anxious, a polite and specific instruction such as "Open the Settings app" is generated. An example of a prompt sentence from this generative AI is as follows:

[0953] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[0954] 7. The server provides audio instructions

[0955] The generated operation instructions are converted into voice data using a Text-to-Speech (TTS) engine, and this voice data is sent to the smart glasses.

[0956] 8. Smart glasses will play voice commands

[0957] The smart glasses then play back the received voice data and provide real-time voice instructions to the user, allowing them to perform operations without stress.

[0958] Specific examples

[0959] Example 1: Deleting a notification

[0960] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[0961] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[0962] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[0963] 4. The server combines the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[0964] 5. The server sends the generated voice data to the smart glasses.

[0965] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[0966] Example 2: Changing settings

[0967] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[0968] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[0969] 3. The server uses an image recognition algorithm to detect icons on the home screen and settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[0970] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[0971] 5. The server sends the generated voice data to the smart glasses.

[0972] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[0973] In this way, the system collects and analyzes real-time user data to help elderly people operate electronic devices safely and efficiently.

[0974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0975] Processing Steps

[0976] Step 1: Smart glasses capture the user's display

[0977] Input: The user wears the smart glasses and operates a smartphone.

[0978] How it works: The device (smart glasses) uses its built-in camera to capture the screen of the user's smartphone in real time.

[0979] Data processing: Compress the captured image data to reduce the data size.

[0980] Output: Send the compressed image data to the server.

[0981] Step 2: The server performs image recognition

[0982] Input: The server receives the compressed image data sent from the smart glasses.

[0983] How it works: The server uses image recognition algorithms such as OpenCV to identify elements on the image (notifications, icons, buttons, etc.).

[0984] Data processing: Analyze image data and identify and classify each element.

[0985] Output: The analysis results are temporarily saved in a database.

[0986] Step 3: Smart glasses capture user's voice

[0987] Input: The user speaks their intention to operate the smart glasses.

[0988] How it works: The device (smart glasses) uses a built-in microphone to capture the user's voice.

[0989] Data processing: Converting captured audio data into a certain format.

[0990] Output: Sends the audio data to the server.

[0991] Step 4: The server performs speech recognition

[0992] Input: The server receives the audio data sent from the smart glasses.

[0993] How it works: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio data into text.

[0994] Data processing: Convert the audio data into text data and analyze the meaning.

[0995] Output: Send the parsed text data to the appropriate processing module in the server.

[0996] Step 5: The emotion engine recognizes the user's emotion

[0997] Input: The smart glasses capture the user's facial expressions and send them to the server, as well as audio data.

[0998] Operation: The server uses an emotion engine to analyze facial and voice data to determine the user's emotional state.

[0999] Data processing: Combining facial expression analysis and voice analysis to identify the user's emotions.

[1000] Output: The results of sentiment analysis are saved in a database on the server.

[1001] Step 6: Generative AI generates instructions

[1002] Input: The results of image recognition, voice recognition, and emotion recognition are stored in a database on the server.

[1003] Action: The server integrates this data and uses a generative AI model to generate appropriate operational instructions.

[1004] Data processing: Various data are input into the generative AI model as prompt sentences to generate instruction sentences.

[1005] Output: The generated operation instructions are output as text data. This text includes specific operation methods.

[1006] Prompt Sentence Examples

[1007] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[1008] Step 7: The server provides spoken instructions

[1009] Input: Text data of operation instructions generated by the generative AI model.

[1010] How it works: The server uses a text-to-speech (TTS) engine to convert text data into audio data (e.g., Google Cloud Text-to-Speech).

[1011] Data processing: Convert text data into audio data.

[1012] Output: The generated audio data is sent to the smart glasses.

[1013] Step 8: Smart glasses play voice commands

[1014] Input: Audio data sent from the server.

[1015] Operation: The device (smart glasses) plays the received audio data and provides real-time operating instructions to the user.

[1016] Data processing: None in particular (data reproduction).

[1017] Output: Provides an environment where the user can listen to instructions and assists with operation.

[1018] (Application example 2)

[1019] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1020] Elderly people often experience difficulties when searching for products in physical stores, completing purchase procedures, and obtaining campaign information. Therefore, there is a need for support systems that allow elderly people to enjoy shopping independently and comfortably. While current technology supports elderly people using voice recognition and image recognition, there is a lack of systems that combine emotion recognition and real-time support, which has resulted in a lack of improvements in the user experience.

[1021] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1022] In this invention, the server includes means for capturing a user's display in real time, means for transmitting the captured image to the server, means for analyzing the received image in the server and performing image recognition, means for capturing the user's voice and transmitting the voice data to the server, means for analyzing the voice data in the server and performing voice recognition, means for generating operation instructions in the server using a generation AI based on the results of the image recognition and the voice recognition, means for converting the generated operation instructions into voice data and providing it to the user, means for receiving a question about a specific product so that the user can search for the product in a physical store and providing voice guidance on the location of the product on the shelf, means for obtaining information on the product's barcode and purchase procedure and providing voice guidance on how to purchase, and means for obtaining campaign information such as discounted products from a database and providing voice guidance in real time. This enables elderly people to enjoy shopping independently and comfortably in physical stores.

[1023] "Capturing a user's display" means taking a real-time picture of the screen of the electronic device being used by the user.

[1024] A "server" is a central control unit that processes data and provides necessary information over a network.

[1025] "Image recognition" is a technology that analyzes captured image data and identifies specific objects, characters, symbols, etc. within it.

[1026] "Voice capture" is the process of recording a user's speech with a microphone and collecting that voice data.

[1027] "Speech recognition" is a technology that analyzes captured voice data and converts it into text data.

[1028] "Generative AI" is an artificial intelligence technology that generates appropriate output based on input data, creating operational instructions based on user questions and requests.

[1029] "Generation of operation instructions" is the process of determining appropriate actions based on the results of image recognition and voice recognition, and generating those instructions.

[1030] "Converting voice data" refers to the process of converting the generated textual instructions into voice using speech synthesis technology.

[1031] A "physical store" is a commercial facility that consumers can physically visit and where goods and services are offered.

[1032] "Product search" is the process of gathering information and providing guidance needed to locate a specific product.

[1033] "Shelf location information guidance" refers to audio notification of where specific products are located within the store.

[1034] "Barcode acquisition" refers to reading the barcode information to identify the uniqueness of the product.

[1035] "Purchase procedure guidance" refers to audio instructions explaining the steps and methods for purchasing a product.

[1036] "Providing campaign information" means informing users in real time about current discounts, special offers, and other information.

[1037] A "database" is an electronic information storage system that organizes and stores information, allowing it to be quickly searched and updated as needed.

[1038] This system aims to alleviate the difficulties that seniors face when searching for products in brick-and-mortar stores, completing purchase procedures, and obtaining campaign information. In particular, it uses smart glasses, smartphones, a central control server, and an emotion engine to help seniors enjoy shopping independently and comfortably.

[1039] Overall system configuration

[1040] The system mainly consists of the user's devices (smart glasses and smartphone), a central server, an emotion engine, and a database. The smart glasses are equipped with a camera and microphone to capture the user's voice and image. The server processes this data and generates the necessary operation instructions, which are then provided via voice.

[1041] Hardware and Software

[1042] Smart glasses: Built-in camera and microphone capture the user's voice and display. Example: A common name for smart wearable devices.

[1043] Server: Processes and analyzes data and generates operational instructions. Example: A common name is a cloud server.

[1044] Speech recognition engine: converts voice data into text. Example: speech processing software.

[1045] Image Recognition Engine: Analyzes image data and extracts information. Example: Image analysis software.

[1046] Emotion engine: Determines the user's emotions from voice and facial expression data. Example: emotion analysis software.

[1047] Database: Stores product information, store maps, and campaign information and updates them in real time. Example: Data storage system.

[1048] Specific functions and operations of the system

[1049] 1. Product search support

[1050] A user puts on smart glasses and asks, "Where is this item?"

[1051] The device (smart glasses) captures images of the store using a camera and sends them to a server. It also captures the user's voice using a microphone and sends it to the server.

[1052] The server analyzes the received data using an image recognition engine and a voice recognition engine to identify the product's location information.

[1053] The server uses generation AI to generate shelf location information as voice instructions, and then converts this into voice data using a TTS engine.

[1054] The terminal (smart glasses) guides the generated voice instructions to the user.

[1055] 2. Purchase procedure assistance

[1056] The user asks the smart glasses, "How do I purchase this product?"

[1057] The device (smart glasses) captures the user's voice and sends it to the server.

[1058] The server converts the voice data into text using a voice recognition engine and obtains information about the purchase procedure.

[1059] The server uses generative AI to generate purchase procedure instructions and converts them into voice data using a TTS engine.

[1060] The terminal (smart glasses) guides the generated voice instructions to the user.

[1061] 3. Providing campaign information

[1062] The user asks the smart glasses, "What are the deals today?"

[1063] The device (smart glasses) captures the user's voice and sends it to the server.

[1064] The server converts the voice data into text using a voice recognition engine and retrieves the latest campaign information from the database in real time.

[1065] The server uses generation AI to generate campaign information as voice instructions and converts them into voice data using a TTS engine.

[1066] The terminal (smart glasses) guides the generated voice instructions to the user.

[1067] Examples of concrete examples and prompts

[1068] Specific examples of product searches

[1069] 1. A user asks, "Where is this item?"

[1070] 2. The device (smart glasses) captures images and audio and sends them to the server.

[1071] 3. The server identifies the product's location and provides a voice message saying, "The product is on the left side of aisle 3."

[1072] Example prompt sentence:

[1073] When you ask, "Where is this product?" the system analyzes camera footage from within the store and provides voice guidance on the location of the product.

[1074] Examples of campaign information provision

[1075] 1. A user asks, "What are the discounted items today?"

[1076] 2. The device (smart glasses) captures the audio and sends it to the server.

[1077] 3. The server retrieves the latest campaign information from the database and announces, "Today's discounted items are apples and oranges."

[1078] Example prompt sentence:

[1079] When you ask, "What are the discounted items today?" the system will query the database and provide you with the latest campaign information via voice.

[1080] The system combines advanced technologies such as real-time image capture, voice recognition, and emotion recognition to help seniors shop independently and comfortably in brick-and-mortar stores.

[1081] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1082] Step 1:

[1083] The user puts on the smart glasses and asks a question.

[1084] Input: User utterance ("Where is this item?").

[1085] Output: Audio data.

[1086] Specific operation: The user asks a question into the microphone of the smart glasses. The voice is captured by the microphone of the smart glasses and collected as voice data.

[1087] Step 2:

[1088] The device (smart glasses) captures images of the inside of the store using a camera and sends them to the server along with audio data.

[1089] Input: User's voice data and store image data.

[1090] Output: Audio and image data sent to the server.

[1091] How it works: The smart glasses' camera captures images of the store in real time, while the smart glasses' microphone simultaneously collects audio data and sends both to a server.

[1092] Step 3:

[1093] The image data received by the server is analyzed using an image analysis engine.

[1094] Input: Image data sent to the server.

[1095] Output: Image analysis results (shelf and item location information).

[1096] Specific operation: The server passes the received image data to an image analysis engine (e.g., image analysis software) for analysis. It extracts the location information of shelves and products in the image and temporarily stores the analysis results.

[1097] Step 4:

[1098] The server analyzes the received voice data using a voice recognition engine.

[1099] Input: The audio data sent to the server.

[1100] Output: Text data (user question).

[1101] Specific operation: The server passes the voice data to a voice recognition engine (e.g., voice processing software) and converts the voice into text, which extracts the question asked by the user as text data.

[1102] Step 5:

[1103] The server integrates the image recognition results and voice recognition results and generates operation instructions using a generative AI model.

[1104] Input: Image analysis results and text data.

[1105] Output: Text data of operation instructions.

[1106] Specific operation: The server integrates the image analysis results and the voice recognition results, and generates appropriate operation instructions for the user's question using a generative AI model (e.g., generative AI). The generated operation instructions are saved as text data.

[1107] Step 6:

[1108] The server converts the generated operation instructions into voice data using a TTS engine.

[1109] Input: Text data of operation instructions.

[1110] Output: Audio data.

[1111] Specific operation: The server passes the text-based operation instructions to a TTS engine (e.g., speech synthesis software) and converts them into voice data, which is then sent to the smart glasses.

[1112] Step 7:

[1113] The device (smart glasses) plays the received audio data to the user.

[1114] Input: Audio data sent from the server.

[1115] Output: Audio instructions provided to the user.

[1116] Specific operation: The smart glasses play the received audio data through a speaker and provide the user with audio instructions, such as guidance on the location of product shelves and the purchase procedure.

[1117] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1118] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1119] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1120] [Third embodiment]

[1121] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1122] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1123] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1124] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1125] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1126] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1127] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1128] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1129] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1130] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1131] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1132] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1133] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures the user's display in real time, sends the image to a server, and uses image and voice recognition on the server to provide operation instructions using generative AI.

[1134] Overall system configuration

[1135] This system mainly consists of the user's devices (smart glasses and smartphone) and a central server. The smart glasses have a built-in camera and microphone, and are responsible for capturing images from the user's smartphone screen and collecting the user's voice. The server is responsible for processing the data and generating operation instructions.

[1136] Program processing overview

[1137] 1. Smart glasses capture the user's display

[1138] The user wears the smart glasses and operates a smartphone.

[1139] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[1140] 2. The server performs image recognition

[1141] It decodes the received image and uses image recognition algorithms to analyze elements on the smartphone screen.

[1142] The analysis results are temporarily stored in a database and the next step is taken.

[1143] 3. Recognize the user's voice

[1144] The smart glasses capture the user's voice with a microphone and send the voice data to a server.

[1145] The server uses a speech recognition engine to convert the voice data into text.

[1146] Analyze text data to identify the user's intent and instructions.

[1147] 4. Generative AI generates instructions

[1148] The results of image recognition and voice recognition are integrated and appropriate operating instructions are generated using generative AI (e.g., large-scale language models).

[1149] For example, if the user's intention is recognized as "I want to delete the notification," a specific instruction such as "Swipe the notification to the left" is generated.

[1150] 5. The server provides audio instructions

[1151] A text-to-speech engine (TTS) is used to convert the generated text instructions into audio data.

[1152] The generated voice data is sent to the smart glasses to provide voice instructions to the user.

[1153] Specific examples

[1154] Example 1: Deleting a notification

[1155] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1156] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1157] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1158] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[1159] 5. The server uses voice recognition to obtain the text "I want to delete this notification."

[1160] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[1161] 7. The server transmits the generated voice data to the smart glasses.

[1162] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1163] Example 2: Changing settings

[1164] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1165] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1166] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1167] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[1168] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[1169] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[1170] 7. The server transmits the generated voice data to the smart glasses.

[1171] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1172] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[1173] The processing flow will be explained below.

[1174] Step 1:

[1175] When a user looks at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and compressed.

[1176] Step 2:

[1177] The device (smart glasses) transmits the captured images to the server in real time using a secure communication protocol (e.g., HTTPS).

[1178] Step 3:

[1179] The server decodes the received image data and analyzes it using image recognition algorithms to identify important elements on the screen (notifications, icons, buttons, etc.).

[1180] Step 4:

[1181] The device (smart glasses) captures the user's voice using a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired and sent to the server.

[1182] Step 5:

[1183] The server passes the received voice data through a speech recognition engine to convert it into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[1184] Step 6:

[1185] The server combines the analyzed image recognition results with the voice recognition results and uses generative AI to generate appropriate instructions for the user, such as "swipe left on the notification."

[1186] Step 7:

[1187] The server uses a text-to-speech (TTS) engine to convert the generated text instructions into audio data, which is then sent to the smart glasses.

[1188] Step 8:

[1189] The device (smart glasses) plays the received audio data and provides the user with audio instructions, such as "Swipe the notification left."

[1190] These are the specific processing steps of the program. Through this process, users can receive appropriate and prompt instructions based on the information displayed in real time. This can eliminate the anxiety and difficulty that seniors may have when operating smartphones and other electronic devices.

[1191] Example 1

[1192] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1193] The purpose of this project is to provide a system that solves the difficulties that elderly people face when operating smartphones and other electronic devices due to visual, hearing, and operational difficulties. In particular, for people with visual or hearing impairments, it is difficult to provide appropriate operational instructions in real time, and these operations need to be intuitive and easy for users to understand.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1195] In this invention, the server includes means for capturing an image of the user's display device in real time, means for transmitting the captured image to a central processing unit, means for analyzing the received image in the central processing unit and performing image recognition, means for recording the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice recognition in the central processing unit, means for generating operation instructions in the central processing unit using a generation AI based on the results of image recognition and voice recognition, and means for converting the generated operation instructions into voice data and providing it to the user. This enables users to receive intuitive and easy-to-understand operation instructions in real time without having difficulty operating their smartphones or other electronic devices.

[1196] "User" refers to the person who operates the system, and in this system includes elderly people who use smart glasses or smartphones.

[1197] "Display device" refers to electronic devices that display visual information, such as smartphones and tablets.

[1198] The term "means for capturing images" refers to a device, such as a camera or an imaging device, that captures the screen of a display device in real time.

[1199] "Central processing unit" refers to a computing resource accessed via a network for data analysis and processing, such as a cloud server or remote server.

[1200] "Transmitting means" refers to technologies and devices for transmitting data to other devices or servers, including wireless communication modules, Wi-Fi, Bluetooth, etc.

[1201] "Means for receiving" refers to the technology and devices for obtaining data via a network.

[1202] "Image recognition" refers to the technology of analyzing captured image data and recognizing specific elements or patterns.

[1203] "Voice recognition" refers to the technology of analyzing recorded voice data and converting what is spoken into text data.

[1204] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to generate text data or operational instructions, including large-scale language models such as GPT-4.

[1205] "Operation instructions" refers to information that instructs the user to perform specific actions to achieve the operation or goal desired by the user.

[1206] "Means for converting into voice data" refers to technology that converts text data into voice using a text-to-speech synthesis engine (TTS) or similar.

[1207] "Means for providing to the user" refers to devices and technologies for conveying specific operating instructions to the user in voice instructions or other forms.

[1208] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures a user's display device in real time, sends the image to a central processing unit, and uses a generation AI to provide operating instructions by performing image and voice recognition in the central processing unit.

[1209] Overall system configuration

[1210] This system mainly consists of the user's devices (smart glasses and smartphone) and a central processing unit. The smart glasses have a built-in camera and microphone, and are responsible for capturing images of the user's smartphone screen and collecting the user's voice. The central processing unit is responsible for processing data and generating operating instructions.

[1211] Hardware and Software Configuration

[1212] 1. Smart Glasses

[1213] The smart glasses are equipped with a built-in camera module, microphone, and wireless communication module, which allows them to capture the user's smartphone screen and transmit it to a central processing unit in real time.The microphone also captures the user's voice and transmits it to the central processing unit.

[1214] 2. Central Processing Unit

[1215] The server used as the central processing unit is built on a high-performance cloud infrastructure. This server uses OpenCV and TensorFlow for image analysis, the Google Speech-to-Text API for voice analysis, GPT-4 for generative AI, and Amazon Polly for text-to-speech (TTS) engine.

[1216] Processing flow

[1217] 1. The user puts on the smart glasses

[1218] The user puts on the smart glasses and prepares to operate the smartphone.

[1219] 2. The device (smart glasses) captures the display of the user's smartphone.

[1220] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[1221] 3. The server processes the received image

[1222] The server decodes the received image and analyzes the elements on the screen using image recognition algorithms such as OpenCV and TensorFlow.

[1223] 4. The device (smart glasses) captures the user's voice.

[1224] The smart glasses' microphone is used to capture the user's voice in real time and send the voice data to a server.

[1225] 5. The server recognizes the voice

[1226] The server analyzes the voice data using the Google Speech-to-Text API and converts it into text data.

[1227] 6. Generative AI generates operation instructions

[1228] Based on the results of image and voice recognition, appropriate operation instructions are generated using generative AI such as GPT-4. For example, if the voice recognizes "I want to change my Wi-Fi settings," the instruction generated is "Open the Settings app."

[1229] 7. The server provides audio instructions

[1230] The generated operation instructions are converted into voice data using TTS such as Amazon Polly and sent to the smart glasses to provide voice instructions to the user.

[1231] Specific examples

[1232] Example 1: Deleting a notification

[1233] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1234] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1235] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1236] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[1237] 5. The server uses voice recognition to obtain the text "I want to dismiss this notification."

[1238] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[1239] 7. The server sends the generated voice data to the smart glasses.

[1240] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1241] Example 2: Changing settings

[1242] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1243] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1244] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1245] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[1246] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[1247] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[1248] 7. The server sends the generated voice data to the smart glasses.

[1249] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1250] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[1251] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1252] Program processing flow

[1253] Step 1: User puts on smart glasses

[1254] Input: User action (wearing smart glasses)

[1255] Output: Smart glasses start

[1256] Specific behavior:

[1257] When a user puts on the smart glasses, the built-in sensors in the smart glasses detect that they are being worn and activate the camera and microphone, preparing the display device for screen and audio capture.

[1258] Step 2: The device (smart glasses) captures the display of the user's smartphone.

[1259] Input: The user's smartphone display

[1260] Output: Captured image data

[1261] Specific behavior:

[1262] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals (for example, 2 frames per second), and the captured image data is compressed in real time and sent to a central processing unit (server) via a wireless communication module.

[1263] Step 3: The server processes the received image

[1264] Input: Capture image data

[1265] Output: Analysis results (element data on the screen)

[1266] Specific behavior:

[1267] The server decodes the image data received via the network and analyzes elements on the smartphone screen (icons, notifications, etc.) using image recognition algorithms such as OpenCV and TensorFlow. The analysis results are temporarily stored in a database.

[1268] Step 4: The device (smart glasses) captures the user's voice

[1269] Input: User's voice

[1270] Output: Audio data

[1271] Specific behavior:

[1272] When a user speaks operational instructions into the smart glasses, the smart glasses' built-in microphone captures the voice in real time and transmits the voice data to a server via a wireless communication module.

[1273] Step 5: The server analyzes the audio data

[1274] Input: Audio data

[1275] Output: Analysis results (text data)

[1276] Specific behavior:

[1277] The server receives the voice data and converts it into text using a speech recognition engine such as the Google Speech-to-Text API. The resulting text data is used as input for the generation AI.

[1278] Step 6: Generative AI generates operation instructions

[1279] Input: Image analysis results, audio analysis results (text data)

[1280] Output: Instruction text

[1281] Specific behavior:

[1282] The server combines the results of image and voice recognition and generates appropriate instructions using a generative AI model such as GPT-4. For example, if the text "I want to change my Wi-Fi settings" is obtained through voice recognition, the server generates the specific instruction "Open the Settings app."

[1283] Step 7: The server converts the instructions into audio data

[1284] Input: Instruction text

[1285] Output: Operation instruction voice data

[1286] Specific behavior:

[1287] The text instructions generated by the AI ​​are converted into voice data using a text-to-speech (TTS) engine such as Amazon Polly, and the generated voice data is sent to the smart glasses via the network.

[1288] Step 8: The device (smart glasses) plays back the operation instructions.

[1289] Input: Operation instruction voice data

[1290] Output: Instructions provided to the user

[1291] Specific behavior:

[1292] The smart glasses play back the audio data received from the server and give the user instructions such as "Open the Settings app," allowing the user to intuitively understand the appropriate operation.

[1293] These are the specific processing steps of this system's program. The specific operations, inputs, and outputs are clearly defined at each step, allowing for a detailed understanding of how the entire system operates.

[1294] (Application example 1)

[1295] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1296] There is an urgent need to address the difficulties that seniors face when using smartphones and other electronic devices. Particularly in brick-and-mortar stores, seniors often experience unfamiliar operational challenges when searching for products or making payments. Complicated procedures and small screen displays can also be a visual burden. The present invention aims to provide a system that provides real-time assistance to facilitate these operations.

[1297] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1298] In this invention, the server includes means for periodically transmitting the captured image of the user's display to the server, means for analyzing the smart device screen using an image recognition algorithm after transmitting the captured image to the server, means for capturing the user's voice with the microphone of the smart glasses and transmitting it to the server, means for analyzing the voice data and performing voice recognition in the server, means for integrating the image recognition results and the voice recognition results and generating specific operation instructions using a generation AI, means for converting the generated operation instructions into voice data using a text-to-speech engine and providing it to the user via the smart glasses, and means for providing purchasing support and payment assistance in physical stores, which enables elderly people to efficiently purchase products in physical stores and smoothly complete payment procedures.

[1299] The "captured user display image" is the display content of the device being operated by the user captured as is and acquired as image data.

[1300] "Transmission means" refers to the technical means by which data is transferred from one device to another.

[1301] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying elements or patterns within it.

[1302] "Means for analyzing the screen of a smart device" refers to a technical means for understanding the display information of a smartphone or other digital device used by a user and extracting the necessary information.

[1303] "Capturing the user's voice with the microphone of the smart glasses" refers to using the microphone built into the smart glasses to obtain the user's speech as voice data.

[1304] "Means for performing speech recognition" refers to the technical means for analyzing captured speech data and expressing its contents in text form.

[1305] "Means for generating specific operating instructions using generative AI" refers to using artificial intelligence technology to create specific operating instructions for users based on the results of voice recognition and image recognition.

[1306] "Text-to-speech engine" refers to the technical means for converting generated text information into speech data.

[1307] "Purchase support and payment assistance in physical stores" refers to smoothly supporting the process of users selecting and purchasing products and making payments in physical stores.

[1308] This invention is a system that allows elderly people to easily purchase products in physical stores and smoothly complete payment procedures. The system uses the user's smart glasses and smartphone as the main terminals and operates in conjunction with a central server.

[1309] Overall system configuration

[1310] The system consists of user devices (smart glasses and smartphones), a central server, and various software modules.

[1311] Main hardware and software used

[1312] Hardware:

[1313] Smart glasses (built-in camera, microphone, and speaker)

[1314] Smartphone

[1315] software:

[1316] Image recognition algorithm (TensorFlow)

[1317] Speech recognition engine (Google Speech-to-Text API)

[1318] Generative AI models (GPT-3 and similar large-scale language models)

[1319] Text-to-speech engine (Google Text-to-Speech API)

[1320] Program Processing Overview

[1321] 1. Capture:

[1322] When a customer (user) speaks to the smart glasses and says, "I would like to purchase this product," the camera in the smart glasses captures the screen of the POS terminal.

[1323] The captured image is compressed and sent to the server.

[1324] 2. Speech Recognition:

[1325] The microphone in the smart glasses captures the customer's voice and sends the voice data to a server.

[1326] The server uses a speech recognition engine to convert the voice data into text and identify the customer's instructions.

[1327] 3. Image Recognition and Instruction Generation:

[1328] The server decodes the received image and uses image recognition algorithms to analyze the screen elements of the POS terminal.

[1329] The image recognition results and voice recognition results are integrated, and the generative AI model generates appropriate operating instructions.

[1330] For example, generate an operation instruction such as "Scan the QR code."

[1331] 4. Audio guide:

[1332] The generated operation instructions are converted into voice data by a text-to-speech synthesis engine.

[1333] Voice data is sent to the smart glasses, and operational instructions are provided to the customer via voice.

[1334] Specific examples

[1335] 1. Product purchase assistance:

[1336] A customer says, "I would like to purchase this product."

[1337] The camera in the smart glasses captures the screen of the POS terminal and sends the image to the server.

[1338] The audio data is also sent to the server at the same time.

[1339] Image and voice recognition is performed on the server side, and the generation AI generates operation instructions.

[1340] Operation instructions are converted into voice and provided to the customer through smart glasses.

[1341] For example, you might hear a voice prompt saying, "Please scan the QR code."

[1342] 2. Payment Assistance:

[1343] The customer asks, "How do I pay?"

[1344] The camera in the smart glasses captures the screen of the payment terminal and sends the image to a server.

[1345] The received voice data is also analyzed by the server.

[1346] The generative AI creates steps and generates specific instructions, such as "insert your card."

[1347] Instructions converted into voice data are communicated to the customer through smart glasses.

[1348] Prompt Sentence Examples

[1349] Based on the user's intent of "I want to purchase this product," analyze the captured image of the POS terminal screen and generate appropriate instructions, including how to scan the QR code.

[1350] In this way, the system of the present invention provides a concrete means for assisting seniors in purchasing and paying for products in brick-and-mortar stores.

[1351] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1352] Step 1:

[1353] The user speaks to the smart glasses, saying, "I want to buy this product." The microphone in the smart glasses captures this voice. The input is the user's speech, and the output is the captured voice data. This voice data is sent from the smart glasses to the server.

[1354] Step 2:

[1355] After capturing the user's speech, the camera on the smart glasses captures the screen of the POS terminal. The input is the screen of the POS terminal, and the output is the captured screen image. This image is also sent from the smart glasses to the server.

[1356] Step 3:

[1357] The server analyzes the received voice data. The input is the voice data, and the output is the speech recognition result in text format. This analysis is performed using a speech recognition engine (Google Speech-to-Text API).

[1358] Step 4:

[1359] The server analyzes the received image data in parallel. The input is the screen image, and the output is the analysis result to identify the screen elements. An image recognition algorithm (TensorFlow) is used to recognize the information on the POS terminal screen.

[1360] Step 5:

[1361] The server integrates the results of speech recognition and image recognition. The input is the speech recognition result and the image recognition result, and the output is the interpretation data that combines them. This determines the appropriate instruction based on the user's intention (e.g., "I want to purchase this product").

[1362] Step 6:

[1363] The server generates instructions using a generative AI model (such as GPT-3). The input is interpretation data, and the output is specific instructions (e.g., "Scan the QR code"). The generative AI determines the optimal operation procedure based on the results of voice recognition and image recognition.

[1364] Step 7:

[1365] The server converts the generated operation instructions into voice data using a text-to-speech synthesis engine (Google Text-to-Speech API). The input is the text data of the operation instructions, and the output is voice data.

[1366] Step 8:

[1367] The device (smart glasses) receives the voice data sent from the server and provides instructions to the user via the speaker. The input is voice data, and the output is voice guidance to the user. The smart glasses provide specific voice instructions such as "Please scan the QR code."

[1368] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1369] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. The system captures the user's display in real time, analyzes image and voice data on a server, and generates and provides appropriate operating instructions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions.

[1370] Overall system configuration

[1371] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server is responsible for processing the data and generating operation instructions, while the emotion engine is responsible for analyzing the user's emotions from the collected data.

[1372] Program processing overview

[1373] 1. Smart glasses capture the user's display

[1374] The user wears the smart glasses and operates a smartphone.

[1375] The camera in the smart glasses captures the screen of the user's smartphone in real time, compresses the image and sends it to a server.

[1376] 2. The server performs image recognition

[1377] The received image data is analyzed and image recognition algorithms are used to identify on-screen elements (notifications, icons, buttons, etc.) The analysis results are temporarily stored in a database.

[1378] 3. Smart glasses will capture your voice

[1379] The smart glasses capture the user's voice with a built-in microphone and send the audio data to a server.

[1380] 4. The server performs voice recognition

[1381] The received voice data is passed through a speech recognition engine to be converted into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[1382] 5. Emotion engine recognizes user emotions

[1383] The smart glasses capture the user's facial expressions and send the image data to a server, which also analyzes the voice data, and an emotion engine determines the user's emotions from the voice and facial expressions.

[1384] 6. Generative AI generates instructions

[1385] It combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operating instructions using generative AI. For example, when the user is feeling anxious, it intends to generate more polite and easy-to-understand instructions.

[1386] 7. The server provides audio instructions

[1387] The generated text instructions are converted into voice data using a text-to-speech engine (TTS), and the generated voice data is sent to the smart glasses.

[1388] 8. Smart glasses will play voice commands

[1389] The smart glasses then play back the received audio data and provide the user with voice instructions, allowing them to perform operations while receiving real-time assistance.

[1390] Specific examples

[1391] Example 1: Deleting a notification

[1392] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1393] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[1394] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[1395] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[1396] 5. The server transmits the generated voice data to the smart glasses.

[1397] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1398] Example 2: Changing settings

[1399] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1400] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[1401] 3. The server uses an image recognition algorithm to detect icons on the home screen and in the settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[1402] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[1403] 5. The server transmits the generated voice data to the smart glasses.

[1404] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1405] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[1406] The processing flow will be explained below.

[1407] Step 1:

[1408] When a user is looking at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and the image data is compressed.

[1409] Step 2:

[1410] The device (smart glasses) transmits the captured image data to the server in real time using a secure communication protocol (e.g., HTTPS).

[1411] Step 3:

[1412] The server decodes the received image data and uses image recognition algorithms to analyze on-screen elements (notifications, icons, buttons, etc.) and temporarily stores the analysis results in a database.

[1413] Step 4:

[1414] The device (smart glasses) captures the user's voice with a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired, compressed, and sent to the server.

[1415] Step 5:

[1416] The server converts the received voice data into text data through a speech recognition engine, and analyzes the converted text data to identify the user's intentions and instructions.

[1417] Step 6:

[1418] The device (smart glasses) uses a built-in camera to capture the user's facial expressions, and transmits the acquired facial expression data to a server.

[1419] Step 7:

[1420] The server runs the facial expression data through an emotion engine to analyze the user's emotions. It also determines emotions from voice data and combines them with the facial expression data to determine the user's current emotional state.

[1421] Step 8:

[1422] The server combines the results of image recognition, voice recognition, and the emotion engine. Based on all of this data, it uses generative AI to generate operational instructions. For example, if the user is feeling anxious, it generates gentle and polite instructions such as "Please remain calm and swipe left on the notification."

[1423] Step 9:

[1424] The server passes the generated text instructions through a text-to-speech engine (TTS) to convert them into voice data, which is then sent to the smart glasses.

[1425] Step 10:

[1426] The device (smart glasses) plays the received voice data and provides the user with voice instructions, allowing the user to perform operations while receiving real-time assistance.

[1427] The following is an explanation using a specific example.

[1428] Example 1: Deleting a notification

[1429] Step 1:

[1430] The user speaks to the smart glasses, saying, "I want to delete this notification."

[1431] Step 2:

[1432] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1433] Step 3:

[1434] The device (smart glasses) captures the user's voice data and sends it to the server.

[1435] Step 4:

[1436] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[1437] Step 5:

[1438] The server uses an image recognition algorithm to detect the notification on the screen.

[1439] Step 6:

[1440] The server uses voice recognition to obtain the text "I want to delete this notification."

[1441] Step 7:

[1442] The server analyzes the user's emotions using an emotion engine.

[1443] Step 8:

[1444] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate instructions such as "Swipe left on the notification," which are then converted into voice data.

[1445] Step 9:

[1446] The server transmits the generated voice data to the smart glasses.

[1447] Step 10:

[1448] The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1449] Example 2: Changing settings

[1450] Step 1:

[1451] The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1452] Step 2:

[1453] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1454] Step 3:

[1455] The device (smart glasses) captures the user's voice data and sends it to the server.

[1456] Step 4:

[1457] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[1458] Step 5:

[1459] The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[1460] Step 6:

[1461] The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[1462] Step 7:

[1463] The server analyzes the user's emotions using an emotion engine.

[1464] Step 8:

[1465] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[1466] Step 9:

[1467] The server transmits the generated voice data to the smart glasses.

[1468] Step 10:

[1469] The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1470] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[1471] Example 2

[1472] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1473] Currently, the difficulties that older people experience when operating smartphones and other electronic devices stem from the learning curve and cognitive overhead. Complex setting changes and notification operations, in particular, pose a significant burden for older people. Furthermore, existing operation assistance systems are unable to provide appropriate assistance that takes into account the user's emotional state. Therefore, there is a need for systems that enable older people to operate electronic devices more easily and without stress.

[1474] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1475] In this invention, the server includes means for capturing a user's display device in real time, means for compressing the captured image and transmitting it to a central processing unit, means for analyzing the received image in the central processing unit and performing object recognition, means for capturing the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice conversion in the central processing unit, means for generating operation instructions in the central processing unit using generative artificial intelligence based on the results of the object recognition and the voice conversion, means for converting the generated operation instructions into voice data and providing them to the user, means for capturing the user's facial expressions and performing emotion analysis, and means for reflecting the results of the emotion analysis in the generation of operation instructions. This reduces the difficulty that elderly people experience when operating electronic devices and makes it possible to provide optimal operation assistance in real time according to the user's emotional state.

[1476] A "display device" is a part of an electronic device that allows a user to visually receive information, specifically a device that includes a screen such as a smartphone or tablet.

[1477] A "central processing unit" is a computer system that receives, analyzes, and generates instructions, and specifically includes a server or a cloud computing platform.

[1478] "Object recognition" is a technology that identifies specific objects or elements (e.g., notifications, icons, buttons, etc.) from image data and determines their presence.

[1479] "Speech conversion" is a technology that converts voice data into text data, and is used to analyze a user's intentions and instructions.

[1480] "Generative artificial intelligence" is an AI technology that generates appropriate operating instructions based on multiple data (for example, the results of image recognition, voice conversion, etc.).

[1481] "Emotion analysis" is a technology that determines a user's emotional state based on the user's voice and facial expression data.

[1482] "Operation assistance" refers to instructions and assistance functions provided in real time when a user operates an electronic device, and is assistance that makes user operations easier and more efficient.

[1483] MODE FOR CARRYING OUT THE INVENTION

[1484] This invention is a system for reducing the difficulty that elderly people have when operating smartphones and other electronic devices. This system captures the user's display device in real time, analyzes the image and voice data, and generates and provides appropriate operating instructions. Furthermore, it has the function of recognizing the user's emotions and flexibly adjusting the operating instructions based on the results.

[1485] Overall system configuration

[1486] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine that performs emotion analysis. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server processes the data and generates operation instructions, while the emotion engine analyzes the user's emotions from the collected data.

[1487] Program processing

[1488] 1. Smart glasses capture the user's display

[1489] The user wears smart glasses and operates a smartphone. The camera in the smart glasses captures the smartphone screen in real time, compresses the image data, and sends it to a server. In this case, the specific hardware used is the smart glasses, and the software used is a software module that can process image data in real time.

[1490] 2. The server performs image recognition

[1491] The server analyzes the image data received from the smart glasses and uses image recognition algorithms such as OpenCV to identify elements in the image (notifications, icons, buttons, etc.). The results are temporarily stored in a database.

[1492] 3. Smart glasses will capture your voice

[1493] The user speaks to the smart glasses, and the built-in microphone in the smart glasses captures the voice and sends the voice data to the server.

[1494] 4. The server performs voice recognition

[1495] The server converts the voice data received from the smart glasses into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), then analyzes the converted text data to determine what the user wants.

[1496] 5. Emotion engine recognizes user emotions

[1497] The user's facial expression data is captured and sent to the server, where it is analyzed together with the voice data, and the emotion engine determines the user's emotion from the voice and facial expression.

[1498] 6. Generative AI generates instructions

[1499] The generative AI model combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operational instructions. For example, if the user feels anxious, a polite and specific instruction such as "Open the Settings app" is generated. An example of a prompt sentence from this generative AI is as follows:

[1500] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[1501] 7. The server provides audio instructions

[1502] The generated operation instructions are converted into voice data using a Text-to-Speech (TTS) engine, and this voice data is sent to the smart glasses.

[1503] 8. Smart glasses will play voice commands

[1504] The smart glasses then play back the received voice data and provide real-time voice instructions to the user, allowing them to perform operations without stress.

[1505] Specific examples

[1506] Example 1: Deleting a notification

[1507] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1508] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[1509] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[1510] 4. The server combines the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[1511] 5. The server sends the generated voice data to the smart glasses.

[1512] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1513] Example 2: Changing settings

[1514] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1515] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[1516] 3. The server uses an image recognition algorithm to detect icons on the home screen and settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[1517] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[1518] 5. The server sends the generated voice data to the smart glasses.

[1519] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1520] In this way, the system collects and analyzes real-time user data to help elderly people operate electronic devices safely and efficiently.

[1521] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1522] Processing Steps

[1523] Step 1: Smart glasses capture the user's display

[1524] Input: The user wears the smart glasses and operates a smartphone.

[1525] How it works: The device (smart glasses) uses its built-in camera to capture the screen of the user's smartphone in real time.

[1526] Data processing: Compress the captured image data to reduce the data size.

[1527] Output: Send the compressed image data to the server.

[1528] Step 2: The server performs image recognition

[1529] Input: The server receives the compressed image data sent from the smart glasses.

[1530] How it works: The server uses image recognition algorithms such as OpenCV to identify elements on the image (notifications, icons, buttons, etc.).

[1531] Data processing: Analyze image data and identify and classify each element.

[1532] Output: The analysis results are temporarily saved in a database.

[1533] Step 3: Smart glasses capture user's voice

[1534] Input: The user speaks their intention to operate the smart glasses.

[1535] How it works: The device (smart glasses) uses a built-in microphone to capture the user's voice.

[1536] Data processing: Converting captured audio data into a certain format.

[1537] Output: Sends the audio data to the server.

[1538] Step 4: The server performs speech recognition

[1539] Input: The server receives the audio data sent from the smart glasses.

[1540] How it works: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio data into text.

[1541] Data processing: Convert the audio data into text data and analyze the meaning.

[1542] Output: Send the parsed text data to the appropriate processing module in the server.

[1543] Step 5: The emotion engine recognizes the user's emotion

[1544] Input: The smart glasses capture the user's facial expressions and send them to the server, as well as audio data.

[1545] Operation: The server uses an emotion engine to analyze facial and voice data to determine the user's emotional state.

[1546] Data processing: Combining facial expression analysis and voice analysis to identify the user's emotions.

[1547] Output: The results of sentiment analysis are saved in a database on the server.

[1548] Step 6: Generative AI generates instructions

[1549] Input: The results of image recognition, voice recognition, and emotion recognition are stored in a database on the server.

[1550] Action: The server integrates this data and uses a generative AI model to generate appropriate operational instructions.

[1551] Data processing: Various data are input into the generative AI model as prompt sentences to generate instruction sentences.

[1552] Output: The generated operation instructions are output as text data. This text includes specific operation methods.

[1553] Prompt Sentence Examples

[1554] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[1555] Step 7: The server provides spoken instructions

[1556] Input: Text data of operation instructions generated by the generative AI model.

[1557] How it works: The server uses a text-to-speech (TTS) engine to convert text data into audio data (e.g., Google Cloud Text-to-Speech).

[1558] Data processing: Convert text data into audio data.

[1559] Output: The generated audio data is sent to the smart glasses.

[1560] Step 8: Smart glasses play voice commands

[1561] Input: Audio data sent from the server.

[1562] Operation: The device (smart glasses) plays the received audio data and provides real-time operating instructions to the user.

[1563] Data processing: None in particular (data reproduction).

[1564] Output: Provides an environment where the user can listen to instructions and assists with operation.

[1565] (Application example 2)

[1566] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1567] Elderly people often experience difficulties when searching for products in physical stores, completing purchase procedures, and obtaining campaign information. Therefore, there is a need for support systems that allow elderly people to enjoy shopping independently and comfortably. While current technology supports elderly people using voice recognition and image recognition, there is a lack of systems that combine emotion recognition and real-time support, which has resulted in a lack of improvements in the user experience.

[1568] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1569] In this invention, the server includes means for capturing a user's display in real time, means for transmitting the captured image to the server, means for analyzing the received image in the server and performing image recognition, means for capturing the user's voice and transmitting the voice data to the server, means for analyzing the voice data and performing voice recognition in the server, means for generating operation instructions in the server using a generation AI based on the results of the image recognition and the voice recognition, means for converting the generated operation instructions into voice data and providing it to the user, means for receiving a question about a specific product so that the user can search for the product in a physical store and providing voice guidance on the location of the product on the shelf, means for obtaining information on the product's barcode and purchase procedure and providing voice guidance on how to purchase, and means for obtaining campaign information such as discounted products from a database and providing voice guidance in real time, thereby enabling elderly people to enjoy shopping independently and comfortably in physical stores.

[1570] "Capturing a user's display" means taking a real-time picture of the screen of the electronic device being used by the user.

[1571] A "server" is a central control unit that processes data and provides necessary information over a network.

[1572] "Image recognition" is a technology that analyzes captured image data and identifies specific objects, characters, symbols, etc. within it.

[1573] "Voice capture" is the process of recording a user's speech with a microphone and collecting that voice data.

[1574] "Speech recognition" is a technology that analyzes captured voice data and converts it into text data.

[1575] "Generative AI" is an artificial intelligence technology that generates appropriate output based on input data, creating operational instructions based on user questions and requests.

[1576] "Generation of operation instructions" is the process of determining appropriate actions based on the results of image recognition and voice recognition, and generating those instructions.

[1577] "Converting voice data" refers to the process of converting the generated textual instructions into voice using speech synthesis technology.

[1578] A "physical store" is a commercial facility that consumers can physically visit and where goods and services are offered.

[1579] "Product search" is the process of gathering information and providing guidance needed to locate a specific product.

[1580] "Shelf location information guidance" refers to audio notification of where specific products are located within the store.

[1581] "Barcode acquisition" refers to reading the barcode information to identify the uniqueness of the product.

[1582] "Purchase procedure guidance" refers to audio instructions explaining the steps and methods for purchasing a product.

[1583] "Providing campaign information" means informing users in real time about current discounts, special offers, and other information.

[1584] A "database" is an electronic information storage system that organizes and stores information, allowing it to be quickly searched and updated as needed.

[1585] This system aims to alleviate the difficulties that seniors face when searching for products in brick-and-mortar stores, completing purchase procedures, and obtaining campaign information. In particular, it uses smart glasses, smartphones, a central control server, and an emotion engine to help seniors enjoy shopping independently and comfortably.

[1586] Overall system configuration

[1587] The system mainly consists of the user's devices (smart glasses and smartphone), a central server, an emotion engine, and a database. The smart glasses are equipped with a camera and microphone to capture the user's voice and image. The server processes this data and generates the necessary operation instructions, which are then provided via voice.

[1588] Hardware and Software

[1589] Smart glasses: Built-in camera and microphone capture the user's voice and display. Example: A common name for smart wearable devices.

[1590] Server: Processes and analyzes data and generates operational instructions. Example: A common name is a cloud server.

[1591] Speech recognition engine: converts voice data into text. Example: speech processing software.

[1592] Image Recognition Engine: Analyzes image data and extracts information. Example: Image analysis software.

[1593] Emotion engine: Determines the user's emotions from voice and facial expression data. Example: emotion analysis software.

[1594] Database: Stores product information, store maps, and campaign information and updates them in real time. Example: Data storage system.

[1595] Specific functions and operations of the system

[1596] 1. Product search support

[1597] A user puts on smart glasses and asks, "Where is this item?"

[1598] The device (smart glasses) captures images of the store using a camera and sends them to a server. It also captures the user's voice using a microphone and sends it to the server.

[1599] The server analyzes the received data using an image recognition engine and a voice recognition engine to identify the product's location information.

[1600] The server uses generation AI to generate shelf location information as voice instructions, and then converts this into voice data using a TTS engine.

[1601] The terminal (smart glasses) guides the generated voice instructions to the user.

[1602] 2. Purchase procedure assistance

[1603] The user asks the smart glasses, "How do I purchase this product?"

[1604] The device (smart glasses) captures the user's voice and sends it to the server.

[1605] The server converts the voice data into text using a voice recognition engine and obtains information about the purchase procedure.

[1606] The server uses generative AI to generate purchase procedure instructions and converts them into voice data using a TTS engine.

[1607] The terminal (smart glasses) guides the generated voice instructions to the user.

[1608] 3. Providing campaign information

[1609] The user asks the smart glasses, "What are the deals today?"

[1610] The device (smart glasses) captures the user's voice and sends it to the server.

[1611] The server converts the voice data into text using a voice recognition engine and retrieves the latest campaign information from the database in real time.

[1612] The server uses generation AI to generate campaign information as voice instructions and converts them into voice data using a TTS engine.

[1613] The terminal (smart glasses) guides the generated voice instructions to the user.

[1614] Examples of concrete examples and prompts

[1615] Specific examples of product searches

[1616] 1. A user asks, "Where is this item?"

[1617] 2. The device (smart glasses) captures images and audio and sends them to the server.

[1618] 3. The server identifies the product's location and provides a voice message saying, "The product is on the left side of aisle 3."

[1619] Example prompt sentence:

[1620] When you ask, "Where is this product?" the system analyzes camera footage from within the store and provides voice guidance on the location of the product.

[1621] Examples of campaign information provision

[1622] 1. A user asks, "What are the discounted items today?"

[1623] 2. The device (smart glasses) captures the audio and sends it to the server.

[1624] 3. The server retrieves the latest campaign information from the database and announces, "Today's discounted items are apples and oranges."

[1625] Example prompt sentence:

[1626] When you ask, "What are the discounted items today?" the system will query the database and provide you with the latest campaign information via voice.

[1627] The system combines advanced technologies such as real-time image capture, voice recognition, and emotion recognition to help seniors shop independently and comfortably in brick-and-mortar stores.

[1628] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1629] Step 1:

[1630] The user puts on the smart glasses and asks a question.

[1631] Input: User utterance ("Where is this item?").

[1632] Output: Audio data.

[1633] Specific operation: The user asks a question into the microphone of the smart glasses. The voice is captured by the microphone of the smart glasses and collected as voice data.

[1634] Step 2:

[1635] The device (smart glasses) captures images of the inside of the store using a camera and sends them to the server along with audio data.

[1636] Input: User's voice data and store image data.

[1637] Output: Audio and image data sent to the server.

[1638] How it works: The smart glasses' camera captures images of the store in real time, while the smart glasses' microphone simultaneously collects audio data and sends both to a server.

[1639] Step 3:

[1640] The image data received by the server is analyzed using an image analysis engine.

[1641] Input: Image data sent to the server.

[1642] Output: Image analysis results (shelf and item location information).

[1643] Specific operation: The server passes the received image data to an image analysis engine (e.g., image analysis software) for analysis. It extracts the location information of shelves and products in the image and temporarily stores the analysis results.

[1644] Step 4:

[1645] The server analyzes the received voice data using a voice recognition engine.

[1646] Input: The audio data sent to the server.

[1647] Output: Text data (user question).

[1648] Specific operation: The server passes the voice data to a voice recognition engine (e.g., voice processing software) and converts the voice into text, which extracts the question asked by the user as text data.

[1649] Step 5:

[1650] The server integrates the image recognition results and voice recognition results and generates operation instructions using a generative AI model.

[1651] Input: Image analysis results and text data.

[1652] Output: Text data of operation instructions.

[1653] Specific operation: The server integrates the image analysis results and the voice recognition results, and generates appropriate operation instructions for the user's question using a generative AI model (e.g., generative AI). The generated operation instructions are saved as text data.

[1654] Step 6:

[1655] The server converts the generated operation instructions into voice data using a TTS engine.

[1656] Input: Text data of operation instructions.

[1657] Output: Audio data.

[1658] Specific operation: The server passes the text-based operation instructions to a TTS engine (e.g., speech synthesis software) and converts them into voice data, which is then sent to the smart glasses.

[1659] Step 7:

[1660] The device (smart glasses) plays the received audio data to the user.

[1661] Input: Audio data sent from the server.

[1662] Output: Audio instructions provided to the user.

[1663] Specific operation: The smart glasses play the received audio data through a speaker and provide the user with audio instructions, such as audio guidance on the location of product shelves and how to proceed with the purchase.

[1664] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1665] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1666] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1667] [Fourth embodiment]

[1668] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1669] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1670] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1671] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1672] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1673] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1674] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1675] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1676] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1677] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1678] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1679] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1680] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1681] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures the user's display in real time, sends the image to a server, and uses image and voice recognition on the server to provide operation instructions using generative AI.

[1682] Overall system configuration

[1683] This system mainly consists of the user's devices (smart glasses and smartphone) and a central server. The smart glasses have a built-in camera and microphone, and are responsible for capturing images from the user's smartphone screen and collecting the user's voice. The server is responsible for processing the data and generating operation instructions.

[1684] Program processing overview

[1685] 1. Smart glasses capture the user's display

[1686] The user wears the smart glasses and operates a smartphone.

[1687] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[1688] 2. The server performs image recognition

[1689] It decodes the received image and uses image recognition algorithms to analyze elements on the smartphone screen.

[1690] The analysis results are temporarily stored in a database and the next step is taken.

[1691] 3. Recognize the user's voice

[1692] The smart glasses capture the user's voice with a microphone and send the voice data to a server.

[1693] The server uses a speech recognition engine to convert the voice data into text.

[1694] Analyze text data to identify the user's intent and instructions.

[1695] 4. Generative AI generates instructions

[1696] The results of image recognition and voice recognition are integrated and appropriate operating instructions are generated using generative AI (e.g., large-scale language models).

[1697] For example, if the user's intention is recognized as "I want to delete the notification," a specific instruction such as "Swipe the notification to the left" is generated.

[1698] 5. The server provides audio instructions

[1699] A text-to-speech engine (TTS) is used to convert the generated text instructions into audio data.

[1700] The generated voice data is sent to the smart glasses to provide voice instructions to the user.

[1701] Specific examples

[1702] Example 1: Deleting a notification

[1703] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1704] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1705] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1706] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[1707] 5. The server uses voice recognition to obtain the text "I want to delete this notification."

[1708] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[1709] 7. The server transmits the generated voice data to the smart glasses.

[1710] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1711] Example 2: Changing settings

[1712] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1713] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1714] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1715] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[1716] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[1717] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[1718] 7. The server transmits the generated voice data to the smart glasses.

[1719] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1720] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[1721] The processing flow will be explained below.

[1722] Step 1:

[1723] When a user looks at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and compressed.

[1724] Step 2:

[1725] The device (smart glasses) transmits the captured images to the server in real time using a secure communication protocol (e.g., HTTPS).

[1726] Step 3:

[1727] The server decodes the received image data and analyzes it using image recognition algorithms to identify important elements on the screen (notifications, icons, buttons, etc.).

[1728] Step 4:

[1729] The device (smart glasses) captures the user's voice using a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired and sent to the server.

[1730] Step 5:

[1731] The server passes the received voice data through a speech recognition engine to convert it into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[1732] Step 6:

[1733] The server combines the analyzed image recognition results with the voice recognition results and uses generative AI to generate appropriate instructions for the user, such as "swipe left on the notification."

[1734] Step 7:

[1735] The server uses a text-to-speech (TTS) engine to convert the generated text instructions into audio data, which is then sent to the smart glasses.

[1736] Step 8:

[1737] The device (smart glasses) plays the received audio data and provides the user with audio instructions, such as "Swipe the notification left."

[1738] These are the specific processing steps of the program. Through this process, users can receive appropriate and prompt instructions based on the information displayed in real time. This can eliminate the anxiety and difficulty that seniors may have when operating smartphones and other electronic devices.

[1739] Example 1

[1740] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1741] The purpose of this project is to provide a system that solves the difficulties that elderly people face when operating smartphones and other electronic devices due to visual, hearing, and operational difficulties. In particular, for people with visual or hearing impairments, it is difficult to provide appropriate operational instructions in real time, and these operations need to be intuitive and easy for users to understand.

[1742] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1743] In this invention, the server includes means for capturing an image of the user's display device in real time, means for transmitting the captured image to a central processing unit, means for analyzing the received image in the central processing unit and performing image recognition, means for recording the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice recognition in the central processing unit, means for generating operation instructions in the central processing unit using a generation AI based on the results of image recognition and voice recognition, and means for converting the generated operation instructions into voice data and providing it to the user. This enables users to receive intuitive and easy-to-understand operation instructions in real time without having difficulty operating their smartphones or other electronic devices.

[1744] "User" refers to the person who operates the system, and in this system includes elderly people who use smart glasses or smartphones.

[1745] "Display device" refers to electronic devices that display visual information, such as smartphones and tablets.

[1746] The term "means for capturing images" refers to a device, such as a camera or an imaging device, that captures the screen of a display device in real time.

[1747] "Central processing unit" refers to a computing resource accessed via a network for data analysis and processing, such as a cloud server or remote server.

[1748] "Transmitting means" refers to technologies and devices for transmitting data to other devices or servers, including wireless communication modules, Wi-Fi, Bluetooth, etc.

[1749] "Means for receiving" refers to the technology and devices for obtaining data via a network.

[1750] "Image recognition" refers to the technology of analyzing captured image data and recognizing specific elements or patterns.

[1751] "Voice recognition" refers to the technology of analyzing recorded voice data and converting what is spoken into text data.

[1752] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to generate text data or operational instructions, including large-scale language models such as GPT-4.

[1753] "Operation instructions" refers to information that instructs the user to perform specific actions to achieve the operation or goal desired by the user.

[1754] "Means for converting into voice data" refers to technology that converts text data into voice using a text-to-speech synthesis engine (TTS) or similar.

[1755] "Means for providing to the user" refers to devices and technologies for conveying specific operating instructions to the user in voice instructions or other forms.

[1756] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. This system captures a user's display device in real time, sends the image to a central processing unit, and uses a generation AI to provide operating instructions by performing image and voice recognition in the central processing unit.

[1757] Overall system configuration

[1758] This system mainly consists of the user's devices (smart glasses and smartphone) and a central processing unit. The smart glasses have a built-in camera and microphone, and are responsible for capturing images of the user's smartphone screen and collecting the user's voice. The central processing unit is responsible for processing data and generating operating instructions.

[1759] Hardware and Software Configuration

[1760] 1. Smart Glasses

[1761] The smart glasses are equipped with a built-in camera module, microphone, and wireless communication module, which allows them to capture the user's smartphone screen and transmit it to a central processing unit in real time.The microphone also captures the user's voice and transmits it to the central processing unit.

[1762] 2. Central Processing Unit

[1763] The server used as the central processing unit is built on a high-performance cloud infrastructure. This server uses OpenCV and TensorFlow for image analysis, the Google Speech-to-Text API for voice analysis, GPT-4 for generative AI, and Amazon Polly for text-to-speech (TTS) engine.

[1764] Processing flow

[1765] 1. The user puts on the smart glasses

[1766] The user puts on the smart glasses and prepares to operate the smartphone.

[1767] 2. The device (smart glasses) captures the display of the user's smartphone.

[1768] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals, compresses the images in real time, and sends them to a server.

[1769] 3. The server processes the received image

[1770] The server decodes the received image and analyzes the elements on the screen using image recognition algorithms such as OpenCV and TensorFlow.

[1771] 4. The device (smart glasses) captures the user's voice.

[1772] The smart glasses' microphone is used to capture the user's voice in real time and send the voice data to a server.

[1773] 5. The server recognizes the voice

[1774] The server analyzes the voice data using the Google Speech-to-Text API and converts it into text data.

[1775] 6. Generative AI generates operation instructions

[1776] Based on the results of image and voice recognition, appropriate operation instructions are generated using generative AI such as GPT-4. For example, if the voice recognizes "I want to change my Wi-Fi settings," the instruction generated is "Open the Settings app."

[1777] 7. The server provides audio instructions

[1778] The generated operation instructions are converted into voice data using TTS such as Amazon Polly and sent to the smart glasses to provide voice instructions to the user.

[1779] Specific examples

[1780] Example 1: Deleting a notification

[1781] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1782] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1783] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1784] 4. The server uses an image recognition algorithm to detect the notification on the screen.

[1785] 5. The server uses voice recognition to obtain the text "I want to dismiss this notification."

[1786] 6. The server uses the generation AI to generate the instruction "Swipe the notification left" and converts it into voice data.

[1787] 7. The server sends the generated voice data to the smart glasses.

[1788] 8. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1789] Example 2: Changing settings

[1790] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1791] 2. The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1792] 3. The device (smart glasses) captures the user's voice data and sends it to the server.

[1793] 4. The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[1794] 5. The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[1795] 6. The server uses the generation AI to generate the instruction "Open the Settings app" and converts it into voice data.

[1796] 7. The server sends the generated voice data to the smart glasses.

[1797] 8. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1798] The above is a specific example of an embodiment of the present invention. The system combines real-time image capture and voice recognition to quickly and effectively solve operational problems faced by users.

[1799] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1800] Program processing flow

[1801] Step 1: User puts on smart glasses

[1802] Input: User action (wearing smart glasses)

[1803] Output: Smart glasses start

[1804] Specific behavior:

[1805] When a user puts on the smart glasses, the built-in sensors in the smart glasses detect that they are being worn and activate the camera and microphone, preparing the display device for screen and audio capture.

[1806] Step 2: The device (smart glasses) captures the display of the user's smartphone.

[1807] Input: The user's smartphone display

[1808] Output: Captured image data

[1809] Specific behavior:

[1810] The camera in the smart glasses captures the screen of the user's smartphone at regular intervals (for example, 2 frames per second), and the captured image data is compressed in real time and sent to a central processing unit (server) via a wireless communication module.

[1811] Step 3: The server processes the received image

[1812] Input: Capture image data

[1813] Output: Analysis results (element data on the screen)

[1814] Specific behavior:

[1815] The server decodes the image data received via the network and analyzes elements on the smartphone screen (icons, notifications, etc.) using image recognition algorithms such as OpenCV and TensorFlow. The analysis results are temporarily stored in a database.

[1816] Step 4: The device (smart glasses) captures the user's voice

[1817] Input: User's voice

[1818] Output: Audio data

[1819] Specific behavior:

[1820] When a user speaks operational instructions into the smart glasses, the smart glasses' built-in microphone captures the voice in real time and transmits the voice data to a server via a wireless communication module.

[1821] Step 5: The server analyzes the audio data

[1822] Input: Audio data

[1823] Output: Analysis results (text data)

[1824] Specific behavior:

[1825] The server receives the voice data and converts it into text using a speech recognition engine such as the Google Speech-to-Text API. The resulting text data is used as input for the generation AI.

[1826] Step 6: Generative AI generates operation instructions

[1827] Input: Image analysis results, audio analysis results (text data)

[1828] Output: Instruction text

[1829] Specific behavior:

[1830] The server combines the results of image and voice recognition and generates appropriate instructions using a generative AI model such as GPT-4. For example, if the text "I want to change my Wi-Fi settings" is obtained through voice recognition, the server generates the specific instruction "Open the Settings app."

[1831] Step 7: The server converts the instructions into audio data

[1832] Input: Instruction text

[1833] Output: Operation instruction voice data

[1834] Specific behavior:

[1835] The text instructions generated by the AI ​​are converted into voice data using a text-to-speech (TTS) engine such as Amazon Polly, and the generated voice data is sent to the smart glasses via the network.

[1836] Step 8: The device (smart glasses) plays back the operation instructions.

[1837] Input: Operation instruction voice data

[1838] Output: Instructions provided to the user

[1839] Specific behavior:

[1840] The smart glasses play back the audio data received from the server and give the user instructions such as "Open the Settings app," allowing the user to intuitively understand the appropriate operation.

[1841] These are the specific processing steps of this system's program. The specific operations, inputs, and outputs are clearly defined at each step, allowing for a detailed understanding of how the entire system operates.

[1842] (Application example 1)

[1843] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1844] There is an urgent need to address the difficulties that seniors face when using smartphones and other electronic devices. Particularly in brick-and-mortar stores, seniors often experience unfamiliar operational challenges when searching for products or making payments. Complicated procedures and small screen displays can also be a visual burden. The present invention aims to provide a system that provides real-time assistance to facilitate these operations.

[1845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1846] In this invention, the server includes means for periodically transmitting the captured image of the user's display to the server, means for analyzing the smart device screen using an image recognition algorithm after transmitting the captured image to the server, means for capturing the user's voice with the microphone of the smart glasses and transmitting it to the server, means for analyzing the voice data and performing voice recognition in the server, means for integrating the image recognition results and the voice recognition results and generating specific operation instructions using a generation AI, means for converting the generated operation instructions into voice data using a text-to-speech engine and providing it to the user via the smart glasses, and means for providing purchasing support and payment assistance in physical stores, which enables elderly people to efficiently purchase products in physical stores and smoothly complete payment procedures.

[1847] The "captured user display image" is the display content of the device being operated by the user captured as is and acquired as image data.

[1848] "Transmission means" refers to the technical means by which data is transferred from one device to another.

[1849] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying elements or patterns within it.

[1850] "Means for analyzing the screen of a smart device" refers to a technical means for understanding the display information of a smartphone or other digital device used by a user and extracting the necessary information.

[1851] "Capturing the user's voice with the microphone of the smart glasses" refers to using the microphone built into the smart glasses to obtain the user's speech as voice data.

[1852] "Means for performing speech recognition" refers to the technical means for analyzing captured speech data and expressing its contents in text form.

[1853] "Means for generating specific operating instructions using generative AI" refers to using artificial intelligence technology to create specific operating instructions for users based on the results of voice recognition and image recognition.

[1854] "Text-to-speech engine" refers to the technical means for converting generated text information into speech data.

[1855] "Purchase support and payment assistance in physical stores" refers to smoothly supporting the process of users selecting and purchasing products and making payments in physical stores.

[1856] This invention is a system that allows elderly people to easily purchase products in physical stores and smoothly complete payment procedures. The system uses the user's smart glasses and smartphone as the main terminals and operates in conjunction with a central server.

[1857] Overall system configuration

[1858] The system consists of user devices (smart glasses and smartphones), a central server, and various software modules.

[1859] Main hardware and software used

[1860] Hardware:

[1861] Smart glasses (built-in camera, microphone, and speaker)

[1862] Smartphone

[1863] software:

[1864] Image recognition algorithm (TensorFlow)

[1865] Speech recognition engine (Google Speech-to-Text API)

[1866] Generative AI models (GPT-3 and similar large-scale language models)

[1867] Text-to-speech engine (Google Text-to-Speech API)

[1868] Program Processing Overview

[1869] 1. Capture:

[1870] When a customer (user) speaks to the smart glasses and says, "I would like to purchase this product," the camera in the smart glasses captures the screen of the POS terminal.

[1871] The captured image is compressed and sent to the server.

[1872] 2. Speech Recognition:

[1873] The microphone in the smart glasses captures the customer's voice and sends the voice data to a server.

[1874] The server uses a speech recognition engine to convert the voice data into text and identify the customer's instructions.

[1875] 3. Image Recognition and Instruction Generation:

[1876] The server decodes the received image and uses image recognition algorithms to analyze the screen elements of the POS terminal.

[1877] The image recognition results and voice recognition results are integrated, and the generative AI model generates appropriate operating instructions.

[1878] For example, generate an operation instruction such as "Scan the QR code."

[1879] 4. Audio guide:

[1880] The generated operation instructions are converted into voice data by a text-to-speech synthesis engine.

[1881] Voice data is sent to the smart glasses, and operational instructions are provided to the customer via voice.

[1882] Specific examples

[1883] 1. Product purchase assistance:

[1884] A customer says, "I would like to purchase this product."

[1885] The camera in the smart glasses captures the screen of the POS terminal and sends the image to the server.

[1886] The audio data is also sent to the server at the same time.

[1887] Image and voice recognition is performed on the server side, and the generation AI generates operation instructions.

[1888] Operation instructions are converted into voice and provided to the customer through smart glasses.

[1889] For example, you might hear a voice prompt saying, "Please scan the QR code."

[1890] 2. Payment Assistance:

[1891] The customer asks, "How do I pay?"

[1892] The camera in the smart glasses captures the screen of the payment terminal and sends the image to a server.

[1893] The received voice data is also analyzed by the server.

[1894] The generative AI creates steps and generates specific instructions, such as "insert your card."

[1895] Instructions converted into voice data are communicated to the customer through smart glasses.

[1896] Prompt Sentence Examples

[1897] Based on the user's intent of "I want to purchase this product," analyze the captured image of the POS terminal screen and generate appropriate instructions, including how to scan the QR code.

[1898] In this way, the system of the present invention provides a concrete means for assisting seniors in purchasing and paying for products in brick-and-mortar stores.

[1899] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1900] Step 1:

[1901] The user speaks to the smart glasses, saying, "I want to buy this product." The microphone in the smart glasses captures this voice. The input is the user's speech, and the output is the captured voice data. This voice data is sent from the smart glasses to the server.

[1902] Step 2:

[1903] After capturing the user's speech, the camera on the smart glasses captures the screen of the POS terminal. The input is the screen of the POS terminal, and the output is the captured screen image. This image is also sent from the smart glasses to the server.

[1904] Step 3:

[1905] The server analyzes the received voice data. The input is the voice data, and the output is the speech recognition result in text format. This analysis is performed using a speech recognition engine (Google Speech-to-Text API).

[1906] Step 4:

[1907] The server analyzes the received image data in parallel. The input is the screen image, and the output is the analysis result to identify the screen elements. An image recognition algorithm (TensorFlow) is used to recognize the information on the POS terminal screen.

[1908] Step 5:

[1909] The server integrates the results of speech recognition and image recognition. The input is the speech recognition result and the image recognition result, and the output is the interpretation data that combines them. This determines the appropriate instruction based on the user's intention (e.g., "I want to purchase this product").

[1910] Step 6:

[1911] The server generates instructions using a generative AI model (such as GPT-3). The input is interpretation data, and the output is specific instructions (e.g., "Scan the QR code"). The generative AI determines the optimal operation procedure based on the results of voice recognition and image recognition.

[1912] Step 7:

[1913] The server converts the generated operation instructions into voice data using a text-to-speech synthesis engine (Google Text-to-Speech API). The input is the text data of the operation instructions, and the output is voice data.

[1914] Step 8:

[1915] The device (smart glasses) receives the voice data sent from the server and provides instructions to the user via the speaker. The input is voice data, and the output is voice guidance to the user. The smart glasses provide specific voice instructions such as "Please scan the QR code."

[1916] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1917] This invention is a system to solve the difficulties that elderly people have when operating smartphones and other electronic devices. The system captures the user's display in real time, analyzes image and voice data on a server, and generates and provides appropriate operating instructions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions.

[1918] Overall system configuration

[1919] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server is responsible for processing the data and generating operation instructions, while the emotion engine is responsible for analyzing the user's emotions from the collected data.

[1920] Program processing overview

[1921] 1. Smart glasses capture the user's display

[1922] The user wears the smart glasses and operates a smartphone.

[1923] The camera in the smart glasses captures the screen of the user's smartphone in real time, compresses the image and sends it to a server.

[1924] 2. The server performs image recognition

[1925] The received image data is analyzed and image recognition algorithms are used to identify on-screen elements (notifications, icons, buttons, etc.) The analysis results are temporarily stored in a database.

[1926] 3. Smart glasses will capture your voice

[1927] The smart glasses capture the user's voice with a built-in microphone and send the audio data to a server.

[1928] 4. The server performs voice recognition

[1929] The received voice data is passed through a speech recognition engine to be converted into text data, which is then analyzed to determine what the user wants to do (intention or instructions).

[1930] 5. Emotion engine recognizes user emotions

[1931] The smart glasses capture the user's facial expressions and send the image data to a server, which also analyzes the voice data, and an emotion engine determines the user's emotions from the voice and facial expressions.

[1932] 6. Generative AI generates instructions

[1933] It combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operating instructions using generative AI. For example, when the user is feeling anxious, it intends to generate more polite and easy-to-understand instructions.

[1934] 7. The server provides audio instructions

[1935] The generated text instructions are converted into voice data using a text-to-speech engine (TTS), and the generated voice data is sent to the smart glasses.

[1936] 8. Smart glasses will play voice commands

[1937] The smart glasses then play back the received audio data and provide the user with voice instructions, allowing them to perform operations while receiving real-time assistance.

[1938] Specific examples

[1939] Example 1: Deleting a notification

[1940] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[1941] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[1942] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[1943] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[1944] 5. The server transmits the generated voice data to the smart glasses.

[1945] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1946] Example 2: Changing settings

[1947] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[1948] 2. The device (smart glasses) captures the screen and audio of the user's smartphone and sends the data to a server. It also captures the user's facial expressions.

[1949] 3. The server uses an image recognition algorithm to detect icons on the home screen and in the settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[1950] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[1951] 5. The server transmits the generated voice data to the smart glasses.

[1952] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[1953] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[1954] The processing flow will be explained below.

[1955] Step 1:

[1956] When a user is looking at the smartphone screen, the device (smart glasses) captures the screen in real time using a built-in camera. Images are acquired at regular intervals (for example, 10 frames per second) and the image data is compressed.

[1957] Step 2:

[1958] The device (smart glasses) transmits the captured image data to the server in real time using a secure communication protocol (e.g., HTTPS).

[1959] Step 3:

[1960] The server decodes the received image data and uses image recognition algorithms to analyze on-screen elements (notifications, icons, buttons, etc.) and temporarily stores the analysis results in a database.

[1961] Step 4:

[1962] The device (smart glasses) captures the user's voice with a built-in microphone. When the user speaks to the smart glasses, the voice data is acquired, compressed, and sent to the server.

[1963] Step 5:

[1964] The server converts the received voice data into text data through a speech recognition engine, and analyzes the converted text data to identify the user's intentions and instructions.

[1965] Step 6:

[1966] The device (smart glasses) uses a built-in camera to capture the user's facial expressions, and transmits the acquired facial expression data to a server.

[1967] Step 7:

[1968] The server runs the facial expression data through an emotion engine to analyze the user's emotions. It also determines emotions from voice data and combines them with the facial expression data to determine the user's current emotional state.

[1969] Step 8:

[1970] The server combines the results of image recognition, voice recognition, and the emotion engine. Based on all of this data, it uses generative AI to generate operational instructions. For example, if the user is feeling anxious, it generates gentle and polite instructions such as "Please remain calm and swipe left on the notification."

[1971] Step 9:

[1972] The server passes the generated text instructions through a text-to-speech engine (TTS) to convert them into voice data, which is then sent to the smart glasses.

[1973] Step 10:

[1974] The device (smart glasses) plays the received voice data and provides the user with voice instructions, allowing the user to perform operations while receiving real-time assistance.

[1975] The following is an explanation using a specific example.

[1976] Example 1: Deleting a notification

[1977] Step 1:

[1978] The user speaks to the smart glasses, saying, "I want to delete this notification."

[1979] Step 2:

[1980] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[1981] Step 3:

[1982] The device (smart glasses) captures the user's voice data and sends it to the server.

[1983] Step 4:

[1984] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[1985] Step 5:

[1986] The server uses an image recognition algorithm to detect the notification on the screen.

[1987] Step 6:

[1988] The server uses voice recognition to obtain the text "I want to delete this notification."

[1989] Step 7:

[1990] The server analyzes the user's emotions using an emotion engine.

[1991] Step 8:

[1992] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate instructions such as "Swipe left on the notification," which are then converted into voice data.

[1993] Step 9:

[1994] The server transmits the generated voice data to the smart glasses.

[1995] Step 10:

[1996] The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[1997] Example 2: Changing settings

[1998] Step 1:

[1999] The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[2000] Step 2:

[2001] The device (smart glasses) captures the screen of the user's smartphone and sends the image to the server.

[2002] Step 3:

[2003] The device (smart glasses) captures the user's voice data and sends it to the server.

[2004] Step 4:

[2005] The device (smart glasses) also captures the user's facial expressions and sends the data to the server.

[2006] Step 5:

[2007] The server uses an image recognition algorithm to detect icons on the home screen and settings app.

[2008] Step 6:

[2009] The server uses voice recognition to obtain the text "I would like to change my Wi-Fi settings."

[2010] Step 7:

[2011] The server analyzes the user's emotions using an emotion engine.

[2012] Step 8:

[2013] The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[2014] Step 9:

[2015] The server transmits the generated voice data to the smart glasses.

[2016] Step 10:

[2017] The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[2018] The above is a specific example of a form for implementing the invention that combines an emotion engine. This system optimizes user operation assistance by combining real-time image capture, voice recognition, and emotion recognition, enabling elderly people to operate smartphones and other electronic devices safely and efficiently.

[2019] Example 2

[2020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2021] Currently, the difficulties that older people experience when operating smartphones and other electronic devices stem from the learning curve and cognitive overhead. Complex setting changes and notification operations, in particular, pose a significant burden for older people. Furthermore, existing operation assistance systems are unable to provide appropriate assistance that takes into account the user's emotional state. Therefore, there is a need for systems that enable older people to operate electronic devices more easily and without stress.

[2022] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2023] In this invention, the server includes means for capturing a user's display device in real time, means for compressing the captured image and transmitting it to a central processing unit, means for analyzing the received image in the central processing unit and performing object recognition, means for capturing the user's voice and transmitting the voice data to the central processing unit, means for analyzing the voice data and performing voice conversion in the central processing unit, means for generating operation instructions in the central processing unit using generative artificial intelligence based on the results of the object recognition and the voice conversion, means for converting the generated operation instructions into voice data and providing them to the user, means for capturing the user's facial expressions and performing emotion analysis, and means for reflecting the results of the emotion analysis in the generation of operation instructions. This reduces the difficulty that elderly people experience when operating electronic devices and makes it possible to provide optimal operation assistance in real time according to the user's emotional state.

[2024] A "display device" is a part of an electronic device that allows a user to visually receive information, specifically a device that includes a screen such as a smartphone or tablet.

[2025] A "central processing unit" is a computer system that receives, analyzes, and generates instructions, and specifically includes a server or a cloud computing platform.

[2026] "Object recognition" is a technology that identifies specific objects or elements (e.g., notifications, icons, buttons, etc.) from image data and determines their presence.

[2027] "Speech conversion" is a technology that converts voice data into text data, and is used to analyze a user's intentions and instructions.

[2028] "Generative artificial intelligence" is an AI technology that generates appropriate operating instructions based on multiple data (for example, the results of image recognition, voice conversion, etc.).

[2029] "Emotion analysis" is a technology that determines a user's emotional state based on the user's voice and facial expression data.

[2030] "Operation assistance" refers to instructions and assistance functions provided in real time when a user operates an electronic device, and is assistance that makes user operations easier and more efficient.

[2031] MODE FOR CARRYING OUT THE INVENTION

[2032] This invention is a system for reducing the difficulty that elderly people have when operating smartphones and other electronic devices. This system captures the user's display device in real time, analyzes the image and voice data, and generates and provides appropriate operating instructions. Furthermore, it has the function of recognizing the user's emotions and flexibly adjusting the operating instructions based on the results.

[2033] Overall system configuration

[2034] This system mainly consists of the user's devices (smart glasses and smartphone), a central server, and an emotion engine that performs emotion analysis. The smart glasses have a built-in camera and microphone that capture the screen of the user's smartphone and collect the user's voice and facial expressions. The server processes the data and generates operation instructions, while the emotion engine analyzes the user's emotions from the collected data.

[2035] Program processing

[2036] 1. Smart glasses capture the user's display

[2037] The user wears smart glasses and operates a smartphone. The camera in the smart glasses captures the smartphone screen in real time, compresses the image data, and sends it to a server. In this case, the specific hardware used is the smart glasses, and the software used is a software module that can process image data in real time.

[2038] 2. The server performs image recognition

[2039] The server analyzes the image data received from the smart glasses and uses image recognition algorithms such as OpenCV to identify elements in the image (notifications, icons, buttons, etc.). The results are temporarily stored in a database.

[2040] 3. Smart glasses will capture your voice

[2041] The user speaks to the smart glasses, and the built-in microphone in the smart glasses captures the voice and sends the voice data to the server.

[2042] 4. The server performs voice recognition

[2043] The server converts the voice data received from the smart glasses into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), then analyzes the converted text data to determine what the user wants.

[2044] 5. Emotion engine recognizes user emotions

[2045] The user's facial expression data is captured and sent to the server, where it is analyzed together with the voice data, and the emotion engine determines the user's emotion from the voice and facial expression.

[2046] 6. Generative AI generates instructions

[2047] The generative AI model combines the results of image recognition, speech recognition, and an emotion engine to generate appropriate operational instructions. For example, if the user feels anxious, a polite and specific instruction such as "Open the Settings app" is generated. An example of a prompt sentence from this generative AI is as follows:

[2048] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[2049] 7. The server provides audio instructions

[2050] The generated operation instructions are converted into voice data using a Text-to-Speech (TTS) engine, and this voice data is sent to the smart glasses.

[2051] 8. Smart glasses will play voice commands

[2052] The smart glasses then play back the received voice data and provide real-time voice instructions to the user, allowing them to perform operations without stress.

[2053] Specific examples

[2054] Example 1: Deleting a notification

[2055] 1. The user speaks to the smart glasses, saying, "I want to dismiss this notification."

[2056] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[2057] 3. The server uses an image recognition algorithm to detect the notification on the screen and voice recognition to obtain the text "I want to delete this notification." The emotion engine then determines the user's emotion from their facial expression and voice.

[2058] 4. The server combines the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Swipe left on the notification," which is then converted into voice data.

[2059] 5. The server sends the generated voice data to the smart glasses.

[2060] 6. The device (smart glasses) plays the received audio data and instructs the user to "swipe left on the notification."

[2061] Example 2: Changing settings

[2062] 1. The user speaks to the smart glasses and says, "I want to change the Wi-Fi settings."

[2063] 2. The device (smart glasses) captures the screen of the user's smartphone, records audio, and sends the data to a server. It also captures the user's facial expressions.

[2064] 3. The server uses an image recognition algorithm to detect icons on the home screen and settings app, and uses voice recognition to obtain the text "I want to change my Wi-Fi settings." The emotion engine then determines the user's emotion from their facial expression and voice.

[2065] 4. The server integrates the results of image recognition, voice recognition, and emotion recognition, and uses generative AI to generate the instruction "Open the Settings app," which is then converted into voice data.

[2066] 5. The server sends the generated voice data to the smart glasses.

[2067] 6. The device (smart glasses) plays the received audio data and instructs the user to "open the Settings app."

[2068] In this way, the system collects and analyzes real-time user data to help elderly people operate electronic devices safely and efficiently.

[2069] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2070] Processing Steps

[2071] Step 1: Smart glasses capture the user's display

[2072] Input: The user wears the smart glasses and operates a smartphone.

[2073] How it works: The device (smart glasses) uses its built-in camera to capture the screen of the user's smartphone in real time.

[2074] Data processing: Compress the captured image data to reduce the data size.

[2075] Output: Send the compressed image data to the server.

[2076] Step 2: The server performs image recognition

[2077] Input: The server receives the compressed image data sent from the smart glasses.

[2078] How it works: The server uses image recognition algorithms such as OpenCV to identify elements on the image (notifications, icons, buttons, etc.).

[2079] Data processing: Analyze image data and identify and classify each element.

[2080] Output: The analysis results are temporarily saved in a database.

[2081] Step 3: Smart glasses capture user's voice

[2082] Input: The user speaks their intention to operate the smart glasses.

[2083] How it works: The device (smart glasses) uses a built-in microphone to capture the user's voice.

[2084] Data processing: Converting captured audio data into a certain format.

[2085] Output: Sends the audio data to the server.

[2086] Step 4: The server performs speech recognition

[2087] Input: The server receives the audio data sent from the smart glasses.

[2088] How it works: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio data into text.

[2089] Data processing: Convert the audio data into text data and analyze the meaning.

[2090] Output: Send the parsed text data to the appropriate processing module in the server.

[2091] Step 5: The emotion engine recognizes the user's emotion

[2092] Input: The smart glasses capture the user's facial expressions and send them to the server, as well as audio data.

[2093] Operation: The server uses an emotion engine to analyze facial and voice data to determine the user's emotional state.

[2094] Data processing: Combining facial expression analysis and voice analysis to identify the user's emotions.

[2095] Output: The results of sentiment analysis are saved in a database on the server.

[2096] Step 6: Generative AI generates instructions

[2097] Input: The results of image recognition, voice recognition, and emotion recognition are stored in a database on the server.

[2098] Action: The server integrates this data and uses a generative AI model to generate appropriate operational instructions.

[2099] Data processing: Various data are input into the generative AI model as prompt sentences to generate instruction sentences.

[2100] Output: The generated operation instructions are output as text data. This text includes specific operation methods.

[2101] Prompt Sentence Examples

[2102] Prompt: "When a user says 'I want to change my Wi-Fi settings,' generate the appropriate action instructions for the Generated AI."

[2103] Step 7: The server provides spoken instructions

[2104] Input: Text data of operation instructions generated by the generative AI model.

[2105] How it works: The server uses a text-to-speech (TTS) engine to convert text data into audio data (e.g., Google Cloud Text-to-Speech).

[2106] Data processing: Convert text data into audio data.

[2107] Output: The generated audio data is sent to the smart glasses.

[2108] Step 8: Smart glasses play voice commands

[2109] Input: Audio data sent from the server.

[2110] Operation: The device (smart glasses) plays the received audio data and provides real-time operating instructions to the user.

[2111] Data processing: None in particular (data reproduction).

[2112] Output: Provides an environment where the user can listen to instructions and assists with operation.

[2113] (Application example 2)

[2114] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2115] Elderly people often experience difficulties when searching for products in physical stores, completing purchase procedures, and obtaining campaign information. Therefore, there is a need for support systems that allow elderly people to enjoy shopping independently and comfortably. While current technology supports elderly people using voice recognition and image recognition, there is a lack of systems that combine emotion recognition and real-time support, which has resulted in a lack of improvements in the user experience.

[2116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2117] In this invention, the server includes means for capturing a user's display in real time, means for transmitting the captured image to the server, means for analyzing the received image in the server and performing image recognition, means for capturing the user's voice and transmitting the voice data to the server, means for analyzing the voice data and performing voice recognition in the server, means for generating operation instructions in the server using a generation AI based on the results of the image recognition and the voice recognition, means for converting the generated operation instructions into voice data and providing it to the user, means for receiving a question about a specific product so that the user can search for the product in a physical store and providing voice guidance on the location of the product on the shelf, means for obtaining information on the product's barcode and purchase procedure and providing voice guidance on how to purchase, and means for obtaining campaign information such as discounted products from a database and providing voice guidance in real time, thereby enabling elderly people to enjoy shopping independently and comfortably in physical stores.

[2118] "Capturing a user's display" means taking a real-time picture of the screen of the electronic device being used by the user.

[2119] A "server" is a central control unit that processes data and provides necessary information over a network.

[2120] "Image recognition" is a technology that analyzes captured image data and identifies specific objects, characters, symbols, etc. within it.

[2121] "Voice capture" is the process of recording a user's speech with a microphone and collecting that voice data.

[2122] "Speech recognition" is a technology that analyzes captured voice data and converts it into text data.

[2123] "Generative AI" is an artificial intelligence technology that generates appropriate output based on input data, creating operational instructions based on user questions and requests.

[2124] "Generation of operation instructions" is the process of determining appropriate actions based on the results of image recognition and voice recognition, and generating those instructions.

[2125] "Converting voice data" refers to the process of converting the generated textual instructions into voice using speech synthesis technology.

[2126] A "physical store" is a commercial facility that consumers can physically visit and where goods and services are offered.

[2127] "Product search" is the process of gathering information and providing guidance needed to locate a specific product.

[2128] "Shelf location information guidance" refers to audio notification of where specific products are located within the store.

[2129] "Barcode acquisition" refers to reading the barcode information to identify the uniqueness of the product.

[2130] "Purchase procedure guidance" refers to audio instructions explaining the steps and methods for purchasing a product.

[2131] "Providing campaign information" means informing users in real time about current discounts, special offers, and other information.

[2132] A "database" is an electronic information storage system that organizes and stores information, allowing it to be quickly searched and updated as needed.

[2133] This system aims to alleviate the difficulties that seniors face when searching for products in brick-and-mortar stores, completing purchase procedures, and obtaining campaign information. In particular, it uses smart glasses, smartphones, a central control server, and an emotion engine to help seniors enjoy shopping independently and comfortably.

[2134] Overall system configuration

[2135] The system mainly consists of the user's devices (smart glasses and smartphone), a central server, an emotion engine, and a database. The smart glasses are equipped with a camera and microphone to capture the user's voice and image. The server processes this data and generates the necessary operation instructions, which are then provided via voice.

[2136] Hardware and Software

[2137] Smart glasses: Built-in camera and microphone capture the user's voice and display. Example: A common name for smart wearable devices.

[2138] Server: Processes and analyzes data and generates operational instructions. Example: A common name is a cloud server.

[2139] Speech recognition engine: converts voice data into text. Example: speech processing software.

[2140] Image Recognition Engine: Analyzes image data and extracts information. Example: Image analysis software.

[2141] Emotion engine: Determines the user's emotions from voice and facial expression data. Example: emotion analysis software.

[2142] Database: Stores product information, store maps, and campaign information and updates them in real time. Example: Data storage system.

[2143] Specific functions and operations of the system

[2144] 1. Product search support

[2145] A user puts on smart glasses and asks, "Where is this item?"

[2146] The device (smart glasses) captures images of the store using a camera and sends them to a server. It also captures the user's voice using a microphone and sends it to the server.

[2147] The server analyzes the received data using an image recognition engine and a voice recognition engine to identify the product's location information.

[2148] The server uses generation AI to generate shelf location information as voice instructions, and then converts this into voice data using a TTS engine.

[2149] The terminal (smart glasses) guides the generated voice instructions to the user.

[2150] 2. Purchase procedure assistance

[2151] The user asks the smart glasses, "How do I purchase this product?"

[2152] The device (smart glasses) captures the user's voice and sends it to the server.

[2153] The server converts the voice data into text using a voice recognition engine and obtains information about the purchase procedure.

[2154] The server uses generative AI to generate purchase procedure instructions and converts them into voice data using a TTS engine.

[2155] The terminal (smart glasses) guides the generated voice instructions to the user.

[2156] 3. Providing campaign information

[2157] The user asks the smart glasses, "What are the deals today?"

[2158] The device (smart glasses) captures the user's voice and sends it to the server.

[2159] The server converts the voice data into text using a voice recognition engine and retrieves the latest campaign information from the database in real time.

[2160] The server uses generation AI to generate campaign information as voice instructions and converts them into voice data using a TTS engine.

[2161] The terminal (smart glasses) guides the generated voice instructions to the user.

[2162] Examples of concrete examples and prompts

[2163] Specific examples of product searches

[2164] 1. A user asks, "Where is this item?"

[2165] 2. The device (smart glasses) captures images and audio and sends them to the server.

[2166] 3. The server identifies the product's location and provides a voice message saying, "The product is on the left side of aisle 3."

[2167] Example prompt sentence:

[2168] When you ask, "Where is this product?" the system analyzes camera footage from within the store and provides voice guidance on the location of the product.

[2169] Examples of campaign information provision

[2170] 1. A user asks, "What are the discounted items today?"

[2171] 2. The device (smart glasses) captures the audio and sends it to the server.

[2172] 3. The server retrieves the latest campaign information from the database and announces, "Today's discounted items are apples and oranges."

[2173] Example prompt sentence:

[2174] When you ask, "What are the discounted items today?" the system will query the database and provide you with the latest campaign information via voice.

[2175] The system combines advanced technologies such as real-time image capture, voice recognition, and emotion recognition to help seniors shop independently and comfortably in brick-and-mortar stores.

[2176] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2177] Step 1:

[2178] The user puts on the smart glasses and asks a question.

[2179] Input: User utterance ("Where is this item?").

[2180] Output: Audio data.

[2181] Specific operation: The user asks a question into the microphone of the smart glasses. The voice is captured by the microphone of the smart glasses and collected as voice data.

[2182] Step 2:

[2183] The device (smart glasses) captures images of the inside of the store using a camera and sends them to the server along with audio data.

[2184] Input: User's voice data and store image data. ...

Claims

1. means for capturing a user's display in real time; means for transmitting the captured image to a server; A means for analyzing the received image in the server and performing image recognition; means for capturing a user's voice and transmitting the voice data to a server; means for analyzing voice data and performing voice recognition in a server; A means for generating operation instructions using a generation AI based on the results of image recognition and voice recognition in the server; The system includes a means for converting the generated operation instructions into voice data and providing the voice data to the user.

2. 10. The system of claim 1, wherein the captured image is compressed and transmitted to a server.

3. The system according to claim 1, wherein the generating AI generates operation instructions based on text data.

Images