System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving voice input from a user, a means for converting received voice data into text data, a means for analyzing the converted text data and specifying the user's intention, a means for retrieving an operation procedure corresponding to the specified user's intention, and a means for guiding the retrieved operation procedure by voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many smartphone users, especially the elderly and those unfamiliar with technology, have difficulty operating their smartphones. Furthermore, these users often feel psychologically burdened by having to repeatedly ask others for instructions. This situation can lead users to avoid using smartphones, potentially resulting in isolation in today's digital society. Furthermore, users' safety is not ensured because appropriate measures are not taken against the risk of fraud and unauthorized access. To solve these challenges, a system is needed that allows users to easily operate smartphones and receive assistance whenever necessary. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system that includes a means for accepting voice input from a user, converting the voice data into text, analyzing it to identify the user's intent, searching for the corresponding operating procedure, and providing voice guidance. The system also includes a means for acquiring a capture of the current screen and analyzing the acquired capture image to identify the currently displayed app or menu. The system also includes a means for periodically collecting operation logs from the device, analyzing the collected operation logs to determine whether the user is unsure of the operation, and offering assistance if the user is unsure of the operation. The system also includes a means for analyzing the currently displayed website or email and displaying a warning message if fraud is suspected. Furthermore, the system includes a means for storing the collected voice data and operation logs in local storage and automatically deleting them after a certain period of time, thereby ensuring privacy protection. This system allows users to use their smartphones with peace of mind and provides immediate support if they encounter operational difficulties.

[0006] "Voice input" is a means by which a user provides data to a system by speaking.

[0007] "Voice data" refers to information about a user's speech acquired through a microphone.

[0008] "Text data" is information obtained by analyzing voice data and converting it into text format.

[0009] "Analysis" is the process by which a system reads data and understands its meaning and content.

[0010] "Operating procedures" are specific steps that users must take to properly operate their smartphones.

[0011] "Screen capture" is a method of capturing an image of the currently displayed smartphone screen.

[0012] An "operation log" is data that records the operation history performed by a user on a smartphone.

[0013] A "warning message" is a notification displayed to inform the user of a risk or problem.

[0014] "Local storage" refers to a storage device within a terminal, and is used to store data within the terminal.

[0015] "Automatic deletion" is a process in which the system automatically deletes data after a certain period of time.

[0016] "Suspected fraud" means that a website or email is suspected to be intended to deceive users. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly elderly people. The following describes an embodiment of the present invention.

[0039] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intent, searches for the corresponding operating procedure, and provides voice guidance. It also has the ability to capture the current screen and analyze the captured image to identify the currently displayed app or menu. Below, we will explain each function and provide specific examples of system operation.

[0040] Voice input and analysis

[0041] Subject: Device

[0042] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[0043] Search and guide instructions

[0044] Subject: Device

[0045] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[0046] Screen capture and image analysis

[0047] Subject: Device

[0048] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open and asks how to perform a specific operation, the device will identify that it is the "new email app screen" from the captured image and provide instructions.

[0049] Operation log monitoring and support proposals

[0050] Subject: Server

[0051] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[0052] Security Warning

[0053] Subject: Server

[0054] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[0055] Data Privacy Protection

[0056] Subject: Device

[0057] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[0058] As a result, the system of the present invention can help users operate smartphones easily while ensuring security and privacy, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[0059] The processing flow will be explained below.

[0060] Voice input and analysis

[0061] Subject: Device

[0062] Step 1:

[0063] The device activates the microphone and collects the user's speech in real time as audio data.

[0064] Step 2:

[0065] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[0066] Step 3:

[0067] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[0068] Step 4:

[0069] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[0070] Step 5:

[0071] The device will guide the user through the operation procedures via voice.

[0072] Screen capture and image analysis

[0073] Subject: Device

[0074] Step 1:

[0075] Capture the screen currently displayed on the device and save it as a still image.

[0076] Step 2:

[0077] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[0078] Step 3:

[0079] Predicts what the user will need to do next based on identified apps and menus.

[0080] Step 4:

[0081] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[0082] Operation log monitoring and support proposals

[0083] Subject: Server

[0084] Step 1:

[0085] The server periodically collects operation logs from the terminal.

[0086] Step 2:

[0087] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[0088] Step 3:

[0089] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[0090] Step 4:

[0091] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[0092] Step 5:

[0093] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[0094] Security Warning

[0095] Subject: Server

[0096] Step 1:

[0097] The server monitors and analyzes the content of websites and emails being displayed in real time.

[0098] Step 2:

[0099] Based on the analysis results, we identify content that may be fraudulent.

[0100] Step 3:

[0101] Generate a warning message if any suspected fraud is identified.

[0102] Step 4:

[0103] Sends server-generated warning messages to the terminal for display to the user.

[0104] Step 5:

[0105] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[0106] Data Privacy Protection

[0107] Subject: Device

[0108] Step 1:

[0109] The voice data and operation logs collected by the device are stored in local storage.

[0110] Step 2:

[0111] Encrypt stored data and set access restrictions to prevent unauthorized access.

[0112] Step 3:

[0113] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[0114] Step 4:

[0115] Make sure the deleted data is completely wiped from your device.

[0116] This allows the system to help users operate their smartphones easily while ensuring security and privacy.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] Today's smartphones integrate a variety of functions in a complex manner, making them difficult to operate, especially for elderly users and those unfamiliar with technology. Furthermore, assistance systems using voice recognition technology are required to accurately understand the user's intentions and guide them to the appropriate operating procedures. However, many existing systems lack the precision to analyze the user's voice commands and are unable to provide the appropriate operating procedures. Furthermore, some systems lack the functionality to provide prompt assistance when the user is unsure of how to operate the device, and some lack data privacy protection.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, and means for analyzing the converted text data to identify the user's intention, thereby enabling the server to accurately analyze the content of a voice instruction given by the user and provide appropriate operating procedures.

[0122] Furthermore, the server includes means for searching for an operation procedure corresponding to the user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, means for analyzing the captured image and identifying the application or menu currently being displayed, means for analyzing operation logs collected from the terminal and determining whether the user is unsure of the operation, means for offering assistance when the user is unsure of the operation, means for encrypting the collected voice data and operation logs, and means for automatically deleting the encrypted data after a certain period of time. This makes it possible to offer assistance immediately when the user is unsure of the operation, while at the same time ensuring the privacy of the collected data.

[0123] "Means for accepting voice input" is a general term for devices and software functions for capturing voice uttered by a user.

[0124] "Means for converting into text data" is a general term for technology and software for converting voice data into character string data.

[0125] "Means for identifying user intent" is a general term for technologies and algorithms that analyze acquired text data and determine what the user is trying to achieve.

[0126] "Means for searching for operating procedures" is a general term for technology and software for searching a recorded database or storage device for an operating method that corresponds to a specified user's intention.

[0127] "Voice guidance means" is a general term for devices and software that notify users of searched operating procedures by voice.

[0128] "Means for capturing the screen" is a general term for technologies and devices for acquiring an image of the currently displayed screen.

[0129] "Means for analyzing a captured image" is a general term for techniques and algorithms for analyzing an acquired captured image and identifying the contents of the currently displayed application or menu.

[0130] "Means for analyzing operation logs" is a general term for technologies and algorithms used to analyze operation history data collected from a terminal and determine the user's operation status.

[0131] "Means for suggesting assistance" is a general term for technology and software that presents appropriate operating procedures and supplementary information when a user is unsure how to operate something.

[0132] "Means for encrypting voice data and operation logs" is a general term for technologies and algorithms used to encrypt and protect collected voice data and operation logs.

[0133] "Means for automatic deletion after a certain period of time" is a general term for technology and software that automatically deletes encrypted data after a set period of time has passed.

[0134] MODE FOR CARRYING OUT THE INVENTION

[0135] This invention relates to an AI-equipped system that assists users, particularly elderly people and those unfamiliar with technology, in easily operating smartphones. The system accepts voice input from the user, converts it into text data, and analyzes it to identify the user's intent. It then searches for operating procedures corresponding to the identified intent and provides voice guidance. It also has the ability to capture the current screen and analyze the image to identify the application or menu currently being displayed. Furthermore, it periodically collects operation logs from the device, determines whether the user is unsure of an operation, and offers assistance. Finally, the collected data is encrypted and stored, and automatically deleted after a certain period of time to protect privacy.

[0136] Hardware and software used

[0137] Speech recognition engine: Google® Speech-to-Text (to convert voice data into text data)

[0138] Generative AI model: GPT-3 (registered trademark) (to analyze text data and identify user intent)

[0139] Database: Search for operating instructions

[0140] Audio output device: Speaker (to provide audio instructions)

[0141] Image analysis engine: Google Vision API (to analyze captured images)

[0142] Encryption technology: AES-256 (to encrypt collected data)

[0143] Specific examples

[0144] 1. Voice to text conversion:

[0145] User: Say "I want to connect to the Internet."

[0146] Device: Audio is captured through the microphone and converted into text data such as "I want to connect to the Internet" using Google Speech-to-Text.

[0147] 2. Intention analysis of text data and search for operation procedures:

[0148] On the device: The generative AI model GPT-3 is used to analyze the "Wi-Fi settings" and the user's intent, and then the corresponding operation steps are searched for in the database.

[0149] Device: A voice prompt will say, "Open the Settings app and go to the Wi-Fi menu."

[0150] 3. Screen capture and analysis:

[0151] User: You have the Mail app open and say, "I want to compose a new email."

[0152] Device: Capture the currently displayed screen and use the Google Vision API to identify it as the "new email app composition screen." The device then prompts the user to "enter the subject of the email."

[0153] 4. Operation log collection and support suggestions:

[0154] Server: Periodically collects operation logs from the device, and if the user does not operate the device for more than 30 seconds, it uses AI to determine that assistance is required.

[0155] Server: Sends a help offer message to the device saying "Can we help you?"

[0156] 5. Security Checks and Warnings:

[0157] User: When you try to click on a suspicious URL.

[0158] Server: Immediately analyzes the URL and sends a warning message to the device saying, "This site may not be secure. Do you want to continue?"

[0159] 6. Data Encryption and Storage:

[0160] Terminal: Collected audio data and operation logs are stored in local storage using AES-256 encryption technology.

[0161] Device: Stored data will be automatically deleted after one month.

[0162] Prompt Sentence Examples

[0163] When a user says they want to perform a specific action, for example, "I want to set up Wi-Fi," how can I guide them to "Open the Settings app and go to the Wi-Fi menu"?

[0164] The system allows even non-technical users to easily perform complex smartphone operations, provides assistance when needed, and protects data privacy.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] System program processing flow

[0167] Step 1:

[0168] The user speaks and performs voice input. For example, the user might say, "I want to connect to the Internet."

[0169] Input: User's voice

[0170] Output: Audio data

[0171] Step 2:

[0172] The device picks up audio through a microphone and generates audio data.

[0173] Input: Audio data

[0174] Output: Raw audio data

[0175] Step 3:

[0176] The device converts the voice data into text using Google Speech-to-Text. For example, the text data might say, "I want to connect to the Internet."

[0177] Input: Raw audio data

[0178] Output: Text data

[0179] Step 4:

[0180] The device uses a generative AI model (GPT-3) to analyze text data and identify the user's intent. For example, it can analyze the intent "Wi-Fi settings" from the text "I want to connect to the Internet."

[0181] Input: Text data

[0182] Output: User intent (e.g., "Wi-Fi settings")

[0183] Step 5:

[0184] The device searches a database for the operation procedure that corresponds to the user's intention. For example, it searches for the procedure for "Wi-Fi settings."

[0185] Input: User intent

[0186] Output: Instructions (e.g. "Open the Settings app and go to the Wi-Fi menu")

[0187] Step 6:

[0188] The device will be searched and the operation procedure will be guided to the user using the voice output device (speaker). For example, it will say, "Open the Settings app and go to the Wi-Fi menu."

[0189] Input: Operation procedure

[0190] Output: Voice guidance

[0191] Step 7:

[0192] Capture the screen that the device is currently displaying. For example, if the user has the email app open, capture that screen.

[0193] Input: Current screen

[0194] Output: Captured image

[0195] Step 8:

[0196] The device analyzes the captured image using the Google Vision API. For example, it identifies the captured image as the "new email app composition screen."

[0197] Input: Captcha image

[0198] Output: Identifying the currently displayed application or menu

[0199] Step 9:

[0200] The server periodically collects operation logs from the device. For example, if the user does not operate the device for 30 seconds or more, a log is collected.

[0201] Input: Operation Log

[0202] Output: Operation log data

[0203] Step 10:

[0204] The server uses AI to analyze the collected operation logs and determine whether the user is unsure of the operation. For example, if the user does not operate the device for more than 30 seconds, it will determine that the user is unsure.

[0205] Input: Operation log data

[0206] Output: Determine whether the user is confused about the operation

[0207] Step 11:

[0208] If the server is unsure of an operation, it sends a message offering assistance to the terminal, such as "Can I help you?"

[0209] Input: Determine if the user is confused

[0210] Output: Support offer message

[0211] Step 12:

[0212] The server analyzes the website or email you are viewing in real time and sends a warning message if it detects any fraudulent activity, such as "This site may not be safe. Do you want to continue?"

[0213] Input: The website or email you are viewing

[0214] Output: Security warning message

[0215] Step 13:

[0216] The voice data and operation logs collected by the device are encrypted using AES-256 encryption technology. For example, the voice data is encrypted and stored in local storage.

[0217] Input: Collected voice data and operation logs

[0218] Output: Encrypted data

[0219] Step 14:

[0220] The device will automatically delete the encrypted data after a certain period of time, for example, after one month.

[0221] Input: Encrypted data

[0222] Output: Automatic data deletion

[0223] (Application example 1)

[0224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] Operating conventional smartphones is often difficult, especially for the elderly. Similarly, using food delivery services is difficult, with many users struggling to understand the ordering process. Therefore, there is a demand for systems that are easy for these users to operate.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0227] In this invention, the server includes means for accepting voice input from a user, means for converting the accepted voice data into text data, means for analyzing the converted text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, analyzing the image, and identifying the displayed content, and means for providing voice guidance for ordering food delivery based on the identified displayed content. This enables even users who have difficulty operating smartphones to easily place food delivery orders.

[0228] "Means for accepting voice input from the user" is a general term for devices and software that recognize the voice spoken by the user and receive the voice data.

[0229] "Means for converting voice data into text data" refers to the technology or function for analyzing received voice information and converting it into text information.

[0230] "Means for identifying user intent" refers to technologies and mechanisms that understand and determine the operations and service content desired by the user from the converted text data.

[0231] A "means for searching for operating procedures" is a technology or system that searches for appropriate operating methods or procedures based on the identified user intent.

[0232] "Voice guidance means" refers to technology or devices that verbally communicate searched operating procedures to the user.

[0233] "Means for capturing the currently displayed screen" refers to the technology or function for acquiring the currently displayed screen content of a device as image data.

[0234] "Means for analyzing images" refers to techniques or algorithms for analyzing captured image data and recognizing the information displayed therein.

[0235] "Means for providing voice guidance for ordering food delivery" refers to technologies and systems that provide voice guidance to users through the food delivery ordering process based on specified display content and the user's intent.

[0236] This invention relates to a voice-operated food delivery system designed for users who have difficulty operating smartphones, particularly the elderly. The system accepts voice input, converts the voice data into text, analyzes it, identifies the user's intention, and provides voice guidance on the appropriate operation procedures.

[0237] System configuration

[0238] The system consists of the following main modules:

[0239] Audio Input Module

[0240] Voice Recognition Module

[0241] Intention Analysis Module

[0242] Operation procedure search module

[0243] Voice Guidance Module

[0244] Screen Capture Module

[0245] Image Analysis Module

[0246] Food delivery information module

[0247] Hardware and software used

[0248] Hardware

[0249] Microphone: A device for capturing the user's voice.

[0250] Camera: A device for capturing the currently displayed screen (using the screenshot function on your smartphone).

[0251] software

[0252] SpeechRecognition Library: A library for converting voice data into text data.

[0253] Pyttsx3 library: A synthetic speech engine for generating voice prompts.

[0254] PIL (Python Imaging Library): A library for processing screen capture images.

[0255] Pytesseract library: An optical character recognition (OCR) engine for extracting text from images.

[0256] System operation explanation

[0257] Voice to text conversion

[0258] A user voice-overs a food delivery order through a microphone, for example, saying, "I'd like to order a pizza." The microphone captures the voice input, and the SpeechRecognition library converts the speech into text data.

[0259] Intention analysis and operation procedure search

[0260] The converted text data is analyzed by an intention analysis module, and is identified as, for example, "ordering a pizza." Next, an operating procedure search module searches for an appropriate operating procedure based on the intention.

[0261] Voice guidance and screen analysis

[0262] The searched operation procedures are then audibly guided to the user by the voice guidance module. For example, instructions such as "Add pizza to cart. Press the continue button" are given by synthesized voice. To determine which screen the user is currently on, the screen capture module captures the screen, and the image analysis module analyzes the captured image.

[0263] Food delivery information

[0264] The voice prompts are repeated until the user completes the food delivery order. For example, the voice prompts the user based on the current screen, such as "Open the pizza menu" on the menu screen or "Confirm your order" on the cart screen.

[0265] Specific examples

[0266] For example, if a user says, "I would like to order sushi," the speech is converted into text and the intent is analyzed as "Order sushi." After image analysis confirms that the screen is a menu screen, a voice prompt will be played saying, "Add sushi to cart. Please press the continue button."

[0267] Example prompts for generative AI models

[0268] To develop a voice-operated food delivery app for the elderly, write a program that converts the user's voice input into text, analyzes the user's actions from screen captures, and provides appropriate guidance.

[0269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0270] Step 1:

[0271] The user inputs voice through the microphone. In this step, the user's voice is picked up through the microphone and input to the terminal as voice data. For example, the user may say, "I'd like to order a pizza."

[0272] Step 2:

[0273] The speech recognition module converts the speech data into text data. The device then analyzes the acquired speech data using the SpeechRecognition library and converts it into text data. For example, a speech saying "I would like to order a pizza" is converted into text data saying "I would like to order a pizza."

[0274] Step 3:

[0275] The intent analysis module analyzes the text data and identifies the user's intent. The device analyzes the converted text data and understands the user's intent. This step uses NLP (natural language processing) technology to analyze and identify the user's intent as "order pizza."

[0276] Step 4:

[0277] The operation procedure search module searches for operation procedures based on the specified intent. The device searches for operation procedures from a database based on the specified intent. For example, in the case of "ordering pizza," the module searches for procedures to open the pizza menu and add it to the cart.

[0278] Step 5:

[0279] The voice guidance module provides voice guidance for the searched operation procedures. The terminal uses a speech synthesis engine (Pyttsx3 library) to guide the user through the searched procedures. For example, the user may hear voice guidance such as "Add pizza to cart. Press the continue button."

[0280] Step 6:

[0281] The screen capture module captures the current screen. The device captures the currently displayed screen of a food delivery app and obtains its image data. For example, if the user is on the menu screen of a food delivery app, that screen will be captured.

[0282] Step 7:

[0283] The image analysis module analyzes the captured image and identifies the displayed content. The device processes the captured image using the PIL library and uses the Pytesseract library to extract and analyze the text in the image. This identifies the screen the user is currently on. For example, it identifies the current screen as the "menu screen."

[0284] Step 8:

[0285] The food delivery guidance module then provides further voice guidance based on the identified display content. The device combines the analyzed display content with the user's intention and provides voice guidance on the next action to take. For example, if the user is on the menu screen, they may be told, "Add pizza to cart. Press the continue button," or if they move to the cart screen, they may be told, "Confirm your order."

[0286] Step 9:

[0287] If the user does not complete an operation, the operation log is periodically collected and analyzed by the server. The server receives the operation log from the device and analyzes it using an algorithm to determine whether the user is confused about the operation. For example, if a specific operation is not performed for a certain period of time, it is determined that the user is confused.

[0288] Step 10:

[0289] If the server determines that the user is unsure of how to operate the device, it will provide a voice message suggesting additional assistance. The server sends a message such as "Are you having trouble?" to the device, which then relays this message to the user via voice guidance.

[0290] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0291] The present invention adds emotion recognition functionality to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly the elderly, making it even easier to use. The following describes an embodiment of the present invention.

[0292] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intention, searches for the corresponding operating procedure, and provides voice guidance.It also has the ability to capture the current screen and analyze the captured image to identify the app or menu currently being displayed.In addition, an emotion recognition engine is incorporated, further improving the user experience by adjusting the assistance method according to the user's emotions.

[0293] Voice input and analysis

[0294] Subject: Device

[0295] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[0296] Search and guide instructions

[0297] Subject: Device

[0298] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[0299] Screen capture and image analysis

[0300] Subject: Device

[0301] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open, the device will identify from the captured image that it is the "new email app screen" and provide instructions on how to use it.

[0302] Operation log monitoring and support proposals

[0303] Subject: Server

[0304] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[0305] Security Warning

[0306] Subject: Server

[0307] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[0308] Data Privacy Protection

[0309] Subject: Device

[0310] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[0311] Emotion recognition engine

[0312] Subject: Device

[0313] The device is equipped with an emotion recognition engine that can recognize emotions from the user's voice and operation logs. For example, if the device detects that the user is frustrated, it will explain the operation procedure more slowly and send an encouraging message to calm the user.

[0314] Adjusting operating procedures according to emotions

[0315] Subject: Device

[0316] The emotion recognition engine analyzes the user's emotions and adjusts the operating procedures and guidance accordingly. For example, if the user is feeling anxious, the device will provide detailed instructions and suggest that additional support is needed.

[0317] Generating emotion-responsive messages

[0318] Subject: Device

[0319] The emotion recognition engine generates encouraging and comforting messages based on the user's emotions, and the device then provides these messages to the user through voice. For example, if the user shows signs of fatigue, the device will provide a message such as, "Take a short break. It's okay, let's do the next step together."

[0320] As a result, the system of the present invention not only helps users operate smartphones easily while ensuring security and privacy, but also provides an advanced user experience through emotion recognition, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[0321] The processing flow will be explained below.

[0322] Voice input and analysis

[0323] Subject: Device

[0324] Step 1:

[0325] The device activates the microphone and collects the user's speech in real time as audio data.

[0326] Step 2:

[0327] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[0328] Step 3:

[0329] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[0330] Step 4:

[0331] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[0332] Step 5:

[0333] The device will guide the user through the operation procedures via voice.

[0334] Screen capture and image analysis

[0335] Subject: Device

[0336] Step 1:

[0337] Capture the screen currently displayed on the device and save it as a still image.

[0338] Step 2:

[0339] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[0340] Step 3:

[0341] Predicts what the user will need to do next based on identified apps and menus.

[0342] Step 4:

[0343] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[0344] Operation log monitoring and support proposals

[0345] Subject: Server

[0346] Step 1:

[0347] The server periodically collects operation logs from the terminal.

[0348] Step 2:

[0349] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[0350] Step 3:

[0351] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[0352] Step 4:

[0353] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[0354] Step 5:

[0355] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[0356] Security Warning

[0357] Subject: Server

[0358] Step 1:

[0359] The server monitors and analyzes the content of websites and emails being displayed in real time.

[0360] Step 2:

[0361] Based on the analysis results, we identify content that may be fraudulent.

[0362] Step 3:

[0363] Generate a warning message if any suspected fraud is identified.

[0364] Step 4:

[0365] Sends server-generated warning messages to the terminal for display to the user.

[0366] Step 5:

[0367] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[0368] Data Privacy Protection

[0369] Subject: Device

[0370] Step 1:

[0371] The voice data and operation logs collected by the device are stored in local storage.

[0372] Step 2:

[0373] Encrypt stored data and set access restrictions to prevent unauthorized access.

[0374] Step 3:

[0375] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[0376] Step 4:

[0377] Make sure the deleted data is completely wiped from your device.

[0378] Emotion recognition engine

[0379] Subject: Device

[0380] Step 1:

[0381] The device collects the user's voice and operation logs and sends them to an emotion recognition engine.

[0382] Step 2:

[0383] The emotion recognition engine analyzes the tone of voice, operation speed, etc. to identify the user's emotions.

[0384] Step 3:

[0385] The analyzed user emotion data is integrated into the current operation support process.

[0386] Adjusting operating procedures according to emotions

[0387] Subject: Device

[0388] Step 1:

[0389] The emotion recognition engine analyzes the user's emotions.

[0390] Step 2:

[0391] The device adjusts the operating procedures and guidance methods according to the results of the emotion analysis.

[0392] Step 3:

[0393] If the user is feeling unsure, the device will provide detailed instructions on how to proceed.

[0394] Generating emotion-responsive messages

[0395] Subject: Device

[0396] Step 1:

[0397] The emotion recognition engine analyzes the user's emotional data.

[0398] Step 2:

[0399] Generate encouraging and comforting messages according to the user's emotional state.

[0400] Step 3:

[0401] If the user appears to be tired of operating the device, the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[0402] Specific examples

[0403] Examples of voice input and analysis:

[0404] When a user says, "I want to connect to the Internet," the device converts the speech into text and interprets it as "Wi-Fi settings." The device then provides voice instructions on how to access the Wi-Fi settings screen.

[0405] Examples of operational log monitoring and support suggestions:

[0406] If the user does not interact with the screen for 30 seconds or more, the server generates a message saying "Can we help you?" and sends it to the device. If the user accepts this message, the server guides them through the appropriate operation procedure.

[0407] Examples of emotionally relevant messages:

[0408] If the user is tired of operating the device, the emotion recognition engine will analyze this and the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[0409] As a result, this system provides operational assistance while taking the user's emotions into consideration, creating an environment in which users can use their smartphones with peace of mind.

[0410] Example 2

[0411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] Elderly people and users unfamiliar with technology often find it difficult to operate smartphones, posing security risks such as getting lost or accidentally accessing phishing sites. Furthermore, existing systems lack support that takes into account the user's emotional state, which can easily cause stress. Therefore, there is a need for a system that allows users to operate smartphones more easily and safely.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0414] In this invention, the server includes: means for accepting voice input from a user; means for converting the accepted voice data into text data; means for analyzing the converted text data and identifying the user's intent; means for searching for an operating procedure corresponding to the identified user's intent; means for providing voice guidance for the searched operating procedure; means for acquiring a screen capture and analyzing the acquired screen capture image to identify the software application or menu currently displayed; means for periodically collecting a user's operation log and analyzing the collected operation log to determine whether the user is unsure of an operation; means for analyzing the user's emotions and adjusting the guidance and operating procedure based on the results; means for suggesting assistance when the user is unsure of an operation or emotionally unstable; means for incorporating an emotion recognition engine and adjusting the assistance method according to the user's emotions; and means for sending a warning of possible fraud to the user's screen. This not only allows the user to easily receive voice guidance on the operation method, but also improves their understanding of the screen and provides appropriate assistance when they are unsure of an operation or feel anxious. Furthermore, security risks are reduced, allowing users to use their smartphones with peace of mind.

[0415] "Means for accepting voice input" refers to devices or software that acquire voice data spoken by a user as a digital signal and input it into the system.

[0416] "Means for converting voice data into text data" refers to devices or software that convert acquired voice data into text information using natural language processing technology.

[0417] "Means for analyzing text data and identifying user intent" refers to algorithms and software that use converted text data to understand user requests and objectives.

[0418] "Means for searching for operating procedures" refers to devices or software that refer to a database for operating methods or procedures that correspond to the specified user's intentions and obtain appropriate information.

[0419] "Means for providing voice guidance on operating procedures" refers to speech synthesis technology or software that explains the searched operating procedures to the user by voice.

[0420] "Means for obtaining a screen capture" refers to a device or software for obtaining the contents of the screen currently displayed on the display as image data.

[0421] "Means for analyzing screen capture images" refers to algorithms or software for analyzing the captured image and identifying the software applications and menus being displayed.

[0422] "Means for collecting operation logs" refers to devices or software that record user operation history and periodically collect that data.

[0423] "Means of analyzing operation logs to determine whether the user is having difficulty operating the system" refers to algorithms or software that evaluate the user's operation status based on collected operation logs and determine whether the user is having difficulty operating the system.

[0424] "Means for analyzing user emotions" refers to algorithms or software that evaluate the user's current emotional state based on their voice and operation history.

[0425] "Means for adjusting guidance and instructions" refers to algorithms or software that change the way instructions and instructions are presented depending on the user's emotional state.

[0426] "Means for suggesting assistance" refers to devices or software that suggest appropriate assistance measures or solutions when a user is unsure about how to operate the device or feels anxious.

[0427] "Means of incorporating an emotion recognition engine" refers to algorithms or software that analyzes user emotions in real time and adjusts the system's behavior based on that information.

[0428] "Potential fraud warning means" refers to a device or software that sends a message to a user warning them that a website they visit or an email they receive may be a phishing scam.

[0429] The present invention provides a system for assisting elderly people and users who are unfamiliar with technology to easily operate a smartphone. Hereinafter, an embodiment of the present invention will be described in detail.

[0430] Voice input and analysis

[0431] Subject: Device

[0432] The device accepts voice input when the user speaks. Voice input is done through the built-in microphone, and the voice data is temporarily stored in the device's memory in WAV format or other formats. This voice data is then converted into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data that reads, "I want to connect to the Internet."

[0433] Search and guide instructions

[0434] Subject: Device

[0435] Based on the text data, the device uses a generative AI model (e.g., GPT-4 (registered trademark)) to analyze the user's intent. This analysis determines that "I want to connect to the Internet" means "Set up Wi-Fi." The device then searches a database to obtain the appropriate operating instructions. Steps such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network" are searched for. These steps are then audibly instructed to the user using a voice output engine (e.g., Google Text-to-Speech). Instructions such as "Open the Settings app, select Wi-Fi, and connect to an available network" are heard.

[0436] Screen capture and image analysis

[0437] Subject: Device

[0438] To help users understand the currently displayed screen, the device takes a screen capture. The captured image is temporarily saved in the device's memory in PNG format, and then an AI image analysis engine such as Google Cloud Vision API is used to identify the software application or menu currently displayed. For example, the captured image can be identified as the "new screen of the Mail app."

[0439] Operation log collection and support proposals

[0440] Subject: Server

[0441] The server periodically collects user operation logs from the device. The operation logs are saved in CSV format and sent to the server. The received operation logs are analyzed using a machine learning model to determine whether the user is unsure of the operation. For example, if a certain operation has not continued for more than 30 seconds, it is determined that the user is unsure of the operation and a support suggestion message asking "Need help?" is sent to the device.

[0442] Security Warning

[0443] Subject: Server

[0444] The server analyzes the website being viewed or the email received and generates a warning message if there is a possibility of a phishing scam. For example, if the server determines that the URL the user clicked on may be fraudulent, it will send a warning message to the device saying, "This site may not be safe. Do you want to continue?"

[0445] Sentiment analysis and response

[0446] Subject: Device

[0447] The device is equipped with an emotion recognition engine (e.g., EmoNet) that analyzes emotions from the user's voice and operation logs. For example, if the device determines from the voice analysis that the user is frustrated, it will explain the operation procedure slowly and generate an encouraging voice message such as "You're almost there. Keep up the great work." If the user is feeling anxious, it will explain the operation procedure in detail and suggest additional support.

[0448] Data Privacy Protection

[0449] Subject: Device

[0450] The device encrypts collected voice data and operation logs using AES encryption and stores them in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and is set to be automatically deleted after that period has elapsed.

[0451] Examples of prompt statements

[0452] For example, when using GPT-4, the prompt might look like this:

[0453] "A user says, 'I want to connect to the Internet.' How can I convert this speech to text and provide the appropriate instructions?"

[0454] By inputting these prompts into a generative AI model, a solution can be obtained that specifically analyzes the voice data and provides guidance on operation procedures. By implementing this system, users can operate their smartphones quickly and safely, and even users who are unfamiliar with technology can use it without stress.

[0455] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0456] Step 1:

[0457] Accepting voice input

[0458] Subject: Device

[0459] Input: User speech

[0460] How it works: The device uses the built-in microphone to capture the user's speech as audio data, which is temporarily stored in the device's memory in WAV format.

[0461] Output: WAV format audio data

[0462] Step 2:

[0463] Converting audio data to text

[0464] Subject: Device

[0465] Input: WAV format audio data

[0466] How it works: The voice data is converted to text data by calling the Google Speech-to-Text API. For example, the voice saying "I want to connect to the Internet" is converted to text data like "I want to connect to the Internet."

[0467] Output: Text data "I want to connect to the Internet"

[0468] Step 3:

[0469] Intent Analysis

[0470] Subject: Device

[0471] Input: Text data "I want to connect to the Internet"

[0472] How it works: The device uses a generative AI model (e.g., GPT-4) to analyze text data and identify user intent. For example, the intent "Wi-Fi settings" is identified from the text "I want to connect to the Internet."

[0473] Output: "Wi-Fi settings"

[0474] Step 4:

[0475] Find instructions

[0476] Subject: Device

[0477] Input: "Wi-Fi settings"

[0478] How it works: The device searches its built-in database to retrieve instructions for "Wi-Fi settings," such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network."

[0479] Output: List of operation steps

[0480] Step 5:

[0481] Voice guidance for operation procedures

[0482] Subject: Device

[0483] Input: List of operating instructions

[0484] How it works: The device uses the Google Text-to-Speech API to provide voice instructions, such as "Open the Settings app, select Wi-Fi, and connect to an available network."

[0485] Output: Voice guidance

[0486] Step 6:

[0487] Taking a screen capture

[0488] Subject: Device

[0489] Input:Currently displayed screen

[0490] What it does: The device captures the current screen image, which is temporarily saved in the device's memory as a PNG image.

[0491] Output: Captured image in PNG format

[0492] Step 7:

[0493] Analysis of captured images

[0494] Subject: Device

[0495] Input: Captcha image in PNG format

[0496] How it works: The device uses the Google Cloud Vision API to analyze the captured image and identify the software application or menu currently displayed. For example, it may identify that the image is the "Mail app's compose screen."

[0497] Output: Identified applications and menus

[0498] Step 8:

[0499] Collecting operation logs

[0500] Subject: Device

[0501] Input: User operation history

[0502] Operation: The device records user operation logs in CSV format and periodically sends them to the server. The operation logs include click and tap locations, operation times, etc.

[0503] Output: Operation log in CSV format

[0504] Step 9:

[0505] Log analysis

[0506] Subject: Server

[0507] Input: CSV format operation log

[0508] How it works: The server analyzes the received operation log using a machine learning model to determine whether the user is having trouble with an operation. For example, if a certain operation has not continued for more than 30 seconds, it determines that the user is having trouble.

[0509] Output: Judgment result of whether the user is confused about the operation

[0510] Step 10:

[0511] Submit a support offer

[0512] Subject: Server

[0513] Input: Determine whether the user is confused about the operation

[0514] Operation: The server generates a help offer message saying "Can I help you?" and sends it to the terminal.

[0515] Output: Support offer message

[0516] Step 11:

[0517] Generate security warnings

[0518] Subject: Server

[0519] Input: Information about the website or email you are viewing

[0520] How it works: The server analyzes the website or email you are viewing and generates a warning message if there is a possibility of a phishing scam, such as "This site may not be secure. Do you want to continue?"

[0521] Output: Security warning message

[0522] Step 12:

[0523] Sending a warning message

[0524] Subject: Server

[0525] Input: Security warning message

[0526] Action: The server generates a security warning message and sends it to the terminal.

[0527] Output: Warning message displayed on the terminal

[0528] Step 13:

[0529] Emotion Analysis

[0530] Subject: Device

[0531] Input: User voice and operation history

[0532] How it works: The device uses an emotion recognition engine (e.g., EmoNet) to analyze the user's emotions. For example, it determines if the user is frustrated based on voice analysis and operation patterns.

[0533] Output: User's emotional state

[0534] Step 14:

[0535] Emotion-based guidance adjustment

[0536] Subject: Device

[0537] Input: User's emotional state

[0538] What it does: If the user is frustrated, the device will explain instructions and directions more slowly and generate encouraging messages, such as "Take a break and we'll walk you through the next steps together."

[0539] Output: Adjusted operating procedures and guidance messages

[0540] Step 15:

[0541] Data Privacy Protection

[0542] Subject: Device

[0543] Input: Collected voice data and operation logs

[0544] Operation: The device encrypts voice data and operation logs using AES encryption and stores them in local storage. After a certain period of time (e.g., one month), this data is automatically deleted.

[0545] Output: Encrypted data and deletion log

[0546] (Application example 2)

[0547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0548] While conventional voice recognition systems have the ability to identify user intent and provide voice guidance, they are still difficult to operate, especially for elderly people who are unfamiliar with technology and users who are unfamiliar with digital devices. They also lack the ability to provide appropriate assistance based on the user's emotions, which hinders the user experience. Another issue is the lack of assistants for in-store navigation and product information provision.

[0549] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, means for analyzing the text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for analyzing the user's emotions, means for adjusting the operation procedure and guidance based on the analysis results, means for analyzing the acquired captured image to identify the displayed information and providing voice guidance based on the result, and means for suggesting assistance if the user is unfamiliar with the technology. This allows even users unfamiliar with the technology to easily operate the system and provides appropriate assistance based on the user's emotions. It also facilitates navigation and product information provision in physical stores.

[0550] A "means for accepting voice input" has the function of capturing speech uttered by a user and transmitting it to the system for processing.

[0551] The "means for converting voice data into text data" has a function of analyzing received voice and converting the content into text information.

[0552] The "means for identifying the user's intent" is a function that analyzes the text data obtained by voice recognition and determines what the user is looking for.

[0553] The "means for searching for operating procedures" has the function of searching a database or program for the optimal operating procedure that corresponds to the specified user's intention.

[0554] "Voice guidance means" means a function that notifies and guides the user through the searched operating procedures by voice.

[0555] "Means for analyzing user emotions" refers to a function that reads emotions from the user's speech and operation logs and recognizes their state.

[0556] The "means for adjusting operating procedures and guidance" has the function of appropriately changing the content and method of guidance based on the results of user emotion analysis, and providing support according to the user's condition.

[0557] The "means for analyzing a captured image" has a function of analyzing an acquired screen capture and identifying the information and application currently being displayed.

[0558] "Means to offer assistance when the user is unfamiliar with the technology" refers to a function that determines whether the user is unsure of how to operate the device and automatically provides the necessary support.

[0559] This invention utilizes AI technology to provide a "smart store assistant" system to assist elderly and technology-challenged customers in brick-and-mortar stores. The system can be installed and used on devices such as smartphones or smart glasses.

[0560] System Configuration

[0561] 1. Voice recognition and analysis functions

[0562] The device accepts voice input from the user and converts the voice data captured through the microphone into text data using a speech recognition engine (e.g., Google's speech recognition API). The converted text data is then analyzed using a generative AI model to identify the user's intent.

[0563] 2. Operation procedures and guidance functions

[0564] The operating procedures corresponding to the identified user intent are searched for in a database (such as a cloud database), and the search results are then audibly guided to the user using a speech synthesis engine (such as pyttsx3).

[0565] 3. Screen capture and analysis functions

[0566] The device takes a screenshot of the currently displayed screen and uses an AI image analysis engine (such as OpenCV or the machine learning library TENSORFLOW (registered trademark)) to identify the information and app being displayed. The results of this analysis are also announced via voice.

[0567] 4. Emotion recognition and response functions

[0568] The device uses an emotion recognition engine (such as the Hugging Face emotion recognition model) to analyze the user's emotions from their speech and operation logs. Based on the analysis results, the device adjusts operation procedures and guidance appropriately to provide support tailored to the user.

[0569] 5. User assistance suggestion function

[0570] The server periodically collects and analyzes operation logs from the device. If the server determines that the user is unsure of how to operate the device, it sends a message suggesting assistance to the device and displays it to the user.

[0571] 6. Security and Privacy Protection Features

[0572] The server analyzes the displayed information and sends a warning message if there is a security risk. The collected data is encrypted and automatically deleted after a certain period of time.

[0573] Specific examples

[0574] Example prompt sentence:

[0575] "When a user says, 'Where is the milk?', the system converts the voice input into text, performs emotion recognition, and then provides in-store navigation guidance."

[0576] Scenario description:

[0577] A user puts on smart glasses and asks, "Where is the milk?" while in a physical store. This voice input is converted into text by a speech recognition engine, and analysis identifies the user's intent as "I'm looking for milk." An emotion recognition engine evaluates the user's emotions and generates more detailed and courteous guidance if, for example, the user feels anxious. Captured image analysis then analyzes the store map and provides a voice guide saying, "The product is on the left side of aisle A."

[0578] As described above, this system allows even users who are unfamiliar with technology to easily operate it, and enables navigation within physical stores and acquisition of product information.

[0579] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0580] Step 1:

[0581] The device receives voice input from the user: the user speaks questions or commands into the smart glasses or smartphone, which are captured as voice data through the device's microphone.

[0582] Step 2:

[0583] The device sends the acquired voice data to a voice recognition engine and converts it into text data. For example, if a user says, "Where is the milk?", a voice recognition engine (such as Google's voice recognition API) is used to create the text data, "Where is the milk?"

[0584] Step 3:

[0585] The device inputs the converted text data into a generative AI model to analyze the user's intent. Based on the analysis results, it determines what the user is looking for. In this case, the user's intent is determined to be "looking for milk."

[0586] Step 4:

[0587] The device sends the analysis results to an emotion recognition engine to analyze the user's emotions. Using Hugging Face's emotion recognition model, the device evaluates the user's emotional state (e.g., anxiety, irritation, calmness, etc.).

[0588] Step 5:

[0589] The device adjusts the operation procedures and guidance methods based on the results of the user's emotion analysis. For example, if an anxious state is detected, it generates more detailed and slower voice guidance. To generate this voice guidance message, the generated text and the emotion analysis results are used as input data.

[0590] Step 6:

[0591] The device uses a capture image analysis engine to capture the current screen and analyze the information on the screen, including a map of the store. Using AI image analysis technology, it identifies the user's location and the location of the product they are looking for (in this case, milk).

[0592] Step 7:

[0593] The device sends the analysis results to a speech synthesis engine and provides the user with appropriate instructions via voice. For example, it generates a specific message such as "The product is on the left side of aisle A" and outputs it aloud. It uses a speech synthesis library such as pyttsx3.

[0594] Step 8:

[0595] The server periodically collects operation logs from the device and determines whether the user is unsure of how to operate the device. Based on the analysis results, it sends messages to the device suggesting additional guidance or support if necessary. If the user does not perform any operation for a long period of time, a support message such as "Are you having trouble?" will be displayed.

[0596] Step 9:

[0597] The server performs security checks on the information being displayed and the web page being accessed, and if it detects any suspected fraud, it sends a warning message to the device, such as "This site may not be secure. Do you want to continue?" This process is done in real time to ensure the safety of the user.

[0598] Step 10:

[0599] The device encrypts and stores collected voice data and operation logs in local storage, and automatically deletes them after a certain period of time. For example, voice data is protected using AES encryption technology, and a system is implemented that automatically deletes it after 30 days, protecting user privacy.

[0600] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0601] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0602] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0603] [Second embodiment]

[0604] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0605] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0606] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0607] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0608] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0609] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0610] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0611] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0612] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0613] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0614] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0615] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0616] The present invention relates to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly elderly people. The following describes an embodiment of the present invention.

[0617] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intent, searches for the corresponding operating procedure, and provides voice guidance. It also has the ability to capture the current screen and analyze the captured image to identify the currently displayed app or menu. Below, we will explain each function and provide specific examples of system operation.

[0618] Voice input and analysis

[0619] Subject: Device

[0620] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[0621] Search and guide instructions

[0622] Subject: Device

[0623] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[0624] Screen capture and image analysis

[0625] Subject: Device

[0626] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open and asks how to perform a specific operation, the device will identify that it is the "new email app screen" from the captured image and provide instructions.

[0627] Operation log monitoring and support proposals

[0628] Subject: Server

[0629] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[0630] Security Warning

[0631] Subject: Server

[0632] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[0633] Data Privacy Protection

[0634] Subject: Device

[0635] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[0636] As a result, the system of the present invention can help users operate smartphones easily while ensuring security and privacy, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[0637] The processing flow will be explained below.

[0638] Voice input and analysis

[0639] Subject: Device

[0640] Step 1:

[0641] The device activates the microphone and collects the user's speech in real time as audio data.

[0642] Step 2:

[0643] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[0644] Step 3:

[0645] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[0646] Step 4:

[0647] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[0648] Step 5:

[0649] The device will guide the user through the operation procedures via voice.

[0650] Screen capture and image analysis

[0651] Subject: Device

[0652] Step 1:

[0653] Capture the screen currently displayed on the device and save it as a still image.

[0654] Step 2:

[0655] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[0656] Step 3:

[0657] Predicts what the user will need to do next based on identified apps and menus.

[0658] Step 4:

[0659] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[0660] Operation log monitoring and support proposals

[0661] Subject: Server

[0662] Step 1:

[0663] The server periodically collects operation logs from the terminal.

[0664] Step 2:

[0665] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[0666] Step 3:

[0667] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[0668] Step 4:

[0669] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[0670] Step 5:

[0671] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[0672] Security Warning

[0673] Subject: Server

[0674] Step 1:

[0675] The server monitors and analyzes the content of websites and emails being displayed in real time.

[0676] Step 2:

[0677] Based on the analysis results, we identify content that may be fraudulent.

[0678] Step 3:

[0679] Generate a warning message if any suspected fraud is identified.

[0680] Step 4:

[0681] Sends server-generated warning messages to the terminal for display to the user.

[0682] Step 5:

[0683] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[0684] Data Privacy Protection

[0685] Subject: Device

[0686] Step 1:

[0687] The voice data and operation logs collected by the device are stored in local storage.

[0688] Step 2:

[0689] Encrypt stored data and set access restrictions to prevent unauthorized access.

[0690] Step 3:

[0691] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[0692] Step 4:

[0693] Make sure the deleted data is completely wiped from your device.

[0694] This allows the system to help users operate their smartphones easily while ensuring security and privacy.

[0695] Example 1

[0696] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Today's smartphones integrate a variety of functions in a complex manner, making them difficult to operate, especially for elderly users and those unfamiliar with technology. Furthermore, assistance systems using voice recognition technology are required to accurately understand the user's intentions and guide them to the appropriate operating procedures. However, many existing systems lack the precision to analyze the user's voice commands and are unable to provide the appropriate operating procedures. Furthermore, some systems lack the functionality to provide prompt assistance when the user is unsure of how to operate the device, and some lack data privacy protection.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0699] In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, and means for analyzing the converted text data to identify the user's intention, thereby enabling the server to accurately analyze the content of a voice instruction given by the user and provide appropriate operating procedures.

[0700] Furthermore, the server includes means for searching for an operation procedure corresponding to the user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, means for analyzing the captured image and identifying the application or menu currently being displayed, means for analyzing operation logs collected from the terminal and determining whether the user is unsure of the operation, means for offering assistance when the user is unsure of the operation, means for encrypting the collected voice data and operation logs, and means for automatically deleting the encrypted data after a certain period of time. This makes it possible to offer assistance immediately when the user is unsure of the operation, while at the same time ensuring the privacy of the collected data.

[0701] "Means for accepting voice input" is a general term for devices and software functions for capturing voice uttered by a user.

[0702] "Means for converting into text data" is a general term for technology and software for converting voice data into character string data.

[0703] "Means for identifying user intent" is a general term for technologies and algorithms that analyze acquired text data and determine what the user is trying to achieve.

[0704] "Means for searching for operating procedures" is a general term for technology and software for searching a recorded database or storage device for an operating method that corresponds to a specified user's intention.

[0705] "Voice guidance means" is a general term for devices and software that notify users of searched operating procedures by voice.

[0706] "Means for capturing the screen" is a general term for technologies and devices for acquiring an image of the currently displayed screen.

[0707] "Means for analyzing a captured image" is a general term for techniques and algorithms for analyzing an acquired captured image and identifying the contents of the currently displayed application or menu.

[0708] "Means for analyzing operation logs" is a general term for technologies and algorithms used to analyze operation history data collected from a terminal and determine the user's operation status.

[0709] "Means for suggesting assistance" is a general term for technology and software that presents appropriate operating procedures and supplementary information when a user is unsure how to operate something.

[0710] "Means for encrypting voice data and operation logs" is a general term for technologies and algorithms used to encrypt and protect collected voice data and operation logs.

[0711] "Means for automatic deletion after a certain period of time" is a general term for technology and software that automatically deletes encrypted data after a set period of time has passed.

[0712] MODE FOR CARRYING OUT THE INVENTION

[0713] This invention relates to an AI-equipped system that assists users, particularly elderly people and those unfamiliar with technology, in easily operating smartphones. The system accepts voice input from the user, converts it into text data, and analyzes it to identify the user's intent. It then searches for operating procedures corresponding to the identified intent and provides voice guidance. It also has the ability to capture the current screen and analyze the image to identify the application or menu currently being displayed. Furthermore, it periodically collects operation logs from the device, determines whether the user is unsure of an operation, and offers assistance. Finally, the collected data is encrypted and stored, and automatically deleted after a certain period of time to protect privacy.

[0714] Hardware and software used

[0715] Speech recognition engine: Google Speech-to-Text (to convert voice data into text data)

[0716] Generative AI model: GPT-3 (to analyze text data and identify user intent)

[0717] Database: Search for operating instructions

[0718] Audio output device: Speaker (to provide audio instructions)

[0719] Image analysis engine: Google Vision API (to analyze captured images)

[0720] Encryption technology: AES-256 (to encrypt collected data)

[0721] Specific examples

[0722] 1. Voice to text conversion:

[0723] User: Say "I want to connect to the Internet."

[0724] Device: Audio is captured through the microphone and converted into text data such as "I want to connect to the Internet" using Google Speech-to-Text.

[0725] 2. Intention analysis of text data and search for operation procedures:

[0726] On the device: The generative AI model GPT-3 is used to analyze the "Wi-Fi settings" and the user's intent, and then the corresponding operation steps are searched for in the database.

[0727] Device: A voice prompt will say, "Open the Settings app and go to the Wi-Fi menu."

[0728] 3. Screen capture and analysis:

[0729] User: You have the Mail app open and say, "I want to compose a new email."

[0730] Device: Capture the currently displayed screen and use the Google Vision API to identify it as the "new email app composition screen." The device then prompts the user to "enter the subject of the email."

[0731] 4. Operation log collection and support suggestions:

[0732] Server: Periodically collects operation logs from the device, and if the user does not operate the device for more than 30 seconds, it uses AI to determine that assistance is required.

[0733] Server: Sends a help offer message to the device saying "Can we help you?"

[0734] 5. Security Checks and Warnings:

[0735] User: When you try to click on a suspicious URL.

[0736] Server: Immediately analyzes the URL and sends a warning message to the device saying, "This site may not be secure. Do you want to continue?"

[0737] 6. Data Encryption and Storage:

[0738] Terminal: Collected audio data and operation logs are stored in local storage using AES-256 encryption technology.

[0739] Device: Stored data will be automatically deleted after one month.

[0740] Prompt Sentence Examples

[0741] When a user says they want to perform a specific action, for example, "I want to set up Wi-Fi," how can I guide them to "Open the Settings app and go to the Wi-Fi menu"?

[0742] The system allows even non-technical users to easily perform complex smartphone operations, provides assistance when needed, and protects data privacy.

[0743] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0744] System program processing flow

[0745] Step 1:

[0746] The user speaks and performs voice input. For example, the user might say, "I want to connect to the Internet."

[0747] Input: User's voice

[0748] Output: Audio data

[0749] Step 2:

[0750] The device picks up audio through a microphone and generates audio data.

[0751] Input: Audio data

[0752] Output: Raw audio data

[0753] Step 3:

[0754] The device converts the voice data into text using Google Speech-to-Text. For example, the text data might say, "I want to connect to the Internet."

[0755] Input: Raw audio data

[0756] Output: Text data

[0757] Step 4:

[0758] The device uses a generative AI model (GPT-3) to analyze text data and identify the user's intent. For example, it can analyze the intent "Wi-Fi settings" from the text "I want to connect to the Internet."

[0759] Input: Text data

[0760] Output: User intent (e.g., "Wi-Fi settings")

[0761] Step 5:

[0762] The device searches a database for the operation procedure that corresponds to the user's intention. For example, it searches for the procedure for "Wi-Fi settings."

[0763] Input: User intent

[0764] Output: Instructions (e.g. "Open the Settings app and go to the Wi-Fi menu")

[0765] Step 6:

[0766] The device will be searched and the operation procedure will be guided to the user using the voice output device (speaker). For example, it will say, "Open the Settings app and go to the Wi-Fi menu."

[0767] Input: Operation procedure

[0768] Output: Voice guidance

[0769] Step 7:

[0770] Capture the screen that the device is currently displaying. For example, if the user has the email app open, capture that screen.

[0771] Input: Current screen

[0772] Output: Captured image

[0773] Step 8:

[0774] The device analyzes the captured image using the Google Vision API. For example, it identifies the captured image as the "new email app composition screen."

[0775] Input: Captcha image

[0776] Output: Identifying the currently displayed application or menu

[0777] Step 9:

[0778] The server periodically collects operation logs from the device. For example, if the user does not operate the device for 30 seconds or more, a log is collected.

[0779] Input: Operation Log

[0780] Output: Operation log data

[0781] Step 10:

[0782] The server uses AI to analyze the collected operation logs and determine whether the user is unsure of the operation. For example, if the user does not operate the device for more than 30 seconds, it will determine that the user is unsure.

[0783] Input: Operation log data

[0784] Output: Determine whether the user is confused about the operation

[0785] Step 11:

[0786] If the server is unsure of an operation, it sends a message offering assistance to the terminal, such as "Can I help you?"

[0787] Input: Determine if the user is confused

[0788] Output: Support offer message

[0789] Step 12:

[0790] The server analyzes the website or email you are viewing in real time and sends a warning message if it detects any fraudulent activity, such as "This site may not be safe. Do you want to continue?"

[0791] Input: The website or email you are viewing

[0792] Output: Security warning message

[0793] Step 13:

[0794] The voice data and operation logs collected by the device are encrypted using AES-256 encryption technology. For example, the voice data is encrypted and stored in local storage.

[0795] Input: Collected voice data and operation logs

[0796] Output: Encrypted data

[0797] Step 14:

[0798] The device will automatically delete the encrypted data after a certain period of time, for example, after one month.

[0799] Input: Encrypted data

[0800] Output: Automatic data deletion

[0801] (Application example 1)

[0802] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0803] Operating conventional smartphones is often difficult, especially for the elderly. Similarly, using food delivery services is difficult, with many users struggling to understand the ordering process. Therefore, there is a demand for systems that are easy for these users to operate.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0805] In this invention, the server includes means for accepting voice input from a user, means for converting the accepted voice data into text data, means for analyzing the converted text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, analyzing the image, and identifying the displayed content, and means for providing voice guidance for ordering food delivery based on the identified displayed content. This enables even users who have difficulty operating smartphones to easily place food delivery orders.

[0806] "Means for accepting voice input from the user" is a general term for devices and software that recognize the voice spoken by the user and receive the voice data.

[0807] "Means for converting voice data into text data" refers to the technology or function for analyzing received voice information and converting it into text information.

[0808] "Means for identifying user intent" refers to technologies and mechanisms that understand and determine the operations and service content desired by the user from the converted text data.

[0809] A "means for searching for operating procedures" is a technology or system that searches for appropriate operating methods or procedures based on the identified user intent.

[0810] "Voice guidance means" refers to technology or devices that verbally communicate searched operating procedures to the user.

[0811] "Means for capturing the currently displayed screen" refers to the technology or function for acquiring the currently displayed screen content of a device as image data.

[0812] "Means for analyzing images" refers to techniques or algorithms for analyzing captured image data and recognizing the information displayed therein.

[0813] "Means for providing voice guidance for ordering food delivery" refers to technologies and systems that provide voice guidance to users through the food delivery ordering process based on specified display content and the user's intent.

[0814] This invention relates to a voice-operated food delivery system designed for users who have difficulty operating smartphones, particularly the elderly. The system accepts voice input, converts the voice data into text, analyzes it, identifies the user's intention, and provides voice guidance on the appropriate operation procedures.

[0815] System configuration

[0816] The system consists of the following main modules:

[0817] Audio Input Module

[0818] Voice Recognition Module

[0819] Intention Analysis Module

[0820] Operation procedure search module

[0821] Voice Guidance Module

[0822] Screen Capture Module

[0823] Image Analysis Module

[0824] Food delivery information module

[0825] Hardware and software used

[0826] Hardware

[0827] Microphone: A device for capturing the user's voice.

[0828] Camera: A device for capturing the currently displayed screen (using the screenshot function on your smartphone).

[0829] software

[0830] SpeechRecognition Library: A library for converting voice data into text data.

[0831] Pyttsx3 library: A synthetic speech engine for generating voice prompts.

[0832] PIL (Python Imaging Library): A library for processing screen capture images.

[0833] Pytesseract library: An optical character recognition (OCR) engine for extracting text from images.

[0834] System operation explanation

[0835] Voice to text conversion

[0836] A user voice-overs a food delivery order through a microphone, for example, saying, "I'd like to order a pizza." The microphone captures the voice input, and the SpeechRecognition library converts the speech into text data.

[0837] Intention analysis and operation procedure search

[0838] The converted text data is analyzed by an intention analysis module, and is identified as, for example, "ordering a pizza." Next, an operating procedure search module searches for an appropriate operating procedure based on the intention.

[0839] Voice guidance and screen analysis

[0840] The searched operation procedures are then audibly guided to the user by the voice guidance module. For example, instructions such as "Add pizza to cart. Press the continue button" are given by synthesized voice. To determine which screen the user is currently on, the screen capture module captures the screen, and the image analysis module analyzes the captured image.

[0841] Food delivery information

[0842] The voice prompts are repeated until the user completes the food delivery order. For example, the voice prompts the user based on the current screen, such as "Open the pizza menu" on the menu screen or "Confirm your order" on the cart screen.

[0843] Specific examples

[0844] For example, if a user says, "I would like to order sushi," the speech is converted into text and the intent is analyzed as "Order sushi." After image analysis confirms that the screen is a menu screen, a voice prompt will be played saying, "Add sushi to cart. Please press the continue button."

[0845] Example prompts for generative AI models

[0846] To develop a voice-operated food delivery app for the elderly, write a program that converts the user's voice input into text, analyzes the user's actions from screen captures, and provides appropriate guidance.

[0847] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0848] Step 1:

[0849] The user inputs voice through the microphone. In this step, the user's voice is picked up through the microphone and input to the terminal as voice data. For example, the user may say, "I'd like to order a pizza."

[0850] Step 2:

[0851] The speech recognition module converts the speech data into text data. The device then analyzes the acquired speech data using the SpeechRecognition library and converts it into text data. For example, a speech saying "I would like to order a pizza" is converted into text data saying "I would like to order a pizza."

[0852] Step 3:

[0853] The intent analysis module analyzes the text data and identifies the user's intent. The device analyzes the converted text data and understands the user's intent. This step uses NLP (natural language processing) technology to analyze and identify the user's intent as "order pizza."

[0854] Step 4:

[0855] The operation procedure search module searches for operation procedures based on the specified intent. The device searches for operation procedures from a database based on the specified intent. For example, in the case of "ordering pizza," the module searches for procedures to open the pizza menu and add it to the cart.

[0856] Step 5:

[0857] The voice guidance module provides voice guidance for the searched operation procedures. The terminal uses a speech synthesis engine (Pyttsx3 library) to guide the user through the searched procedures. For example, the user may hear voice guidance such as "Add pizza to cart. Press the continue button."

[0858] Step 6:

[0859] The screen capture module captures the current screen. The device captures the currently displayed screen of a food delivery app and obtains its image data. For example, if the user is on the menu screen of a food delivery app, that screen will be captured.

[0860] Step 7:

[0861] The image analysis module analyzes the captured image and identifies the displayed content. The device processes the captured image using the PIL library and uses the Pytesseract library to extract and analyze the text in the image. This identifies the screen the user is currently on. For example, it identifies the current screen as the "menu screen."

[0862] Step 8:

[0863] The food delivery guidance module then provides further voice guidance based on the identified display content. The device combines the analyzed display content with the user's intention and provides voice guidance on the next action to take. For example, if the user is on the menu screen, they may be told, "Add pizza to cart. Press the continue button," or if they move to the cart screen, they may be told, "Confirm your order."

[0864] Step 9:

[0865] If the user does not complete an operation, the operation log is periodically collected and analyzed by the server. The server receives the operation log from the device and analyzes it using an algorithm to determine whether the user is confused about the operation. For example, if a specific operation is not performed for a certain period of time, it is determined that the user is confused.

[0866] Step 10:

[0867] If the server determines that the user is unsure of how to operate the device, it will provide a voice message suggesting additional assistance. The server sends a message such as "Are you having trouble?" to the device, which then relays this message to the user via voice guidance.

[0868] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0869] The present invention adds emotion recognition functionality to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly the elderly, making it even easier to use. The following describes an embodiment of the present invention.

[0870] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intention, searches for the corresponding operating procedure, and provides voice guidance.It also has the ability to capture the current screen and analyze the captured image to identify the app or menu currently being displayed.In addition, an emotion recognition engine is incorporated, further improving the user experience by adjusting the assistance method according to the user's emotions.

[0871] Voice input and analysis

[0872] Subject: Device

[0873] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[0874] Search and guide instructions

[0875] Subject: Device

[0876] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[0877] Screen capture and image analysis

[0878] Subject: Device

[0879] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open, the device will identify from the captured image that it is the "new email app screen" and provide instructions on how to use it.

[0880] Operation log monitoring and support proposals

[0881] Subject: Server

[0882] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[0883] Security Warning

[0884] Subject: Server

[0885] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[0886] Data Privacy Protection

[0887] Subject: Device

[0888] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[0889] Emotion recognition engine

[0890] Subject: Device

[0891] The device is equipped with an emotion recognition engine that can recognize emotions from the user's voice and operation logs. For example, if the device detects that the user is frustrated, it will explain the operation procedure more slowly and send an encouraging message to calm the user.

[0892] Adjusting operating procedures according to emotions

[0893] Subject: Device

[0894] The emotion recognition engine analyzes the user's emotions and adjusts the operating procedures and guidance accordingly. For example, if the user is feeling anxious, the device will provide detailed instructions and suggest that additional support is needed.

[0895] Generating emotion-responsive messages

[0896] Subject: Device

[0897] The emotion recognition engine generates encouraging and comforting messages based on the user's emotions, and the device then provides these messages to the user through voice. For example, if the user shows signs of fatigue, the device will provide a message such as, "Take a short break. It's okay, let's do the next step together."

[0898] As a result, the system of the present invention not only helps users operate smartphones easily while ensuring security and privacy, but also provides an advanced user experience through emotion recognition, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[0899] The processing flow will be explained below.

[0900] Voice input and analysis

[0901] Subject: Device

[0902] Step 1:

[0903] The device activates the microphone and collects the user's speech in real time as audio data.

[0904] Step 2:

[0905] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[0906] Step 3:

[0907] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[0908] Step 4:

[0909] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[0910] Step 5:

[0911] The device will guide the user through the operation procedures via voice.

[0912] Screen capture and image analysis

[0913] Subject: Device

[0914] Step 1:

[0915] Capture the screen currently displayed on the device and save it as a still image.

[0916] Step 2:

[0917] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[0918] Step 3:

[0919] Predicts what the user will need to do next based on identified apps and menus.

[0920] Step 4:

[0921] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[0922] Operation log monitoring and support proposals

[0923] Subject: Server

[0924] Step 1:

[0925] The server periodically collects operation logs from the terminal.

[0926] Step 2:

[0927] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[0928] Step 3:

[0929] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[0930] Step 4:

[0931] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[0932] Step 5:

[0933] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[0934] Security Warning

[0935] Subject: Server

[0936] Step 1:

[0937] The server monitors and analyzes the content of websites and emails being displayed in real time.

[0938] Step 2:

[0939] Based on the analysis results, we identify content that may be fraudulent.

[0940] Step 3:

[0941] Generate a warning message if any suspected fraud is identified.

[0942] Step 4:

[0943] Sends server-generated warning messages to the terminal for display to the user.

[0944] Step 5:

[0945] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[0946] Data Privacy Protection

[0947] Subject: Device

[0948] Step 1:

[0949] The voice data and operation logs collected by the device are stored in local storage.

[0950] Step 2:

[0951] Encrypt stored data and set access restrictions to prevent unauthorized access.

[0952] Step 3:

[0953] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[0954] Step 4:

[0955] Make sure the deleted data is completely wiped from your device.

[0956] Emotion recognition engine

[0957] Subject: Device

[0958] Step 1:

[0959] The device collects the user's voice and operation logs and sends them to an emotion recognition engine.

[0960] Step 2:

[0961] The emotion recognition engine analyzes the tone of voice, operation speed, etc. to identify the user's emotions.

[0962] Step 3:

[0963] The analyzed user emotion data is integrated into the current operation support process.

[0964] Adjusting operating procedures according to emotions

[0965] Subject: Device

[0966] Step 1:

[0967] The emotion recognition engine analyzes the user's emotions.

[0968] Step 2:

[0969] The device adjusts the operating procedures and guidance methods according to the results of the emotion analysis.

[0970] Step 3:

[0971] If the user is feeling unsure, the device will provide detailed instructions on how to proceed.

[0972] Generating emotion-responsive messages

[0973] Subject: Device

[0974] Step 1:

[0975] The emotion recognition engine analyzes the user's emotional data.

[0976] Step 2:

[0977] Generate encouraging and comforting messages according to the user's emotional state.

[0978] Step 3:

[0979] If the user appears to be tired of operating the device, the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[0980] Specific examples

[0981] Examples of voice input and analysis:

[0982] When a user says, "I want to connect to the Internet," the device converts the speech into text and interprets it as "Wi-Fi settings." The device then provides voice instructions on how to access the Wi-Fi settings screen.

[0983] Examples of operational log monitoring and support suggestions:

[0984] If the user does not interact with the screen for 30 seconds or more, the server generates a message saying "Can we help you?" and sends it to the device. If the user accepts this message, the server guides them through the appropriate operation procedure.

[0985] Examples of emotionally relevant messages:

[0986] If the user is tired of operating the device, the emotion recognition engine will analyze this and the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[0987] As a result, this system provides operational assistance while taking the user's emotions into consideration, creating an environment in which users can use their smartphones with peace of mind.

[0988] Example 2

[0989] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0990] Elderly people and users unfamiliar with technology often find it difficult to operate smartphones, posing security risks such as getting lost or accidentally accessing phishing sites. Furthermore, existing systems lack support that takes into account the user's emotional state, which can easily cause stress. Therefore, there is a need for a system that allows users to operate smartphones more easily and safely.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0992] In this invention, the server includes: means for accepting voice input from a user; means for converting the accepted voice data into text data; means for analyzing the converted text data and identifying the user's intent; means for searching for an operating procedure corresponding to the identified user's intent; means for providing voice guidance for the searched operating procedure; means for acquiring a screen capture and analyzing the acquired screen capture image to identify the software application or menu currently displayed; means for periodically collecting a user's operation log and analyzing the collected operation log to determine whether the user is unsure of an operation; means for analyzing the user's emotions and adjusting the guidance and operating procedure based on the results; means for suggesting assistance when the user is unsure of an operation or emotionally unstable; means for incorporating an emotion recognition engine and adjusting the assistance method according to the user's emotions; and means for sending a warning of possible fraud to the user's screen. This not only allows the user to easily receive voice guidance on the operation method, but also improves their understanding of the screen and provides appropriate assistance when they are unsure of an operation or feel anxious. Furthermore, security risks are reduced, allowing users to use their smartphones with peace of mind.

[0993] "Means for accepting voice input" refers to devices or software that acquire voice data spoken by a user as a digital signal and input it into the system.

[0994] "Means for converting voice data into text data" refers to devices or software that convert acquired voice data into text information using natural language processing technology.

[0995] "Means for analyzing text data and identifying user intent" refers to algorithms and software that use converted text data to understand user requests and objectives.

[0996] "Means for searching for operating procedures" refers to devices or software that refer to a database for operating methods or procedures that correspond to the specified user's intentions and obtain appropriate information.

[0997] "Means for providing voice guidance on operating procedures" refers to speech synthesis technology or software that explains the searched operating procedures to the user by voice.

[0998] "Means for obtaining a screen capture" refers to a device or software for obtaining the contents of the screen currently displayed on the display as image data.

[0999] "Means for analyzing screen capture images" refers to algorithms or software for analyzing the captured image and identifying the software applications and menus being displayed.

[1000] "Means for collecting operation logs" refers to devices or software that record user operation history and periodically collect that data.

[1001] "Means of analyzing operation logs to determine whether the user is having difficulty operating the system" refers to algorithms or software that evaluate the user's operation status based on collected operation logs and determine whether the user is having difficulty operating the system.

[1002] "Means for analyzing user emotions" refers to algorithms or software that evaluate the user's current emotional state based on their voice and operation history.

[1003] "Means for adjusting guidance and instructions" refers to algorithms or software that change the way instructions and instructions are presented depending on the user's emotional state.

[1004] "Means for suggesting assistance" refers to devices or software that suggest appropriate assistance measures or solutions when a user is unsure about how to operate the device or feels anxious.

[1005] "Means of incorporating an emotion recognition engine" refers to algorithms or software that analyzes user emotions in real time and adjusts the system's behavior based on that information.

[1006] "Potential fraud warning means" refers to a device or software that sends a message to a user warning them that a website they visit or an email they receive may be a phishing scam.

[1007] The present invention provides a system for assisting elderly people and users who are unfamiliar with technology to easily operate a smartphone. Hereinafter, an embodiment of the present invention will be described in detail.

[1008] Voice input and analysis

[1009] Subject: Device

[1010] The device accepts voice input when the user speaks. Voice input is done through the built-in microphone, and the voice data is temporarily stored in the device's memory in WAV format or other formats. This voice data is then converted into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data that reads, "I want to connect to the Internet."

[1011] Search and guide instructions

[1012] Subject: Device

[1013] Based on the text data, the device uses a generative AI model (e.g., GPT-4) to analyze the user's intent. This analysis determines that "I want to connect to the Internet" means "Set up Wi-Fi." The device then searches a database to obtain the appropriate operating instructions. Steps such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network" are searched for. These steps are then audibly instructed to the user using a speech output engine (e.g., Google Text-to-Speech). Instructions such as "Open the Settings app, select Wi-Fi, and connect to an available network" are provided.

[1014] Screen capture and image analysis

[1015] Subject: Device

[1016] To help users understand the currently displayed screen, the device takes a screen capture. The captured image is temporarily saved in the device's memory in PNG format, and then an AI image analysis engine such as Google Cloud Vision API is used to identify the software application or menu currently displayed. For example, the captured image can be identified as the "new screen of the Mail app."

[1017] Operation log collection and support proposals

[1018] Subject: Server

[1019] The server periodically collects user operation logs from the device. The operation logs are saved in CSV format and sent to the server. The received operation logs are analyzed using a machine learning model to determine whether the user is unsure of the operation. For example, if a certain operation has not continued for more than 30 seconds, it is determined that the user is unsure of the operation and a support suggestion message asking "Need help?" is sent to the device.

[1020] Security Warning

[1021] Subject: Server

[1022] The server analyzes the website being viewed or the email received and generates a warning message if there is a possibility of a phishing scam. For example, if the server determines that the URL the user clicked on may be fraudulent, it will send a warning message to the device saying, "This site may not be safe. Do you want to continue?"

[1023] Sentiment analysis and response

[1024] Subject: Device

[1025] The device is equipped with an emotion recognition engine (e.g., EmoNet) that analyzes emotions from the user's voice and operation logs. For example, if the device determines from the voice analysis that the user is frustrated, it will explain the operation procedure slowly and generate an encouraging voice message such as "You're almost there. Keep up the great work." If the user is feeling anxious, it will explain the operation procedure in detail and suggest additional support.

[1026] Data Privacy Protection

[1027] Subject: Device

[1028] The device encrypts collected voice data and operation logs using AES encryption and stores them in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and is set to be automatically deleted after that period has elapsed.

[1029] Examples of prompt statements

[1030] For example, when using GPT-4, the prompt might look like this:

[1031] "A user says, 'I want to connect to the Internet.' How can I convert this speech to text and provide the appropriate instructions?"

[1032] By inputting these prompts into a generative AI model, a solution can be obtained that specifically analyzes the voice data and provides guidance on operation procedures. By implementing this system, users can operate their smartphones quickly and safely, and even users who are unfamiliar with technology can use it without stress.

[1033] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1034] Step 1:

[1035] Accepting voice input

[1036] Subject: Device

[1037] Input: User speech

[1038] How it works: The device uses the built-in microphone to capture the user's speech as audio data, which is temporarily stored in the device's memory in WAV format.

[1039] Output: WAV format audio data

[1040] Step 2:

[1041] Converting audio data to text

[1042] Subject: Device

[1043] Input: WAV format audio data

[1044] How it works: The voice data is converted to text data by calling the Google Speech-to-Text API. For example, the voice saying "I want to connect to the Internet" is converted to text data like "I want to connect to the Internet."

[1045] Output: Text data "I want to connect to the Internet"

[1046] Step 3:

[1047] Intent Analysis

[1048] Subject: Device

[1049] Input: Text data "I want to connect to the Internet"

[1050] How it works: The device uses a generative AI model (e.g., GPT-4) to analyze text data and identify user intent. For example, the intent "Wi-Fi settings" is identified from the text "I want to connect to the Internet."

[1051] Output: "Wi-Fi settings"

[1052] Step 4:

[1053] Find instructions

[1054] Subject: Device

[1055] Input: "Wi-Fi settings"

[1056] How it works: The device searches its built-in database to retrieve instructions for "Wi-Fi settings," such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network."

[1057] Output: List of operation steps

[1058] Step 5:

[1059] Voice guidance for operation procedures

[1060] Subject: Device

[1061] Input: List of operating instructions

[1062] How it works: The device uses the Google Text-to-Speech API to provide voice instructions, such as "Open the Settings app, select Wi-Fi, and connect to an available network."

[1063] Output: Voice guidance

[1064] Step 6:

[1065] Taking a screen capture

[1066] Subject: Device

[1067] Input:Currently displayed screen

[1068] What it does: The device captures the current screen image, which is temporarily saved in the device's memory as a PNG image.

[1069] Output: Captured image in PNG format

[1070] Step 7:

[1071] Analysis of captured images

[1072] Subject: Device

[1073] Input: Captcha image in PNG format

[1074] How it works: The device uses the Google Cloud Vision API to analyze the captured image and identify the software application or menu currently displayed. For example, it may identify that the image is the "Mail app's compose screen."

[1075] Output: Identified applications and menus

[1076] Step 8:

[1077] Collecting operation logs

[1078] Subject: Device

[1079] Input: User operation history

[1080] Operation: The device records user operation logs in CSV format and periodically sends them to the server. The operation logs include click and tap locations, operation times, etc.

[1081] Output: Operation log in CSV format

[1082] Step 9:

[1083] Log analysis

[1084] Subject: Server

[1085] Input: CSV format operation log

[1086] How it works: The server analyzes the received operation log using a machine learning model to determine whether the user is having trouble with an operation. For example, if a certain operation has not continued for more than 30 seconds, it determines that the user is having trouble.

[1087] Output: Judgment result of whether the user is confused about the operation

[1088] Step 10:

[1089] Submit a support offer

[1090] Subject: Server

[1091] Input: Determine whether the user is confused about the operation

[1092] Operation: The server generates a help offer message saying "Can I help you?" and sends it to the terminal.

[1093] Output: Support offer message

[1094] Step 11:

[1095] Generate security warnings

[1096] Subject: Server

[1097] Input: Information about the website or email you are viewing

[1098] How it works: The server analyzes the website or email you are viewing and generates a warning message if there is a possibility of a phishing scam, such as "This site may not be secure. Do you want to continue?"

[1099] Output: Security warning message

[1100] Step 12:

[1101] Sending a warning message

[1102] Subject: Server

[1103] Input: Security warning message

[1104] Action: The server generates a security warning message and sends it to the terminal.

[1105] Output: Warning message displayed on the terminal

[1106] Step 13:

[1107] Emotion Analysis

[1108] Subject: Device

[1109] Input: User voice and operation history

[1110] How it works: The device uses an emotion recognition engine (e.g., EmoNet) to analyze the user's emotions. For example, it determines if the user is frustrated based on voice analysis and operation patterns.

[1111] Output: User's emotional state

[1112] Step 14:

[1113] Emotion-based guidance adjustment

[1114] Subject: Device

[1115] Input: User's emotional state

[1116] What it does: If the user is frustrated, the device will explain instructions and directions more slowly and generate encouraging messages, such as "Take a break and we'll walk you through the next steps together."

[1117] Output: Adjusted operating procedures and guidance messages

[1118] Step 15:

[1119] Data Privacy Protection

[1120] Subject: Device

[1121] Input: Collected voice data and operation logs

[1122] Operation: The device encrypts voice data and operation logs using AES encryption and stores them in local storage. After a certain period of time (e.g., one month), this data is automatically deleted.

[1123] Output: Encrypted data and deletion log

[1124] (Application example 2)

[1125] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1126] While conventional voice recognition systems have the ability to identify user intent and provide voice guidance, they are still difficult to operate, especially for elderly people who are unfamiliar with technology and users who are unfamiliar with digital devices. They also lack the ability to provide appropriate assistance based on the user's emotions, which hinders the user experience. Another issue is the lack of assistants for in-store navigation and product information provision.

[1127] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, means for analyzing the text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for analyzing the user's emotions, means for adjusting the operation procedure and guidance based on the analysis results, means for analyzing the acquired captured image to identify the displayed information and providing voice guidance based on the result, and means for suggesting assistance if the user is unfamiliar with the technology. This allows even users unfamiliar with the technology to easily operate the system and provides appropriate assistance based on the user's emotions. It also facilitates navigation and product information provision in physical stores.

[1128] A "means for accepting voice input" has the function of capturing speech uttered by a user and transmitting it to the system for processing.

[1129] The "means for converting voice data into text data" has a function of analyzing received voice and converting the content into text information.

[1130] The "means for identifying the user's intent" is a function that analyzes the text data obtained by voice recognition and determines what the user is looking for.

[1131] The "means for searching for operating procedures" has the function of searching a database or program for the optimal operating procedure that corresponds to the specified user's intention.

[1132] "Voice guidance means" means a function that notifies and guides the user through the searched operating procedures by voice.

[1133] "Means for analyzing user emotions" refers to a function that reads emotions from the user's speech and operation logs and recognizes their state.

[1134] The "means for adjusting operating procedures and guidance" has the function of appropriately changing the content and method of guidance based on the results of user emotion analysis, and providing support according to the user's condition.

[1135] The "means for analyzing a captured image" has a function of analyzing an acquired screen capture and identifying the information and application currently being displayed.

[1136] "Means to offer assistance when the user is unfamiliar with the technology" refers to a function that determines whether the user is unsure of how to operate the device and automatically provides the necessary support.

[1137] This invention utilizes AI technology to provide a "smart store assistant" system to assist elderly and technology-challenged customers in brick-and-mortar stores. The system can be installed and used on devices such as smartphones or smart glasses.

[1138] System Configuration

[1139] 1. Voice recognition and analysis functions

[1140] The device accepts voice input from the user and converts the voice data captured through the microphone into text data using a speech recognition engine (e.g., Google's speech recognition API). The converted text data is then analyzed using a generative AI model to identify the user's intent.

[1141] 2. Operation procedures and guidance functions

[1142] The operating procedures corresponding to the identified user intent are searched for in a database (such as a cloud database), and the search results are then audibly guided to the user using a speech synthesis engine (such as pyttsx3).

[1143] 3. Screen capture and analysis functions

[1144] The device captures the currently displayed screen and uses an AI image analysis engine (such as OpenCV or the machine learning library TensorFlow) to identify the information and app being displayed. The results of this analysis are also announced via voice.

[1145] 4. Emotion recognition and response functions

[1146] The device uses an emotion recognition engine (such as the Hugging Face emotion recognition model) to analyze the user's emotions from their speech and operation logs. Based on the analysis results, the device adjusts operation procedures and guidance appropriately to provide support tailored to the user.

[1147] 5. User assistance suggestion function

[1148] The server periodically collects and analyzes operation logs from the device. If the server determines that the user is unsure of how to operate the device, it sends a message suggesting assistance to the device and displays it to the user.

[1149] 6. Security and Privacy Protection Features

[1150] The server analyzes the displayed information and sends a warning message if there is a security risk. The collected data is encrypted and automatically deleted after a certain period of time.

[1151] Specific examples

[1152] Example prompt sentence:

[1153] "When a user says, 'Where is the milk?', the system converts the voice input into text, performs emotion recognition, and then provides in-store navigation guidance."

[1154] Scenario description:

[1155] A user puts on smart glasses and asks, "Where is the milk?" while in a physical store. This voice input is converted into text by a speech recognition engine, and analysis identifies the user's intent as "I'm looking for milk." An emotion recognition engine evaluates the user's emotions and generates more detailed and courteous guidance if, for example, the user feels anxious. Captured image analysis then analyzes the store map and provides a voice guide saying, "The product is on the left side of aisle A."

[1156] As described above, this system allows even users who are unfamiliar with technology to easily operate it, and enables navigation within physical stores and acquisition of product information.

[1157] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1158] Step 1:

[1159] The device receives voice input from the user: the user speaks questions or commands into the smart glasses or smartphone, which are captured as voice data through the device's microphone.

[1160] Step 2:

[1161] The device sends the acquired voice data to a voice recognition engine and converts it into text data. For example, if a user says, "Where is the milk?", a voice recognition engine (such as Google's voice recognition API) is used to create the text data, "Where is the milk?"

[1162] Step 3:

[1163] The device inputs the converted text data into a generative AI model to analyze the user's intent. Based on the analysis results, it determines what the user is looking for. In this case, the user's intent is determined to be "looking for milk."

[1164] Step 4:

[1165] The device sends the analysis results to an emotion recognition engine to analyze the user's emotions. Using Hugging Face's emotion recognition model, the device evaluates the user's emotional state (e.g., anxiety, irritation, calmness, etc.).

[1166] Step 5:

[1167] The device adjusts the operation procedures and guidance methods based on the results of the user's emotion analysis. For example, if an anxious state is detected, it generates more detailed and slower voice guidance. To generate this voice guidance message, the generated text and the emotion analysis results are used as input data.

[1168] Step 6:

[1169] The device uses a capture image analysis engine to capture the current screen and analyze the information on the screen, including a map of the store. Using AI image analysis technology, it identifies the user's location and the location of the product they are looking for (in this case, milk).

[1170] Step 7:

[1171] The device sends the analysis results to a speech synthesis engine and provides the user with appropriate instructions via voice. For example, it generates a specific message such as "The product is on the left side of aisle A" and outputs it aloud. It uses a speech synthesis library such as pyttsx3.

[1172] Step 8:

[1173] The server periodically collects operation logs from the device and determines whether the user is unsure of how to operate the device. Based on the analysis results, it sends messages to the device suggesting additional guidance or support if necessary. If the user does not perform any operation for a long period of time, a support message such as "Are you having trouble?" will be displayed.

[1174] Step 9:

[1175] The server performs security checks on the information being displayed and the web page being accessed, and if it detects any suspected fraud, it sends a warning message to the device, such as "This site may not be secure. Do you want to continue?" This process is done in real time to ensure the safety of the user.

[1176] Step 10:

[1177] The device encrypts and stores collected voice data and operation logs in local storage, and automatically deletes them after a certain period of time. For example, voice data is protected using AES encryption technology, and a system is implemented that automatically deletes it after 30 days, protecting user privacy.

[1178] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1179] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1180] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1181] [Third embodiment]

[1182] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1183] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1184] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1185] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1186] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1187] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1188] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1189] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1190] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1191] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1192] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1193] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1194] The present invention relates to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly elderly people. The following describes an embodiment of the present invention.

[1195] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intent, searches for the corresponding operating procedure, and provides voice guidance. It also has the ability to capture the current screen and analyze the captured image to identify the currently displayed app or menu. Below, we will explain each function and provide specific examples of system operation.

[1196] Voice input and analysis

[1197] Subject: Device

[1198] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[1199] Search and guide instructions

[1200] Subject: Device

[1201] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[1202] Screen capture and image analysis

[1203] Subject: Device

[1204] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open and asks how to perform a specific operation, the device will identify that it is the "new email app screen" from the captured image and provide instructions.

[1205] Operation log monitoring and support proposals

[1206] Subject: Server

[1207] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[1208] Security Warning

[1209] Subject: Server

[1210] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[1211] Data Privacy Protection

[1212] Subject: Device

[1213] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[1214] As a result, the system of the present invention can help users operate smartphones easily while ensuring security and privacy, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[1215] The processing flow will be explained below.

[1216] Voice input and analysis

[1217] Subject: Device

[1218] Step 1:

[1219] The device activates the microphone and collects the user's speech in real time as audio data.

[1220] Step 2:

[1221] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[1222] Step 3:

[1223] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[1224] Step 4:

[1225] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[1226] Step 5:

[1227] The device will guide the user through the operation procedures via voice.

[1228] Screen capture and image analysis

[1229] Subject: Device

[1230] Step 1:

[1231] Capture the screen currently displayed on the device and save it as a still image.

[1232] Step 2:

[1233] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[1234] Step 3:

[1235] Predicts what the user will need to do next based on identified apps and menus.

[1236] Step 4:

[1237] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[1238] Operation log monitoring and support proposals

[1239] Subject: Server

[1240] Step 1:

[1241] The server periodically collects operation logs from the terminal.

[1242] Step 2:

[1243] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[1244] Step 3:

[1245] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[1246] Step 4:

[1247] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[1248] Step 5:

[1249] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[1250] Security Warning

[1251] Subject: Server

[1252] Step 1:

[1253] The server monitors and analyzes the content of websites and emails being displayed in real time.

[1254] Step 2:

[1255] Based on the analysis results, we identify content that may be fraudulent.

[1256] Step 3:

[1257] Generate a warning message if any suspected fraud is identified.

[1258] Step 4:

[1259] Sends server-generated warning messages to the terminal for display to the user.

[1260] Step 5:

[1261] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[1262] Data Privacy Protection

[1263] Subject: Device

[1264] Step 1:

[1265] The voice data and operation logs collected by the device are stored in local storage.

[1266] Step 2:

[1267] Encrypt stored data and set access restrictions to prevent unauthorized access.

[1268] Step 3:

[1269] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[1270] Step 4:

[1271] Make sure the deleted data is completely wiped from your device.

[1272] This allows the system to help users operate their smartphones easily while ensuring security and privacy.

[1273] Example 1

[1274] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1275] Today's smartphones integrate a variety of functions in a complex manner, making them difficult to operate, especially for elderly users and those unfamiliar with technology. Furthermore, assistance systems using voice recognition technology are required to accurately understand the user's intentions and guide them to the appropriate operating procedures. However, many existing systems lack the precision to analyze the user's voice commands and are unable to provide the appropriate operating procedures. Furthermore, some systems lack the functionality to provide prompt assistance when the user is unsure of how to operate the device, and some lack data privacy protection.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1277] In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, and means for analyzing the converted text data to identify the user's intention, thereby enabling the server to accurately analyze the content of a voice instruction given by the user and provide appropriate operating procedures.

[1278] Furthermore, the server includes means for searching for an operation procedure corresponding to the user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, means for analyzing the captured image and identifying the application or menu currently being displayed, means for analyzing operation logs collected from the terminal and determining whether the user is unsure of the operation, means for offering assistance when the user is unsure of the operation, means for encrypting the collected voice data and operation logs, and means for automatically deleting the encrypted data after a certain period of time. This makes it possible to offer assistance immediately when the user is unsure of the operation, while at the same time ensuring the privacy of the collected data.

[1279] "Means for accepting voice input" is a general term for devices and software functions for capturing voice uttered by a user.

[1280] "Means for converting into text data" is a general term for technology and software for converting voice data into character string data.

[1281] "Means for identifying user intent" is a general term for technologies and algorithms that analyze acquired text data and determine what the user is trying to achieve.

[1282] "Means for searching for operating procedures" is a general term for technology and software for searching a recorded database or storage device for an operating method that corresponds to a specified user's intention.

[1283] "Voice guidance means" is a general term for devices and software that notify users of searched operating procedures by voice.

[1284] "Means for capturing the screen" is a general term for technologies and devices for acquiring an image of the currently displayed screen.

[1285] "Means for analyzing a captured image" is a general term for techniques and algorithms for analyzing an acquired captured image and identifying the contents of the currently displayed application or menu.

[1286] "Means for analyzing operation logs" is a general term for technologies and algorithms used to analyze operation history data collected from a terminal and determine the user's operation status.

[1287] "Means for suggesting assistance" is a general term for technology and software that presents appropriate operating procedures and supplementary information when a user is unsure how to operate something.

[1288] "Means for encrypting voice data and operation logs" is a general term for technologies and algorithms used to encrypt and protect collected voice data and operation logs.

[1289] "Means for automatic deletion after a certain period of time" is a general term for technology and software that automatically deletes encrypted data after a set period of time has passed.

[1290] MODE FOR CARRYING OUT THE INVENTION

[1291] This invention relates to an AI-equipped system that assists users, particularly elderly people and those unfamiliar with technology, in easily operating smartphones. The system accepts voice input from the user, converts it into text data, and analyzes it to identify the user's intent. It then searches for operating procedures corresponding to the identified intent and provides voice guidance. It also has the ability to capture the current screen and analyze the image to identify the application or menu currently being displayed. Furthermore, it periodically collects operation logs from the device, determines whether the user is unsure of an operation, and offers assistance. Finally, the collected data is encrypted and stored, and automatically deleted after a certain period of time to protect privacy.

[1292] Hardware and software used

[1293] Speech recognition engine: Google Speech-to-Text (to convert voice data into text data)

[1294] Generative AI model: GPT-3 (to analyze text data and identify user intent)

[1295] Database: Search for operating instructions

[1296] Audio output device: Speaker (to provide audio instructions)

[1297] Image analysis engine: Google Vision API (to analyze captured images)

[1298] Encryption technology: AES-256 (to encrypt collected data)

[1299] Specific examples

[1300] 1. Voice to text conversion:

[1301] User: Say "I want to connect to the Internet."

[1302] Device: Audio is captured through the microphone and converted into text data such as "I want to connect to the Internet" using Google Speech-to-Text.

[1303] 2. Intention analysis of text data and search for operation procedures:

[1304] On the device: The generative AI model GPT-3 is used to analyze the "Wi-Fi settings" and the user's intent, and then the corresponding operation steps are searched for in the database.

[1305] Device: A voice prompt will say, "Open the Settings app and go to the Wi-Fi menu."

[1306] 3. Screen capture and analysis:

[1307] User: You have the Mail app open and say, "I want to compose a new email."

[1308] Device: Capture the currently displayed screen and use the Google Vision API to identify it as the "new email app composition screen." The device then prompts the user to "enter the subject of the email."

[1309] 4. Operation log collection and support suggestions:

[1310] Server: Periodically collects operation logs from the device, and if the user does not operate the device for more than 30 seconds, it uses AI to determine that assistance is required.

[1311] Server: Sends a help offer message to the device saying "Can we help you?"

[1312] 5. Security Checks and Warnings:

[1313] User: When you try to click on a suspicious URL.

[1314] Server: Immediately analyzes the URL and sends a warning message to the device saying, "This site may not be secure. Do you want to continue?"

[1315] 6. Data Encryption and Storage:

[1316] Terminal: Collected audio data and operation logs are stored in local storage using AES-256 encryption technology.

[1317] Device: Stored data will be automatically deleted after one month.

[1318] Prompt Sentence Examples

[1319] When a user says they want to perform a specific action, for example, "I want to set up Wi-Fi," how can I guide them to "Open the Settings app and go to the Wi-Fi menu"?

[1320] The system allows even non-technical users to easily perform complex smartphone operations, provides assistance when needed, and protects data privacy.

[1321] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1322] System program processing flow

[1323] Step 1:

[1324] The user speaks and performs voice input. For example, the user might say, "I want to connect to the Internet."

[1325] Input: User's voice

[1326] Output: Audio data

[1327] Step 2:

[1328] The device picks up audio through a microphone and generates audio data.

[1329] Input: Audio data

[1330] Output: Raw audio data

[1331] Step 3:

[1332] The device converts the voice data into text using Google Speech-to-Text. For example, the text data might say, "I want to connect to the Internet."

[1333] Input: Raw audio data

[1334] Output: Text data

[1335] Step 4:

[1336] The device uses a generative AI model (GPT-3) to analyze text data and identify the user's intent. For example, it can analyze the intent "Wi-Fi settings" from the text "I want to connect to the Internet."

[1337] Input: Text data

[1338] Output: User intent (e.g., "Wi-Fi settings")

[1339] Step 5:

[1340] The device searches a database for the operation procedure that corresponds to the user's intention. For example, it searches for the procedure for "Wi-Fi settings."

[1341] Input: User intent

[1342] Output: Instructions (e.g. "Open the Settings app and go to the Wi-Fi menu")

[1343] Step 6:

[1344] The device will be searched and the operation procedure will be guided to the user using the voice output device (speaker). For example, it will say, "Open the Settings app and go to the Wi-Fi menu."

[1345] Input: Operation procedure

[1346] Output: Voice guidance

[1347] Step 7:

[1348] Capture the screen that the device is currently displaying. For example, if the user has the email app open, capture that screen.

[1349] Input: Current screen

[1350] Output: Captured image

[1351] Step 8:

[1352] The device analyzes the captured image using the Google Vision API. For example, it identifies the captured image as the "new email app composition screen."

[1353] Input: Captcha image

[1354] Output: Identifying the currently displayed application or menu

[1355] Step 9:

[1356] The server periodically collects operation logs from the device. For example, if the user does not operate the device for 30 seconds or more, a log is collected.

[1357] Input: Operation Log

[1358] Output: Operation log data

[1359] Step 10:

[1360] The server uses AI to analyze the collected operation logs and determine whether the user is unsure of the operation. For example, if the user does not operate the device for more than 30 seconds, it will determine that the user is unsure.

[1361] Input: Operation log data

[1362] Output: Determine whether the user is confused about the operation

[1363] Step 11:

[1364] If the server is unsure of an operation, it sends a message offering assistance to the terminal, such as "Can I help you?"

[1365] Input: Determine if the user is confused

[1366] Output: Support offer message

[1367] Step 12:

[1368] The server analyzes the website or email you are viewing in real time and sends a warning message if it detects any fraudulent activity, such as "This site may not be safe. Do you want to continue?"

[1369] Input: The website or email you are viewing

[1370] Output: Security warning message

[1371] Step 13:

[1372] The voice data and operation logs collected by the device are encrypted using AES-256 encryption technology. For example, the voice data is encrypted and stored in local storage.

[1373] Input: Collected voice data and operation logs

[1374] Output: Encrypted data

[1375] Step 14:

[1376] The device will automatically delete the encrypted data after a certain period of time, for example, after one month.

[1377] Input: Encrypted data

[1378] Output: Automatic data deletion

[1379] (Application example 1)

[1380] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1381] Operating conventional smartphones is often difficult, especially for the elderly. Similarly, using food delivery services is difficult, with many users struggling to understand the ordering process. Therefore, there is a demand for systems that are easy for these users to operate.

[1382] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1383] In this invention, the server includes means for accepting voice input from a user, means for converting the accepted voice data into text data, means for analyzing the converted text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, analyzing the image, and identifying the displayed content, and means for providing voice guidance for ordering food delivery based on the identified displayed content. This enables even users who have difficulty operating smartphones to easily place food delivery orders.

[1384] "Means for accepting voice input from the user" is a general term for devices and software that recognize the voice spoken by the user and receive the voice data.

[1385] "Means for converting voice data into text data" refers to the technology or function for analyzing received voice information and converting it into text information.

[1386] "Means for identifying user intent" refers to technologies and mechanisms that understand and determine the operations and service content desired by the user from the converted text data.

[1387] A "means for searching for operating procedures" is a technology or system that searches for appropriate operating methods or procedures based on the identified user intent.

[1388] "Voice guidance means" refers to technology or devices that verbally communicate searched operating procedures to the user.

[1389] "Means for capturing the currently displayed screen" refers to the technology or function for acquiring the currently displayed screen content of a device as image data.

[1390] "Means for analyzing images" refers to techniques or algorithms for analyzing captured image data and recognizing the information displayed therein.

[1391] "Means for providing voice guidance for ordering food delivery" refers to technologies and systems that provide voice guidance to users through the food delivery ordering process based on specified display content and the user's intent.

[1392] This invention relates to a voice-operated food delivery system designed for users who have difficulty operating smartphones, particularly the elderly. The system accepts voice input, converts the voice data into text, analyzes it, identifies the user's intention, and provides voice guidance on the appropriate operation procedures.

[1393] System configuration

[1394] The system consists of the following main modules:

[1395] Audio Input Module

[1396] Voice Recognition Module

[1397] Intention Analysis Module

[1398] Operation procedure search module

[1399] Voice Guidance Module

[1400] Screen Capture Module

[1401] Image Analysis Module

[1402] Food delivery information module

[1403] Hardware and software used

[1404] Hardware

[1405] Microphone: A device for capturing the user's voice.

[1406] Camera: A device for capturing the currently displayed screen (using the screenshot function on your smartphone).

[1407] software

[1408] SpeechRecognition Library: A library for converting voice data into text data.

[1409] Pyttsx3 library: A synthetic speech engine for generating voice prompts.

[1410] PIL (Python Imaging Library): A library for processing screen capture images.

[1411] Pytesseract library: An optical character recognition (OCR) engine for extracting text from images.

[1412] System operation explanation

[1413] Voice to text conversion

[1414] A user voice-overs a food delivery order through a microphone, for example, saying, "I'd like to order a pizza." The microphone captures the voice input, and the SpeechRecognition library converts the speech into text data.

[1415] Intention analysis and operation procedure search

[1416] The converted text data is analyzed by an intention analysis module, and is identified as, for example, "ordering a pizza." Next, an operating procedure search module searches for an appropriate operating procedure based on the intention.

[1417] Voice guidance and screen analysis

[1418] The searched operation procedures are then audibly guided to the user by the voice guidance module. For example, instructions such as "Add pizza to cart. Press the continue button" are given by synthesized voice. To determine which screen the user is currently on, the screen capture module captures the screen, and the image analysis module analyzes the captured image.

[1419] Food delivery information

[1420] The voice prompts are repeated until the user completes the food delivery order. For example, the voice prompts the user based on the current screen, such as "Open the pizza menu" on the menu screen or "Confirm your order" on the cart screen.

[1421] Specific examples

[1422] For example, if a user says, "I would like to order sushi," the speech is converted into text and the intent is analyzed as "Order sushi." After image analysis confirms that the screen is a menu screen, a voice prompt will be played saying, "Add sushi to cart. Please press the continue button."

[1423] Example prompts for generative AI models

[1424] To develop a voice-operated food delivery app for the elderly, write a program that converts the user's voice input into text, analyzes the user's actions from screen captures, and provides appropriate guidance.

[1425] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1426] Step 1:

[1427] The user inputs voice through the microphone. In this step, the user's voice is picked up through the microphone and input to the terminal as voice data. For example, the user may say, "I'd like to order a pizza."

[1428] Step 2:

[1429] The speech recognition module converts the speech data into text data. The device then analyzes the acquired speech data using the SpeechRecognition library and converts it into text data. For example, a speech saying "I would like to order a pizza" is converted into text data saying "I would like to order a pizza."

[1430] Step 3:

[1431] The intent analysis module analyzes the text data and identifies the user's intent. The device analyzes the converted text data and understands the user's intent. This step uses NLP (natural language processing) technology to analyze and identify the user's intent as "order pizza."

[1432] Step 4:

[1433] The operation procedure search module searches for operation procedures based on the specified intent. The device searches for operation procedures from a database based on the specified intent. For example, in the case of "ordering pizza," the module searches for procedures to open the pizza menu and add it to the cart.

[1434] Step 5:

[1435] The voice guidance module provides voice guidance for the searched operation procedures. The terminal uses a speech synthesis engine (Pyttsx3 library) to guide the user through the searched procedures. For example, the user may hear voice guidance such as "Add pizza to cart. Press the continue button."

[1436] Step 6:

[1437] The screen capture module captures the current screen. The device captures the currently displayed screen of a food delivery app and obtains its image data. For example, if the user is on the menu screen of a food delivery app, that screen will be captured.

[1438] Step 7:

[1439] The image analysis module analyzes the captured image and identifies the displayed content. The device processes the captured image using the PIL library and uses the Pytesseract library to extract and analyze the text in the image. This identifies the screen the user is currently on. For example, it identifies the current screen as the "menu screen."

[1440] Step 8:

[1441] The food delivery guidance module then provides further voice guidance based on the identified display content. The device combines the analyzed display content with the user's intention and provides voice guidance on the next action to take. For example, if the user is on the menu screen, they may be told, "Add pizza to cart. Press the continue button," or if they move to the cart screen, they may be told, "Confirm your order."

[1442] Step 9:

[1443] If the user does not complete an operation, the operation log is periodically collected and analyzed by the server. The server receives the operation log from the device and analyzes it using an algorithm to determine whether the user is confused about the operation. For example, if a specific operation is not performed for a certain period of time, it is determined that the user is confused.

[1444] Step 10:

[1445] If the server determines that the user is unsure of how to operate the device, it will provide a voice message suggesting additional assistance. The server sends a message such as "Are you having trouble?" to the device, which then relays this message to the user via voice guidance.

[1446] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1447] The present invention adds emotion recognition functionality to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly the elderly, making it even easier to use. The following describes an embodiment of the present invention.

[1448] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intention, searches for the corresponding operating procedure, and provides voice guidance.It also has the ability to capture the current screen and analyze the captured image to identify the app or menu currently being displayed.In addition, an emotion recognition engine is incorporated, further improving the user experience by adjusting the assistance method according to the user's emotions.

[1449] Voice input and analysis

[1450] Subject: Device

[1451] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[1452] Search and guide instructions

[1453] Subject: Device

[1454] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[1455] Screen capture and image analysis

[1456] Subject: Device

[1457] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open, the device will identify from the captured image that it is the "new email app screen" and provide instructions on how to use it.

[1458] Operation log monitoring and support proposals

[1459] Subject: Server

[1460] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[1461] Security Warning

[1462] Subject: Server

[1463] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[1464] Data Privacy Protection

[1465] Subject: Device

[1466] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[1467] Emotion recognition engine

[1468] Subject: Device

[1469] The device is equipped with an emotion recognition engine that can recognize emotions from the user's voice and operation logs. For example, if the device detects that the user is frustrated, it will explain the operation procedure more slowly and send an encouraging message to calm the user.

[1470] Adjusting operating procedures according to emotions

[1471] Subject: Device

[1472] The emotion recognition engine analyzes the user's emotions and adjusts the operating procedures and guidance accordingly. For example, if the user is feeling anxious, the device will provide detailed instructions and suggest that additional support is needed.

[1473] Generating emotion-responsive messages

[1474] Subject: Device

[1475] The emotion recognition engine generates encouraging and comforting messages based on the user's emotions, and the device then provides these messages to the user through voice. For example, if the user shows signs of fatigue, the device will provide a message such as, "Take a short break. It's okay, let's do the next step together."

[1476] As a result, the system of the present invention not only helps users operate smartphones easily while ensuring security and privacy, but also provides an advanced user experience through emotion recognition, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[1477] The processing flow will be explained below.

[1478] Voice input and analysis

[1479] Subject: Device

[1480] Step 1:

[1481] The device activates the microphone and collects the user's speech in real time as audio data.

[1482] Step 2:

[1483] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[1484] Step 3:

[1485] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[1486] Step 4:

[1487] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[1488] Step 5:

[1489] The device will guide the user through the operation procedures via voice.

[1490] Screen capture and image analysis

[1491] Subject: Device

[1492] Step 1:

[1493] Capture the screen currently displayed on the device and save it as a still image.

[1494] Step 2:

[1495] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[1496] Step 3:

[1497] Predicts what the user will need to do next based on identified apps and menus.

[1498] Step 4:

[1499] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[1500] Operation log monitoring and support proposals

[1501] Subject: Server

[1502] Step 1:

[1503] The server periodically collects operation logs from the terminal.

[1504] Step 2:

[1505] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[1506] Step 3:

[1507] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[1508] Step 4:

[1509] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[1510] Step 5:

[1511] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[1512] Security Warning

[1513] Subject: Server

[1514] Step 1:

[1515] The server monitors and analyzes the content of websites and emails being displayed in real time.

[1516] Step 2:

[1517] Based on the analysis results, we identify content that may be fraudulent.

[1518] Step 3:

[1519] Generate a warning message if any suspected fraud is identified.

[1520] Step 4:

[1521] Sends server-generated warning messages to the terminal for display to the user.

[1522] Step 5:

[1523] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[1524] Data Privacy Protection

[1525] Subject: Device

[1526] Step 1:

[1527] The voice data and operation logs collected by the device are stored in local storage.

[1528] Step 2:

[1529] Encrypt stored data and set access restrictions to prevent unauthorized access.

[1530] Step 3:

[1531] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[1532] Step 4:

[1533] Make sure the deleted data is completely wiped from your device.

[1534] Emotion recognition engine

[1535] Subject: Device

[1536] Step 1:

[1537] The device collects the user's voice and operation logs and sends them to an emotion recognition engine.

[1538] Step 2:

[1539] The emotion recognition engine analyzes the tone of voice, operation speed, etc. to identify the user's emotions.

[1540] Step 3:

[1541] The analyzed user emotion data is integrated into the current operation support process.

[1542] Adjusting operating procedures according to emotions

[1543] Subject: Device

[1544] Step 1:

[1545] The emotion recognition engine analyzes the user's emotions.

[1546] Step 2:

[1547] The device adjusts the operating procedures and guidance methods according to the results of the emotion analysis.

[1548] Step 3:

[1549] If the user is feeling unsure, the device will provide detailed instructions on how to proceed.

[1550] Generating emotion-responsive messages

[1551] Subject: Device

[1552] Step 1:

[1553] The emotion recognition engine analyzes the user's emotional data.

[1554] Step 2:

[1555] Generate encouraging and comforting messages according to the user's emotional state.

[1556] Step 3:

[1557] If the user appears to be tired of operating the device, the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[1558] Specific examples

[1559] Examples of voice input and analysis:

[1560] When a user says, "I want to connect to the Internet," the device converts the speech into text and interprets it as "Wi-Fi settings." The device then provides voice instructions on how to access the Wi-Fi settings screen.

[1561] Examples of operational log monitoring and support suggestions:

[1562] If the user does not interact with the screen for 30 seconds or more, the server generates a message saying "Can we help you?" and sends it to the device. If the user accepts this message, the server guides them through the appropriate operation procedure.

[1563] Examples of emotionally relevant messages:

[1564] If the user is tired of operating the device, the emotion recognition engine will analyze this and the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[1565] As a result, this system provides operational assistance while taking the user's emotions into consideration, creating an environment in which users can use their smartphones with peace of mind.

[1566] Example 2

[1567] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1568] Elderly people and users unfamiliar with technology often find it difficult to operate smartphones, posing security risks such as getting lost or accidentally accessing phishing sites. Furthermore, existing systems lack support that takes into account the user's emotional state, which can easily cause stress. Therefore, there is a need for a system that allows users to operate smartphones more easily and safely.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1570] In this invention, the server includes: means for accepting voice input from a user; means for converting the accepted voice data into text data; means for analyzing the converted text data and identifying the user's intent; means for searching for an operating procedure corresponding to the identified user's intent; means for providing voice guidance for the searched operating procedure; means for acquiring a screen capture and analyzing the acquired screen capture image to identify the software application or menu currently displayed; means for periodically collecting a user's operation log and analyzing the collected operation log to determine whether the user is unsure of an operation; means for analyzing the user's emotions and adjusting the guidance and operating procedure based on the results; means for suggesting assistance when the user is unsure of an operation or emotionally unstable; means for incorporating an emotion recognition engine and adjusting the assistance method according to the user's emotions; and means for sending a warning of possible fraud to the user's screen. This not only allows the user to easily receive voice guidance on the operation method, but also improves their understanding of the screen and provides appropriate assistance when they are unsure of an operation or feel anxious. Furthermore, security risks are reduced, allowing users to use their smartphones with peace of mind.

[1571] "Means for accepting voice input" refers to devices or software that acquire voice data spoken by a user as a digital signal and input it into the system.

[1572] "Means for converting voice data into text data" refers to devices or software that convert acquired voice data into text information using natural language processing technology.

[1573] "Means for analyzing text data and identifying user intent" refers to algorithms and software that use converted text data to understand user requests and objectives.

[1574] "Means for searching for operating procedures" refers to devices or software that refer to a database for operating methods or procedures that correspond to the specified user's intentions and obtain appropriate information.

[1575] "Means for providing voice guidance on operating procedures" refers to speech synthesis technology or software that explains the searched operating procedures to the user by voice.

[1576] "Means for obtaining a screen capture" refers to a device or software for obtaining the contents of the screen currently displayed on the display as image data.

[1577] "Means for analyzing screen capture images" refers to algorithms or software for analyzing the captured image and identifying the software applications and menus being displayed.

[1578] "Means for collecting operation logs" refers to devices or software that record user operation history and periodically collect that data.

[1579] "Means of analyzing operation logs to determine whether the user is having difficulty operating the system" refers to algorithms or software that evaluate the user's operation status based on collected operation logs and determine whether the user is having difficulty operating the system.

[1580] "Means for analyzing user emotions" refers to algorithms or software that evaluate the user's current emotional state based on their voice and operation history.

[1581] "Means for adjusting guidance and instructions" refers to algorithms or software that change the way instructions and instructions are presented depending on the user's emotional state.

[1582] "Means for suggesting assistance" refers to devices or software that suggest appropriate assistance measures or solutions when a user is unsure about how to operate the device or feels anxious.

[1583] "Means of incorporating an emotion recognition engine" refers to algorithms or software that analyzes user emotions in real time and adjusts the system's behavior based on that information.

[1584] "Potential fraud warning means" refers to a device or software that sends a message to a user warning them that a website they visit or an email they receive may be a phishing scam.

[1585] The present invention provides a system for assisting elderly people and users who are unfamiliar with technology to easily operate a smartphone. Hereinafter, an embodiment of the present invention will be described in detail.

[1586] Voice input and analysis

[1587] Subject: Device

[1588] The device accepts voice input when the user speaks. Voice input is done through the built-in microphone, and the voice data is temporarily stored in the device's memory in WAV format or other formats. This voice data is then converted into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data that reads, "I want to connect to the Internet."

[1589] Search and guide instructions

[1590] Subject: Device

[1591] Based on the text data, the device uses a generative AI model (e.g., GPT-4) to analyze the user's intent. This analysis determines that "I want to connect to the Internet" means "Set up Wi-Fi." The device then searches a database to obtain the appropriate operating instructions. Steps such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network" are searched for. These steps are then audibly instructed to the user using a speech output engine (e.g., Google Text-to-Speech). Instructions such as "Open the Settings app, select Wi-Fi, and connect to an available network" are provided.

[1592] Screen capture and image analysis

[1593] Subject: Device

[1594] To help users understand the currently displayed screen, the device takes a screen capture. The captured image is temporarily saved in the device's memory in PNG format, and then an AI image analysis engine such as Google Cloud Vision API is used to identify the software application or menu currently displayed. For example, the captured image can be identified as the "new screen of the Mail app."

[1595] Operation log collection and support proposals

[1596] Subject: Server

[1597] The server periodically collects user operation logs from the device. The operation logs are saved in CSV format and sent to the server. The received operation logs are analyzed using a machine learning model to determine whether the user is unsure of the operation. For example, if a certain operation has not continued for more than 30 seconds, it is determined that the user is unsure of the operation and a support suggestion message asking "Need help?" is sent to the device.

[1598] Security Warning

[1599] Subject: Server

[1600] The server analyzes the website being viewed or the email received and generates a warning message if there is a possibility of a phishing scam. For example, if the server determines that the URL the user clicked on may be fraudulent, it will send a warning message to the device saying, "This site may not be safe. Do you want to continue?"

[1601] Sentiment analysis and response

[1602] Subject: Device

[1603] The device is equipped with an emotion recognition engine (e.g., EmoNet) that analyzes emotions from the user's voice and operation logs. For example, if the device determines from the voice analysis that the user is frustrated, it will explain the operation procedure slowly and generate an encouraging voice message such as "You're almost there. Keep up the great work." If the user is feeling anxious, it will explain the operation procedure in detail and suggest additional support.

[1604] Data Privacy Protection

[1605] Subject: Device

[1606] The device encrypts collected voice data and operation logs using AES encryption and stores them in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and is set to be automatically deleted after that period has elapsed.

[1607] Examples of prompt statements

[1608] For example, when using GPT-4, the prompt might look like this:

[1609] "A user says, 'I want to connect to the Internet.' How can I convert this speech to text and provide the appropriate instructions?"

[1610] By inputting these prompts into a generative AI model, a solution can be obtained that specifically analyzes the voice data and provides guidance on operation procedures. By implementing this system, users can operate their smartphones quickly and safely, and even users who are unfamiliar with technology can use it without stress.

[1611] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1612] Step 1:

[1613] Accepting voice input

[1614] Subject: Device

[1615] Input: User speech

[1616] How it works: The device uses the built-in microphone to capture the user's speech as audio data, which is temporarily stored in the device's memory in WAV format.

[1617] Output: WAV format audio data

[1618] Step 2:

[1619] Converting audio data to text

[1620] Subject: Device

[1621] Input: WAV format audio data

[1622] How it works: The voice data is converted to text data by calling the Google Speech-to-Text API. For example, the voice saying "I want to connect to the Internet" is converted to text data like "I want to connect to the Internet."

[1623] Output: Text data "I want to connect to the Internet"

[1624] Step 3:

[1625] Intent Analysis

[1626] Subject: Device

[1627] Input: Text data "I want to connect to the Internet"

[1628] How it works: The device uses a generative AI model (e.g., GPT-4) to analyze text data and identify user intent. For example, the intent "Wi-Fi settings" is identified from the text "I want to connect to the Internet."

[1629] Output: "Wi-Fi settings"

[1630] Step 4:

[1631] Find instructions

[1632] Subject: Device

[1633] Input: "Wi-Fi settings"

[1634] How it works: The device searches its built-in database to retrieve instructions for "Wi-Fi settings," such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network."

[1635] Output: List of operation steps

[1636] Step 5:

[1637] Voice guidance for operation procedures

[1638] Subject: Device

[1639] Input: List of operating instructions

[1640] How it works: The device uses the Google Text-to-Speech API to provide voice instructions, such as "Open the Settings app, select Wi-Fi, and connect to an available network."

[1641] Output: Voice guidance

[1642] Step 6:

[1643] Taking a screen capture

[1644] Subject: Device

[1645] Input:Currently displayed screen

[1646] What it does: The device captures the current screen image, which is temporarily saved in the device's memory as a PNG image.

[1647] Output: Captured image in PNG format

[1648] Step 7:

[1649] Analysis of captured images

[1650] Subject: Device

[1651] Input: Captcha image in PNG format

[1652] How it works: The device uses the Google Cloud Vision API to analyze the captured image and identify the software application or menu currently displayed. For example, it may identify that the image is the "Mail app's compose screen."

[1653] Output: Identified applications and menus

[1654] Step 8:

[1655] Collecting operation logs

[1656] Subject: Device

[1657] Input: User operation history

[1658] Operation: The device records user operation logs in CSV format and periodically sends them to the server. The operation logs include click and tap locations, operation times, etc.

[1659] Output: Operation log in CSV format

[1660] Step 9:

[1661] Log analysis

[1662] Subject: Server

[1663] Input: CSV format operation log

[1664] How it works: The server analyzes the received operation log using a machine learning model to determine whether the user is having trouble with an operation. For example, if a certain operation has not continued for more than 30 seconds, it determines that the user is having trouble.

[1665] Output: Judgment result of whether the user is confused about the operation

[1666] Step 10:

[1667] Submit a support offer

[1668] Subject: Server

[1669] Input: Determine whether the user is confused about the operation

[1670] Operation: The server generates a help offer message saying "Can I help you?" and sends it to the terminal.

[1671] Output: Support offer message

[1672] Step 11:

[1673] Generate security warnings

[1674] Subject: Server

[1675] Input: Information about the website or email you are viewing

[1676] How it works: The server analyzes the website or email you are viewing and generates a warning message if there is a possibility of a phishing scam, such as "This site may not be secure. Do you want to continue?"

[1677] Output: Security warning message

[1678] Step 12:

[1679] Sending a warning message

[1680] Subject: Server

[1681] Input: Security warning message

[1682] Action: The server generates a security warning message and sends it to the terminal.

[1683] Output: Warning message displayed on the terminal

[1684] Step 13:

[1685] Emotion Analysis

[1686] Subject: Device

[1687] Input: User voice and operation history

[1688] How it works: The device uses an emotion recognition engine (e.g., EmoNet) to analyze the user's emotions. For example, it determines if the user is frustrated based on voice analysis and operation patterns.

[1689] Output: User's emotional state

[1690] Step 14:

[1691] Emotion-based guidance adjustment

[1692] Subject: Device

[1693] Input: User's emotional state

[1694] What it does: If the user is frustrated, the device will explain instructions and directions more slowly and generate encouraging messages, such as "Take a break and we'll walk you through the next steps together."

[1695] Output: Adjusted operating procedures and guidance messages

[1696] Step 15:

[1697] Data Privacy Protection

[1698] Subject: Device

[1699] Input: Collected voice data and operation logs

[1700] Operation: The device encrypts voice data and operation logs using AES encryption and stores them in local storage. After a certain period of time (e.g., one month), this data is automatically deleted.

[1701] Output: Encrypted data and deletion log

[1702] (Application example 2)

[1703] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1704] While conventional voice recognition systems have the ability to identify user intent and provide voice guidance, they are still difficult to operate, especially for elderly people who are unfamiliar with technology and users who are unfamiliar with digital devices. They also lack the ability to provide appropriate assistance based on the user's emotions, which hinders the user experience. Another issue is the lack of assistants for in-store navigation and product information provision.

[1705] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, means for analyzing the text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for analyzing the user's emotions, means for adjusting the operation procedure and guidance based on the analysis results, means for analyzing the acquired captured image to identify the displayed information and providing voice guidance based on the result, and means for suggesting assistance if the user is unfamiliar with the technology. This allows even users unfamiliar with the technology to easily operate the system and provides appropriate assistance based on the user's emotions. It also facilitates navigation and product information provision in physical stores.

[1706] A "means for accepting voice input" has the function of capturing speech uttered by a user and transmitting it to the system for processing.

[1707] The "means for converting voice data into text data" has a function of analyzing received voice and converting the content into text information.

[1708] The "means for identifying the user's intent" is a function that analyzes the text data obtained by voice recognition and determines what the user is looking for.

[1709] The "means for searching for operating procedures" has the function of searching a database or program for the optimal operating procedure that corresponds to the specified user's intention.

[1710] "Voice guidance means" means a function that notifies and guides the user through the searched operating procedures by voice.

[1711] "Means for analyzing user emotions" refers to a function that reads emotions from the user's speech and operation logs and recognizes their state.

[1712] The "means for adjusting operating procedures and guidance" has the function of appropriately changing the content and method of guidance based on the results of user emotion analysis, and providing support according to the user's condition.

[1713] The "means for analyzing a captured image" has a function of analyzing an acquired screen capture and identifying the information and application currently being displayed.

[1714] "Means to offer assistance when the user is unfamiliar with the technology" refers to a function that determines whether the user is unsure of how to operate the device and automatically provides the necessary support.

[1715] This invention utilizes AI technology to provide a "smart store assistant" system to assist elderly and technology-challenged customers in brick-and-mortar stores. The system can be installed and used on devices such as smartphones or smart glasses.

[1716] System Configuration

[1717] 1. Voice recognition and analysis functions

[1718] The device accepts voice input from the user and converts the voice data captured through the microphone into text data using a speech recognition engine (e.g., Google's speech recognition API). The converted text data is then analyzed using a generative AI model to identify the user's intent.

[1719] 2. Operation procedures and guidance functions

[1720] The operating procedures corresponding to the identified user intent are searched for in a database (such as a cloud database), and the search results are then audibly guided to the user using a speech synthesis engine (such as pyttsx3).

[1721] 3. Screen capture and analysis functions

[1722] The device captures the currently displayed screen and uses an AI image analysis engine (such as OpenCV or the machine learning library TensorFlow) to identify the information and app being displayed. The results of this analysis are also announced via voice.

[1723] 4. Emotion recognition and response functions

[1724] The device uses an emotion recognition engine (such as the Hugging Face emotion recognition model) to analyze the user's emotions from their speech and operation logs. Based on the analysis results, the device adjusts operation procedures and guidance appropriately to provide support tailored to the user.

[1725] 5. User assistance suggestion function

[1726] The server periodically collects and analyzes operation logs from the device. If the server determines that the user is unsure of how to operate the device, it sends a message suggesting assistance to the device and displays it to the user.

[1727] 6. Security and Privacy Protection Features

[1728] The server analyzes the displayed information and sends a warning message if there is a security risk. The collected data is encrypted and automatically deleted after a certain period of time.

[1729] Specific examples

[1730] Example prompt sentence:

[1731] "When a user says, 'Where is the milk?', the system converts the voice input into text, performs emotion recognition, and then provides in-store navigation guidance."

[1732] Scenario description:

[1733] A user puts on smart glasses and asks, "Where is the milk?" while in a physical store. This voice input is converted into text by a speech recognition engine, and analysis identifies the user's intent as "I'm looking for milk." An emotion recognition engine evaluates the user's emotions and generates more detailed and courteous guidance if, for example, the user feels anxious. Captured image analysis then analyzes the store map and provides a voice guide saying, "The product is on the left side of aisle A."

[1734] As described above, this system allows even users who are unfamiliar with technology to easily operate it, and enables navigation within physical stores and acquisition of product information.

[1735] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1736] Step 1:

[1737] The device receives voice input from the user: the user speaks questions or commands into the smart glasses or smartphone, which are captured as voice data through the device's microphone.

[1738] Step 2:

[1739] The device sends the acquired voice data to a voice recognition engine and converts it into text data. For example, if a user says, "Where is the milk?", a voice recognition engine (such as Google's voice recognition API) is used to create the text data, "Where is the milk?"

[1740] Step 3:

[1741] The device inputs the converted text data into a generative AI model to analyze the user's intent. Based on the analysis results, it determines what the user is looking for. In this case, the user's intent is determined to be "looking for milk."

[1742] Step 4:

[1743] The device sends the analysis results to an emotion recognition engine to analyze the user's emotions. Using Hugging Face's emotion recognition model, the device evaluates the user's emotional state (e.g., anxiety, irritation, calmness, etc.).

[1744] Step 5:

[1745] The device adjusts the operation procedures and guidance methods based on the results of the user's emotion analysis. For example, if an anxious state is detected, it generates more detailed and slower voice guidance. To generate this voice guidance message, the generated text and the emotion analysis results are used as input data.

[1746] Step 6:

[1747] The device uses a capture image analysis engine to capture the current screen and analyze the information on the screen, including a map of the store. Using AI image analysis technology, it identifies the user's location and the location of the product they are looking for (in this case, milk).

[1748] Step 7:

[1749] The device sends the analysis results to a speech synthesis engine and provides the user with appropriate instructions via voice. For example, it generates a specific message such as "The product is on the left side of aisle A" and outputs it aloud. It uses a speech synthesis library such as pyttsx3.

[1750] Step 8:

[1751] The server periodically collects operation logs from the device and determines whether the user is unsure of how to operate the device. Based on the analysis results, it sends messages to the device suggesting additional guidance or support if necessary. If the user does not perform any operation for a long period of time, a support message such as "Are you having trouble?" will be displayed.

[1752] Step 9:

[1753] The server performs security checks on the information being displayed and the web page being accessed, and if it detects any suspected fraud, it sends a warning message to the device, such as "This site may not be secure. Do you want to continue?" This process is done in real time to ensure the safety of the user.

[1754] Step 10:

[1755] The device encrypts and stores collected voice data and operation logs in local storage, and automatically deletes them after a certain period of time. For example, voice data is protected using AES encryption technology, and a system is implemented that automatically deletes it after 30 days, protecting user privacy.

[1756] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1757] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1758] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1759] [Fourth embodiment]

[1760] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1761] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1762] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1763] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1764] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1765] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1766] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1767] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1768] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1769] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1770] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1771] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1772] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1773] The present invention relates to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly elderly people. The following describes an embodiment of the present invention.

[1774] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intent, searches for the corresponding operating procedure, and provides voice guidance. It also has the ability to capture the current screen and analyze the captured image to identify the currently displayed app or menu. Below, we will explain each function and provide specific examples of system operation.

[1775] Voice input and analysis

[1776] Subject: Device

[1777] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[1778] Search and guide instructions

[1779] Subject: Device

[1780] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[1781] Screen capture and image analysis

[1782] Subject: Device

[1783] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open and asks how to perform a specific operation, the device will identify that it is the "new email app screen" from the captured image and provide instructions.

[1784] Operation log monitoring and support proposals

[1785] Subject: Server

[1786] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[1787] Security Warning

[1788] Subject: Server

[1789] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[1790] Data Privacy Protection

[1791] Subject: Device

[1792] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[1793] As a result, the system of the present invention can help users operate smartphones easily while ensuring security and privacy, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[1794] The processing flow will be explained below.

[1795] Voice input and analysis

[1796] Subject: Device

[1797] Step 1:

[1798] The device activates the microphone and collects the user's speech in real time as audio data.

[1799] Step 2:

[1800] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[1801] Step 3:

[1802] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[1803] Step 4:

[1804] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[1805] Step 5:

[1806] The device will guide the user through the operation procedures via voice.

[1807] Screen capture and image analysis

[1808] Subject: Device

[1809] Step 1:

[1810] Capture the screen currently displayed on the device and save it as a still image.

[1811] Step 2:

[1812] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[1813] Step 3:

[1814] Predicts what the user will need to do next based on identified apps and menus.

[1815] Step 4:

[1816] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[1817] Operation log monitoring and support proposals

[1818] Subject: Server

[1819] Step 1:

[1820] The server periodically collects operation logs from the terminal.

[1821] Step 2:

[1822] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[1823] Step 3:

[1824] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[1825] Step 4:

[1826] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[1827] Step 5:

[1828] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[1829] Security Warning

[1830] Subject: Server

[1831] Step 1:

[1832] The server monitors and analyzes the content of websites and emails being displayed in real time.

[1833] Step 2:

[1834] Based on the analysis results, we identify content that may be fraudulent.

[1835] Step 3:

[1836] Generate a warning message if any suspected fraud is identified.

[1837] Step 4:

[1838] Sends server-generated warning messages to the terminal for display to the user.

[1839] Step 5:

[1840] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[1841] Data Privacy Protection

[1842] Subject: Device

[1843] Step 1:

[1844] The voice data and operation logs collected by the device are stored in local storage.

[1845] Step 2:

[1846] Encrypt stored data and set access restrictions to prevent unauthorized access.

[1847] Step 3:

[1848] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[1849] Step 4:

[1850] Make sure the deleted data is completely wiped from your device.

[1851] This allows the system to help users operate their smartphones easily while ensuring security and privacy.

[1852] Example 1

[1853] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1854] Today's smartphones integrate a variety of functions in a complex manner, making them difficult to operate, especially for elderly users and those unfamiliar with technology. Furthermore, assistance systems using voice recognition technology are required to accurately understand the user's intentions and guide them to the appropriate operating procedures. However, many existing systems lack the precision to analyze the user's voice commands and are unable to provide the appropriate operating procedures. Furthermore, some systems lack the functionality to provide prompt assistance when the user is unsure of how to operate the device, and some lack data privacy protection.

[1855] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1856] In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, and means for analyzing the converted text data to identify the user's intention, thereby enabling the server to accurately analyze the content of a voice instruction given by the user and provide appropriate operating procedures.

[1857] Furthermore, the server includes means for searching for an operation procedure corresponding to the user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, means for analyzing the captured image and identifying the application or menu currently being displayed, means for analyzing operation logs collected from the terminal and determining whether the user is unsure of the operation, means for offering assistance when the user is unsure of the operation, means for encrypting the collected voice data and operation logs, and means for automatically deleting the encrypted data after a certain period of time. This makes it possible to offer assistance immediately when the user is unsure of the operation, while at the same time ensuring the privacy of the collected data.

[1858] "Means for accepting voice input" is a general term for devices and software functions for capturing voice uttered by a user.

[1859] "Means for converting into text data" is a general term for technology and software for converting voice data into character string data.

[1860] "Means for identifying user intent" is a general term for technologies and algorithms that analyze acquired text data and determine what the user is trying to achieve.

[1861] "Means for searching for operating procedures" is a general term for technology and software for searching a recorded database or storage device for an operating method that corresponds to a specified user's intention.

[1862] "Voice guidance means" is a general term for devices and software that notify users of searched operating procedures by voice.

[1863] "Means for capturing the screen" is a general term for technologies and devices for acquiring an image of the currently displayed screen.

[1864] "Means for analyzing a captured image" is a general term for techniques and algorithms for analyzing an acquired captured image and identifying the contents of the currently displayed application or menu.

[1865] "Means for analyzing operation logs" is a general term for technologies and algorithms used to analyze operation history data collected from a terminal and determine the user's operation status.

[1866] "Means for suggesting assistance" is a general term for technology and software that presents appropriate operating procedures and supplementary information when a user is unsure how to operate something.

[1867] "Means for encrypting voice data and operation logs" is a general term for technologies and algorithms used to encrypt and protect collected voice data and operation logs.

[1868] "Means for automatic deletion after a certain period of time" is a general term for technology and software that automatically deletes encrypted data after a set period of time has passed.

[1869] MODE FOR CARRYING OUT THE INVENTION

[1870] This invention relates to an AI-equipped system that assists users, particularly elderly people and those unfamiliar with technology, in easily operating smartphones. The system accepts voice input from the user, converts it into text data, and analyzes it to identify the user's intent. It then searches for operating procedures corresponding to the identified intent and provides voice guidance. It also has the ability to capture the current screen and analyze the image to identify the application or menu currently being displayed. Furthermore, it periodically collects operation logs from the device, determines whether the user is unsure of an operation, and offers assistance. Finally, the collected data is encrypted and stored, and automatically deleted after a certain period of time to protect privacy.

[1871] Hardware and software used

[1872] Speech recognition engine: Google Speech-to-Text (to convert voice data into text data)

[1873] Generative AI model: GPT-3 (to analyze text data and identify user intent)

[1874] Database: Search for operating instructions

[1875] Audio output device: Speaker (to provide audio instructions)

[1876] Image analysis engine: Google Vision API (to analyze captured images)

[1877] Encryption technology: AES-256 (to encrypt collected data)

[1878] Specific examples

[1879] 1. Voice to text conversion:

[1880] User: Say "I want to connect to the Internet."

[1881] Device: Audio is captured through the microphone and converted into text data such as "I want to connect to the Internet" using Google Speech-to-Text.

[1882] 2. Intention analysis of text data and search for operation procedures:

[1883] On the device: The generative AI model GPT-3 is used to analyze the "Wi-Fi settings" and the user's intent, and then the corresponding operation steps are searched for in the database.

[1884] Device: A voice prompt will say, "Open the Settings app and go to the Wi-Fi menu."

[1885] 3. Screen capture and analysis:

[1886] User: You have the Mail app open and say, "I want to compose a new email."

[1887] Device: Capture the currently displayed screen and use the Google Vision API to identify it as the "new email app composition screen." The device then prompts the user to "enter the subject of the email."

[1888] 4. Operation log collection and support suggestions:

[1889] Server: Periodically collects operation logs from the device, and if the user does not operate the device for more than 30 seconds, it uses AI to determine that assistance is required.

[1890] Server: Sends a help offer message to the device saying "Can we help you?"

[1891] 5. Security Checks and Warnings:

[1892] User: When you try to click on a suspicious URL.

[1893] Server: Immediately analyzes the URL and sends a warning message to the device saying, "This site may not be secure. Do you want to continue?"

[1894] 6. Data Encryption and Storage:

[1895] Terminal: Collected audio data and operation logs are stored in local storage using AES-256 encryption technology.

[1896] Device: Stored data will be automatically deleted after one month.

[1897] Prompt Sentence Examples

[1898] When a user says they want to perform a specific action, for example, "I want to set up Wi-Fi," how can I guide them to "Open the Settings app and go to the Wi-Fi menu"?

[1899] The system allows even non-technical users to easily perform complex smartphone operations, provides assistance when needed, and protects data privacy.

[1900] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1901] System program processing flow

[1902] Step 1:

[1903] The user speaks and performs voice input. For example, the user might say, "I want to connect to the Internet."

[1904] Input: User's voice

[1905] Output: Audio data

[1906] Step 2:

[1907] The device picks up audio through a microphone and generates audio data.

[1908] Input: Audio data

[1909] Output: Raw audio data

[1910] Step 3:

[1911] The device converts the voice data into text using Google Speech-to-Text. For example, the text data might say, "I want to connect to the Internet."

[1912] Input: Raw audio data

[1913] Output: Text data

[1914] Step 4:

[1915] The device uses a generative AI model (GPT-3) to analyze text data and identify the user's intent. For example, it can analyze the intent "Wi-Fi settings" from the text "I want to connect to the Internet."

[1916] Input: Text data

[1917] Output: User intent (e.g., "Wi-Fi settings")

[1918] Step 5:

[1919] The device searches a database for the operation procedure that corresponds to the user's intention. For example, it searches for the procedure for "Wi-Fi settings."

[1920] Input: User intent

[1921] Output: Instructions (e.g. "Open the Settings app and go to the Wi-Fi menu")

[1922] Step 6:

[1923] The device will be searched and the operation procedure will be guided to the user using the voice output device (speaker). For example, it will say, "Open the Settings app and go to the Wi-Fi menu."

[1924] Input: Operation procedure

[1925] Output: Voice guidance

[1926] Step 7:

[1927] Capture the screen that the device is currently displaying. For example, if the user has the email app open, capture that screen.

[1928] Input: Current screen

[1929] Output: Captured image

[1930] Step 8:

[1931] The device analyzes the captured image using the Google Vision API. For example, it identifies the captured image as the "new email app composition screen."

[1932] Input: Captcha image

[1933] Output: Identifying the currently displayed application or menu

[1934] Step 9:

[1935] The server periodically collects operation logs from the device. For example, if the user does not operate the device for 30 seconds or more, a log is collected.

[1936] Input: Operation Log

[1937] Output: Operation log data

[1938] Step 10:

[1939] The server uses AI to analyze the collected operation logs and determine whether the user is unsure of the operation. For example, if the user does not operate the device for more than 30 seconds, it will determine that the user is unsure.

[1940] Input: Operation log data

[1941] Output: Determine whether the user is confused about the operation

[1942] Step 11:

[1943] If the server is unsure of an operation, it sends a message offering assistance to the terminal, such as "Can I help you?"

[1944] Input: Determine if the user is confused

[1945] Output: Support offer message

[1946] Step 12:

[1947] The server analyzes the website or email you are viewing in real time and sends a warning message if it detects any fraudulent activity, such as "This site may not be safe. Do you want to continue?"

[1948] Input: The website or email you are viewing

[1949] Output: Security warning message

[1950] Step 13:

[1951] The voice data and operation logs collected by the device are encrypted using AES-256 encryption technology. For example, the voice data is encrypted and stored in local storage.

[1952] Input: Collected voice data and operation logs

[1953] Output: Encrypted data

[1954] Step 14:

[1955] The device will automatically delete the encrypted data after a certain period of time, for example, after one month.

[1956] Input: Encrypted data

[1957] Output: Automatic data deletion

[1958] (Application example 1)

[1959] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1960] Operating conventional smartphones is often difficult, especially for the elderly. Similarly, using food delivery services is difficult, with many users struggling to understand the ordering process. Therefore, there is a demand for systems that are easy for these users to operate.

[1961] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1962] In this invention, the server includes means for accepting voice input from a user, means for converting the accepted voice data into text data, means for analyzing the converted text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for capturing the currently displayed screen, analyzing the image, and identifying the displayed content, and means for providing voice guidance for ordering food delivery based on the identified displayed content. This enables even users who have difficulty operating smartphones to easily place food delivery orders.

[1963] "Means for accepting voice input from the user" is a general term for devices and software that recognize the voice spoken by the user and receive the voice data.

[1964] "Means for converting voice data into text data" refers to the technology or function for analyzing received voice information and converting it into text information.

[1965] "Means for identifying user intent" refers to technologies and mechanisms that understand and determine the operations and service content desired by the user from the converted text data.

[1966] A "means for searching for operating procedures" is a technology or system that searches for appropriate operating methods or procedures based on the identified user intent.

[1967] "Voice guidance means" refers to technology or devices that verbally communicate searched operating procedures to the user.

[1968] "Means for capturing the currently displayed screen" refers to the technology or function for acquiring the currently displayed screen content of a device as image data.

[1969] "Means for analyzing images" refers to techniques or algorithms for analyzing captured image data and recognizing the information displayed therein.

[1970] "Means for providing voice guidance for ordering food delivery" refers to technologies and systems that provide voice guidance to users through the food delivery ordering process based on specified display content and the user's intent.

[1971] This invention relates to a voice-operated food delivery system designed for users who have difficulty operating smartphones, particularly the elderly. The system accepts voice input, converts the voice data into text, analyzes it, identifies the user's intention, and provides voice guidance on the appropriate operation procedures.

[1972] System configuration

[1973] The system consists of the following main modules:

[1974] Audio Input Module

[1975] Voice Recognition Module

[1976] Intention Analysis Module

[1977] Operation procedure search module

[1978] Voice Guidance Module

[1979] Screen Capture Module

[1980] Image Analysis Module

[1981] Food delivery information module

[1982] Hardware and software used

[1983] Hardware

[1984] Microphone: A device for capturing the user's voice.

[1985] Camera: A device for capturing the currently displayed screen (using the screenshot function on your smartphone).

[1986] software

[1987] SpeechRecognition Library: A library for converting voice data into text data.

[1988] Pyttsx3 library: A synthetic speech engine for generating voice prompts.

[1989] PIL (Python Imaging Library): A library for processing screen capture images.

[1990] Pytesseract library: An optical character recognition (OCR) engine for extracting text from images.

[1991] System operation explanation

[1992] Voice to text conversion

[1993] A user voice-overs a food delivery order through a microphone, for example, saying, "I'd like to order a pizza." The microphone captures the voice input, and the SpeechRecognition library converts the speech into text data.

[1994] Intention analysis and operation procedure search

[1995] The converted text data is analyzed by an intention analysis module, and is identified as, for example, "ordering a pizza." Next, an operating procedure search module searches for an appropriate operating procedure based on the intention.

[1996] Voice guidance and screen analysis

[1997] The searched operation procedures are then audibly guided to the user by the voice guidance module. For example, instructions such as "Add pizza to cart. Press the continue button" are given by synthesized voice. To determine which screen the user is currently on, the screen capture module captures the screen, and the image analysis module analyzes the captured image.

[1998] Food delivery information

[1999] The voice prompts are repeated until the user completes the food delivery order. For example, the voice prompts the user based on the current screen, such as "Open the pizza menu" on the menu screen or "Confirm your order" on the cart screen.

[2000] Specific examples

[2001] For example, if a user says, "I would like to order sushi," the speech is converted into text and the intent is analyzed as "Order sushi." After image analysis confirms that the screen is a menu screen, a voice prompt will be played saying, "Add sushi to cart. Please press the continue button."

[2002] Example prompts for generative AI models

[2003] To develop a voice-operated food delivery app for the elderly, write a program that converts the user's voice input into text, analyzes the user's actions from screen captures, and provides appropriate guidance.

[2004] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2005] Step 1:

[2006] The user inputs voice through the microphone. In this step, the user's voice is picked up through the microphone and input to the terminal as voice data. For example, the user may say, "I'd like to order a pizza."

[2007] Step 2:

[2008] The speech recognition module converts the speech data into text data. The device then analyzes the acquired speech data using the SpeechRecognition library and converts it into text data. For example, a speech saying "I would like to order a pizza" is converted into text data saying "I would like to order a pizza."

[2009] Step 3:

[2010] The intent analysis module analyzes the text data and identifies the user's intent. The device analyzes the converted text data and understands the user's intent. This step uses NLP (natural language processing) technology to analyze and identify the user's intent as "order pizza."

[2011] Step 4:

[2012] The operation procedure search module searches for operation procedures based on the specified intent. The device searches for operation procedures from a database based on the specified intent. For example, in the case of "ordering pizza," the module searches for procedures to open the pizza menu and add it to the cart.

[2013] Step 5:

[2014] The voice guidance module provides voice guidance for the searched operation procedures. The terminal uses a speech synthesis engine (Pyttsx3 library) to guide the user through the searched procedures. For example, the user may hear voice guidance such as "Add pizza to cart. Press the continue button."

[2015] Step 6:

[2016] The screen capture module captures the current screen. The device captures the currently displayed screen of a food delivery app and obtains its image data. For example, if the user is on the menu screen of a food delivery app, that screen will be captured.

[2017] Step 7:

[2018] The image analysis module analyzes the captured image and identifies the displayed content. The device processes the captured image using the PIL library and uses the Pytesseract library to extract and analyze the text in the image. This identifies the screen the user is currently on. For example, it identifies the current screen as the "menu screen."

[2019] Step 8:

[2020] The food delivery guidance module then provides further voice guidance based on the identified display content. The device combines the analyzed display content with the user's intention and provides voice guidance on the next action to take. For example, if the user is on the menu screen, they may be told, "Add pizza to cart. Press the continue button," or if they move to the cart screen, they may be told, "Confirm your order."

[2021] Step 9:

[2022] If the user does not complete an operation, the operation log is periodically collected and analyzed by the server. The server receives the operation log from the device and analyzes it using an algorithm to determine whether the user is confused about the operation. For example, if a specific operation is not performed for a certain period of time, it is determined that the user is confused.

[2023] Step 10:

[2024] If the server determines that the user is unsure of how to operate the device, it will provide a voice message suggesting additional assistance. The server sends a message such as "Are you having trouble?" to the device, which then relays this message to the user via voice guidance.

[2025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2026] The present invention adds emotion recognition functionality to an AI-equipped smartphone system designed for users who are not good at operating smartphones, particularly the elderly, making it even easier to use. The following describes an embodiment of the present invention.

[2027] This system accepts voice input from the user, converts the voice data into text, analyzes it to identify the user's intention, searches for the corresponding operating procedure, and provides voice guidance.It also has the ability to capture the current screen and analyze the captured image to identify the app or menu currently being displayed.In addition, an emotion recognition engine is incorporated, further improving the user experience by adjusting the assistance method according to the user's emotions.

[2028] Voice input and analysis

[2029] Subject: Device

[2030] The device accepts voice input when the user speaks. Voice data acquired through the microphone is converted into text data by a voice recognition engine. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data and analyzed as "Wi-Fi settings."

[2031] Search and guide instructions

[2032] Subject: Device

[2033] Based on the text data, the device uses an AI model to analyze the user's intent and searches the database for the appropriate operation procedure, which is then output to the user via voice, instructing them to open the "Settings" app and then go to the "Wi-Fi" menu.

[2034] Screen capture and image analysis

[2035] Subject: Device

[2036] To help users understand the currently displayed screen in an easy-to-understand way, the device takes a screen capture and uses an AI image analysis engine to identify the app or menu currently displayed. For example, if a user has the email app open, the device will identify from the captured image that it is the "new email app screen" and provide instructions on how to use it.

[2037] Operation log monitoring and support proposals

[2038] Subject: Server

[2039] The server periodically collects operation logs from the device and uses AI to determine whether the user is unsure of how to operate the device. For example, if the user does not operate the screen for more than 30 seconds, the server determines that assistance is required and sends a message to the device suggesting assistance, asking, "Are you having trouble?"

[2040] Security Warning

[2041] Subject: Server

[2042] The server analyzes the website or email being viewed and sends a warning message to the device if it suspects fraud. For example, if a user clicks on a suspicious URL, the server detects that the URL may be fraudulent and displays a warning message saying, "This site may not be safe. Do you want to continue?"

[2043] Data Privacy Protection

[2044] Subject: Device

[2045] The device encrypts and stores collected voice data and operation logs in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and then automatically deleted after that period has elapsed.

[2046] Emotion recognition engine

[2047] Subject: Device

[2048] The device is equipped with an emotion recognition engine that can recognize emotions from the user's voice and operation logs. For example, if the device detects that the user is frustrated, it will explain the operation procedure more slowly and send an encouraging message to calm the user.

[2049] Adjusting operating procedures according to emotions

[2050] Subject: Device

[2051] The emotion recognition engine analyzes the user's emotions and adjusts the operating procedures and guidance accordingly. For example, if the user is feeling anxious, the device will provide detailed instructions and suggest that additional support is needed.

[2052] Generating emotion-responsive messages

[2053] Subject: Device

[2054] The emotion recognition engine generates encouraging and comforting messages based on the user's emotions, and the device then provides these messages to the user through voice. For example, if the user shows signs of fatigue, the device will provide a message such as, "Take a short break. It's okay, let's do the next step together."

[2055] As a result, the system of the present invention not only helps users operate smartphones easily while ensuring security and privacy, but also provides an advanced user experience through emotion recognition, providing an environment where even users who are unfamiliar with technology, such as the elderly, can use smartphones with confidence.

[2056] The processing flow will be explained below.

[2057] Voice input and analysis

[2058] Subject: Device

[2059] Step 1:

[2060] The device activates the microphone and collects the user's speech in real time as audio data.

[2061] Step 2:

[2062] The voice data collected by the device is sent to a voice recognition engine and converted into text data.

[2063] Step 3:

[2064] Text data is sent to an analysis engine, which uses AI to identify the user's intent.

[2065] Step 4:

[2066] Based on the analysis results, the appropriate operating procedures corresponding to the user's questions and instructions are searched from the database.

[2067] Step 5:

[2068] The device will guide the user through the operation procedures via voice.

[2069] Screen capture and image analysis

[2070] Subject: Device

[2071] Step 1:

[2072] Capture the screen currently displayed on the device and save it as a still image.

[2073] Step 2:

[2074] The captured image acquired by the device is sent to an image analysis engine to identify the app or menu being displayed.

[2075] Step 3:

[2076] Predicts what the user will need to do next based on identified apps and menus.

[2077] Step 4:

[2078] Determine whether the predicted operation is appropriate, and if not, prompt the user to re-enter the information.

[2079] Operation log monitoring and support proposals

[2080] Subject: Server

[2081] Step 1:

[2082] The server periodically collects operation logs from the terminal.

[2083] Step 2:

[2084] The collected operation logs are analyzed using an AI analysis engine to understand the user's operation status.

[2085] Step 3:

[2086] If it is determined that the user is unsure of an operation, the server generates a message offering assistance.

[2087] Step 4:

[2088] The server generates a support suggestion message and sends it to the terminal, where it is displayed to the user.

[2089] Step 5:

[2090] If the user accepts the assistance proposal, the server sends instructions to the terminal guiding the user through the appropriate operating procedures.

[2091] Security Warning

[2092] Subject: Server

[2093] Step 1:

[2094] The server monitors and analyzes the content of websites and emails being displayed in real time.

[2095] Step 2:

[2096] Based on the analysis results, we identify content that may be fraudulent.

[2097] Step 3:

[2098] Generate a warning message if any suspected fraud is identified.

[2099] Step 4:

[2100] Sends server-generated warning messages to the terminal for display to the user.

[2101] Step 5:

[2102] If the user attempts to continue interacting with the fraudulent content, a strong warning message will be sent again.

[2103] Data Privacy Protection

[2104] Subject: Device

[2105] Step 1:

[2106] The voice data and operation logs collected by the device are stored in local storage.

[2107] Step 2:

[2108] Encrypt stored data and set access restrictions to prevent unauthorized access.

[2109] Step 3:

[2110] The device periodically checks the stored data and runs a program that automatically deletes it after a certain period of time.

[2111] Step 4:

[2112] Make sure the deleted data is completely wiped from your device.

[2113] Emotion recognition engine

[2114] Subject: Device

[2115] Step 1:

[2116] The device collects the user's voice and operation logs and sends them to an emotion recognition engine.

[2117] Step 2:

[2118] The emotion recognition engine analyzes the tone of voice, operation speed, etc. to identify the user's emotions.

[2119] Step 3:

[2120] The analyzed user emotion data is integrated into the current operation support process.

[2121] Adjusting operating procedures according to emotions

[2122] Subject: Device

[2123] Step 1:

[2124] The emotion recognition engine analyzes the user's emotions.

[2125] Step 2:

[2126] The device adjusts the operating procedures and guidance methods according to the results of the emotion analysis.

[2127] Step 3:

[2128] If the user is feeling unsure, the device will provide detailed instructions on how to proceed.

[2129] Generating emotion-responsive messages

[2130] Subject: Device

[2131] Step 1:

[2132] The emotion recognition engine analyzes the user's emotional data.

[2133] Step 2:

[2134] Generate encouraging and comforting messages according to the user's emotional state.

[2135] Step 3:

[2136] If the user appears to be tired of operating the device, the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[2137] Specific examples

[2138] Examples of voice input and analysis:

[2139] When a user says, "I want to connect to the Internet," the device converts the speech into text and interprets it as "Wi-Fi settings." The device then provides voice instructions on how to access the Wi-Fi settings screen.

[2140] Examples of operational log monitoring and support suggestions:

[2141] If the user does not interact with the screen for 30 seconds or more, the server generates a message saying "Can we help you?" and sends it to the device. If the user accepts this message, the server guides them through the appropriate operation procedure.

[2142] Examples of emotionally relevant messages:

[2143] If the user is tired of operating the device, the emotion recognition engine will analyze this and the device will provide a voice message such as, "Take a short break. It's okay, let's do the next step together."

[2144] As a result, this system provides operational assistance while taking the user's emotions into consideration, creating an environment in which users can use their smartphones with peace of mind.

[2145] Example 2

[2146] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2147] Elderly people and users unfamiliar with technology often find it difficult to operate smartphones, posing security risks such as getting lost or accidentally accessing phishing sites. Furthermore, existing systems lack support that takes into account the user's emotional state, which can easily cause stress. Therefore, there is a need for a system that allows users to operate smartphones more easily and safely.

[2148] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2149] In this invention, the server includes: means for accepting voice input from a user; means for converting the accepted voice data into text data; means for analyzing the converted text data and identifying the user's intent; means for searching for an operating procedure corresponding to the identified user's intent; means for providing voice guidance for the searched operating procedure; means for acquiring a screen capture and analyzing the acquired screen capture image to identify the software application or menu currently displayed; means for periodically collecting a user's operation log and analyzing the collected operation log to determine whether the user is unsure of an operation; means for analyzing the user's emotions and adjusting the guidance and operating procedure based on the results; means for suggesting assistance when the user is unsure of an operation or emotionally unstable; means for incorporating an emotion recognition engine and adjusting the assistance method according to the user's emotions; and means for sending a warning of possible fraud to the user's screen. This not only allows the user to easily receive voice guidance on the operation method, but also improves their understanding of the screen and provides appropriate assistance when they are unsure of an operation or feel anxious. Furthermore, security risks are reduced, allowing users to use their smartphones with peace of mind.

[2150] "Means for accepting voice input" refers to devices or software that acquire voice data spoken by a user as a digital signal and input it into the system.

[2151] "Means for converting voice data into text data" refers to devices or software that convert acquired voice data into text information using natural language processing technology.

[2152] "Means for analyzing text data and identifying user intent" refers to algorithms and software that use converted text data to understand user requests and objectives.

[2153] "Means for searching for operating procedures" refers to devices or software that refer to a database for operating methods or procedures that correspond to the specified user's intentions and obtain appropriate information.

[2154] "Means for providing voice guidance on operating procedures" refers to speech synthesis technology or software that explains the searched operating procedures to the user by voice.

[2155] "Means for obtaining a screen capture" refers to a device or software for obtaining the contents of the screen currently displayed on the display as image data.

[2156] "Means for analyzing screen capture images" refers to algorithms or software for analyzing the captured image and identifying the software applications and menus being displayed.

[2157] "Means for collecting operation logs" refers to devices or software that record user operation history and periodically collect that data.

[2158] "Means of analyzing operation logs to determine whether the user is having difficulty operating the system" refers to algorithms or software that evaluate the user's operation status based on collected operation logs and determine whether the user is having difficulty operating the system.

[2159] "Means for analyzing user emotions" refers to algorithms or software that evaluate the user's current emotional state based on their voice and operation history.

[2160] "Means for adjusting guidance and instructions" refers to algorithms or software that change the way instructions and instructions are presented depending on the user's emotional state.

[2161] "Means for suggesting assistance" refers to devices or software that suggest appropriate assistance measures or solutions when a user is unsure about how to operate the device or feels anxious.

[2162] "Means of incorporating an emotion recognition engine" refers to algorithms or software that analyzes user emotions in real time and adjusts the system's behavior based on that information.

[2163] "Potential fraud warning means" refers to a device or software that sends a message to a user warning them that a website they visit or an email they receive may be a phishing scam.

[2164] The present invention provides a system for assisting elderly people and users who are unfamiliar with technology to easily operate a smartphone. Hereinafter, an embodiment of the present invention will be described in detail.

[2165] Voice input and analysis

[2166] Subject: Device

[2167] The device accepts voice input when the user speaks. Voice input is done through the built-in microphone, and the voice data is temporarily stored in the device's memory in WAV format or other formats. This voice data is then converted into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, if a user says, "I want to connect to the Internet," the voice is converted into text data that reads, "I want to connect to the Internet."

[2168] Search and guide instructions

[2169] Subject: Device

[2170] Based on the text data, the device uses a generative AI model (e.g., GPT-4) to analyze the user's intent. This analysis determines that "I want to connect to the Internet" means "Set up Wi-Fi." The device then searches a database to obtain the appropriate operating instructions. Steps such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network" are searched for. These steps are then audibly instructed to the user using a speech output engine (e.g., Google Text-to-Speech). Instructions such as "Open the Settings app, select Wi-Fi, and connect to an available network" are provided.

[2171] Screen capture and image analysis

[2172] Subject: Device

[2173] To help users understand the currently displayed screen, the device takes a screen capture. The captured image is temporarily saved in the device's memory in PNG format, and then an AI image analysis engine such as Google Cloud Vision API is used to identify the software application or menu currently displayed. For example, the captured image can be identified as the "new screen of the Mail app."

[2174] Operation log collection and support proposals

[2175] Subject: Server

[2176] The server periodically collects user operation logs from the device. The operation logs are saved in CSV format and sent to the server. The received operation logs are analyzed using a machine learning model to determine whether the user is unsure of the operation. For example, if a certain operation has not continued for more than 30 seconds, it is determined that the user is unsure of the operation and a support suggestion message asking "Need help?" is sent to the device.

[2177] Security Warning

[2178] Subject: Server

[2179] The server analyzes the website being viewed or the email received and generates a warning message if there is a possibility of a phishing scam. For example, if the server determines that the URL the user clicked on may be fraudulent, it will send a warning message to the device saying, "This site may not be safe. Do you want to continue?"

[2180] Sentiment analysis and response

[2181] Subject: Device

[2182] The device is equipped with an emotion recognition engine (e.g., EmoNet) that analyzes emotions from the user's voice and operation logs. For example, if the device determines from the voice analysis that the user is frustrated, it will explain the operation procedure slowly and generate an encouraging voice message such as "You're almost there. Keep up the great work." If the user is feeling anxious, it will explain the operation procedure in detail and suggest additional support.

[2183] Data Privacy Protection

[2184] Subject: Device

[2185] The device encrypts collected voice data and operation logs using AES encryption and stores them in local storage. Access to the stored data is restricted and it is automatically deleted after a certain period of time. For example, voice data collected for support purposes is stored in the device for one month and is set to be automatically deleted after that period has elapsed.

[2186] Examples of prompt statements

[2187] For example, when using GPT-4, the prompt might look like this:

[2188] "A user says, 'I want to connect to the Internet.' How can I convert this speech to text and provide the appropriate instructions?"

[2189] By inputting these prompts into a generative AI model, a solution can be obtained that specifically analyzes the voice data and provides guidance on operation procedures. By implementing this system, users can operate their smartphones quickly and safely, and even users who are unfamiliar with technology can use it without stress.

[2190] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2191] Step 1:

[2192] Accepting voice input

[2193] Subject: Device

[2194] Input: User speech

[2195] How it works: The device uses the built-in microphone to capture the user's speech as audio data, which is temporarily stored in the device's memory in WAV format.

[2196] Output: WAV format audio data

[2197] Step 2:

[2198] Converting audio data to text

[2199] Subject: Device

[2200] Input: WAV format audio data

[2201] How it works: The voice data is converted to text data by calling the Google Speech-to-Text API. For example, the voice saying "I want to connect to the Internet" is converted to text data like "I want to connect to the Internet."

[2202] Output: Text data "I want to connect to the Internet"

[2203] Step 3:

[2204] Intent Analysis

[2205] Subject: Device

[2206] Input: Text data "I want to connect to the Internet"

[2207] How it works: The device uses a generative AI model (e.g., GPT-4) to analyze text data and identify user intent. For example, the intent "Wi-Fi settings" is identified from the text "I want to connect to the Internet."

[2208] Output: "Wi-Fi settings"

[2209] Step 4:

[2210] Find instructions

[2211] Subject: Device

[2212] Input: "Wi-Fi settings"

[2213] How it works: The device searches its built-in database to retrieve instructions for "Wi-Fi settings," such as "Open the Settings app," "Select Wi-Fi," and "Connect to an available network."

[2214] Output: List of operation steps

[2215] Step 5:

[2216] Voice guidance for operation procedures

[2217] Subject: Device

[2218] Input: List of operating instructions

[2219] How it works: The device uses the Google Text-to-Speech API to provide voice instructions, such as "Open the Settings app, select Wi-Fi, and connect to an available network."

[2220] Output: Voice guidance

[2221] Step 6:

[2222] Taking a screen capture

[2223] Subject: Device

[2224] Input:Currently displayed screen

[2225] What it does: The device captures the current screen image, which is temporarily saved in the device's memory as a PNG image.

[2226] Output: Captured image in PNG format

[2227] Step 7:

[2228] Analysis of captured images

[2229] Subject: Device

[2230] Input: Captcha image in PNG format

[2231] How it works: The device uses the Google Cloud Vision API to analyze the captured image and identify the software application or menu currently displayed. For example, it may identify that the image is the "Mail app's compose screen."

[2232] Output: Identified applications and menus

[2233] Step 8:

[2234] Collecting operation logs

[2235] Subject: Device

[2236] Input: User operation history

[2237] Operation: The device records user operation logs in CSV format and periodically sends them to the server. The operation logs include click and tap locations, operation times, etc.

[2238] Output: Operation log in CSV format

[2239] Step 9:

[2240] Log analysis

[2241] Subject: Server

[2242] Input: CSV format operation log

[2243] How it works: The server analyzes the received operation log using a machine learning model to determine whether the user is having trouble with an operation. For example, if a certain operation has not continued for more than 30 seconds, it determines that the user is having trouble.

[2244] Output: Judgment result of whether the user is confused about the operation

[2245] Step 10:

[2246] Submit a support offer

[2247] Subject: Server

[2248] Input: Determine whether the user is confused about the operation

[2249] Operation: The server generates a help offer message saying "Can I help you?" and sends it to the terminal.

[2250] Output: Support offer message

[2251] Step 11:

[2252] Generate security warnings

[2253] Subject: Server

[2254] Input: Information about the website or email you are viewing

[2255] How it works: The server analyzes the website or email you are viewing and generates a warning message if there is a possibility of a phishing scam, such as "This site may not be secure. Do you want to continue?"

[2256] Output: Security warning message

[2257] Step 12:

[2258] Sending a warning message

[2259] Subject: Server

[2260] Input: Security warning message

[2261] Action: The server generates a security warning message and sends it to the terminal.

[2262] Output: Warning message displayed on the terminal

[2263] Step 13:

[2264] Emotion Analysis

[2265] Subject: Device

[2266] Input: User voice and operation history

[2267] How it works: The device uses an emotion recognition engine (e.g., EmoNet) to analyze the user's emotions. For example, it determines if the user is frustrated based on voice analysis and operation patterns.

[2268] Output: User's emotional state

[2269] Step 14:

[2270] Emotion-based guidance adjustment

[2271] Subject: Device

[2272] Input: User's emotional state

[2273] What it does: If the user is frustrated, the device will explain instructions and directions more slowly and generate encouraging messages, such as "Take a break and we'll walk you through the next steps together."

[2274] Output: Adjusted operating procedures and guidance messages

[2275] Step 15:

[2276] Data Privacy Protection

[2277] Subject: Device

[2278] Input: Collected voice data and operation logs

[2279] Operation: The device encrypts voice data and operation logs using AES encryption and stores them in local storage. After a certain period of time (e.g., one month), this data is automatically deleted.

[2280] Output: Encrypted data and deletion log

[2281] (Application example 2)

[2282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2283] While conventional voice recognition systems have the ability to identify user intent and provide voice guidance, they are still difficult to operate, especially for elderly people who are unfamiliar with technology and users who are unfamiliar with digital devices. They also lack the ability to provide appropriate assistance based on the user's emotions, which hinders the user experience. Another issue is the lack of assistants for in-store navigation and product information provision.

[2284] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input from the user, means for converting the accepted voice data into text data, means for analyzing the text data and identifying the user's intention, means for searching for an operation procedure corresponding to the identified user's intention, means for providing voice guidance for the searched operation procedure, means for analyzing the user's emotions, means for adjusting the operation procedure and guidance based on the analysis results, means for analyzing the acquired captured image to identify the displayed information and providing voice guidance based on the result, and means for suggesting assistance if the user is unfamiliar with the technology. This allows even users unfamiliar with the technology to easily operate the system and provides appropriate assistance based on the user's emotions. It also facilitates navigation and product information provision in physical stores.

[2285] A "means for accepting voice input" has the function of capturing speech uttered by a user and transmitting it to the system for processing.

[2286] The "means for converting voice data into text data" has a function of analyzing received voice and converting the content into text information.

[2287] The "means for identifying the user's intent" is a function that analyzes the text data obtained by voice recognition and determines what the user is looking for.

[2288] The "means for searching for operating procedures" has the function of searching a database or program for the optimal operating procedure that corresponds to the specified user's intention.

[2289] "Voice guidance means" means a function that notifies and guides the user through the searched operating procedures by voice.

[2290] "Means for analyzing user emotions" refers to a function that reads emotions from the user's speech and operation logs and recognizes their state.

[2291] The "means for adjusting operating procedures and guidance" has the function of appropriately changing the content and method of guidance based on the results of user emotion analysis, and providing support according to the user's condition.

[2292] The "means for analyzing a captured image" has a function of analyzing an acquired screen capture and identifying the information and application currently being displayed.

[2293] "Means to offer assistance when the user is unfamiliar with the technology" refers to a function that determines whether the user is unsure of how to operate the device and automatically provides the necessary support.

[2294] This invention utilizes AI technology to provide a "smart store assistant" system to assist elderly and technology-challenged customers in brick-and-mortar stores. The system can be installed and used on devices such as smartphones or smart glasses.

[2295] System Configuration

[2296] 1. Voice recognition and analysis functions

[2297] The device accepts voice input from the user and converts the voice data captured through the microphone into text data using a speech recognition engine (e.g., Google's speech recognition API). The converted text data is then analyzed using a generative AI model to identify the user's intent.

[2298] 2. Operation procedures and guidance functions

[2299] The operating procedures corresponding to the identified user intent are searched for in a database (such as a cloud database), and the search results are then audibly guided to the user using a speech synthesis engine (such as pyttsx3).

[2300] 3. Screen capture and analysis functions

[2301] The device captures the currently displayed screen and uses an AI image analysis engine (such as OpenCV or the machine learning library TensorFlow) to identify the information and app being displayed. The results of this analysis are also announced via voice.

[2302] 4. Emotion recognition and response functions

[2303] The device uses an emotion recognition engine (such as the Hugging Face emotion recognition model) to analyze the user's emotions from their speech and operation logs. Based on the analysis results, the device adjusts operation procedures and guidance appropriately to provide support tailored to the user.

[2304] 5. User assistance suggestion function

[2305] The server periodically collects and analyzes operation logs from the device. If the server determines that the user is unsure of how to operate the device, it sends a message suggesting assistance to the device and displays it to the user.

[2306] 6. Security and Privacy Protection Features

[2307] The server analyzes the displayed information and sends a warning message if there is a security risk. The collected data is encrypted and automatically deleted after a certain period of time.

[2308] Specific examples

[2309] Example prompt sentence:

[2310] "When a user says, 'Where is the milk?', the system converts the voice input into text, performs emotion recognition, and then provides in-store navigation guidance."

[2311] Scenario description:

[2312] A user puts on smart glasses and asks, "Where is the milk?" while in a physical store. This voice input is converted into text by a speech recognition engine, and analysis identifies the user's intent as "I'm looking for milk." An emotion recognition engine evaluates the user's emotions and generates more detailed and courteous guidance if, for example, the user feels anxious. Captured image analysis then analyzes the store map and provides a voice guide saying, "The product is on the left side of aisle A."

[2313] As described above, this system allows even users who are unfamiliar with technology to easily operate it, and enables navigation within physical stores and acquisition of product information.

[2314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2315] Step 1:

[2316] The device receives voice input from the user: the user speaks questions or commands into the smart glasses or smartphone, which are captured as voice data through the device's microphone.

[2317] Step 2:

[2318] The device sends the acquired voice data to a voice recognition engine and converts it into text data. For example, if a user says, "Where is the milk?", a voice recognition engine (such as Google's voice recognition API) is used to create the text data, "Where is the milk?"

[2319] Step 3:

[2320] The device inputs the converted text data into a generative AI model to analyze the user's intent. Based on the analysis results, it determines what the user is looking for. In this case, the user's intent is determined to be "looking for milk."

[2321] Step 4:

[2322] The device sends the analysis results to an emotion recognition engine to analyze the user's emotions. Using Hugging Face's emotion recognition model, the device evaluates the user's emotional state (e.g., anxiety, irritation, calmness, etc.).

[2323] Step 5:

[2324] The device adjusts the operation procedures and guidance methods based on the results of the user's emotion analysis. For example, if an anxious state is detected, it generates more detailed and slower voice guidance. To generate this voice guidance message, the generated text and the emotion analysis results are used as input data.

[2325] Step 6:

[2326] The device uses a capture image analysis engine to capture the current screen and analyze the information on the screen, including a map of the store. Using AI image analysis technology, it identifies the user's location and the location of the product they are looking for (in this case, milk).

[2327] Step 7:

[2328] The device sends the analysis results to a speech synthesis engine and provides the user with appropriate instructions via voice. For example, it generates a specific message such as "The product is on the left side of aisle A" and outputs it aloud. It uses a speech synthesis library such as pyttsx3.

[2329] Step 8:

[2330] The server periodically collects operation logs from the device and determines whether the user is unsure of how to operate the device. Based on the analysis results, it sends messages to the device suggesting additional guidance or support if necessary. If the user does not perform any operation for a long period of time, a support message such as "Are you having trouble?" will be displayed.

[2331] Step 9:

[2332] The server performs security checks on the information being displayed and the web page being accessed, and if it detects any suspected fraud, it sends a warning message to the device, such as "This site may not be secure. Do you want to continue?" This process is done in real time to ensure the safety of the user.

[2333] Step 10:

[2334] The device encrypts and stores collected voice data and operation logs in local storage, and automatically deletes them after a certain period of time. For example, voice data is protected using AES encryption technology, and a system is implemented that automatically deletes it after 30 days, protecting user privacy.

[2335] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2336] The data generation model 58 is a so-call...

Claims

1. means for accepting voice input from a user; means for converting the received voice data into text data; A means for analyzing the converted text data and identifying the user's intent; A means for searching for an operating procedure corresponding to the identified user intent; The system includes a means for providing voice guidance for the searched operating procedures.

2. 10. The system of claim 1, The system further includes means for acquiring a capture of the current screen and analyzing the acquired capture image to identify the app or menu being displayed.

3. 10. The system of claim 1, A method for periodically collecting operation logs from the terminal and analyzing the collected operation logs to determine whether the user is having trouble with the operation. The system further includes means for offering assistance when the user is unsure of an operation.

4. 10. The system of claim 1, A system that includes a means to analyze the website or email being viewed and display a warning message if fraud is suspected.

5. 10. The system of claim 1, A system that includes a means to store collected voice data and operation logs in local storage and automatically delete them after a certain period of time.

Images